Prototyping a fully offline CHT Server environment

@mrjones The response am getting is as below:

Running the same curl command from the host machine where cht is installed I get the response of 200 OK

@andrineM - ah ha! While this is bad in that the DESKTOP-VS7DR70 host has an error - this hopefully is the same reason that the mobile devices can’t connect. Now we have an easier way to test and debug the problem.

Can you please run this command on both the server and the server and the DESKTOP-VS7DR70 host?

curl -vI https://cht.my.local-ip.co

By adding v to the command we can confirm that curl is trying to talk to the same IP on both hosts. Further, we can compare the results of the two calls to see if there’s any important differences.

Below I’ve mocked up a test where my CHT instance is at 192.168.68.17. You can see the first call is when I’ve intentionally stopped the docker containers and it gives the same error, but with more information, as you’re seeing:

clear;curl -vI https://cht.my.local-ip.co
*   Trying 192.168.68.17:443...
* TCP_NODELAY set     
* connect to 192.168.68.17 port 443 failed: Connection refused
* Failed to connect to cht.my.local-ip.co port 443: Connection refused 
* Closing connection 0                                                
curl: (7) Failed to connect to cht.my.local-ip.co port 443: Connection refused

Then, when I restart the containers, here’s what a successful call looks like:

curl -vI https://cht.my.local-ip.co   
*   Trying 192.168.68.17:443...                                       
* TCP_NODELAY set                                                     
* Connected to cht.my.local-ip.co (192.168.68.17) port 443 (#0)
* ALPN, offering h2                                                   
* ALPN, offering http/1.1                                             
* successfully set certificate verify locations:           
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs                                              
* TLSv1.3 (OUT), TLS handshake, Client hello (1):    
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):                                                                                 
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-SHA
* ALPN, server accepted to use http/1.1
* Server certificate:    
*  subject: CN=*.my.local-ip.co
*  start date: Apr 27 09:42:25 2021 GMT
*  expire date: Jul 26 09:42:25 2021 GMT
*  subjectAltName: host "cht.my.local-ip.co" matched cert's "*.my.local-ip.co"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
> HEAD / HTTP/1.1   
> Host: cht.my.local-ip.co
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
HTTP/1.1 404 Not Found
< Server: nginx/1.13.6
Server: nginx/1.13.6
< Date: Thu, 13 May 2021 17:01:50 GMT
Date: Thu, 13 May 2021 17:01:50 GMT
< Content-Type: text/html
Content-Type: text/html
< Content-Length: 169
Content-Length: 169
< Connection: keep-alive
Connection: keep-alive
< Vary: Accept-Encoding
Vary: Accept-Encoding

< 
* Connection #0 to host cht.my.local-ip.co left intact

@mrjones Here is the response I get from running when i run the curl from remote DESKTOP-VS7DR70 host

  • Trying 64:ff9b::413f:0…
  • Trying 65.63.0.0…
  • connect to 64:ff9b::413f:0 port 443 failed: Connection refused
  • connect to 65.63.0.0 port 443 failed: Connection refused
  • Failed to connect to cht.my.local-ip.co port 443: Connection refused
  • Closing connection 0
    curl: (7) Failed to connect to cht.my.local-ip.co port 443: Connection refused

When I run the same command from the Host where CHT is running am able to get the response as below;

@andrineM - Very interesting! On DESKTOP-VS7DR70 the dig call is telling you that it resolves cht.my.local-ip.co to the correct IP of 192.168.0.138. But when we use curl on the same machine it’s actually resolving to 65.63.0.0. This 65 IP is what happens when the DNS request goes out to the Internet, skipping Pi-hole all together.

Can we go back a step to the nmap call again? Can you run nmap --script broadcast-dhcp-discover on the DESKTOP-VS7DR70 computer? I suspect this is not actually getting the Pi-hole as it’s DNS server which is why the curl command fails, but the dig command succeeds.

@mrjones Please see the response from re-running the command *nmap --script broadcast-dhcp-discover*:
Have used a remote machine within the network with ip: 192.168.0.137
Have included the results from our previous commands as well.

@andrineM - Ah yes, this makes sense - thanks. Here’s what I understand of your LAN configuration, please let me know if anything is incorrect. You have a LAN with at least 5 devices:

  • Device 1 at 192.168.0.138: Ubuntu desktop with running docker-compose with Pi-hole and CHT
  • Device 2: DESKTOP-VS7DR70 Ubuntu Desktop
  • Device 3: Mobile Android device
  • Device 4 at 192.168.0.137: Andrine’s linux computer
  • Device 5 at 192.168.0.1: A router with a DHCP and DNS server

We discovered device 5 just now - this is a big help to know about - when you ran the last nmap --script broadcast-dhcp-discover command. A network can not have two devices running a DHCP server. Step 5 in the LAN & Sever section calls this out:

Set up a router that has DHCP turned off

So, to fix it, just turn of DHCP on 192.168.0.1 - hopefully an easy fix! While we’re at it, if you have an AP that is a stand alone device, ensure it is not running a DHCP server either.

After you have a LAN with just one DHCP server (the Pi-hole server), re-run the dig, curl and nmap commands an your three hosts (Andrine, DESKTOP-VS7DR70 and server) and they should all match up! You may need to reboot the Andrine and DESKTOP-VS7DR70 computers for DHCP change to take effect.

Let me know what you find when making the above changes.

@mrjones Thanks so much for this.The one challenge that we may experience is that majority of our facilities have a replica of the setup I have which you have described above.
Another thing have also noticed is the routers have a Dynamic DHCP as default with no option to disable it (Turn on/off at router level).
Please have a look at the router setting we have.

@andrineM - yes, that does sound challenging if you can’t turn off DHCP. Maybe you have an option to unplug the router from the LAN? For my set up, the router/Internet was just a way to get apt and docker-compose updates while I was doing set up - it is not needed for day to day operation.

From this screenshot it looks like you’re using a TP-Link M7200 which is a 4g hotspot type device which may mean all devices are wireless on the LAN and so you do not have, for example, a stand alone Ethernet switch and stand alone AP. If so, you’ll have to figure another solution than using the TP-Link M7200.

Best of luck and please do report back if you have more questions!

@mrjones Thanks so much for the support this far. Kindly allow me to give my feedback upon disabling of DHCP on the router. I was successfully able to curl the CHT endpoint and could access the application remotely on both the the desktop browser and the mobile phone browser. I however wasn’t able to use the custom url option on the App as I got the error: unable to contact server.

Below please see the result when using the phone browser and while using the app mobile phone option. Let me know what could be the cause of this.

@andrineM - This one took some thinking to solve - and I hope I have indeed solved it. Let’s find out!

I think you have a space in your URL. I was able to reproduce your exact problem, as well as solve it, as see in these side by side screenshots:

On the left you see a what looks to be a valid URL but has a space in it. On the right is the unbranded app working with the “same” URL, but it has no spaces. In my testing, I wasn’t unable to reproduce the problem in the mobile browser as it always trimmed the spaces before submitting it. I’ve filed a ticket to get this fixed in future version of Medic’s unbranded app - thanks for pointing out the issue!

Please try the URL again in the unbranded app, being sure there are no spaces before or after the URL and let us know how it works! As well, please be sure you’re testing in the latest version of the unbranded app, v 0.8.0.

@mrjones Have confirmed am able to access from NEON_RAY (Android 10) using the custom url. Still unsuccessful on Samsung A71 and Sumsung A30. Both on android 10 and ensuring no spaces before and after.

@andrineM - great to hear that you were able to connect with a different phone! This shows that everything should be working, but for some reason it’s not for your two phones (Samsung A71 and Sumsung A30)

Let’s try removing your custom offline deployment from our testing to see if it’s something to do with the phones specifically. On the two problem phones, can you please try:

  1. Uninstall the “Medic Mobile” Android App and re-install it from the Play Store
  2. Launch the app
  3. Click “Custom” to enter a new URL
  4. Enter https://gamma.dev.medicmobile.org

https://gamma.dev.medicmobile.org is a public test instance. You won’t be able to login, but you can see if the URL works which is what we care about now. Be sure not to click the existing link for this instance as we want to try and reproduce the error of entering your custom URL.

If you have problems with that URL, then it is a problem with either the phone itself or our Android app - either way great to know. Then we can open a ticket on the Android Medic repo to track the issue separately.

@mrjones Am able to access the instance above .We also have an online instance which am able to access and login from the two Samsung phones. Only issue I experience is when accessing https://cht.my.local-ip.co

@andrineM - I’m so sorry you’re still having issues with the Samsung A71 and Sumsung A30 phones - you must find this really frustrating! Thanks for your patience.

Can you confirm this accurately describes the issue you’re facing?

Title: “unable to contact server” error on self hosted CHT
Description: On a Samsung A71 or Sumsung A30 running Android 10, we can not access our CHT instance running in Docker. Using the unbranded app, we get a “unable to contact server” error.
Steps to reproduce:

  1. Install the CHT using docker-compose and a local only DNS entry and *.my.local-ip.co TLS certs per this forum post.
  2. Using a Samsung A71 or Sumsung A30 phone, download the unbranded 0.8.0 Android app from the Play Store
  3. Connect to the LAN where the CHT is hosted
  4. In the app, enter the URL https://cht.my.local-ip.co

Expected: The app connects to the CHT instace
Actual: The app shows a “unable to contact server” error and doesn’t connect

Related: Going to https://cht.my.local-ip.co in a browser on either phone works. Going to https://cht.my.local-ip.co on a desktop on the same LAN works. Using the test instance https://gamma.dev.medicmobile.org in the app and the phones’ browser works.

@mrjones I confirm. The above is the correct description of the issue.

@andrineM - Sorry for the delay in getting back to you!

Thank you for confirming the above is correct. I’ve filed a ticket to track this issue. Hopefully we’ll be able to get further in solving it.

@andrineM - I wanted to double check: can you confirm which version of the Android APK you’re running on the Samsung A71 and Sumsung A30? For example, I can see in this screenshot of my app that I’m running v0.8.0-webview:

@mrjones Am on Android 11 on both

@mrjones Allow me to thank you and the CHT team for the support accorded to Palladium Kenya (KeHMIS 3 Project) as we run the prototype for the deployment of CHT on both online and offline environments. Below please see the progress that we have made.

  • Under the leadership of @ojwangantony from our development team, enockrugut has managed to further automated the process of setting up and configuring both pihole and CHT. This has made deployment a seamless process and we are now ready to carryout a mass rollout in our facilities.

  • Lavatsaleo has also worked closely with AbdulhakimRajab in testing out the scripts and giving feedback from our pilot facilities.

We look forward to further collaborations with the community. We shall continuously offer feedback and support. Thank you

6 Likes

@andrineM - what a wonderful update to read after a good long while - thank you!

If there’s any of your work that is open source, or if you’ve published a more detailed write up of some the automaton work, it would be wonderful to see. Please share if possible!

Otherwise, best of luck with the scale-up!

Cheers,
-adj

4 Likes