Using sudo with docker helper, installing TLS certs

stephenmhuli · September 9, 2024, 3:29pm

Hi, @mrjones I have a simple question regarding this topic as we discussed a couple of months back, I used sudo to create and run a new project as shown below:

smhuli@HPx:~/Documents/CHT-Projects/docker-helper-4.x$ sudo ./cht-docker-compose.sh config_dtree_4_4.env

homedir: /root/.medic/cht-docker/config_dtree_4_4-dir
[+] Running 1/1
 ✔ Container config_dtree_4_4-dir-cht-upgrade-service-1  Started                                                                       0.0s 
Starting project "config_dtree_4_4". First run takes a while. Will try for up to five minutes.....2024/09/09 14:51:38 [warn] 63#63: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/nginx.conf:37
nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/nginx.conf:37
2024/09/09 14:51:38 [warn] 63#63: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/nginx.conf:38
nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/nginx.conf:38
2024/09/09 14:51:38 [notice] 63#63: signal process started


 -------------------------------------------------------- 

  Success! "config_dtree_4_4" is set up:

    https://192-168-124-76.local-ip.medicmobile.org:10443/ (CHT)
    https://192-168-124-76.local-ip.medicmobile.org:10443/_utils/ (Fauxton)

    Login: medic
    Password: password

 -------------------------------------------------------- 

Start existing project
    ./cht-docker-compose.sh ENV-FILE.env

Stop and keep project:
    ./cht-docker-compose.sh ENV-FILE.env stop

Stop and destroy all project data:
    ./cht-docker-compose.sh ENV-FILE.env destroy

https://docs.communityhealthtoolkit.org/apps/guides/hosting/4.x/app-developer/


 Have a great day!

It always finishes so fast when using sudo to create a new project compared to when I’m not using, so even when updating the cert it requires to use root user privileges for it to show below message, i.e

smhuli@HPx:~/Documents/CHT-Projects/cht-core$ sudo ./scripts/add-local-ip-certs-to-docker-4.x.sh config_dtree_4_4-nginx-1
config_dtree_4_4-nginx-1

If just container name is shown above, a fresh local-ip.medicmobile.org certificate was downloaded fresh local-ip.medicmobile.org.

However, after doing so, the web page remains insecure.
What might be the issue? I prefer using sudo for my initial projects to start faster because I tried to initiate a new project without sudo failing(I tried more than 10 times, but no containers were created).

mrjones · September 9, 2024, 6:07pm

Hi @stephenmhuli !

If you’re running into issues with rootless docker, be sure to check the docs on how to do this.

I’ve got a couple of thoughts on your issue which are broken down here:

ERROR `dial unix /var/run/docker.sock: connect: permission denied`

Semi-related, I recently did a clean install of Ubuntu 24.04 and had an issue where I could create the upgrade service, but none of the other services like API or sentinel would start. In my upgrade service I would see these errors:

Running cmd: docker compose -p 4_clean_clean -f /docker-compose/cht-core.yml -f /docker-compose/couchdb.yml up -d --remove-orphans
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.project%3D4_clean_clean%22%3Atrue%7D%7D&limit=0": dial unix /var/run/docker.sock: connect: permission denied

Error while starting containers Error: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.project%3D4_clean_clean%22%3Atrue%7D%7D&limit=0": dial unix /var/run/docker.sock: connect: permission denied

    at ChildProcess.<anonymous> (/app/src/docker-compose-cli.js:35:25)
    at ChildProcess.emit (node:events:513:28)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:293:12)

If you’re not seeing this error, please continue on to the next section!

The fix for me was to edit the upgrade service compose file in vim ~/.medic/cht-docker/PROJECT_NAME/upgrade-service.yml and change the volumes section to use the non-root accessible docker socket. So I changed this:

volumes:
  - /var/run/docker.sock:/var/run/docker.sock

and instead change it to this:

volumes:
  - /run/user/1000/docker.sock:/var/run/docker.sock

Root vs Rootless

When you run docker with sudo, everything is completely separate from the the docker commands run as your user. The services started, the images downloaded and all other docker actions run as root will be separate from your user.

I suspect that when you run add-local-ip-certs-to-docker-4.x.sh as your user, it’s not seeing the containers that are running as root. Can you try switching to the root user with sudo su - and then try running add-local-ip-certs-to-docker-4.x.sh without sudo?

Docker Helper updates

Docker helper has been updated in Feb, Jun, Jul and Aug. Can you please ensure you’re running the latest docker helper?

Do this by:

cding into your cht-core directory
running git pull origin

Really, the TLS cert should be installed the first time, every time!

stephenmhuli · September 10, 2024, 1:35pm

@mrjones I followed the instructions you provided by editing the upgrade service compose file, adding:

volumes:
  - /run/user/1000/docker.sock:/var/run/docker.sock

This worked just fine thank you for that, but the issue(TLS certs not installed) is still the same when I run:

smhuli@HPx:~/Documents/CHT-Projects/cht-core/scripts$ ./add-local-ip-certs-to-docker-4.x.sh cht_dev_4_5_2-nginx-1
cht_dev_4_5_2-nginx-1

If just container name is shown above, a fresh local-ip.medicmobile.org certificate was downloaded fresh local-ip.medicmobile.org.

I had to run the above command multiple times to ensure I got a secure web page, is there any other possible solution for this?

mrjones · September 11, 2024, 10:18pm

@stephenmhuli - can you confirm which version of the docker helper script you have? If you pull in the latest in the cht-core repo, the MD5 has should match mine (on latest). If you run md5sum scripts/docker-helper-4.x/cht-docker-compose.sh it should be:

57e3371b90f8f6cb31423ae930d4b8f0  scripts/docker-helper-4.x/cht-docker-compose.sh

The script has been updated a number of times to account for TLS certs specifically.

stephenmhuli · September 12, 2024, 6:09am

@mrjones Version looks the same as yours:

smhuli@HPx:~/Documents/CHT-Projects/cht-core/scripts/docker-helper-4.x$ md5sum cht-docker-compose.sh
57e3371b90f8f6cb31423ae930d4b8f0  cht-docker-compose.sh

I also checked the TLS certs script, I hope they are the same:

smhuli@HPx:~/Documents/CHT-Projects/cht-core/scripts$ md5sum ./add-local-ip-certs-to-docker-4.x.sh 
b34a42665a87f6650a01b33fc1677483  ./add-local-ip-certs-to-docker-4.x.sh

This issue doesn’t happen only on my machine, but also for the other three developers installing cht locally as part of capacity building.

Another thing is, every time I run a new cht project I have to comment out or remove the version: '3.9' line in /home/smhuli/.medic/cht-docker/PROJECT-NAME-dir/upgrade-service.yml file to resolve below warning:

smhuli@HPx:~/Documents/CHT-Projects/cht-core/scripts/docker-helper-4.x$ ./cht-docker-compose.sh test_cht.env 

homedir: /home/smhuli/.medic/cht-docker/test_cht-dir
WARN[0000] /home/smhuli/.medic/cht-docker/PROJECT-NAME-dir/upgrade-service.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion

Also, update the volumes as per your previous instructions.

mrjones · September 12, 2024, 5:42pm

@stephenmhuli - thanks for confirming you’re running the same version of Docker Helper. I confirm you have the latest version of add-local-ip-certs-to-docker-4.x.sh as well.

The the attribute 'version' is obsolete warning is annoying, but is otherwise benign. You’re welcome to both ignore or or remove the version: '3.9' from the compose files. They both won’t impact if the compose file works or not.

And I’m so sorry to hear the TLS issue persists! Can you please list out the version of your OS, your version of docker and docker compose, the version of the CHT you’re trying to run? As well, can you please delete all CHT docker images, networks and containers and then list out the exact steps you take to reproduce the error of the TLS cert not being installed? With this, hopefully I’ll be able to reproduce the issue.

Finally, do I understand you correctly that repeatedly running the script fixes the problem such that it eventually works? This is odd! I wonder if curl is failing inside the container to successfully get the certificate?!

You can test this by:

running the add local ip certs script or docker helper
if you see a self signed cert in the browser, find a list of CHT nginx instances with: docker ps --format '{{ .Names }}' | grep '.*-nginx-1'
using the name from the prior step for NAME-OF-CONTAINER, check the md5sums of the files: docker exec NAME-OF-CONTAINER bash -c "md5sum /etc/nginx/private/*.pem"

they should match mine:

a11eab1b4c8fb9a6a1da9c15d59c59ed  /etc/nginx/private/cert.pem
70fe68adb5398299e8664ef40f7fb2b5  /etc/nginx/private/key.pem

If they do not match it means curl failed to download the latest certs for some reason - we can debug this further once I hear back from your findings!

stephenmhuli · September 13, 2024, 10:11am

@mrjones, Thank you for your guidance. I have gathered the requested information regarding my environment and followed the steps to troubleshoot the TLS certificate issue.

Versions:

OS Version:

smhuli@HPx:~/Documents/CHT-Projects/cht-core/scripts$ lsb_release -a
No LSB modules are available.
Distributor ID:   Ubuntu
Description:      Ubuntu 24.04.1 LTS
Release:          24.04
Codename:         noble

Docker Version:

smhuli@HPx:~/Documents/CHT-Projects/cht-core/scripts$ docker --version
Docker version 27.2.1, build 9e34c9b

Docker Compose Version:

smhuli@HPx:~/Documents/CHT-Projects/cht-core/scripts$ docker-compose --version
Docker Compose version v2.22.0

CHT Version: 4.5.2

Steps Taken:

I deleted all CHT Docker images, containers, and networks as per your instructions. I then proceeded with setting up a new CHT project and installing the TLS certificates using the following command:

smhuli@HPx:~/Documents/CHT-Projects/cht-core/scripts/docker-helper-4.x$ ./cht-docker-compose.sh test_cht.env

Output:

homedir: /home/smhuli/.medic/cht-docker/test_cht-dir
[+] Running 1/0
 ✔ Container test_cht-dir-cht-upgrade-service-1  Running                                                                 0.0s 
Starting project "test_cht". First run takes a while. Will try for up to five minutes...

 -------------------------------------------------------- 

  Success! "test_cht" is set up:

    https://192-168-124-76.local-ip.medicmobile.org:10444/ (CHT)
    https://192-168-124-76.local-ip.medicmobile.org:10444/_utils/ (Fauxton)

    Login: medic
    Password: password

 -------------------------------------------------------- 

Start existing project
    ./cht-docker-compose.sh ENV-FILE.env

Stop and keep project:
    ./cht-docker-compose.sh ENV-FILE.env stop

Stop and destroy all project data:
    ./cht-docker-compose.sh ENV-FILE.env destroy

https://docs.communityhealthtoolkit.org/apps/guides/hosting/4.x/app-developer/


 Have a great day!

Next, I identified the running NGINX container:

smhuli@HPx:~/Documents/CHT-Projects/cht-core/scripts$ docker ps --filter "name=nginx" --format '{{ .Names }}'
test_cht-nginx-1

I then ran the local IP certs script as follows:

smhuli@HPx:~/Documents/CHT-Projects/cht-core/scripts$ ./add-local-ip-certs-to-docker-4.x.sh test_cht-nginx-1
test_cht-nginx-1

If just container name is shown above, a fresh local-ip.medicmobile.org certificate was downloaded fresh local-ip.medicmobile.org.

Despite following these steps, the “Connection is not private” error still persists after refreshing the webpage.

MD5 Checksum Verification:

I checked the MD5 sums of the certificate files inside the NGINX container, and they do not match the expected values:

smhuli@HPx:~/Documents/CHT-Projects/cht-core/scripts/docker-helper-4.x$ docker ps --format '{{ .Names }}' | grep '.*-nginx-1'
test_cht-nginx-1
smhuli@HPx:~/Documents/CHT-Projects/cht-core/scripts/docker-helper-4.x$ docker exec test_cht-nginx-1 bash -c "md5sum /etc/nginx/private/*.pem"
c2e16252a3e42892f390529d8217f9a7  /etc/nginx/private/cert.pem
9546303951274e66bbcbccab00db78fe  /etc/nginx/private/key.pem

The MD5 checksums do not match the expected values, which suggests that the curl command may have failed to download the latest certificates correctly.

Please let me know if I should take any further steps to resolve this issue. Thank you for your support.

mrjones · September 13, 2024, 10:23pm

@stephenmhuli - thanks so much for all the testing. I realize the certs often change because they’re regularly renewed, so a better way to test than md5sum is to run openssl in the nginx container. Assuming your nginx container is called test_cht-nginx-1, that would be:

docker exec test_cht-nginx-1 bash -c "openssl  x509 -noout -text -in /etc/nginx/private/cert.pem  | head -n 15"

If you see these lines:

Issuer: C = US, O = Let's Encrypt, CN = R11
Subject: CN = *.local-ip.medicmobile.org

then you have the valid local-ip cert. If you see these lines, then you have the self signed cert:

Issuer: C = US, ST = Oregon, L = Portland, O = Medic Mobile, 
CN = *.medicmobile.org,

More importantly, I’ve managed to reproduced your issue - but only some of the time! Sometimes it fails (on the local cafe’s wifi, in the nginx container), sometimes it works (at my home office’s wifi, inside OR outside the container). Quite frustrating!

To see if your curl calls are failing, assuming your nginx container is still called test_cht-nginx-1, run this code:

docker exec -it test_cht-nginx-1 bash -c "curl -v  https://local-ip.medicmobile.org/fullchain"

You should see the output of the full chain. Instead I get this error:

* processing: https://local-ip.medicmobile.org/fullchaina
*   Trying 15.188.129.97:443...
* connect to 15.188.129.97 port 443 failed: Connection refused
* Failed to connect to local-ip.medicmobile.org port 443 after 255 ms: Couldn't connect to server
* Closing connection
curl: (7) Failed to connect to local-ip.medicmobile.org port 443 after 255 ms: Couldn't connect to server

A successful curl command looks like this:

 curl https://local-ip.medicmobile.org/fullchain   
-----BEGIN CERTIFICATE-----
MIIFBTCCA+2gAwIBAgISA2gfNrwE/rDT1VxEo298J6XCMA0GCSqGSIb3DQEBCwUA
MDMxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQwwCgYDVQQD
[CUT SHORT FOR BREVITY]
R0UIYy3kE56zEI3hJmJfaHQmRZ4XpPQmrqUe/g3MWTrHMKB4auNZ3HFrPW1ao9mU
cPwMPfpU2GI8aHzSOnMYxDoIyeegxUJ+B9mVMnleM2JLLw0Q+/GFNJU=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFBjCCAu6gAwIBAgIRAIp9PhPWLzDvI4a9KQdrNPgwDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
[CUT SHORT FOR BREVITY]
6FlWkWYtbt4pgdamlwVeZEW+LM7qZEJEsMNPrfC03APKmZsJgpWCDWOKZvkZcvjV
uYkQ4omYCTX5ohy+knMjdOmdH9c7SpqEWBDC86fiNex+O0XOMEZSa8DA
-----END CERTIFICATE-----

As a work around, it may work to download them on your workstation, copy them into nginx and then reload nginx. Here’s how I did that, again still assuming your container is called test_cht-nginx-1 :

download the key and fullchain on your workstation (not inside nginx container):

curl -o fullchain https://local-ip.medicmobile.org/fullchain
curl -o key https://local-ip.medicmobile.org/key

copy them into your nginx container:

docker cp key test_cht-nginx-1:/etc/nginx/private/key.pem                                            
docker cp fullchain test_cht-nginx-1:/etc/nginx/private/cert.pem

reload nginx:

docker exec -it test_cht-nginx-1 bash -c "nginx -s reload"

I’ll be out of office and back Tue Sep 17th when I’ll pick up working on this again. I hope the work around I’ve provided are successful!

stephenmhuli · September 14, 2024, 5:10am

Hi @mrjones, Thank you for your detailed instructions and support. I followed the steps you provided, and I was able to resolve the TLS issue successfully. Below are the exact steps I took:

1. Verified Certificate with OpenSSL
I first checked the certificate inside the nginx container:

docker exec test_cht-nginx-1 bash -c "openssl x509 -noout -text -in /etc/nginx/private/cert.pem | head -n 15"

The output confirmed the current certificate details:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1a:9c:65:a2:56:ce:5b:b0:0b:2b:dc:17:1b:ba:74:6b:db:54:30:f4
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: emailAddress = domains@medic.org, C = US, ST = California, L = San_Francisco, O = medic, OU = Information_Security, CN = test-nginx.dev.medicmobile.org

2. Tested Curl Call for Fullchain
I then ran the curl command inside the container to check the fullchain:

docker exec -it test_cht-nginx-1 bash -c "curl -v https://local-ip.medicmobile.org/fullchain"

However, this failed with the error:

curl: (6) Could not resolve host: local-ip.medicmobile.org

3. Downloaded Key and Fullchain on Workstation
I created a directory on my workstation and downloaded the required certificates:

mkdir keyfullchain-workstation
cd keyfullchain-workstation
curl -o fullchain https://local-ip.medicmobile.org/fullchain
curl -o key https://local-ip.medicmobile.org/key

4. Copied Certificates to Nginx Container
I then copied the downloaded certificates into the nginx container:

docker cp key test_cht-nginx-1:/etc/nginx/private/key.pem
docker cp fullchain test_cht-nginx-1:/etc/nginx/private/cert.pem

5. Reloaded nginx
Finally, I reloaded the nginx server to apply the changes:

docker exec -it test_cht-nginx-1 bash -c "nginx -s reload"

After following these steps, the TLS issue was resolved.

Thank you once again @mrjones for your great support!

mrjones · September 14, 2024, 9:50pm

@stephenmhuli - yay! I’m so happy that worked.

As well, thanks again for reporting the issue with loading the certificates. I’m so sorry it didn’t work for you, but by raising an issue here on the forum allows us to help you and for others to find this issue as well.

I’ll work to resolve this in the docker helper itself in a future release!

mrjones · September 26, 2024, 9:40pm

@stephenmhuli et al. - We’ve just released an upgrade version of the Docker Helper today. You can get it by downloading it directly or by using git to pull the latest changes into your CHT Core repo you have checked out. The Add Certificate script was also updated.

I’d love to know if it solves the problems you were facing without any additional work by you of copying certificates around!

mrjones · September 30, 2024, 2:38am

Hello again! I’m posting another follow up that the latest release of CHT Upgrade Service fixes the warning about version: the attribute 'version' is obsolete, it will be ignored, please remove it to avoid potential confusion

the Docker Helper pulls in the latest compose files any time you create a new project, so only new projects will get the fixt - but older versions of Docker Helper will do this correctly (but you should totally upgrade per my prior comment).