Using Docker helper with the latest version of CHT v4.5.1 results in a 502 Bad Gateway error and the couchdb docker container (public.ecr.aws/medic/cht-couchdb:4.5.1) keeps restarting. It was working fine before with v4.2.4. I am using an M1 chip Macbook Air locally and following this documentation.
Launching CHT using Docker helper with the latest version (4.5.1) results in a 502 Bad Gateway Error
Hi @sookwalinga
Can you please share logs from your couchdb docker container? It’s impossible to know what the problem is without some hints. Thanks!
Hi @diana,
2024-02-05 12:26:50 Waiting for cht couchdb
2024-02-05 12:26:51 qemu: uncaught target signal 11 (Segmentation fault) - core dumped
2024-02-05 12:26:51 Segmentation fault
2024-02-05 12:26:58 /docker-entrypoint.sh: line 38: warning: command substitution: ignored null byte in input
2024-02-05 12:26:58 /docker-entrypoint.sh: line 38: [: too many arguments
2024-02-05 12:26:58 /docker-entrypoint.sh: line 48: warning: command substitution: ignored null byte in input
2024-02-05 12:26:58 /docker-entrypoint.sh: line 48: [: too many arguments
Thanks for sharing the logs @sookwalinga
That segmentation fault seems suspicious, it looks like it’s something about docker compose and MacOs. I’m linking some resources I found with an online search:
I’m not fluent in MacOs at all, so I’m hoping MacOs users can pitch in to help.
@Esther_Moturi or @andrablaj - I know you both have Apple Silicon Macs - are you able to reproduce @sookwalinga issue with Docker Helper and CHT Core 4.5.1?
I confirm I encounter the same error as @sookwalinga (502 Bad Gateway) on my Mac machine with Apple Silicon chip, by using the master branch of cht-core.
CouchDB logs below:
2024-02-05 17:28:46 Segmentation fault
2024-02-05 17:29:46 /docker-entrypoint.sh: line 38: warning: command substitution: ignored null byte in input
2024-02-05 17:29:46 /docker-entrypoint.sh: line 38: [: too many arguments
2024-02-05 17:29:46 /docker-entrypoint.sh: line 48: warning: command substitution: ignored null byte in input
2024-02-05 17:29:46 /docker-entrypoint.sh: line 48: [: too many arguments
2024-02-05 17:29:46 Waiting for cht couchdb
2024-02-05 17:29:47 Segmentation fault
Yikes! That’s no good to hear @andrablaj - but good to confirm. I’ve re-opened and updated the issue @sookwalinga created to track this.
@andrablaj - do you have a moment to check where the regression came in? If you could try and run a Docker Helper instance on 4.3.0 and 4.4.0 we can hopefully narrow it down. So far we have:
- 4.5.1 - fails
- 4.4.0 - ?
- 4.3.0 - ?
- 4.2.4 - succeeds
My guess is one of the upgrade in there caused it to fail. I’ll check the updates to the compose and Dockerfiles to see if we’re pulling in a new docker image upstream.
Thanks @mrjones! I am testing the suggested releases one-by-one.
I started with 4.2.4, and I have a clarifying question. When running Docker Helper with 4.2.4, I still get the 502 error, and I see that the images downloaded are tagged with x:4.5.1. See a screenshot of Docker Desktop below:
Is that expected? Shouldn’t that be x:4.2.4?
Ah - right - sorry for the confusion @andrablaj ! This isn’t about which version of Docker Helper, but which version of CHT Core, thus which images we’re using, thus which upstream images we’re using. So when prompted Do you want to run the latest CHT Core version (4.5.1) [Y/n]? answer n. When you do, you can then select the CHT Core version to run as shown below.
➜ docker-helper-4.x git:(master) ✗ ./cht-docker-compose.sh
Would you like to initialize a new project [y/N]? y
Do you want to run the latest CHT Core version (4.5.1) [Y/n]? n
Which version to you want to run? (ctrl + c to quit)
1) 4.0.0 72) 8433-add-telemetry-events-to-contact-forms
2) 4.0.1 73) 8462-export-user-devices-details-to-csv
3) 4.1.0 74) 8462-jkuester
4) 4.1.0-FR-supervisor-chw-create-beta.1 75) 8462-jkuester-again
5) 4.1.0-FR-supervisor-chw-create-beta.4 76) 8462-raw-query
6) 4.1.1 77) 8462-smart-query
7) 4.1.2 78) 8501-Remove_upgrade_e2e_temp_selector
8) 4.2.0 79) 8513-update-target-component
9) 4.2.1 80) 8585-fix-Asterisk-for-required-field-in-enketo
10) 4.2.2 81) 8621-update-wdio
11) 4.2.3 82) 8621-update-wdio-test
12) 4.2.4 83) 8644-concurrent-haproxy-healthcheck
13) 4.3.0 84) 8669-enketo-label-fix
14) 4.3.1 85) 8671-fix-test
15) 4.3.2 86) 8679_bump_version
16) 4.4.0 87) 8686-update-target-ui-to-match-figma
17) 4.4.0-beta.1 88) 8689-low-hanging-fruit
18) 4.4.1 89) 8692-flaky-navigation-hamburguer-menu-test
19) 4.4.1-FR-barcode 90) 8697-consolidate-enketo-common-selectors
20) 4.4.1-FR-barcode-beta.1 91) 8701-clean-and-organize-enketo-folder
21) 4.4.1-FR-enketo-7 92) 8706-split-integration-test-job
22) 4.4.1-FR-enketo-7-beta.1 93) 8707-compose-links-as-comment
23) 4.4.1-FR-enketo-7-beta.2 94) 8707-make-compose-urls-more-reliable-2
24) 4.4.1-FR-enketo-7-with-async-compute 95) 8708-merge-docker-unit-test-build
25) 4.4.1-FR-enketo-7-with-async-compute-beta.1 96) 8727-upgrade-for-macos
26) 4.4.1-FR-enketo-7-with-async-compute-beta.2 97) 8730-fix-actionbar-permissions
27) 4.4.2 98) 8743-couch-bump
28) 4.4.2-beta.1 99) 8745-fix-task-form-context-4.4.x
29) 4-4-cares 100) 8745-for-4.5.x
30) 4.4.x 101) 8757-remove-standard-config
Oh! Gotcha. yes, it was introduced in master of CHT Core on Aug 31, 2023. You should just need to git pull origin master in cht-core repo to get the latest. Sorry again!
Got it, thanks @mrjones! I tested several releases as below:
4.2.4 - Worked
4.3.0 - Worked
4.3.1 - Worked
4.3.2 - Worked
4.4.0 - Failed
4.4.0 CouchDB error logs similar to 4.5.1:
2024-02-05 19:50:19 /docker-entrypoint.sh: line 38: warning: command substitution: ignored null byte in input
2024-02-05 19:50:19 /docker-entrypoint.sh: line 38: [: too many arguments
2024-02-05 19:50:19 /docker-entrypoint.sh: line 48: warning: command substitution: ignored null byte in input
2024-02-05 19:50:19 /docker-entrypoint.sh: line 48: [: too many arguments
2024-02-05 19:50:19 Waiting for cht couchdb
2024-02-05 19:50:20 Segmentation fault
4.4.0 nginx container outputs errors too:
2024-02-05 19:50:18 Launching Nginx
2024-02-05 19:50:18 2024/02/05 19:50:18 [warn] 1#1: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/nginx.conf:37
2024-02-05 19:50:18 nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/nginx.conf:37
2024-02-05 19:50:18 2024/02/05 19:50:18 [warn] 1#1: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/nginx.conf:38
2024-02-05 19:50:18 nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/nginx.conf:38
2024-02-05 19:50:18 2024/02/05 19:50:18 [emerg] 1#1: SSL_CTX_use_PrivateKey("/etc/nginx/private/key.pem") failed (SSL: error:05800074:x509 certificate routines::key values mismatch)
2024-02-05 19:50:18 nginx: [emerg] SSL_CTX_use_PrivateKey("/etc/nginx/private/key.pem") failed (SSL: error:05800074:x509 certificate routines::key values mismatch)
2024-02-05 19:50:44 Running SSL certificate checks
2024-02-05 19:50:44 self signed SSL cert already exists.
2024-02-05 19:50:44 Launching Nginx
2024-02-05 19:50:44 2024/02/05 19:50:44 [warn] 1#1: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/nginx.conf:37
2024-02-05 19:50:44 nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/nginx.conf:37
2024-02-05 19:50:44 2024/02/05 19:50:44 [warn] 1#1: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/nginx.conf:38
2024-02-05 19:50:44 nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/nginx.conf:38
2024-02-05 19:50:44 2024/02/05 19:50:44 [emerg] 1#1: SSL_CTX_use_PrivateKey("/etc/nginx/private/key.pem") failed (SSL: error:05800074:x509 certificate routines::key values mismatch)
2024-02-05 19:50:44 nginx: [emerg] SSL_CTX_use_PrivateKey("/etc/nginx/private/key.pem") failed (SSL: error:05800074:x509 certificate routines::key values mismatch)
2024-02-05 19:50:44 /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
2024-02-05 19:50:44 /docker-entrypoint.sh: Configuration complete; ready for start up
4.4.1 - Failed
4.4.1 CouchDB error logs:
2024-02-05 20:06:12 /docker-entrypoint.sh: line 38: warning: command substitution: ignored null byte in input
2024-02-05 20:06:12 /docker-entrypoint.sh: line 38: [: too many arguments
2024-02-05 20:06:12 /docker-entrypoint.sh: line 48: warning: command substitution: ignored null byte in input
2024-02-05 20:06:12 /docker-entrypoint.sh: line 48: [: too many arguments
2024-02-05 20:06:12 Waiting for cht couchdb
2024-02-05 20:06:12 Segmentation fault
To conclude, it looks like the problem was introduced with 4.4.0 (maybe related to CouchDB 3 upgrade?).
Additionally, it seems like the not secure warning was introduced with 4.4.1 (couldn’t see it in 4.4.0 as the container didn’t start properly, but I guess it was a 4.4.0 change).
1 Like
Thanks all for looking into this, at first I thought I was just doing something wrong on my end, I appreciate all this help.
1 Like