Error 502 when running local CHT instance

Esther_Moturi · January 19, 2023, 9:28am

Hallo,

I am getting error 502 Bad Gateway when I try to follow this guide on how to install CHT local instance. I had run localhost earlier today and it worked fine. I am not sure what the issue could be.

diana · January 19, 2023, 9:38am

Hi @Esther_Moturi

Can you please run docker ps and share the output?
I’m looking to get logs from some of the services, and want to see what is actually running on your machine. Thanks!

Esther_Moturi · January 19, 2023, 9:53am

Here is the output

diana · January 19, 2023, 9:56am

Thanks for sharing.

Can you please run docker logs cht_nginx_1 -n 50 and share the output?

Esther_Moturi · January 19, 2023, 9:58am

esthermoturi@Esthers-MacBook-Pro cht-local-setup % docker logs cht_nginx_1 -n 50
2023/01/19 09:12:11 [error] 70#70: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://172.19.0.6:5988/favicon.ico", host: "localhost", referrer: "https://localhost/"
172.19.0.1 - - [19/Jan/2023:09:12:11 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "https://localhost/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"
2023/01/19 09:12:15 [error] 70#70: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.19.0.6:5988/", host: "localhost", referrer: "https://docs.communityhealthtoolkit.org/"
172.19.0.1 - - [19/Jan/2023:09:12:15 +0000] "GET / HTTP/1.1" 502 559 "https://docs.communityhealthtoolkit.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"
2023/01/19 09:12:15 [error] 70#70: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://172.19.0.6:5988/favicon.ico", host: "localhost", referrer: "https://localhost/"
172.19.0.1 - - [19/Jan/2023:09:12:15 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "https://localhost/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: IPv6 listen already enabled
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/server.conf.template to /etc/nginx/conf.d/server.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/ssl-install.sh
Running SSL certificate checks
CERTIFICATE MODE = SELF_SIGNED
self signed SSL cert already exists.
Launching Nginx
/docker-entrypoint.sh: Configuration complete; ready for start up
2023/01/19 09:14:25 [emerg] 67#67: io_setup() failed (38: Function not implemented)
2023/01/19 09:14:25 [emerg] 69#69: io_setup() failed (38: Function not implemented)
2023/01/19 09:14:25 [emerg] 73#73: io_setup() failed (38: Function not implemented)
2023/01/19 09:14:25 [emerg] 71#71: io_setup() failed (38: Function not implemented)
2023/01/19 09:15:49 [error] 67#67: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.19.0.3:5988/", host: "localhost", referrer: "https://docs.communityhealthtoolkit.org/"
172.19.0.1 - - [19/Jan/2023:09:15:49 +0000] "GET / HTTP/1.1" 502 559 "https://docs.communityhealthtoolkit.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"
2023/01/19 09:15:49 [error] 67#67: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://172.19.0.3:5988/favicon.ico", host: "localhost", referrer: "https://localhost/"
172.19.0.1 - - [19/Jan/2023:09:15:49 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "https://localhost/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"
2023/01/19 09:29:38 [error] 69#69: *8 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.19.0.3:5988/", host: "localhost", referrer: "https://docs.communityhealthtoolkit.org/"
172.19.0.1 - - [19/Jan/2023:09:29:38 +0000] "GET / HTTP/1.1" 502 559 "https://docs.communityhealthtoolkit.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"
2023/01/19 09:29:38 [error] 69#69: *8 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://172.19.0.3:5988/favicon.ico", host: "localhost", referrer: "https://localhost/"
172.19.0.1 - - [19/Jan/2023:09:29:38 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "https://localhost/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: IPv6 listen already enabled
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/server.conf.template to /etc/nginx/conf.d/server.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/ssl-install.sh
Running SSL certificate checks
CERTIFICATE MODE = SELF_SIGNED
self signed SSL cert already exists.
Launching Nginx
/docker-entrypoint.sh: Configuration complete; ready for start up
2023/01/19 09:49:20 [emerg] 71#71: io_setup() failed (38: Function not implemented)
2023/01/19 09:49:20 [emerg] 67#67: io_setup() failed (38: Function not implemented)
2023/01/19 09:49:20 [emerg] 69#69: io_setup() failed (38: Function not implemented)
2023/01/19 09:49:20 [emerg] 70#70: io_setup() failed (38: Function not implemented)
2023/01/19 09:49:57 [error] 69#69: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.19.0.2:5988/", host: "localhost", referrer: "https://docs.communityhealthtoolkit.org/"
172.19.0.1 - - [19/Jan/2023:09:49:57 +0000] "GET / HTTP/1.1" 502 559 "https://docs.communityhealthtoolkit.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"
2023/01/19 09:49:57 [error] 69#69: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://172.19.0.2:5988/favicon.ico", host: "localhost", referrer: "https://localhost/"
172.19.0.1 - - [19/Jan/2023:09:49:57 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "https://localhost/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"
esthermoturi@Esthers-MacBook-Pro cht-local-setup %

diana · January 19, 2023, 10:21am

Thanks @Esther_Moturi

It looks like your API service is not responding.
Can you please run docker logs cht_api_1 -n 50 and share the output?

Esther_Moturi · January 19, 2023, 10:31am

'</body></html>\n'

}
StatusCodeError: 503 - “

503 Service Unavailable

\nNo server is available to handle this request.\n\n”
at new StatusCodeError (/api/node_modules/request-promise-core/lib/errors.js:32:15)
at Request.plumbing.callback (/api/node_modules/request-promise-core/lib/plumbing.js:104:33)
at Request.RP$callback [as _callback] (/api/node_modules/request-promise-core/lib/plumbing.js:46:31)
at Request.self.callback (/api/node_modules/request/request.js:185:22)
at Request.emit (node:events:513:28)
at Request. (/api/node_modules/request/request.js:1154:10)
at Request.emit (node:events:513:28)
at IncomingMessage. (/api/node_modules/request/request.js:1076:12)
at Object.onceWrapper (node:events:627:28)
at IncomingMessage.emit (node:events:525:35) {
statusCode: 503,
error: ‘

503 Service Unavailable

\n’ +
‘No server is available to handle this request.\n’ +
‘\n’
}
StatusCodeError: 503 - “

503 Service Unavailable

\nNo server is available to handle this request.\n\n”
at new StatusCodeError (/api/node_modules/request-promise-core/lib/errors.js:32:15)
at Request.plumbing.callback (/api/node_modules/request-promise-core/lib/plumbing.js:104:33)
at Request.RP$callback [as _callback] (/api/node_modules/request-promise-core/lib/plumbing.js:46:31)
at Request.self.callback (/api/node_modules/request/request.js:185:22)
at Request.emit (node:events:513:28)
at Request. (/api/node_modules/request/request.js:1154:10)
at Request.emit (node:events:513:28)
at IncomingMessage. (/api/node_modules/request/request.js:1076:12)
at Object.onceWrapper (node:events:627:28)
at IncomingMessage.emit (node:events:525:35) {
statusCode: 503,
error: ‘

503 Service Unavailable

\n’ +
‘No server is available to handle this request.\n’ +
‘\n’
}
StatusCodeError: 503 - “

503 Service Unavailable

\nNo server is available to handle this request.\n\n”
at new StatusCodeError (/api/node_modules/request-promise-core/lib/errors.js:32:15)
at Request.plumbing.callback (/api/node_modules/request-promise-core/lib/plumbing.js:104:33)
at Request.RP$callback [as _callback] (/api/node_modules/request-promise-core/lib/plumbing.js:46:31)
at Request.self.callback (/api/node_modules/request/request.js:185:22)
at Request.emit (node:events:513:28)
at Request. (/api/node_modules/request/request.js:1154:10)
at Request.emit (node:events:513:28)
at IncomingMessage. (/api/node_modules/request/request.js:1076:12)
at Object.onceWrapper (node:events:627:28)
at IncomingMessage.emit (node:events:525:35) {
statusCode: 503,
error: ‘

503 Service Unavailable

\n’ +
‘No server is available to handle this request.\n’ +
‘\n’
}

Esther_Moturi · January 19, 2023, 10:49am

@diana This is the error I get when I go to the couch_url indicated in docker

diana · January 19, 2023, 1:42pm

Hi @Esther_Moturi

It’s not expected that CouchDb is directly accessible from the host, so getting that error in the browser is normal.
According to the logs you shared above, it looks like haproxy is unable to make a connection with Couchdb.

Can you please check the logs of both the haproxy container and the couchdb container and share if you see any errors?

Thanks!

Esther_Moturi · January 19, 2023, 6:35pm

Hi @diana I am getting this for haproxy logs : Unrecognized input header: 99 and this for couchdb: No couchdb end point Found at http://medic:password@couchdb:5984
Waiting for cht couchdb

diana · January 19, 2023, 6:47pm

Cool, so we’ve likely narrowed it down to CouchDb not responding, although I am seeing the container running in your docker ps call.
Would you mind repeating the docker logs command, but this time for cht_couchdb_1 please?

Esther_Moturi · January 19, 2023, 7:06pm

@diana

mrjones · January 19, 2023, 10:08pm

@Esther_Moturi - we were privately chatting about how to restart your containers in case that might help. Here’s what I shared based on the set up in the local-setup docs you’re using:

ctrl + c in the terminal window where you first ran docker. It should have a bunch of upgrade-cht-upgrade-service-1 log entries. Since we don’t start the service detached (-d ), you need stop the upgrade service as a first step
docker kill $(docker ps --all --filter "name=cht_" --quiet) - This forcibly kills all containers that start with cht_, which are all the ones we just started
cd ~/cht-local-setup/upgrade/ - make sure you’re still in the right directory
docker-compose up - restart the services we just stopped

Side note - docker-compose down won’t work to stop all related service. This because of the way CHT Upgrade Service is architected. It is intentionally meant to be autonomous from the CHT instances it is managing upgrades for.

mrjones · March 21, 2023, 11:54pm

A post was split to a new topic: Error 502 on local development instance