CHT Instance crashing and failing to sync

Technical Support
13

Hi Everyone
Yesterday, I noted that some services were not running, we are still down as below:
docker exec -it medic-os /boot/supervisor-inspect
Package ‘horticulturalist’:
Service ‘horticulturalist’:
Status: Up
Up: 2 seconds, Restarts: 227
Attributes: watched, running, expected
Service PID: 54161, Supervisor PID: 344

Package ‘medic-api’:
Service ‘medic-api’:
Status: Failure
Up: 4 seconds, Restarts: 213
Attributes: watched, expected
Service PID: None, Supervisor PID: 387

Package ‘medic-core’:
Service ‘couchdb’:
Status: Up
Up: 2 seconds, Restarts: 279
Attributes: watched, running, expected
Service PID: 54192, Supervisor PID: 428
Service ‘nginx’:
Status: Up
Up: 1238 seconds, Restarts: 1
Attributes: watched, running, expected
Service PID: 4823, Supervisor PID: 475
Service ‘openssh’:
Status: Disabled
Up: 0 seconds, Restarts: 0
Attributes: disabled
Service PID: None, Supervisor PID: None

Package ‘medic-couch2pg’:
Service ‘medic-couch2pg’:
Status: Disabled
Up: 0 seconds, Restarts: 270
Attributes: disabled
Service PID: None, Supervisor PID: None

Package ‘medic-rdbms’:
Service ‘postgresql’:
Status: Disabled
Up: 0 seconds, Restarts: 338
Attributes: disabled
Service PID: None, Supervisor PID: None

Package ‘medic-sentinel’:
Service ‘medic-sentinel’:
Status: Up
Up: 0 seconds, Restarts: 237
Attributes: watched, running, expected
Service PID: 54774, Supervisor PID: 691

Package ‘system-services’:
Service ‘cron’:
Status: Up
Up: 1307 seconds, Restarts: 0
Attributes: watched, running, expected
Service PID: 757, Supervisor PID: 740
Service ‘syslog’:
Status: Up
Up: 1307 seconds, Restarts: 0
Attributes: watched, running, expected
Service PID: 784, Supervisor PID: 772

docker exec -it medic-os head -n70 /srv/storage/medic-api/logs/medic-api.log
[2023-01-09 22:02:45] 2023-01-09 22:02:45 INFO: Running server checks…
[2023-01-09 22:02:45] Node Environment Options: ‘–max_old_space_size=8192’
[2023-01-09 22:02:45] Node Version: 8.11.4 in production mode
[2023-01-09 22:02:45] COUCH_URL http://haproxy:5984/medic
[2023-01-09 22:02:45] COUCH_NODE_NAME couchdb@127.0.0.1
Expected a 401 when accessing db without authentication.
Instead we got a 503
[2023-01-09 22:02:48] 2023-01-09 22:02:48 ERROR: Fatal error initialising medic-api
[2023-01-09 22:02:48] 2023-01-09 22:02:48 ERROR: { Error: CouchDB security seems to be misconfigured, see: cht-core/DEVELOPMENT.md at master · medic/cht-core · GitHub
[2023-01-09 22:02:48] at ClientRequest.net.get (/srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/@medic/server-checks/src/checks.js:63:16)
[2023-01-09 22:02:48] at Object.onceWrapper (events.js:315:30)
[2023-01-09 22:02:48] at emitOne (events.js:116:13)
[2023-01-09 22:02:48] at ClientRequest.emit (events.js:211:7)
[2023-01-09 22:02:48] at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:543:21)
[2023-01-09 22:02:48] at HTTPParser.parserOnHeadersComplete (_http_common.js:112:17)
[2023-01-09 22:02:48] at Socket.socketOnData (_http_client.js:440:20)
[2023-01-09 22:02:48] at emitOne (events.js:116:13)
[2023-01-09 22:02:48] at Socket.emit (events.js:211:7)
[2023-01-09 22:02:48] at addChunk (_stream_readable.js:263:12)
[2023-01-09 22:02:48] [stack]: ‘Error: CouchDB security seems to be misconfigured, see: https://github.com/medic/cht-core/blob/master/DEVELOPMENT.md#enabling-a-secure-couchdb\n at ClientRequest.net.get (/srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/@medic/server-checks/src/checks.js:63:16)\n at Object.onceWrapper (events.js:315:30)\n at emitOne (events.js:116:13)\n at ClientRequest.emit (events.js:211:7)\n at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:543:21)\n at HTTPParser.parserOnHeadersComplete (_http_common.js:112:17)\n at Socket.socketOnData (_http_client.js:440:20)\n at emitOne (events.js:116:13)\n at Socket.emit (events.js:211:7)\n at addChunk (_stream_readable.js:263:12)’,
[2023-01-09 22:02:48] [message]: ‘CouchDB security seems to be misconfigured, see: cht-core/DEVELOPMENT.md at master · medic/cht-core · GitHub’ }
[2023-01-09 22:02:51] 2023-01-09 22:02:51 INFO: Running server checks…
[2023-01-09 22:02:51] Node Environment Options: ‘–max_old_space_size=8192’
[2023-01-09 22:02:51] Node Version: 8.11.4 in production mode
[2023-01-09 22:02:51] COUCH_URL http://haproxy:5984/medic
[2023-01-09 22:02:51] COUCH_NODE_NAME couchdb@127.0.0.1
[2023-01-09 22:02:54] 2023-01-09 22:02:54 ERROR: Error watching sentinel changes, restarting: { FetchError: invalid json response body at http://haproxy:5984/medic-sentinel/ reason: Unexpected token < in JSON at position 0
[2023-01-09 22:02:54] at /srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/node-fetch/lib/index.js:272:32
[2023-01-09 22:02:54] at
[2023-01-09 22:02:54] at process._tickCallback (internal/process/next_tick.js:188:7)
[2023-01-09 22:02:54] message: ‘invalid json response body at http://haproxy:5984/medic-sentinel/ reason: Unexpected token < in JSON at position 0’,
[2023-01-09 22:02:54] type: ‘invalid-json’,
[2023-01-09 22:02:54] [stack]: ‘FetchError: invalid json response body at http://haproxy:5984/medic-sentinel/ reason: Unexpected token < in JSON at position 0\n at /srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/node-fetch/lib/index.js:272:32\n at \n at process._tickCallback (internal/process/next_tick.js:188:7)’ }
Expected a 401 when accessing db without authentication.
Instead we got a 503
[2023-01-09 22:02:54] 2023-01-09 22:02:54 ERROR: Fatal error initialising medic-api
[2023-01-09 22:02:54] 2023-01-09 22:02:54 ERROR: { Error: CouchDB security seems to be misconfigured, see: cht-core/DEVELOPMENT.md at master · medic/cht-core · GitHub
[2023-01-09 22:02:54] at ClientRequest.net.get (/srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/@medic/server-checks/src/checks.js:63:16)
[2023-01-09 22:02:54] at Object.onceWrapper (events.js:315:30)
[2023-01-09 22:02:54] at emitOne (events.js:116:13)
[2023-01-09 22:02:54] at ClientRequest.emit (events.js:211:7)
[2023-01-09 22:02:54] at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:543:21)
[2023-01-09 22:02:54] at HTTPParser.parserOnHeadersComplete (_http_common.js:112:17)
[2023-01-09 22:02:54] at Socket.socketOnData (_http_client.js:440:20)
[2023-01-09 22:02:54] at emitOne (events.js:116:13)
[2023-01-09 22:02:54] at Socket.emit (events.js:211:7)
[2023-01-09 22:02:54] at addChunk (_stream_readable.js:263:12)
[2023-01-09 22:02:54] [stack]: ‘Error: CouchDB security seems to be misconfigured, see: https://github.com/medic/cht-core/blob/master/DEVELOPMENT.md#enabling-a-secure-couchdb\n at ClientRequest.net.get (/srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/@medic/server-checks/src/checks.js:63:16)\n at Object.onceWrapper (events.js:315:30)\n at emitOne (events.js:116:13)\n at ClientRequest.emit (events.js:211:7)\n at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:543:21)\n at HTTPParser.parserOnHeadersComplete (_http_common.js:112:17)\n at Socket.socketOnData (_http_client.js:440:20)\n at emitOne (events.js:116:13)\n at Socket.emit (events.js:211:7)\n at addChunk (_stream_readable.js:263:12)’,
[2023-01-09 22:02:54] [message]: ‘CouchDB security seems to be misconfigured, see: cht-core/DEVELOPMENT.md at master · medic/cht-core · GitHub’ }
[2023-01-09 22:02:57] 2023-01-09 22:02:57 INFO: Running server checks…
[2023-01-09 22:02:57] Node Environment Options: ‘–max_old_space_size=8192’
[2023-01-09 22:02:57] Node Version: 8.11.4 in production mode
[2023-01-09 22:02:57] COUCH_URL http://haproxy:5984/medic
[2023-01-09 22:02:57] COUCH_NODE_NAME couchdb@127.0.0.1
[2023-01-09 22:03:00] 2023-01-09 22:03:01 ERROR: Error watching sentinel changes, restarting: { FetchError: invalid json response body at http://haproxy:5984/medic-sentinel/ reason: Unexpected token < in JSON at position 0
[2023-01-09 22:03:00] at /srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/node-fetch/lib/index.js:272:32
[2023-01-09 22:03:01] at
[2023-01-09 22:03:01] at process._tickCallback (internal/process/next_tick.js:188:7)
[2023-01-09 22:03:01] message: ‘invalid json response body at http://haproxy:5984/medic-sentinel/ reason: Unexpected token < in JSON at position 0’,
[2023-01-09 22:03:01] type: ‘invalid-json’,
[2023-01-09 22:03:01] [stack]: ‘FetchError: invalid json response body at http://haproxy:5984/medic-sentinel/ reason: Unexpected token < in JSON at position 0\n at /srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/node-fetch/lib/index.js:272:32\n at \n at process._tickCallback (internal/process/next_tick.js:188:7)’ }
Expected a 401 when accessing db without authentication.
Instead we got a 503
[2023-01-09 22:03:01] 2023-01-09 22:03:01 ERROR: Fatal error initialising medic-api
[2023-01-09 22:03:01] 2023-01-09 22:03:01 ERROR: { Error: CouchDB security seems to be misconfigured, see: cht-core/DEVELOPMENT.md at master · medic/cht-core · GitHub
[2023-01-09 22:03:01] at ClientRequest.net.get (/srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/@medic/server-checks/src/checks.js:63:16)
[2023-01-09 22:03:01] at Object.onceWrapper (events.js:315:30)
[2023-01-09 22:03:01] at emitOne (events.js:116:13)
[2023-01-09 22:03:01] at ClientRequest.emit (events.js:211:7)
[2023-01-09 22:03:01] at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:543:21)
root@boda-project:/var/community-testing-config-test# docker exec -it medic-os head -n70 /srv/storage/medic-api/logs/medic-api.log
[2023-01-09 22:02:45] 2023-01-09 22:02:45 INFO: Running server checks…
[2023-01-09 22:02:45] Node Environment Options: ‘–max_old_space_size=8192’
[2023-01-09 22:02:45] Node Version: 8.11.4 in production mode
[2023-01-09 22:02:45] COUCH_URL http://haproxy:5984/medic
[2023-01-09 22:02:45] COUCH_NODE_NAME couchdb@127.0.0.1
Expected a 401 when accessing db without authentication.
Instead we got a 503
[2023-01-09 22:02:48] 2023-01-09 22:02:48 ERROR: Fatal error initialising medic-api
[2023-01-09 22:02:48] 2023-01-09 22:02:48 ERROR: { Error: CouchDB security seems to be misconfigured, see: cht-core/DEVELOPMENT.md at master · medic/cht-core · GitHub
[2023-01-09 22:02:48] at ClientRequest.net.get (/srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/@medic/server-checks/src/checks.js:63:16)
[2023-01-09 22:02:48] at Object.onceWrapper (events.js:315:30)
[2023-01-09 22:02:48] at emitOne (events.js:116:13)
[2023-01-09 22:02:48] at ClientRequest.emit (events.js:211:7)
[2023-01-09 22:02:48] at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:543:21)
[2023-01-09 22:02:48] at HTTPParser.parserOnHeadersComplete (_http_common.js:112:17)
[2023-01-09 22:02:48] at Socket.socketOnData (_http_client.js:440:20)
[2023-01-09 22:02:48] at emitOne (events.js:116:13)
[2023-01-09 22:02:48] at Socket.emit (events.js:211:7)
[2023-01-09 22:02:48] at addChunk (_stream_readable.js:263:12)
[2023-01-09 22:02:48] [stack]: ‘Error: CouchDB security seems to be misconfigured, see: https://github.com/medic/cht-core/blob/master/DEVELOPMENT.md#enabling-a-secure-couchdb\n at ClientRequest.net.get (/srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/@medic/server-checks/src/checks.js:63:16)\n at Object.onceWrapper (events.js:315:30)\n at emitOne (events.js:116:13)\n at ClientRequest.emit (events.js:211:7)\n at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:543:21)\n at HTTPParser.parserOnHeadersComplete (_http_common.js:112:17)\n at Socket.socketOnData (_http_client.js:440:20)\n at emitOne (events.js:116:13)\n at Socket.emit (events.js:211:7)\n at addChunk (_stream_readable.js:263:12)’,
[2023-01-09 22:02:48] [message]: ‘CouchDB security seems to be misconfigured, see: cht-core/DEVELOPMENT.md at master · medic/cht-core · GitHub’ }
[2023-01-09 22:02:51] 2023-01-09 22:02:51 INFO: Running server checks…
[2023-01-09 22:02:51] Node Environment Options: ‘–max_old_space_size=8192’
[2023-01-09 22:02:51] Node Version: 8.11.4 in production mode
[2023-01-09 22:02:51] COUCH_URL http://haproxy:5984/medic
[2023-01-09 22:02:51] COUCH_NODE_NAME couchdb@127.0.0.1
[2023-01-09 22:02:54] 2023-01-09 22:02:54 ERROR: Error watching sentinel changes, restarting: { FetchError: invalid json response body at http://haproxy:5984/medic-sentinel/ reason: Unexpected token < in JSON at position 0
[2023-01-09 22:02:54] at /srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/node-fetch/lib/index.js:272:32
[2023-01-09 22:02:54] at
[2023-01-09 22:02:54] at process._tickCallback (internal/process/next_tick.js:188:7)
[2023-01-09 22:02:54] message: ‘invalid json response body at http://haproxy:5984/medic-sentinel/ reason: Unexpected token < in JSON at position 0’,
[2023-01-09 22:02:54] type: ‘invalid-json’,
[2023-01-09 22:02:54] [stack]: ‘FetchError: invalid json response body at http://haproxy:5984/medic-sentinel/ reason: Unexpected token < in JSON at position 0\n at /srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/node-fetch/lib/index.js:272:32\n at \n at process._tickCallback (internal/process/next_tick.js:188:7)’ }
Expected a 401 when accessing db without authentication.
Instead we got a 503
[2023-01-09 22:02:54] 2023-01-09 22:02:54 ERROR: Fatal error initialising medic-api
[2023-01-09 22:02:54] 2023-01-09 22:02:54 ERROR: { Error: CouchDB security seems to be misconfigured, see: cht-core/DEVELOPMENT.md at master · medic/cht-core · GitHub
[2023-01-09 22:02:54] at ClientRequest.net.get (/srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/@medic/server-checks/src/checks.js:63:16)
[2023-01-09 22:02:54] at Object.onceWrapper (events.js:315:30)
[2023-01-09 22:02:54] at emitOne (events.js:116:13)
[2023-01-09 22:02:54] at ClientRequest.emit (events.js:211:7)
[2023-01-09 22:02:54] at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:543:21)
[2023-01-09 22:02:54] at HTTPParser.parserOnHeadersComplete (_http_common.js:112:17)
[2023-01-09 22:02:54] at Socket.socketOnData (_http_client.js:440:20)
[2023-01-09 22:02:54] at emitOne (events.js:116:13)
[2023-01-09 22:02:54] at Socket.emit (events.js:211:7)
[2023-01-09 22:02:54] at addChunk (_stream_readable.js:263:12)
[2023-01-09 22:02:54] [stack]: ‘Error: CouchDB security seems to be misconfigured, see: https://github.com/medic/cht-core/blob/master/DEVELOPMENT.md#enabling-a-secure-couchdb\n at ClientRequest.net.get (/srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/@medic/server-checks/src/checks.js:63:16)\n at Object.onceWrapper (events.js:315:30)\n at emitOne (events.js:116:13)\n at ClientRequest.emit (events.js:211:7)\n at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:543:21)\n at HTTPParser.parserOnHeadersComplete (_http_common.js:112:17)\n at Socket.socketOnData (_http_client.js:440:20)\n at emitOne (events.js:116:13)\n at Socket.emit (events.js:211:7)\n at addChunk (_stream_readable.js:263:12)’,
[2023-01-09 22:02:54] [message]: ‘CouchDB security seems to be misconfigured, see: cht-core/DEVELOPMENT.md at master · medic/cht-core · GitHub’ }
[2023-01-09 22:02:57] 2023-01-09 22:02:57 INFO: Running server checks…
[2023-01-09 22:02:57] Node Environment Options: ‘–max_old_space_size=8192’
[2023-01-09 22:02:57] Node Version: 8.11.4 in production mode
[2023-01-09 22:02:57] COUCH_URL http://haproxy:5984/medic
[2023-01-09 22:02:57] COUCH_NODE_NAME couchdb@127.0.0.1
[2023-01-09 22:03:00] 2023-01-09 22:03:01 ERROR: Error watching sentinel changes, restarting: { FetchError: invalid json response body at http://haproxy:5984/medic-sentinel/ reason: Unexpected token < in JSON at position 0
[2023-01-09 22:03:00] at /srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/node-fetch/lib/index.js:272:32
[2023-01-09 22:03:01] at
[2023-01-09 22:03:01] at process._tickCallback (internal/process/next_tick.js:188:7)
[2023-01-09 22:03:01] message: ‘invalid json response body at http://haproxy:5984/medic-sentinel/ reason: Unexpected token < in JSON at position 0’,
[2023-01-09 22:03:01] type: ‘invalid-json’,
[2023-01-09 22:03:01] [stack]: ‘FetchError: invalid json response body at http://haproxy:5984/medic-sentinel/ reason: Unexpected token < in JSON at position 0\n at /srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/node-fetch/lib/index.js:272:32\n at \n at process._tickCallback (internal/process/next_tick.js:188:7)’ }
Expected a 401 when accessing db without authentication.
Instead we got a 503
[2023-01-09 22:03:01] 2023-01-09 22:03:01 ERROR: Fatal error initialising medic-api
[2023-01-09 22:03:01] 2023-01-09 22:03:01 ERROR: { Error: CouchDB security seems to be misconfigured, see: cht-core/DEVELOPMENT.md at master · medic/cht-core · GitHub
[2023-01-09 22:03:01] at ClientRequest.net.get (/srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/@medic/server-checks/src/checks.js:63:16)
[2023-01-09 22:03:01] at Object.onceWrapper (events.js:315:30)
[2023-01-09 22:03:01] at emitOne (events.js:116:13)
[2023-01-09 22:03:01] at ClientRequest.emit (events.js:211:7)
[2023-01-09 22:03:01] at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:543:21)

I have tried to follow the online troubleshooting sessions to have our app, including https://forum.communityhealthtoolkit.org/t/error-failed-to-get-a-response/2294/5, so far none has worked, looks like something has failed, trying to have everything up this morning, and not lose the data, any ideas?