@hareet
We have managed to provide more space to the volume, which I realized was provisioned but but not added by the LLVM as below, now the app has refused to wake up as follows:
Package ‘horticulturalist’:
Service ‘horticulturalist’:
Status: Failure
Up: 1 seconds, Restarts: 150
Attributes: watched, expected
Service PID: None, Supervisor PID: 344
Package ‘medic-api’:
Service ‘medic-api’:
Status: Up
Up: 1 seconds, Restarts: 130
Attributes: watched, expected
Service PID: None, Supervisor PID: 387
Package ‘medic-core’:
Service ‘couchdb’:
Status: Up
Up: 428 seconds, Restarts: 0
Attributes: watched, running, expected
Service PID: 444, Supervisor PID: 427
Service ‘nginx’:
Status: Down
Up: 0 seconds, Restarts: -1
Attributes: watched, down
Service PID: None, Supervisor PID: 466
Service ‘openssh’:
Status: Up
Up: 428 seconds, Restarts: 0
Attributes: watched, running, expected
Service PID: 516, Supervisor PID: 501
Package ‘medic-couch2pg’:
Service ‘medic-couch2pg’:
Status: Failure
Up: 0 seconds, Restarts: 166
Attributes: watched, expected
Service PID: None, Supervisor PID: 562
Package ‘medic-rdbms’:
Service ‘postgresql’:
Status: Up
Up: 429 seconds, Restarts: 0
Attributes: watched, running, expected
Service PID: 632, Supervisor PID: 610
Package ‘medic-sentinel’:
Service ‘medic-sentinel’:
Status: Up
Up: 1 seconds, Restarts: 164
Attributes: watched, running, expected
Service PID: 19022, Supervisor PID: 659
Package ‘system-services’:
Service ‘cron’:
Status: Up
Up: 429 seconds, Restarts: 0
Attributes: watched, running, expected
Service PID: 725, Supervisor PID: 700
Service ‘syslog’:
Status: Up
Up: 429 seconds, Restarts: 0
Attributes: watched, running, expected
Service PID: 749, Supervisor PID: 737
And error logs as below:
[2023-01-10 06:47:56] 2023-01-10 06:47:56 ERROR: Error watching sentinel changes, restarting: { FetchError: invalid json response body at http://haproxy:5984/medic-sentinel/ reason: Unexpected token < in JSON at position 0
[2023-01-10 06:47:56] at /srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/node-fetch/lib/index.js:272:32
[2023-01-10 06:47:56] at
[2023-01-10 06:47:56] at process._tickCallback (internal/process/next_tick.js:188:7)
[2023-01-10 06:47:56] message: ‘invalid json response body at http://haproxy:5984/medic-sentinel/ reason: Unexpected token < in JSON at position 0’,
[2023-01-10 06:47:56] type: ‘invalid-json’,
[2023-01-10 06:47:56] [stack]: ‘FetchError: invalid json response body at http://haproxy:5984/medic-sentinel/ reason: Unexpected token < in JSON at position 0\n at /srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/node-fetch/lib/index.js:272:32\n at \n at process._tickCallback (internal/process/next_tick.js:188:7)’ }
Expected a 401 when accessing db without authentication.
Instead we got a 503
[2023-01-10 06:47:56] 2023-01-10 06:47:56 ERROR: Fatal error initialising medic-api
[2023-01-10 06:47:56] 2023-01-10 06:47:56 ERROR: { Error: CouchDB security seems to be misconfigured, see: cht-core/DEVELOPMENT.md at master · medic/cht-core · GitHub
[2023-01-10 06:47:56] at ClientRequest.net.get (/srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/@medic/server-checks/src/checks.js:63:16)
[2023-01-10 06:47:56] at Object.onceWrapper (events.js:315:30)
[2023-01-10 06:47:56] at emitOne (events.js:116:13)
[2023-01-10 06:47:56] at ClientRequest.emit (events.js:211:7)
[2023-01-10 06:47:56] at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:543:21)
[2023-01-10 06:47:56] at HTTPParser.parserOnHeadersComplete (_http_common.js:112:17)
[2023-01-10 06:47:56] at Socket.socketOnData (_http_client.js:440:20)
[2023-01-10 06:47:56] at emitOne (events.js:116:13)
[2023-01-10 06:47:56] at Socket.emit (events.js:211:7)
[2023-01-10 06:47:56] at addChunk (_stream_readable.js:263:12)
[2023-01-10 06:47:56] [stack]: ‘Error: CouchDB security seems to be misconfigured, see: cht-core/DEVELOPMENT.md at master · medic/cht-core · GitHub at ClientRequest.net.get (/srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/@medic/server-checks/src/checks.js:63:16)\n at Object.onceWrapper (events.js:315:30)\n at emitOne (events.js:116:13)\n at ClientRequest.emit (events.js:211:7)\n at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:543:21)\n at HTTPParser.parserOnHeadersComplete (_http_common.js:112:17)\n at Socket.socketOnData (_http_client.js:440:20)\n at emitOne (events.js:116:13)\n at Socket.emit (events.js:211:7)\n at addChunk (_stream_readable.js:263:12)’,
[2023-01-10 06:47:56] [message]: ‘CouchDB security seems to be misconfigured, see: cht-core/DEVELOPMENT.md at master · medic/cht-core · GitHub’ }
[2023-01-10 06:47:59] 2023-01-10 06:47:59 INFO: Running server checks…
[2023-01-10 06:47:59] Node Environment Options: ‘–max_old_space_size=8192’
[2023-01-10 06:47:59] Node Version: 8.11.4 in production mode
[2023-01-10 06:47:59] COUCH_URL http://haproxy:5984/medic
[2023-01-10 06:47:59] COUCH_NODE_NAME couchdb@127.0.0.1
Expected a 401 when accessing db without authentication.
Instead we got a 503
[2023-01-10 06:48:02] 2023-01-10 06:48:02 ERROR: Fatal error initialising medic-api
[2023-01-10 06:48:02] 2023-01-10 06:48:02 ERROR: { Error: CouchDB security seems to be misconfigured, see: cht-core/DEVELOPMENT.md at master · medic/cht-core · GitHub
[2023-01-10 06:48:02] at ClientRequest.net.get (/srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/@medic/server-checks/src/checks.js:63:16)
[2023-01-10 06:48:02] at Object.onceWrapper (events.js:315:30)
[2023-01-10 06:48:02] at emitOne (events.js:116:13)
[2023-01-10 06:48:02] at ClientRequest.emit (events.js:211:7)
[2023-01-10 06:48:02] at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:543:21)
[2023-01-10 06:48:02] at HTTPParser.parserOnHeadersComplete (_http_common.js:112:17)
[2023-01-10 06:48:02] at Socket.socketOnData (_http_client.js:440:20)
[2023-01-10 06:48:02] at emitOne (events.js:116:13)
[2023-01-10 06:48:02] at Socket.emit (events.js:211:7)
[2023-01-10 06:48:02] at addChunk (_stream_readable.js:263:12)
[2023-01-10 06:48:02] [stack]: ‘Error: CouchDB security seems to be misconfigured, see: cht-core/DEVELOPMENT.md at master · medic/cht-core · GitHub at ClientRequest.net.get (/srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/@medic/server-checks/src/checks.js:63:16)\n at Object.onceWrapper (events.js:315:30)\n at emitOne (events.js:116:13)\n at ClientRequest.emit (events.js:211:7)\n at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:543:21)\n at HTTPParser.parserOnHeadersComplete (_http_common.js:112:17)\n at Socket.socketOnData (_http_client.js:440:20)\n at emitOne (events.js:116:13)\n at Socket.emit (events.js:211:7)\n at addChunk (_stream_readable.js:263:12)’,
[2023-01-10 06:48:02] [message]: ‘CouchDB security seems to be misconfigured, see: cht-core/DEVELOPMENT.md at master · medic/cht-core · GitHub’ }
[2023-01-10 06:48:05] 2023-01-10 06:48:05 INFO: Running server checks…
[2023-01-10 06:48:05] Node Environment Options: ‘–max_old_space_size=8192’
[2023-01-10 06:48:05] Node Version: 8.11.4 in production mode
[2023-01-10 06:48:05] COUCH_URL http://haproxy:5984/medic
[2023-01-10 06:48:05] COUCH_NODE_NAME couchdb@127.0.0.1
Expected a 401 when accessing db without authentication.
Instead we got a 503
[2023-01-10 06:48:08] 2023-01-10 06:48:08 ERROR: Fatal error initialising medic-api
[2023-01-10 06:48:08] 2023-01-10 06:48:08 ERROR: { Error: CouchDB security seems to be misconfigured, see: cht-core/DEVELOPMENT.md at master · medic/cht-core · GitHub
[2023-01-10 06:48:08] at ClientRequest.net.get (/srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/@medic/server-checks/src/checks.js:63:16)
[2023-01-10 06:48:08] at Object.onceWrapper (events.js:315:30)
[2023-01-10 06:48:08] at emitOne (events.js:116:13)
[2023-01-10 06:48:08] at ClientRequest.emit (events.js:211:7)
[2023-01-10 06:48:08] at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:543:21)
[2023-01-10 06:48:08] at HTTPParser.parserOnHeadersComplete (_http_common.js:112:17)
[2023-01-10 06:48:08] at Socket.socketOnData (_http_client.js:440:20)
[2023-01-10 06:48:08] at emitOne (events.js:116:13)
[2023-01-10 06:48:08] at Socket.emit (events.js:211:7)
[2023-01-10 06:48:08] at addChunk (_stream_readable.js:263:12)
[2023-01-10 06:48:08] [stack]: ‘Error: CouchDB security seems to be misconfigured, see: cht-core/DEVELOPMENT.md at master · medic/cht-core · GitHub at ClientRequest.net.get (/srv/software/medic-api/md5-IaUJW7p4rpaMwbWhKl5C1A==/node_modules/@medic/server-checks/src/checks.js:63:16)\n at Object.onceWrapper (events.js:315:30)\n at emitOne (events.js:116:13)\n at ClientRequest.emit (events.js:211:7)\n at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:543:21)\n at HTTPParser.parserOnHeadersComplete (_http_common.js:112:17)\n at Socket.socketOnData (_http_client.js:440:20)\n at emitOne (events.js:116:13)\n at Socket.emit (events.js:211:7)\n at addChunk (_stream_readable.js:263:12)’,
[2023-01-10 06:48:08] [message]: ‘CouchDB security seems to be misconfigured, see: cht-core/DEVELOPMENT.md at master · medic/cht-core · GitHub’ }
[2023-01-10 06:48:11] 2023-01-10 06:48:11 INFO: Running server checks…
[2023-01-10 06:48:11] Node Environment Options: ‘–max_old_space_size=8192’
[2023-01-10 06:48:11] Node Version: 8.11.4 in production mode
[2023-01-10 06:48:11] COUCH_URL http://haproxy:5984/medic
[2023-01-10 06:48:11] COUCH_NODE_NAME couchdb@127.0.0.1