4.x CHT api service failling to start

Technical Support Development
1

Hello @diana @mrjones

I did setup 4.x on aws and did try to connect it to the rapidpro instance by adding these settings

rapidpro.app = <cht_api_token_generated>
rapidpro:incoming = <rapidpro_api_token_generated>
rapidpro:outgoing = <cht_api_token_generated>

I restarted the docker services but compose-api-1 was failing to restart as seen below

ubuntu@ip-172-51-32-17:~/cht/upgrade-service$ sudo docker ps
CONTAINER ID   IMAGE                                                 COMMAND                   CREATED       STATUS                          PORTS                          NAMES
90ce1eb5e02b   public.ecr.aws/s5s3h4s7/cht-upgrade-service:latest    "node /app/src/index…"    4 days ago    Up About an hour                                               upgrade-service-cht-upgrade-service-1
6d59815ac11a   public.ecr.aws/medic/cht-api:4.10.0                   "/bin/bash /service/…"    4 weeks ago   Restarting (1) 57 seconds ago                                  compose-api-1
15ce065055dc   public.ecr.aws/medic/cht-sentinel:4.10.0              "/bin/bash /service/…"    4 weeks ago   Up About an hour                                               compose-sentinel-1
c2070468a224   public.ecr.aws/medic/cht-haproxy:4.10.0               "/entrypoint.sh"          4 weeks ago   Up About an hour                5984/tcp                       compose-haproxy-1
4609c13ca507   public.ecr.aws/medic/cht-haproxy-healthcheck:4.10.0   "/bin/sh -c \"/app/ch…"   4 weeks ago   Up About an hour                                               compose-healthcheck-1
42097909f51f   public.ecr.aws/medic/cht-couchdb:4.10.0               "tini -- /docker-ent…"    4 weeks ago   Up About an hour                4369/tcp, 5984/tcp, 9100/tcp   compose-couchdb-1

check the logs and got

Info: Starting CHT API
2024-09-06T17:03:47.310 INFO: Running server checks… 
Node Version: 20.15.1
Node Mode: "development"
Node Environment Options: 'undefined'
StatusCodeError: 503 - {"error":"503 Service Unavailable","reason":"No server is available to handle this request","server":"haproxy"}
    at new StatusCodeError (/service/api/node_modules/request-promise-core/lib/errors.js:32:15)
    at plumbing.callback (/service/api/node_modules/request-promise-core/lib/plumbing.js:104:33)
    at Request.RP$callback [as _callback] (/service/api/node_modules/request-promise-core/lib/plumbing.js:46:31)
    at self.callback (/service/api/node_modules/request/request.js:185:22)
    at Request.emit (node:events:519:28)
    at Request.<anonymous> (/service/api/node_modules/request/request.js:1154:10)
    at Request.emit (node:events:519:28)
    at IncomingMessage.<anonymous> (/service/api/node_modules/request/request.js:1076:12)
    at Object.onceWrapper (node:events:633:28)
    at IncomingMessage.emit (node:events:531:35) {
  statusCode: 503,
  error: {
    error: '503 Service Unavailable',
    reason: 'No server is available to handle this request',
    server: 'haproxy'
  }
}
2024-09-06T17:03:55.256 ERROR: Transitions are disabled until the above configuration errors are fixed. Error: Transitions are disabled until the above configuration errors are fixed.
    at Object.loadTransitions (/service/shared-libs/transitions/src/transitions/index.js:173:11)
    at initTransitionLib (/service/api/src/services/config-watcher.js:49:20)
    at /service/api/src/services/config-watcher.js:150:17
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

2024-09-06T17:40:04.492 INFO: Sending 0 messages 
2024-09-06T17:40:04.506 ERROR: Error while polling message states: 'No api key configured. Refer to the RapidPro configuration documentation.' 
2024-09-06T17:41:04.484 DEBUG: Checking for a configured outgoing message service 
2024-09-06T17:41:04.484 DEBUG: Checking for pending outgoing messages
2

Hi @cliff.

It looks like haproxy can’t connect to CouchDb. Can you try restarting all containers? If that doesn’t resolve the problem, can you please check CouchDb logs for possible issues?
Also, can you please share your docker-compose files and environment variables (don’t need share passwords and ids, but do need couchdb / haproxy vars if you changed any).

1 Like
3

thanks @diana
part of the initial couchDB logs

sudo docker logs compose-couchdb-1

Waiting for cht couchdb

[info] 2024-09-06T17:03:47.612986Z couchdb@127.0.0.1 <0.254.0> -------- Preflight check: Checking For Monsters

[info] 2024-09-06T17:03:47.613147Z couchdb@127.0.0.1 <0.254.0> -------- Preflight check: Asserting Admin Account

[info] 2024-09-06T17:03:47.621079Z couchdb@127.0.0.1 <0.254.0> -------- Apache CouchDB 3.3.3 is starting.

[info] 2024-09-06T17:03:47.621194Z couchdb@127.0.0.1 <0.255.0> -------- Starting couch_sup

[notice] 2024-09-06T17:03:47.645039Z couchdb@127.0.0.1 <0.109.0> -------- config: [admins] medic set to '****' for reason nil

[info] 2024-09-06T17:03:47.739579Z couchdb@127.0.0.1 <0.254.0> -------- Apache CouchDB has started. Time to relax.

[notice] 2024-09-06T17:03:47.757143Z couchdb@127.0.0.1 <0.293.0> -------- rexi_server : started servers

[notice] 2024-09-06T17:03:47.760907Z couchdb@127.0.0.1 <0.297.0> -------- rexi_buffer : started servers

[notice] 2024-09-06T17:03:47.939078Z couchdb@127.0.0.1 <0.328.0> -------- mem3_reshard_dbdoc start init()

[notice] 2024-09-06T17:03:47.946157Z couchdb@127.0.0.1 <0.330.0> -------- mem3_reshard start init()

[notice] 2024-09-06T17:03:47.946232Z couchdb@127.0.0.1 <0.331.0> -------- mem3_reshard db monitor <0.331.0> starting

[notice] 2024-09-06T17:03:47.948564Z couchdb@127.0.0.1 <0.330.0> -------- mem3_reshard starting reloading jobs

[notice] 2024-09-06T17:03:47.948731Z couchdb@127.0.0.1 <0.330.0> -------- mem3_reshard finished reloading jobs

[info] 2024-09-06T17:03:47.969346Z couchdb@127.0.0.1 <0.337.0> -------- Apache CouchDB has started. Time to relax.

[info] 2024-09-06T17:03:47.969506Z couchdb@127.0.0.1 <0.337.0> -------- Apache CouchDB has started on http://0.0.0.0:5984/

[notice] 2024-09-06T17:03:48.163526Z couchdb@127.0.0.1 <0.465.0> -------- All system databases exist.

[notice] 2024-09-06T17:03:50.260285Z couchdb@127.0.0.1 <0.534.0> 47894f33e6 couchdb:5984 172.18.0.2 medic GET / 200 ok 81

couchdb is ready

[notice] 2024-09-06T17:03:50.306255Z couchdb@127.0.0.1 <0.535.0> 61bdb26fea couchdb:5984 172.18.0.2 medic PUT /_users 412 ok 26

ERROR: Failed to create system database '_users', continuing without creating database

[notice] 2024-09-06T17:03:50.325467Z couchdb@127.0.0.1 <0.536.0> 62ad5080eb couchdb:5984 172.18.0.2 medic PUT /_replicator 412 ok 3

ERROR: Failed to create system database '_replicator', continuing without creating database

[notice] 2024-09-06T17:03:50.342272Z couchdb@127.0.0.1 <0.537.0> 7ac565933d couchdb:5984 172.18.0.2 medic PUT /_global_changes 412 ok 1

ERROR: Failed to create system database '_global_changes', continuing without creating database

[notice] 2024-09-06T17:03:52.855227Z couchdb@127.0.0.1 <0.563.0> 5a5873070f couchdb:5984 172.18.0.4 medic GET /_membership 200 ok 2

[notice] 2024-09-06T17:03:53.024506Z couchdb@127.0.0.1 <0.387.0> -------- couch_replicator_clustering : cluster stable

[notice] 2024-09-06T17:03:53.027731Z couchdb@127.0.0.1 <0.449.0> -------- Started replicator db changes listener <0.580.0>

[notice] 2024-09-06T17:03:53.314203Z couchdb@127.0.0.1 <0.564.0> 51f0153cb9 haproxy:5984 172.18.0.5 medic GET / 200 ok 1

[notice] 2024-09-06T17:03:53.317387Z couchdb@127.0.0.1 <0.564.0> fb9a292fc1 haproxy:5984 172.18.0.5 undefined GET / 401 ok 0

[notice] 2024-09-06T17:03:53.321078Z couchdb@127.0.0.1 <0.564.0> 4f9e18da11 haproxy:5984 172.18.0.5 medic GET /_membership 200 ok 0

[notice] 2024-09-06T17:03:53.324293Z couchdb@127.0.0.1 <0.564.0> 88adfeb11d haproxy:5984 172.18.0.5 medic GET /_membership 200 ok 0

[notice] 2024-09-06T17:03:53.327196Z couchdb@127.0.0.1 <0.564.0> 363181ac52 haproxy:5984 172.18.0.5 medic GET /_membership 200 ok 0

[notice] 2024-09-06T17:03:53.333644Z couchdb@127.0.0.1 <0.564.0> dae4c42d23 haproxy:5984 172.18.0.5 medic GET /_users 200 ok 4

[notice] 2024-09-06T17:03:53.339260Z couchdb@127.0.0.1 <0.564.0> a1fa8383e8 haproxy:5984 172.18.0.5 medic GET /_replicator 200 ok 2

[notice] 2024-09-06T17:03:53.349346Z couchdb@127.0.0.1 <0.564.0> 4c383fa9c3 haproxy:5984 172.18.0.5 medic GET /_global_changes 200 ok 8

[notice] 2024-09-06T17:03:53.510782Z couchdb@127.0.0.1 <0.564.0> f778aeb82d haproxy:5984 172.18.0.6 medic GET / 200 ok 1

[notice] 2024-09-06T17:03:53.513818Z couchdb@127.0.0.1 <0.564.0> 2c8a9074bf haproxy:5984 172.18.0.6 undefined GET / 401 ok 0

[notice] 2024-09-06T17:03:53.518214Z couchdb@127.0.0.1 <0.564.0> 28c1c0e787 haproxy:5984 172.18.0.6 medic GET /_membership 200 ok 0

[notice] 2024-09-06T17:03:53.521488Z couchdb@127.0.0.1 <0.564.0> 90dfc716f5 haproxy:5984 172.18.0.6 medic GET /_membership 200 ok 0

[notice] 2024-09-06T17:03:53.524677Z couchdb@127.0.0.1 <0.564.0> 75f34e7bc7 haproxy:5984 172.18.0.6 medic GET /_membership 200 ok 0

[notice] 2024-09-06T17:03:53.530197Z couchdb@127.0.0.1 <0.564.0> df9011a999 haproxy:5984 172.18.0.6 medic GET /_users 200 ok 3

[notice] 2024-09-06T17:03:53.535346Z couchdb@127.0.0.1 <0.564.0> 50508e47b5 haproxy:5984 172.18.0.6 medic GET /_replicator 200 ok 2

[notice] 2024-09-06T17:03:53.539414Z couchdb@127.0.0.1 <0.564.0> e4c9c9089b haproxy:5984 172.18.0.6 medic GET /_global_changes 200 ok 1

[notice] 2024-09-06T17:03:53.703226Z couchdb@127.0.0.1 <0.564.0> e463a445a3 haproxy:5984 172.18.0.6 medic POST /_session 200 ok 5

[notice] 2024-09-06T17:03:53.728399Z couchdb@127.0.0.1 <0.564.0> b65c8ef416 haproxy:5984 172.18.0.6 medic GET /medic/ 200 ok 11

[notice] 2024-09-06T17:03:53.738564Z couchdb@127.0.0.1 <0.564.0> 6b60ca1500 haproxy:5984 172.18.0.6 medic GET /medic/settings? 200 ok 5

[notice] 2024-09-06T17:03:53.907776Z couchdb@127.0.0.1 <0.564.0> 2540434abf haproxy:5984 172.18.0.6 medic GET /medic/_all_docs?include_docs=true&startkey=%22messages-%22&endkey=%22messages-%EF%BF%B0%22 200 ok 37

[notice] 2024-09-06T17:03:54.935346Z couchdb@127.0.0.1 <0.564.0> f44b2fb4d7 haproxy:5984 172.18.0.6 medic GET /medic/_all_docs?include_docs=true&startkey=%22messages-%22&endkey=%22messages-%EF%BF%B0%22 200 ok 26

[notice] 2024-09-06T17:03:54.988039Z couchdb@127.0.0.1 <0.564.0> fc5f637cb6 haproxy:5984 172.18.0.6 medic GET /medic/_all_docs?include_docs=true&startkey=%22_design%2F%22&endkey=%22_design%2F%EF%BF%B0%22 200 ok 8

[notice] 2024-09-06T17:03:55.004008Z couchdb@127.0.0.1 <0.564.0> 4fe3f76898 haproxy:5984 172.18.0.6 medic GET /medic-sentinel/ 200 ok 11

[notice] 2024-09-06T17:03:55.011120Z couchdb@127.0.0.1 <0.564.0> e5bccb046f haproxy:5984 172.18.0.6 medic GET /medic-sentinel/_all_docs?include_docs=true&startkey=%22_design%2F%22&endkey=%22_design%2F%EF%BF%B0%22 200 ok 5

[notice] 2024-09-06T17:03:55.024232Z couchdb@127.0.0.1 <0.564.0> 5b582d49c8 haproxy:5984 172.18.0.6 medic GET /medic-logs/ 200 ok 9

[notice] 2024-09-06T17:03:55.032160Z couchdb@127.0.0.1 <0.564.0> ac6d642653 haproxy:5984 172.18.0.6 medic GET /medic-logs/_all_docs?include_docs=true&startkey=%22_design%2F%22&endkey=%22_design%2F%EF%BF%B0%22 200 ok 4

[notice] 2024-09-06T17:03:55.045808Z couchdb@127.0.0.1 <0.564.0> faf50a7a7f haproxy:5984 172.18.0.6 medic GET /medic-users-meta/ 200 ok 9

[notice] 2024-09-06T17:03:55.054187Z couchdb@127.0.0.1 <0.564.0> 83978eb3a7 haproxy:5984 172.18.0.6 medic GET /medic-users-meta/_all_docs?include_docs=true&startkey=%22_design%2F%22&endkey=%22_design%2F%EF%BF%B0%22 200 ok 5

[notice] 2024-09-06T17:03:55.063472Z couchdb@127.0.0.1 <0.564.0> cff4ce6bad haproxy:5984 172.18.0.6 medic GET /_users/ 200 ok 5

[notice] 2024-09-06T17:03:55.076866Z couchdb@127.0.0.1 <0.564.0> d49bbf1ac5 haproxy:5984 172.18.0.6 medic GET /_users/_all_docs?include_docs=true&startkey=%22_design%2F%22&endkey=%22_design%2F%EF%BF%B0%22 200 ok 11

[notice] 2024-09-06T17:03:55.087708Z couchdb@127.0.0.1 <0.564.0> 7982304be6 haproxy:5984 172.18.0.6 medic GET /medic-logs/_all_docs?descending=true&include_docs=true&startkey=%22upgrade_log%3A1725642235079%3A%22&endkey=%22upgrade_log%3A0%3A%22&limit=1 200 ok 5

[notice] 2024-09-06T17:03:55.095649Z couchdb@127.0.0.1 <0.564.0> 44a3487c98 haproxy:5984 172.18.0.6 medic GET /medic/_design/medic? 200 ok 2

[notice] 2024-09-06T17:03:55.118285Z couchdb@127.0.0.1 <0.646.0> b7db915ac7 haproxy:5984 172.18.0.6 medic GET /medic/_all_docs?include_docs=true&startkey=%22messages-%22&endkey=%22messages-%EF%BF%B0%22 200 ok 23

[notice] 2024-09-06T17:03:55.145345Z couchdb@127.0.0.1 <0.646.0> 169fb0741a haproxy:5984 172.18.0.6 medic GET /medic/settings? 200 ok 4

[notice] 2024-09-06T17:03:55.161280Z couchdb@127.0.0.1 <0.646.0> 081312c734 haproxy:5984 172.18.0.6 medic GET /medic/settings? 200 ok 4

[notice] 2024-09-06T17:03:55.168176Z couchdb@127.0.0.1 <0.646.0> d6f71ac465 haproxy:5984 172.18.0.6 medic GET /medic/_security 200 ok 1

[notice] 2024-09-06T17:03:55.171812Z couchdb@127.0.0.1 <0.646.0> 7223a33b97 haproxy:5984 172.18.0.6 medic GET /medic/_security 200 ok 1

[notice] 2024-09-06T17:03:55.174799Z couchdb@127.0.0.1 <0.646.0> b77efd9365 haproxy:5984 172.18.0.6 medic GET /medic/_security 200 ok 1

[notice] 2024-09-06T17:03:55.177700Z couchdb@127.0.0.1 <0.646.0> 9a735c06d8 haproxy:5984 172.18.0.6 medic GET /medic/_security 200 ok 1

[notice] 2024-09-06T17:03:55.180616Z couchdb@127.0.0.1 <0.646.0> efbbf0c3aa haproxy:5984 172.18.0.6 medic GET /medic/_security 200 ok 1

[notice] 2024-09-06T17:03:55.183246Z couchdb@127.0.0.1 <0.646.0> 59ffb4c2bb haproxy:5984 172.18.0.6 medic GET /medic/_security 200 ok 1

[notice] 2024-09-06T17:03:55.185843Z couchdb@127.0.0.1 <0.646.0> 72237efd57 haproxy:5984 172.18.0.6 medic GET /medic/_security 200 ok 1

[notice] 2024-09-06T17:03:55.189523Z couchdb@127.0.0.1 <0.646.0> d92fde0ad7 haproxy:5984 172.18.0.6 medic GET /medic/_security 200 ok 1

[notice] 2024-09-06T17:03:55.192281Z couchdb@127.0.0.1 <0.646.0> e68b41a818 haproxy:5984 172.18.0.6 medic GET /medic/_security 200 ok 1

[notice] 2024-09-06T17:03:55.194852Z couchdb@127.0.0.1 <0.646.0> c5b8b2423e haproxy:5984 172.18.0.6 medic GET /medic/_security 200 ok 1

[notice] 2024-09-06T17:03:55.197291Z couchdb@127.0.0.1 <0.646.0> 87d0eef458 haproxy:5984 172.18.0.6 medic GET /medic/_security 200 ok 1

[notice] 2024-09-06T17:03:55.199834Z couchdb@127.0.0.1 <0.646.0> 850e88b39e haproxy:5984 172.18.0.6 medic GET /medic/_security 200 ok 1

[notice] 2024-09-06T17:03:55.202280Z couchdb@127.0.0.1 <0.646.0> 574a7cb170 haproxy:5984 172.18.0.6 medic GET /medic/_security 200 ok 1

[notice] 2024-09-06T17:03:55.204695Z couchdb@127.0.0.1 <0.646.0> 7d51914519 haproxy:5984 172.18.0.6 medic GET /medic/_security 200 ok 1

[notice] 2024-09-06T17:03:55.207068Z couchdb@127.0.0.1 <0.646.0> 9934f61b53 haproxy:5984 172.18.0.6 medic GET /medic/_security 200 ok 1

[notice] 2024-09-06T17:03:55.209406Z couchdb@127.0.0.1 <0.646.0> 6de6c5b0dd haproxy:5984 172.18.0.6 medic GET /medic/_security 200 ok 1

[notice] 2024-09-06T17:03:55.211850Z couchdb@127.0.0.1 <0.646.0> 1aae22323f haproxy:5984 172.18.0.6 medic GET /medic/_security 200 ok 1

[notice] 2024-09-06T17:03:55.275171Z couchdb@127.0.0.1 <0.646.0> 21449576b9 haproxy:5984 172.18.0.6 medic GET /medic-vault/ 200 ok 16

[notice] 2024-09-06T17:03:55.281758Z couchdb@127.0.0.1 <0.646.0> a03db0d185 haproxy:5984 172.18.0.6 medic GET /medic-vault/ 200 ok 3

[notice] 2024-09-06T17:03:55.294457Z couchdb@127.0.0.1 <0.646.0> 9a187eab7c haproxy:5984 172.18.0.6 medic GET /medic/ 200 ok 5

[notice] 2024-09-06T17:03:55.295073Z couchdb@127.0.0.1 <0.564.0> 4eda938384 haproxy:5984 172.18.0.6 medic GET /medic-sentinel/ 200 ok 6

[notice] 2024-09-06T17:03:55.319853Z couchdb@127.0.0.1 <0.845.0> 1729c3be16 haproxy:5984 172.18.0.6 medic GET /_users/ 200 ok 5

[notice] 2024-09-06T17:03:55.321470Z couchdb@127.0.0.1 <0.846.0> aeca4863ae haproxy:5984 172.18.0.6 medic GET /medic/migration-log? 200 ok 6

[notice] 2024-09-06T17:03:55.336479Z couchdb@127.0.0.1 <0.846.0> a437d3f776 haproxy:5984 172.18.0.6 medic GET /medic/branding? 200 ok 5

[notice] 2024-09-06T17:03:55.366864Z couchdb@127.0.0.1 <0.846.0> fb117de76a haproxy:5984 172.18.0.6 medic GET /medic/branding/medic-logo-light-full.svg?rev=2-6a570422b93374dfabc622de82c23747 200 ok 22

[notice] 2024-09-06T17:03:55.367262Z couchdb@127.0.0.1 <0.847.0> 55df47b493 haproxy:5984 172.18.0.6 medic GET /medic/branding/favicon.ico?rev=2-6a570422b93374dfabc622de82c23747 200 ok 19

[notice] 2024-09-06T17:03:55.368521Z couchdb@127.0.0.1 <0.848.0> 7eaccc191c haproxy:5984 172.18.0.6 medic GET /medic/branding/cht-logo.png?rev=2-6a570422b93374dfabc622de82c23747 200 ok 19

[notice] 2024-09-06T17:03:55.390628Z couchdb@127.0.0.1 <0.846.0> 567588f89b haproxy:5984 172.18.0.6 medic GET /medic/branding? 200 ok 11

[notice] 2024-09-06T17:03:55.392369Z couchdb@127.0.0.1 <0.848.0> 4f4f589c01 haproxy:5984 172.18.0.6 medic GET /medic/settings? 200 ok 13

[notice] 2024-09-06T17:03:55.394922Z couchdb@127.0.0.1 <0.849.0> 3d7e049108 haproxy:5984 172.18.0.6 medic GET /medic/privacy-policies? 404 ok 7

/cht/upgrade-service$ cat docker-compose.yml

version: '3.9'

services:
  cht-upgrade-service:
    restart: always
    image: public.ecr.aws/s5s3h4s7/cht-upgrade-service:latest
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ${DOCKER_CONFIG_PATH}:/config
      - ${CHT_COMPOSE_PATH}:/docker-compose
    networks:
      - cht-net
    environment:
      - COUCHDB_USER
      - COUCHDB_PASSWORD
      - COUCHDB_SECRET
      - COUCHDB_UUID
      - COUCHDB_DATA
      - COUCHDB_SERVERS
      - CLUSTER_PEER_IPS
      - SVC_NAME
      - SVC1_NAME
      - SVC2_NAME
      - SVC3_NAME
      - COUCHDB_LOG_LEVEL
      - MARKET_URL_READ
      - BUILDS_SERVER
      - NGINX_HTTP_PORT
      - NGINX_HTTPS_PORT
      - CERTIFICATE_MODE
      - SSL_VOLUME_MOUNT_PATH
      - SSL_CERT_FILE_PATH
      - SSL_KEY_FILE_PATH
      - COMMON_NAME
      - EMAIL
      - COUNTRY
      - STATE
      - LOCALITY
      - ORGANISATION
      - DEPARTMENT
      - DOCKER_CONFIG=/config
      - CHT_COMPOSE_PROJECT_NAME=${CHT_COMPOSE_PROJECT_NAME:-cht}
      - CHT_NETWORK=${CHT_NETWORK:-cht-net}
      - DOCKER_CONFIG_PATH
      - CHT_COMPOSE_PATH
      - HEALTHCHECK_LOG_LEVEL

networks:
  cht-net:
    name: ${CHT_NETWORK:-cht-net}
4

Hi @cliff

I’m not sure what to make of your logs. These entries are from one month ago (2024-09-06) for both API and CouchDb.

What is the issue you are facing with your instance?
Can you share the latest logs from all services?

@mrjones suggested you could use this script to gather logs: Sharing 4.x Logs | Community Health Toolkit

1 Like
5

hello @diana
These are the last 50 lines form the couch DB logs

/cht/upgrade-service$ sudo docker logs compose-couchdb-1 --tail 50
  last msg: redacted
     state: [{data,[{"State",{file,{file_descriptor,prim_file,#{handle => #Ref<0.2384282298.345112584.122162>,owner => <0.21520.3>,r_ahead_size => 0,r_buffer => #Ref<0.2384282298.345374722.138490>}},false,303316,#Ref<0.2384282298.345243650.138501>,infinity}},{"InitialFilePath","./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}]}]
    extra: [<0.21522.3>,[{gen,do_call,4,[{file,"gen.erl"},{line,214}]},{gen_server,call,3,[{file,"gen_server.erl"},{line,243}]},{couch_file,pread_iolist,2,[{file,"src/couch_file.erl"},{line,170}]},{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,166}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,155}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,474}]},{couch_btree,stream_node,8,[{file,"src/couch_btree.erl"},{line,1069}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,242}]}]]
[error] 2024-10-08T14:35:45.142258Z couchdb@127.0.0.1 <0.21520.3> -------- gen_server <0.21520.3> terminated with reason: bad return value {file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}
  last msg: redacted
     state: [{data,[{"State",{file,{file_descriptor,prim_file,#{handle => #Ref<0.2384282298.345112584.122162>,owner => <0.21520.3>,r_ahead_size => 0,r_buffer => #Ref<0.2384282298.345374722.138490>}},false,303316,#Ref<0.2384282298.345243650.138501>,infinity}},{"InitialFilePath","./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}]}]
    extra: [<0.21522.3>,[{gen,do_call,4,[{file,"gen.erl"},{line,214}]},{gen_server,call,3,[{file,"gen_server.erl"},{line,243}]},{couch_file,pread_iolist,2,[{file,"src/couch_file.erl"},{line,170}]},{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,166}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,155}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,474}]},{couch_btree,stream_node,8,[{file,"src/couch_btree.erl"},{line,1069}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,242}]}]]
[error] 2024-10-08T14:35:45.142382Z couchdb@127.0.0.1 <0.21520.3> -------- CRASH REPORT Process  (<0.21520.3>) with 1 neighbors exited with reason: bad return value {file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"} at gen_server:handle_common_reply/8(line:815) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_file,init,['Argument__1']}, ancestors: [<0.21519.3>,<0.21518.3>], message_queue_len: 0, links: [<0.21519.3>], dictionary: [{couch_file_fd,{{file_descriptor,prim_file,#{handle => #Ref<0.2384282298.345112584.122162>,...}},...}},...], trap_exit: false, status: running, heap_size: 2586, stack_size: 29, reductions: 1658
[error] 2024-10-08T14:35:45.142466Z couchdb@127.0.0.1 <0.21520.3> -------- CRASH REPORT Process  (<0.21520.3>) with 1 neighbors exited with reason: bad return value {file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"} at gen_server:handle_common_reply/8(line:815) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_file,init,['Argument__1']}, ancestors: [<0.21519.3>,<0.21518.3>], message_queue_len: 0, links: [<0.21519.3>], dictionary: [{couch_file_fd,{{file_descriptor,prim_file,#{handle => #Ref<0.2384282298.345112584.122162>,...}},...}},...], trap_exit: false, status: running, heap_size: 2586, stack_size: 29, reductions: 1658
[error] 2024-10-08T14:35:45.149490Z couchdb@127.0.0.1 <0.21540.3> b7bc1acef3 rexi_server: from: couchdb@127.0.0.1(<0.11954.3>) mfa: fabric_rpc:all_docs/3 exit:{{bad_return_value,{file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}},{gen_server,call,[<0.21538.3>,{pread_iolist,295293},infinity]}} [{gen_server,call,3,[{file,"gen_server.erl"},{line,247}]},{couch_file,pread_iolist,2,[{file,"src/couch_file.erl"},{line,170}]},{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,166}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,155}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,474}]},{couch_btree,stream_node,8,[{file,"src/couch_btree.erl"},{line,1069}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,242}]},{couch_bt_engine,fold_docs_int,5,[{file,"src/couch_bt_engine.erl"},{line,1129}]}]
[error] 2024-10-08T14:35:45.149697Z couchdb@127.0.0.1 <0.21538.3> -------- gen_server <0.21538.3> terminated with reason: bad return value {file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}
  last msg: redacted
     state: [{data,[{"State",{file,{file_descriptor,prim_file,#{handle => #Ref<0.2384282298.345112587.122043>,owner => <0.21538.3>,r_ahead_size => 0,r_buffer => #Ref<0.2384282298.345374722.138641>}},false,303316,#Ref<0.2384282298.345505793.158836>,infinity}},{"InitialFilePath","./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}]}]
    extra: [<0.21540.3>,[{gen,do_call,4,[{file,"gen.erl"},{line,214}]},{gen_server,call,3,[{file,"gen_server.erl"},{line,243}]},{couch_file,pread_iolist,2,[{file,"src/couch_file.erl"},{line,170}]},{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,166}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,155}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,474}]},{couch_btree,stream_node,8,[{file,"src/couch_btree.erl"},{line,1069}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,242}]}]]
[info] 2024-10-08T14:35:45.149757Z couchdb@127.0.0.1 <0.260.0> -------- db shards/eaaaaaa7-ffffffff/medic.1721926601 died with reason {bad_return_value,{file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}}
[error] 2024-10-08T14:35:45.149755Z couchdb@127.0.0.1 <0.11954.3> b7bc1acef3 req_err(4070966599) {bad_return_value,
    {file_truncate_error,eof,
        "./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}} : {gen_server,call,[<0.21538.3>,{pread_iolist,295293},infinity]}
    [<<"gen_server:call/3 L247">>,<<"couch_file:pread_iolist/2 L170">>,<<"couch_file:pread_binary/2 L166">>,<<"couch_file:pread_term/2 L155">>,<<"couch_btree:get_node/2 L474">>,<<"couch_btree:stream_node/8 L1069">>,<<"couch_btree:fold/4 L242">>,<<"couch_bt_engine:fold_docs_int/5 L1129">>]
[notice] 2024-10-08T14:35:45.149991Z couchdb@127.0.0.1 <0.11954.3> b7bc1acef3 haproxy:5984 172.18.0.7 medic GET /medic/_all_docs?include_docs=true&startkey=%22messages-%22&endkey=%22messages-%EF%BF%B0%22 500 ok 9
[error] 2024-10-08T14:35:45.150994Z couchdb@127.0.0.1 <0.21538.3> -------- gen_server <0.21538.3> terminated with reason: bad return value {file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}
  last msg: redacted
     state: [{data,[{"State",{file,{file_descriptor,prim_file,#{handle => #Ref<0.2384282298.345112587.122043>,owner => <0.21538.3>,r_ahead_size => 0,r_buffer => #Ref<0.2384282298.345374722.138641>}},false,303316,#Ref<0.2384282298.345505793.158836>,infinity}},{"InitialFilePath","./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}]}]
    extra: [<0.21540.3>,[{gen,do_call,4,[{file,"gen.erl"},{line,214}]},{gen_server,call,3,[{file,"gen_server.erl"},{line,243}]},{couch_file,pread_iolist,2,[{file,"src/couch_file.erl"},{line,170}]},{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,166}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,155}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,474}]},{couch_btree,stream_node,8,[{file,"src/couch_btree.erl"},{line,1069}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,242}]}]]
[error] 2024-10-08T14:35:45.151429Z couchdb@127.0.0.1 <0.21538.3> -------- CRASH REPORT Process  (<0.21538.3>) with 1 neighbors exited with reason: bad return value {file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"} at gen_server:handle_common_reply/8(line:815) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_file,init,['Argument__1']}, ancestors: [<0.21537.3>,<0.21536.3>], message_queue_len: 0, links: [<0.21537.3>], dictionary: [{couch_file_fd,{{file_descriptor,prim_file,#{handle => #Ref<0.2384282298.345112587.122043>,...}},...}},...], trap_exit: false, status: running, heap_size: 2586, stack_size: 29, reductions: 1662
[error] 2024-10-08T14:35:45.151573Z couchdb@127.0.0.1 <0.21538.3> -------- CRASH REPORT Process  (<0.21538.3>) with 1 neighbors exited with reason: bad return value {file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"} at gen_server:handle_common_reply/8(line:815) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_file,init,['Argument__1']}, ancestors: [<0.21537.3>,<0.21536.3>], message_queue_len: 0, links: [<0.21537.3>], dictionary: [{couch_file_fd,{{file_descriptor,prim_file,#{handle => #Ref<0.2384282298.345112587.122043>,...}},...}},...], trap_exit: false, status: running, heap_size: 2586, stack_size: 29, reductions: 1662
[error] 2024-10-08T14:35:45.154463Z couchdb@127.0.0.1 <0.21556.3> -------- gen_server <0.21556.3> terminated with reason: bad return value {file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}
  last msg: redacted
     state: [{data,[{"State",{file,{file_descriptor,prim_file,#{handle => #Ref<0.2384282298.345112588.122330>,owner => <0.21556.3>,r_ahead_size => 0,r_buffer => #Ref<0.2384282298.345636865.158948>}},false,303316,#Ref<0.2384282298.345505793.158960>,infinity}},{"InitialFilePath","./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}]}]
    extra: [<0.21558.3>,[{gen,do_call,4,[{file,"gen.erl"},{line,214}]},{gen_server,call,3,[{file,"gen_server.erl"},{line,243}]},{couch_file,pread_iolist,2,[{file,"src/couch_file.erl"},{line,170}]},{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,166}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,155}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,474}]},{couch_btree,stream_node,8,[{file,"src/couch_btree.erl"},{line,1069}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,242}]}]]
[error] 2024-10-08T14:35:45.154675Z couchdb@127.0.0.1 <0.21556.3> -------- gen_server <0.21556.3> terminated with reason: bad return value {file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}
  last msg: redacted
     state: [{data,[{"State",{file,{file_descriptor,prim_file,#{handle => #Ref<0.2384282298.345112588.122330>,owner => <0.21556.3>,r_ahead_size => 0,r_buffer => #Ref<0.2384282298.345636865.158948>}},false,303316,#Ref<0.2384282298.345505793.158960>,infinity}},{"InitialFilePath","./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}]}]
    extra: [<0.21558.3>,[{gen,do_call,4,[{file,"gen.erl"},{line,214}]},{gen_server,call,3,[{file,"gen_server.erl"},{line,243}]},{couch_file,pread_iolist,2,[{file,"src/couch_file.erl"},{line,170}]},{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,166}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,155}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,474}]},{couch_btree,stream_node,8,[{file,"src/couch_btree.erl"},{line,1069}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,242}]}]]
[error] 2024-10-08T14:35:45.155237Z couchdb@127.0.0.1 <0.21556.3> -------- CRASH REPORT Process  (<0.21556.3>) with 1 neighbors exited with reason: bad return value {file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"} at gen_server:handle_common_reply/8(line:815) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_file,init,['Argument__1']}, ancestors: [<0.21555.3>,<0.21554.3>], message_queue_len: 0, links: [<0.21555.3>], dictionary: [{couch_file_fd,{{file_descriptor,prim_file,#{handle => #Ref<0.2384282298.345112588.122330>,...}},...}},...], trap_exit: false, status: running, heap_size: 2586, stack_size: 29, reductions: 1658
[info] 2024-10-08T14:35:45.155327Z couchdb@127.0.0.1 <0.260.0> -------- db shards/eaaaaaa7-ffffffff/medic.1721926601 died with reason {bad_return_value,{file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}}
[error] 2024-10-08T14:35:45.155350Z couchdb@127.0.0.1 <0.21558.3> ae5f98a4ce rexi_server: from: couchdb@127.0.0.1(<0.11954.3>) mfa: fabric_rpc:all_docs/3 exit:{{bad_return_value,{file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}},{gen_server,call,[<0.21556.3>,{pread_iolist,295293},infinity]}} [{gen_server,call,3,[{file,"gen_server.erl"},{line,247}]},{couch_file,pread_iolist,2,[{file,"src/couch_file.erl"},{line,170}]},{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,166}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,155}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,474}]},{couch_btree,stream_node,8,[{file,"src/couch_btree.erl"},{line,1069}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,242}]},{couch_bt_engine,fold_docs_int,5,[{file,"src/couch_bt_engine.erl"},{line,1129}]}]
[error] 2024-10-08T14:35:45.155473Z couchdb@127.0.0.1 <0.21556.3> -------- CRASH REPORT Process  (<0.21556.3>) with 1 neighbors exited with reason: bad return value {file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"} at gen_server:handle_common_reply/8(line:815) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_file,init,['Argument__1']}, ancestors: [<0.21555.3>,<0.21554.3>], message_queue_len: 0, links: [<0.21555.3>], dictionary: [{couch_file_fd,{{file_descriptor,prim_file,#{handle => #Ref<0.2384282298.345112588.122330>,...}},...}},...], trap_exit: false, status: running, heap_size: 2586, stack_size: 29, reductions: 1658
[error] 2024-10-08T14:35:45.155835Z couchdb@127.0.0.1 <0.11954.3> ae5f98a4ce req_err(4070966599) {bad_return_value,
    {file_truncate_error,eof,
        "./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}} : {gen_server,call,[<0.21556.3>,{pread_iolist,295293},infinity]}
    [<<"gen_server:call/3 L247">>,<<"couch_file:pread_iolist/2 L170">>,<<"couch_file:pread_binary/2 L166">>,<<"couch_file:pread_term/2 L155">>,<<"couch_btree:get_node/2 L474">>,<<"couch_btree:stream_node/8 L1069">>,<<"couch_btree:fold/4 L242">>,<<"couch_bt_engine:fold_docs_int/5 L1129">>]
[notice] 2024-10-08T14:35:45.156497Z couchdb@127.0.0.1 <0.11954.3> ae5f98a4ce haproxy:5984 172.18.0.7 medic GET /medic/_all_docs?include_docs=true&startkey=%22messages-%22&endkey=%22messages-%EF%BF%B0%22 500 ok 6
[notice] 2024-10-08T14:35:46.053010Z couchdb@127.0.0.1 <0.21362.3> 607e7d7be8 couchdb:5984 172.18.0.3 medic GET /_membership 200 ok 0
[notice] 2024-10-08T14:35:51.324650Z couchdb@127.0.0.1 <0.21604.3> f7acbd4ae1 couchdb:5984 172.18.0.3 medic GET /_membership 200 ok 0
[notice] 2024-10-08T14:35:56.580428Z couchdb@127.0.0.1 <0.21646.3> c09acce18c couchdb:5984 172.18.0.3 medic GET /_membership 200 ok 0
[notice] 2024-10-08T14:36:01.819965Z couchdb@127.0.0.1 <0.21685.3> 10d94d1a94 couchdb:5984 172.18.0.3 medic GET /_membership 200 ok 0
[notice] 2024-10-08T14:36:07.001402Z couchdb@127.0.0.1 <0.21729.3> 21afda1392 couchdb:5984 172.18.0.3 medic GET /_membership 200 ok 0
[notice] 2024-10-08T14:36:12.015172Z couchdb@127.0.0.1 <0.21766.3> 6048e7fafe couchdb:5984 172.18.0.3 medic GET /_membership 200 ok 0
[notice] 2024-10-08T14:36:17.180448Z couchdb@127.0.0.1 <0.21810.3> 78f17cce44 couchdb:5984 172.18.0.3 medic GET /_membership 200 ok 0

These are the last 100 lines form the api logs

/cht/upgrade-service$ sudo docker logs compose-api-1 --tail 100
2024-10-08T14:28:37.882 INFO: Extracting initial documents completed successfully 
2024-10-08T14:28:37.882 INFO: Merging translations… 
2024-10-08T14:28:38.048 ERROR: Fatal error initialising API 
2024-10-08T14:28:38.051 ERROR: {
  error: '{bad_return_value,\n' +
    '    {file_truncate_error,eof,\n' +
    '        "./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}}',
  reason: '{gen_server,call,[<0.16514.3>,{pread_iolist,295293},infinity]}',
  ref: 4070966599,
  status: 500,
  name: '{bad_return_value,\n' +
    '    {file_truncate_error,eof,\n' +
    '        "./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}}',
  message: '{gen_server,call,[<0.16514.3>,{pread_iolist,295293},infinity]}',
  stack: 'Error\n' +
    '    at Object.generateErrorFromResponse (/service/api/node_modules/pouchdb-errors/lib/index.js:104:18)\n' +
    '    at /service/api/node_modules/pouchdb-adapter-http/lib/index.js:254:33\n' +
    '    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)'
} 
Info: Starting CHT API
2024-10-08T14:29:38.652 INFO: Running server checks… 
Node Version: 20.15.1
Node Mode: "development"
Node Environment Options: 'undefined'
CouchDB Version: 3.3.3
2024-10-08T14:29:38.744 INFO: Checks passed successfully 
2024-10-08T14:29:38.822 INFO: Extracting initial documents… 
2024-10-08T14:29:38.916 INFO: Extracting initial documents completed successfully 
2024-10-08T14:29:38.916 INFO: Merging translations… 
2024-10-08T14:29:39.078 ERROR: Fatal error initialising API 
2024-10-08T14:29:39.081 ERROR: {
  error: '{bad_return_value,\n' +
    '    {file_truncate_error,eof,\n' +
    '        "./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}}',
  reason: '{gen_server,call,[<0.17231.3>,{pread_iolist,295293},infinity]}',
  ref: 4070966599,
  status: 500,
  name: '{bad_return_value,\n' +
    '    {file_truncate_error,eof,\n' +
    '        "./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}}',
  message: '{gen_server,call,[<0.17231.3>,{pread_iolist,295293},infinity]}',
  stack: 'Error\n' +
    '    at Object.generateErrorFromResponse (/service/api/node_modules/pouchdb-errors/lib/index.js:104:18)\n' +
    '    at /service/api/node_modules/pouchdb-adapter-http/lib/index.js:254:33\n' +
    '    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)'
} 
Info: Starting CHT API
2024-10-08T14:30:39.669 INFO: Running server checks… 
Node Version: 20.15.1
Node Mode: "development"
Node Environment Options: 'undefined'
CouchDB Version: 3.3.3
2024-10-08T14:30:39.766 INFO: Checks passed successfully 
2024-10-08T14:30:39.843 INFO: Extracting initial documents… 
2024-10-08T14:30:39.937 INFO: Extracting initial documents completed successfully 
2024-10-08T14:30:39.937 INFO: Merging translations… 
2024-10-08T14:30:40.098 ERROR: Fatal error initialising API 
2024-10-08T14:30:40.102 ERROR: {
  error: '{bad_return_value,\n' +
    '    {file_truncate_error,eof,\n' +
    '        "./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}}',
  reason: '{gen_server,call,[<0.17958.3>,{pread_iolist,295293},infinity]}',
  ref: 4070966599,
  status: 500,
  name: '{bad_return_value,\n' +
    '    {file_truncate_error,eof,\n' +
    '        "./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}}',
  message: '{gen_server,call,[<0.17958.3>,{pread_iolist,295293},infinity]}',
  stack: 'Error\n' +
    '    at Object.generateErrorFromResponse (/service/api/node_modules/pouchdb-errors/lib/index.js:104:18)\n' +
    '    at /service/api/node_modules/pouchdb-adapter-http/lib/index.js:254:33\n' +
    '    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)'
} 
Info: Starting CHT API
2024-10-08T14:31:40.688 INFO: Running server checks… 
Node Version: 20.15.1
Node Mode: "development"
Node Environment Options: 'undefined'
CouchDB Version: 3.3.3
2024-10-08T14:31:40.779 INFO: Checks passed successfully 
2024-10-08T14:31:40.857 INFO: Extracting initial documents… 
2024-10-08T14:31:40.950 INFO: Extracting initial documents completed successfully 
2024-10-08T14:31:40.950 INFO: Merging translations… 
2024-10-08T14:31:41.110 ERROR: Fatal error initialising API 
2024-10-08T14:31:41.115 ERROR: {
  error: '{bad_return_value,\n' +
    '    {file_truncate_error,eof,\n' +
    '        "./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}}',
  reason: '{gen_server,call,[<0.18687.3>,{pread_iolist,295293},infinity]}',
  ref: 4070966599,
  status: 500,
  name: '{bad_return_value,\n' +
    '    {file_truncate_error,eof,\n' +
    '        "./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}}',
  message: '{gen_server,call,[<0.18687.3>,{pread_iolist,295293},infinity]}',
  stack: 'Error\n' +
    '    at Object.generateErrorFromResponse (/service/api/node_modules/pouchdb-errors/lib/index.js:104:18)\n' +
    '    at /service/api/node_modules/pouchdb-adapter-http/lib/index.js:254:33\n' +
    '    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)'
}

the instance is not powering in the browser ie says This site can’t be reached

6

@diana here is a view of the logs for all the services


~/cht/upgrade-service$ sudo docker ps -q | xargs -I {} sudo docker logs --tail 40 {}
      COUCHDB_USER: medic
      COUCHDB_UUID: 39f3718d-92fc-471b-9e7b-5d5590edae8a
      SVC_NAME: couchdb
    image: public.ecr.aws/medic/cht-couchdb:4.10.0
    logging:
      driver: local
      options:
        max-file: "20"
        max-size: 50m
    networks:
      cht-net: null
    restart: always
    volumes:
    - type: bind
      source: /home/ubuntu/cht/couchdb
      target: /opt/couchdb/data
      bind:
        create_host_path: true
    - type: volume
      source: cht-credentials
      target: /opt/couchdb/etc/local.d/
      volume: {}
networks:
  cht-net:
    name: cht-net
volumes:
  cht-credentials:
    name: cht_cht-credentials

Running cmd: docker compose -p cht -f /docker-compose/cht-core.yml -f /docker-compose/cht-couchdb.yml up -d --remove-orphans
Container ea0efd135791_ea0efd135791_ea0efd135791_ea0efd135791_ea0efd135791_ea0efd135791_cht-nginx-1  Recreate

Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /docker-compose/certs

Error while starting containers Error: Container ea0efd135791_ea0efd135791_ea0efd135791_ea0efd135791_ea0efd135791_ea0efd135791_cht-nginx-1  Recreate
Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /docker-compose/certs

    at ChildProcess.<anonymous> (/app/src/docker-compose-cli.js:35:25)
    at ChildProcess.emit (node:events:513:28)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:293:12)
    '        "./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}}',
  reason: '{gen_server,call,[<0.30903.3>,{pread_iolist,295293},infinity]}',
  ref: 4070966599,
  status: 500,
  name: '{bad_return_value,\n' +
    '    {file_truncate_error,eof,\n' +
    '        "./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}}',
  message: '{gen_server,call,[<0.30903.3>,{pread_iolist,295293},infinity]}',
  stack: 'Error\n' +
    '    at Object.generateErrorFromResponse (/service/api/node_modules/pouchdb-errors/lib/index.js:104:18)\n' +
    '    at /service/api/node_modules/pouchdb-adapter-http/lib/index.js:254:33\n' +
    '    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)'
} 
Info: Starting CHT API
2024-10-08T14:49:59.004 INFO: Running server checks… 
Node Version: 20.15.1
Node Mode: "development"
Node Environment Options: 'undefined'
CouchDB Version: 3.3.3
2024-10-08T14:49:59.099 INFO: Checks passed successfully 
2024-10-08T14:49:59.184 INFO: Extracting initial documents… 
2024-10-08T14:49:59.279 INFO: Extracting initial documents completed successfully 
2024-10-08T14:49:59.279 INFO: Merging translations… 
2024-10-08T14:49:59.439 ERROR: Fatal error initialising API 
2024-10-08T14:49:59.442 ERROR: {
  error: '{bad_return_value,\n' +
    '    {file_truncate_error,eof,\n' +
    '        "./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}}',
  reason: '{gen_server,call,[<0.31633.3>,{pread_iolist,295293},infinity]}',
  ref: 4070966599,
  status: 500,
  name: '{bad_return_value,\n' +
    '    {file_truncate_error,eof,\n' +
    '        "./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}}',
  message: '{gen_server,call,[<0.31633.3>,{pread_iolist,295293},infinity]}',
  stack: 'Error\n' +
    '    at Object.generateErrorFromResponse (/service/api/node_modules/pouchdb-errors/lib/index.js:104:18)\n' +
    '    at /service/api/node_modules/pouchdb-adapter-http/lib/index.js:254:33\n' +
    '    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)'
} 
2024-10-08T14:40:10.194 INFO: Waiting for API to be ready... 
2024-10-08T14:40:25.211 INFO: Waiting for API to be ready... 
2024-10-08T14:40:40.226 INFO: Waiting for API to be ready... 
2024-10-08T14:40:50.239 INFO: Waiting for API to be ready... 
2024-10-08T14:41:05.251 INFO: Waiting for API to be ready... 
2024-10-08T14:41:20.267 INFO: Waiting for API to be ready... 
2024-10-08T14:41:35.283 INFO: Waiting for API to be ready... 
2024-10-08T14:41:50.299 INFO: Waiting for API to be ready... 
2024-10-08T14:42:05.313 INFO: Waiting for API to be ready... 
2024-10-08T14:42:20.324 INFO: Waiting for API to be ready... 
2024-10-08T14:42:35.341 INFO: Waiting for API to be ready... 
2024-10-08T14:42:50.349 INFO: Waiting for API to be ready... 
2024-10-08T14:43:05.366 INFO: Waiting for API to be ready... 
2024-10-08T14:43:20.381 INFO: Waiting for API to be ready... 
2024-10-08T14:43:35.398 INFO: Waiting for API to be ready... 
2024-10-08T14:43:50.414 INFO: Waiting for API to be ready... 
2024-10-08T14:44:05.430 INFO: Waiting for API to be ready... 
2024-10-08T14:44:20.445 INFO: Waiting for API to be ready... 
2024-10-08T14:44:35.461 INFO: Waiting for API to be ready... 
2024-10-08T14:44:50.478 INFO: Waiting for API to be ready... 
2024-10-08T14:45:05.493 INFO: Waiting for API to be ready... 
2024-10-08T14:45:20.509 INFO: Waiting for API to be ready... 
2024-10-08T14:45:35.525 INFO: Waiting for API to be ready... 
2024-10-08T14:45:50.541 INFO: Waiting for API to be ready... 
2024-10-08T14:46:05.549 INFO: Waiting for API to be ready... 
2024-10-08T14:46:20.565 INFO: Waiting for API to be ready... 
2024-10-08T14:46:35.581 INFO: Waiting for API to be ready... 
2024-10-08T14:46:50.597 INFO: Waiting for API to be ready... 
2024-10-08T14:47:05.613 INFO: Waiting for API to be ready... 
2024-10-08T14:47:20.629 INFO: Waiting for API to be ready... 
2024-10-08T14:47:35.645 INFO: Waiting for API to be ready... 
2024-10-08T14:47:50.662 INFO: Waiting for API to be ready... 
2024-10-08T14:48:05.679 INFO: Waiting for API to be ready... 
2024-10-08T14:48:20.696 INFO: Waiting for API to be ready... 
2024-10-08T14:48:35.712 INFO: Waiting for API to be ready... 
2024-10-08T14:48:50.728 INFO: Waiting for API to be ready... 
2024-10-08T14:49:05.744 INFO: Waiting for API to be ready... 
2024-10-08T14:49:20.760 INFO: Waiting for API to be ready... 
2024-10-08T14:49:35.777 INFO: Waiting for API to be ready... 
2024-10-08T14:49:50.793 INFO: Waiting for API to be ready... 
<150>Oct  8 14:46:56 haproxy[12]: 172.18.0.7,couchdb,200,1,141,0,POST,/_session,api,medic,'{"name":"medic","password":"***"}',399,1,54,'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)'
<150>Oct  8 14:46:56 haproxy[12]: 172.18.0.7,couchdb,200,6,13,0,GET,/medic/,api,0,'-',1116,6,893,'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)'
<150>Oct  8 14:46:56 haproxy[12]: 172.18.0.7,couchdb,200,3,3,0,GET,/medic/settings?,api,0,'-',124008,3,123742,'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)'
<150>Oct  8 14:46:56 haproxy[12]: 172.18.0.7,couchdb,500,45,117,0,GET,/medic/_all_docs?include_docs=true&startkey=%22messages-%22&endkey=%22messages-%EF%BF%B0%22,api,0,'-',493,11,223,'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)'
<150>Oct  8 14:47:57 haproxy[12]: 172.18.0.7,couchdb,200,2,0,0,GET,/,-,medic,'-',474,2,251,'-'
<150>Oct  8 14:47:57 haproxy[12]: 172.18.0.7,couchdb,401,0,5,0,GET,/,-,-,'-',348,0,61,'-'
<150>Oct  8 14:47:57 haproxy[12]: 172.18.0.7,couchdb,200,1,0,0,GET,/_membership,-,medic,'-',248,1,74,'-'
<150>Oct  8 14:47:57 haproxy[12]: 172.18.0.7,couchdb,200,1,1,1,GET,/_membership,-,medic,'-',248,0,74,'-'
<150>Oct  8 14:47:57 haproxy[12]: 172.18.0.7,couchdb,200,1,1,0,GET,/_membership,-,medic,'-',248,1,74,'-'
<150>Oct  8 14:47:57 haproxy[12]: 172.18.0.7,couchdb,200,4,1,0,GET,/_users,-,medic,'-',1092,4,869,'-'
<150>Oct  8 14:47:57 haproxy[12]: 172.18.0.7,couchdb,200,7,1,1,GET,/_replicator,-,medic,'-',1084,6,861,'-'
<150>Oct  8 14:47:57 haproxy[12]: 172.18.0.7,couchdb,200,6,1,0,GET,/_global_changes,-,medic,'-',1122,6,899,'-'
<150>Oct  8 14:47:57 haproxy[12]: 172.18.0.7,couchdb,200,1,141,0,POST,/_session,api,medic,'{"name":"medic","password":"***"}',399,1,54,'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)'
<150>Oct  8 14:47:57 haproxy[12]: 172.18.0.7,couchdb,200,6,13,0,GET,/medic/,api,0,'-',1116,6,893,'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)'
<150>Oct  8 14:47:57 haproxy[12]: 172.18.0.7,couchdb,200,3,3,0,GET,/medic/settings?,api,0,'-',124008,3,123742,'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)'
<150>Oct  8 14:47:57 haproxy[12]: 172.18.0.7,couchdb,500,45,116,0,GET,/medic/_all_docs?include_docs=true&startkey=%22messages-%22&endkey=%22messages-%EF%BF%B0%22,api,0,'-',493,4,223,'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)'
<150>Oct  8 14:48:58 haproxy[12]: 172.18.0.7,couchdb,200,6,4,1,GET,/,-,medic,'-',474,5,251,'-'
<150>Oct  8 14:48:58 haproxy[12]: 172.18.0.7,couchdb,401,1,5,0,GET,/,-,-,'-',348,1,61,'-'
<150>Oct  8 14:48:58 haproxy[12]: 172.18.0.7,couchdb,200,1,0,0,GET,/_membership,-,medic,'-',248,1,74,'-'
<150>Oct  8 14:48:58 haproxy[12]: 172.18.0.7,couchdb,200,1,1,1,GET,/_membership,-,medic,'-',248,0,74,'-'
<150>Oct  8 14:48:58 haproxy[12]: 172.18.0.7,couchdb,200,1,1,0,GET,/_membership,-,medic,'-',248,1,74,'-'
<150>Oct  8 14:48:58 haproxy[12]: 172.18.0.7,couchdb,200,4,1,0,GET,/_users,-,medic,'-',1092,4,869,'-'
<150>Oct  8 14:48:58 haproxy[12]: 172.18.0.7,couchdb,200,6,1,1,GET,/_replicator,-,medic,'-',1084,5,861,'-'
<150>Oct  8 14:48:58 haproxy[12]: 172.18.0.7,couchdb,200,3,1,0,GET,/_global_changes,-,medic,'-',1122,3,899,'-'
<150>Oct  8 14:48:58 haproxy[12]: 172.18.0.7,couchdb,200,2,141,0,POST,/_session,api,medic,'{"name":"medic","password":"***"}',399,2,54,'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)'
<150>Oct  8 14:48:58 haproxy[12]: 172.18.0.7,couchdb,200,3,13,0,GET,/medic/,api,0,'-',1116,3,893,'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)'
<150>Oct  8 14:48:58 haproxy[12]: 172.18.0.7,couchdb,200,3,5,0,GET,/medic/settings?,api,0,'-',124008,3,123742,'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)'
<150>Oct  8 14:48:58 haproxy[12]: 172.18.0.7,couchdb,500,42,118,0,GET,/medic/_all_docs?include_docs=true&startkey=%22messages-%22&endkey=%22messages-%EF%BF%B0%22,api,0,'-',493,6,223,'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)'
<150>Oct  8 14:49:59 haproxy[12]: 172.18.0.7,couchdb,200,3,6,1,GET,/,-,medic,'-',474,2,251,'-'
<150>Oct  8 14:49:59 haproxy[12]: 172.18.0.7,couchdb,401,0,7,0,GET,/,-,-,'-',348,0,61,'-'
<150>Oct  8 14:49:59 haproxy[12]: 172.18.0.7,couchdb,200,1,0,0,GET,/_membership,-,medic,'-',248,1,74,'-'
<150>Oct  8 14:49:59 haproxy[12]: 172.18.0.7,couchdb,200,1,1,1,GET,/_membership,-,medic,'-',248,0,74,'-'
<150>Oct  8 14:49:59 haproxy[12]: 172.18.0.7,couchdb,200,1,1,0,GET,/_membership,-,medic,'-',248,1,74,'-'
<150>Oct  8 14:49:59 haproxy[12]: 172.18.0.7,couchdb,200,4,1,1,GET,/_users,-,medic,'-',1092,3,869,'-'
<150>Oct  8 14:49:59 haproxy[12]: 172.18.0.7,couchdb,200,6,1,0,GET,/_replicator,-,medic,'-',1084,6,861,'-'
<150>Oct  8 14:49:59 haproxy[12]: 172.18.0.7,couchdb,200,8,1,1,GET,/_global_changes,-,medic,'-',1122,7,899,'-'
<150>Oct  8 14:49:59 haproxy[12]: 172.18.0.7,couchdb,200,2,148,0,POST,/_session,api,medic,'{"name":"medic","password":"***"}',399,2,54,'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)'
<150>Oct  8 14:49:59 haproxy[12]: 172.18.0.7,couchdb,200,6,13,0,GET,/medic/,api,0,'-',1116,6,893,'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)'
<150>Oct  8 14:49:59 haproxy[12]: 172.18.0.7,couchdb,200,3,3,0,GET,/medic/settings?,api,0,'-',124008,3,123742,'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)'
<150>Oct  8 14:49:59 haproxy[12]: 172.18.0.7,couchdb,500,44,119,0,GET,/medic/_all_docs?include_docs=true&startkey=%22messages-%22&endkey=%22messages-%EF%BF%B0%22,api,0,'-',493,10,223,'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)'
2024-09-06 17:03:45,614 - healthcheck - WARNING - Serving on ('0.0.0.0', 5555)
2024-09-19 19:47:05,776 - healthcheck - WARNING - Serving on ('0.0.0.0', 5555)
2024-09-23 09:13:35,819 - healthcheck - WARNING - Serving on ('0.0.0.0', 5555)
2024-09-23 09:18:33,524 - healthcheck - WARNING - Serving on ('0.0.0.0', 5555)
2024-10-07 13:25:10,808 - healthcheck - WARNING - Serving on ('0.0.0.0', 5555)
2024-10-08 11:48:26,121 - healthcheck - WARNING - Serving on ('0.0.0.0', 5555)
[error] 2024-10-08T14:49:59.422396Z couchdb@127.0.0.1 <0.24947.3> 8d30c1db7d req_err(4070966599) {bad_return_value,
    {file_truncate_error,eof,
        "./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}} : {gen_server,call,[<0.31597.3>,{pread_iolist,295293},infinity]}
    [<<"gen_server:call/3 L247">>,<<"couch_file:pread_iolist/2 L170">>,<<"couch_file:pread_binary/2 L166">>,<<"couch_file:pread_term/2 L155">>,<<"couch_btree:get_node/2 L474">>,<<"couch_btree:stream_node/8 L1069">>,<<"couch_btree:fold/4 L242">>,<<"couch_bt_engine:fold_docs_int/5 L1129">>]
[notice] 2024-10-08T14:49:59.422913Z couchdb@127.0.0.1 <0.24947.3> 8d30c1db7d haproxy:5984 172.18.0.7 medic GET /medic/_all_docs?include_docs=true&startkey=%22messages-%22&endkey=%22messages-%EF%BF%B0%22 500 ok 8
[info] 2024-10-08T14:49:59.427973Z couchdb@127.0.0.1 <0.260.0> -------- db shards/eaaaaaa7-ffffffff/medic.1721926601 died with reason {bad_return_value,{file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}}
[error] 2024-10-08T14:49:59.428000Z couchdb@127.0.0.1 <0.31617.3> ffd56e5fe0 rexi_server: from: couchdb@127.0.0.1(<0.24947.3>) mfa: fabric_rpc:all_docs/3 exit:{{bad_return_value,{file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}},{gen_server,call,[<0.31615.3>,{pread_iolist,295293},infinity]}} [{gen_server,call,3,[{file,"gen_server.erl"},{line,247}]},{couch_file,pread_iolist,2,[{file,"src/couch_file.erl"},{line,170}]},{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,166}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,155}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,474}]},{couch_btree,stream_node,8,[{file,"src/couch_btree.erl"},{line,1069}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,242}]},{couch_bt_engine,fold_docs_int,5,[{file,"src/couch_bt_engine.erl"},{line,1129}]}]
[error] 2024-10-08T14:49:59.428165Z couchdb@127.0.0.1 <0.24947.3> ffd56e5fe0 req_err(4070966599) {bad_return_value,
    {file_truncate_error,eof,
        "./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}} : {gen_server,call,[<0.31615.3>,{pread_iolist,295293},infinity]}
    [<<"gen_server:call/3 L247">>,<<"couch_file:pread_iolist/2 L170">>,<<"couch_file:pread_binary/2 L166">>,<<"couch_file:pread_term/2 L155">>,<<"couch_btree:get_node/2 L474">>,<<"couch_btree:stream_node/8 L1069">>,<<"couch_btree:fold/4 L242">>,<<"couch_bt_engine:fold_docs_int/5 L1129">>]
[error] 2024-10-08T14:49:59.428894Z couchdb@127.0.0.1 <0.31615.3> -------- gen_server <0.31615.3> terminated with reason: bad return value {file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}
  last msg: redacted
     state: [{data,[{"State",{file,{file_descriptor,prim_file,#{handle => #Ref<0.2384282298.345112582.122247>,owner => <0.31615.3>,r_ahead_size => 0,r_buffer => #Ref<0.2384282298.345636865.212641>}},false,303316,#Ref<0.2384282298.345505793.212647>,infinity}},{"InitialFilePath","./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}]}]
    extra: [<0.31617.3>,[{gen,do_call,4,[{file,"gen.erl"},{line,214}]},{gen_server,call,3,[{file,"gen_server.erl"},{line,243}]},{couch_file,pread_iolist,2,[{file,"src/couch_file.erl"},{line,170}]},{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,166}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,155}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,474}]},{couch_btree,stream_node,8,[{file,"src/couch_btree.erl"},{line,1069}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,242}]}]]
[error] 2024-10-08T14:49:59.429219Z couchdb@127.0.0.1 <0.31615.3> -------- gen_server <0.31615.3> terminated with reason: bad return value {file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}
  last msg: redacted
     state: [{data,[{"State",{file,{file_descriptor,prim_file,#{handle => #Ref<0.2384282298.345112582.122247>,owner => <0.31615.3>,r_ahead_size => 0,r_buffer => #Ref<0.2384282298.345636865.212641>}},false,303316,#Ref<0.2384282298.345505793.212647>,infinity}},{"InitialFilePath","./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}]}]
    extra: [<0.31617.3>,[{gen,do_call,4,[{file,"gen.erl"},{line,214}]},{gen_server,call,3,[{file,"gen_server.erl"},{line,243}]},{couch_file,pread_iolist,2,[{file,"src/couch_file.erl"},{line,170}]},{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,166}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,155}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,474}]},{couch_btree,stream_node,8,[{file,"src/couch_btree.erl"},{line,1069}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,242}]}]]
[error] 2024-10-08T14:49:59.429441Z couchdb@127.0.0.1 <0.31615.3> -------- CRASH REPORT Process  (<0.31615.3>) with 1 neighbors exited with reason: bad return value {file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"} at gen_server:handle_common_reply/8(line:815) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_file,init,['Argument__1']}, ancestors: [<0.31614.3>,<0.31613.3>], message_queue_len: 0, links: [<0.31614.3>], dictionary: [{couch_file_fd,{{file_descriptor,prim_file,#{handle => #Ref<0.2384282298.345112582.122247>,...}},...}},...], trap_exit: false, status: running, heap_size: 2586, stack_size: 29, reductions: 1658
[error] 2024-10-08T14:49:59.429656Z couchdb@127.0.0.1 <0.31615.3> -------- CRASH REPORT Process  (<0.31615.3>) with 1 neighbors exited with reason: bad return value {file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"} at gen_server:handle_common_reply/8(line:815) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_file,init,['Argument__1']}, ancestors: [<0.31614.3>,<0.31613.3>], message_queue_len: 0, links: [<0.31614.3>], dictionary: [{couch_file_fd,{{file_descriptor,prim_file,#{handle => #Ref<0.2384282298.345112582.122247>,...}},...}},...], trap_exit: false, status: running, heap_size: 2586, stack_size: 29, reductions: 1658
[notice] 2024-10-08T14:49:59.431085Z couchdb@127.0.0.1 <0.24947.3> ffd56e5fe0 haproxy:5984 172.18.0.7 medic GET /medic/_all_docs?include_docs=true&startkey=%22messages-%22&endkey=%22messages-%EF%BF%B0%22 500 ok 8
[error] 2024-10-08T14:49:59.435795Z couchdb@127.0.0.1 <0.31633.3> -------- gen_server <0.31633.3> terminated with reason: bad return value {file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}
  last msg: redacted
     state: [{data,[{"State",{file,{file_descriptor,prim_file,#{handle => #Ref<0.2384282298.345112586.122230>,owner => <0.31633.3>,r_ahead_size => 0,r_buffer => #Ref<0.2384282298.345374722.155941>}},false,303316,#Ref<0.2384282298.345505793.212756>,infinity}},{"InitialFilePath","./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}]}]
    extra: [<0.31634.3>,[{gen,do_call,4,[{file,"gen.erl"},{line,214}]},{gen_server,call,3,[{file,"gen_server.erl"},{line,243}]},{couch_file,pread_iolist,2,[{file,"src/couch_file.erl"},{line,170}]},{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,166}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,155}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,474}]},{couch_btree,stream_node,8,[{file,"src/couch_btree.erl"},{line,1069}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,242}]}]]
[info] 2024-10-08T14:49:59.435897Z couchdb@127.0.0.1 <0.260.0> -------- db shards/eaaaaaa7-ffffffff/medic.1721926601 died with reason {bad_return_value,{file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}}
[error] 2024-10-08T14:49:59.436062Z couchdb@127.0.0.1 <0.31633.3> -------- gen_server <0.31633.3> terminated with reason: bad return value {file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}
  last msg: redacted
     state: [{data,[{"State",{file,{file_descriptor,prim_file,#{handle => #Ref<0.2384282298.345112586.122230>,owner => <0.31633.3>,r_ahead_size => 0,r_buffer => #Ref<0.2384282298.345374722.155941>}},false,303316,#Ref<0.2384282298.345505793.212756>,infinity}},{"InitialFilePath","./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}]}]
    extra: [<0.31634.3>,[{gen,do_call,4,[{file,"gen.erl"},{line,214}]},{gen_server,call,3,[{file,"gen_server.erl"},{line,243}]},{couch_file,pread_iolist,2,[{file,"src/couch_file.erl"},{line,170}]},{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,166}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,155}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,474}]},{couch_btree,stream_node,8,[{file,"src/couch_btree.erl"},{line,1069}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,242}]}]]
[error] 2024-10-08T14:49:59.436097Z couchdb@127.0.0.1 <0.31634.3> b50349e84e rexi_server: from: couchdb@127.0.0.1(<0.24947.3>) mfa: fabric_rpc:all_docs/3 exit:{{bad_return_value,{file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}},{gen_server,call,[<0.31633.3>,{pread_iolist,295293},infinity]}} [{gen_server,call,3,[{file,"gen_server.erl"},{line,247}]},{couch_file,pread_iolist,2,[{file,"src/couch_file.erl"},{line,170}]},{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,166}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,155}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,474}]},{couch_btree,stream_node,8,[{file,"src/couch_btree.erl"},{line,1069}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,242}]},{couch_bt_engine,fold_docs_int,5,[{file,"src/couch_bt_engine.erl"},{line,1129}]}]
[error] 2024-10-08T14:49:59.436209Z couchdb@127.0.0.1 <0.31633.3> -------- CRASH REPORT Process  (<0.31633.3>) with 1 neighbors exited with reason: bad return value {file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"} at gen_server:handle_common_reply/8(line:815) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_file,init,['Argument__1']}, ancestors: [<0.31632.3>,<0.31631.3>], message_queue_len: 0, links: [<0.31632.3>], dictionary: [{couch_file_fd,{{file_descriptor,prim_file,#{handle => #Ref<0.2384282298.345112586.122230>,...}},...}},...], trap_exit: false, status: running, heap_size: 2586, stack_size: 29, reductions: 1658
[error] 2024-10-08T14:49:59.436719Z couchdb@127.0.0.1 <0.31633.3> -------- CRASH REPORT Process  (<0.31633.3>) with 1 neighbors exited with reason: bad return value {file_truncate_error,eof,"./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"} at gen_server:handle_common_reply/8(line:815) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_file,init,['Argument__1']}, ancestors: [<0.31632.3>,<0.31631.3>], message_queue_len: 0, links: [<0.31632.3>], dictionary: [{couch_file_fd,{{file_descriptor,prim_file,#{handle => #Ref<0.2384282298.345112586.122230>,...}},...}},...], trap_exit: false, status: running, heap_size: 2586, stack_size: 29, reductions: 1658
[error] 2024-10-08T14:49:59.437691Z couchdb@127.0.0.1 <0.24947.3> b50349e84e req_err(4070966599) {bad_return_value,
    {file_truncate_error,eof,
        "./data/shards/eaaaaaa7-ffffffff/medic.1721926601.couch"}} : {gen_server,call,[<0.31633.3>,{pread_iolist,295293},infinity]}
    [<<"gen_server:call/3 L247">>,<<"couch_file:pread_iolist/2 L170">>,<<"couch_file:pread_binary/2 L166">>,<<"couch_file:pread_term/2 L155">>,<<"couch_btree:get_node/2 L474">>,<<"couch_btree:stream_node/8 L1069">>,<<"couch_btree:fold/4 L242">>,<<"couch_bt_engine:fold_docs_int/5 L1129">>]
[notice] 2024-10-08T14:49:59.438162Z couchdb@127.0.0.1 <0.24947.3> b50349e84e haproxy:5984 172.18.0.7 medic GET /medic/_all_docs?include_docs=true&startkey=%22messages-%22&endkey=%22messages-%EF%BF%B0%22 500 ok 7
[notice] 2024-10-08T14:50:00.695953Z couchdb@127.0.0.1 <0.31415.3> 89edb9690b couchdb:5984 172.18.0.3 medic GET /_membership 200 ok 0
7

It looks like your CouchDb data somehow became corrupted.
Can you try re-doing the migration from 3.x to 4.x, it’s possible copying data over caused the corruption.

8

thanks @diana
am not doing the migration from 3.x to 4.x, i did just set up a 4.x instance , may be fresh set up with deleted volumes could fix this ?

9

So what’s the history of this instance then? I’m curious.
You set it up with empty volumes? Did you replicate data to it?
What type of volume are you using for CouchDb data?

10

thanks @diana
its on aws ,
yes we did set up with empty volumes , was following these steps

could you kindly clarify on this question

11

@diana we had basically set i up a testing pilot 4.x instance

12

hello @diana we decide to put down everything a do a fresh clean slate setup again, now able to access the system in the UI .
However when i login with username = medic and password gotten from running grep COUCHDB_PASSWORD /home/ubuntu/cht/upgrade-service/.env | cut -d'=' -f2 its failing to login .
i checked the couchDB logs and saw
[warning] 2024-10-09T08:59:15.017271Z couchdb@127.0.0.1 <0.23543.1> c829e2656c couch_httpd_auth: Authentication failed for user medic from 172.16.0.6

Wondering if you have encountered this before

13

hi @cliff

If your instance is up, it means that the password in the env file is the correct one, because API uses that password to connect to CouchDb.
So unless you’re somehow reusing old volumes and old containers, that should be the correct password.

Without knowing exactly what your steps were, it’s impossible to say what is happening.

14

thanks @diana i think it could be around the volumes because i have tried to use the old password and it still works yet the new one from

**ubuntu@ip-172-51-61-13**:**~/cht**$ grep COUCHDB_PASSWORD /home/ubuntu/cht/upgrade-service/.env | cut -d'=' -f2

allergists-foray-Gienah-aardvarks-watchmakers-list-dodge-scarifies

is not working

15

Then you’re for sure using some old volume. You can do a volume deletion and restart, or you could change the admin password manually. I believe this can be done through Fauxton and it will update in the volume as well.

image
image655Ă—504 39.8 KB

1 Like