Stuck in Initialization Loop - FetchError: Invalid JSON Response Body

Technical Support
1

Hello @diana @jkuester

I’m encountering an issue and have been stuck in this loop for the past 48 hours. The instance is on version 3.9 has approximately 40 million documents.

Here are the details of the error from medic-api logs:

[2024-08-02 04:24:27] CouchDB Version: 2.3.1
[2024-08-02 04:24:29] 2024-08-02 04:24:29 INFO: Extracting ddoc…
[2024-08-02 04:24:29] 2024-08-02 04:24:29 INFO: DDoc extraction completed successfully
[2024-08-02 04:24:29] 2024-08-02 04:24:29 INFO: Extracting resources…
[2024-08-02 04:24:30] 2024-08-02 04:24:30 INFO: Extracting resources completed successfully
[2024-08-02 04:24:30] 2024-08-02 04:24:30 INFO: Extracting initial documents…
[2024-08-02 04:24:30] 2024-08-02 04:24:30 INFO: Extracting initial documents completed successfully
[2024-08-02 04:24:30] 2024-08-02 04:24:30 INFO: Loading configuration…
[2024-08-02 04:24:30] 2024-08-02 04:24:30 INFO: Not updating settings - the existing settings are already up to date
[2024-08-02 05:24:30] 2024-08-02 05:24:30 ERROR: Error loading translations - starting up anyway: { FetchError: invalid json response body at http://haproxy:5984/medic/_design/medic-client/_view/doc_by_type?include_docs=true&key=%5B%22translations%22%2Ctrue%5D reason: Unexpected token < in JSON at position 0
[2024-08-02 05:24:30]     at /srv/software/medic-api/md5-O3yLnjz8HcvO7g+wuf0kiw==/node_modules/node-fetch/lib/index.js:275:32
[2024-08-02 05:24:30]     at <anonymous>
[2024-08-02 05:24:30]     at process._tickCallback (internal/process/next_tick.js:188:7)
[2024-08-02 05:24:30]   message: 'invalid json response body at http://haproxy:5984/medic/_design/medic-client/_view/doc_by_type?include_docs=true&key=%5B%22translations%22%2Ctrue%5D reason: Unexpected token < in JSON at position 0',
[2024-08-02 05:24:30]   type: 'invalid-json',
[2024-08-02 05:24:30]   [stack]: 'FetchError: invalid json response body at http://haproxy:5984/medic/_design/medic-client/_view/doc_by_type?include_docs=true&key=%5B%22translations%22%2Ctrue%5D reason: Unexpected token < in JSON at position 0\n    at /srv/software/medic-api/md5-O3yLnjz8HcvO7g+wuf0kiw==/node_modules/node-fetch/lib/index.js:275:32\n    at <anonymous>\n    at process._tickCallback (internal/process/next_tick.js:188:7)' }
[2024-08-02 05:24:30] 2024-08-02 05:24:30 WARN: Disabled transition "update_clinics"
[2024-08-02 05:24:30] 2024-08-02 05:24:30 WARN: Disabled transition "self_report"
[2024-08-02 05:24:30] 2024-08-02 05:24:30 WARN: Disabled transition "registration"
[2024-08-02 05:24:30] 2024-08-02 05:24:30 WARN: Disabled transition "accept_patient_reports"
[2024-08-02 05:24:30] 2024-08-02 05:24:30 WARN: Disabled transition "accept_case_reports"
[2024-08-02 05:24:30] 2024-08-02 05:24:30 WARN: Disabled transition "generate_shortcode_on_contacts"
[2024-08-02 05:24:30] 2024-08-02 05:24:30 INFO: Loading transition "generate_patient_id_on_people"
[2024-08-02 05:24:30] 2024-08-02 05:24:30 INFO: Skipping asynchronous transition "generate_patient_id_on_people"
[2024-08-02 05:24:30] 2024-08-02 05:24:30 WARN: Disabled transition "default_responses"
[2024-08-02 05:24:30] 2024-08-02 05:24:30 WARN: Disabled transition "update_sent_by"
[2024-08-02 05:24:30] 2024-08-02 05:24:30 WARN: Disabled transition "update_sent_forms"
[2024-08-02 05:24:30] 2024-08-02 05:24:30 INFO: Loading transition "death_reporting"
[2024-08-02 05:24:30] 2024-08-02 05:24:30 WARN: Disabled transition "conditional_alerts"
[2024-08-02 05:24:30] 2024-08-02 05:24:30 WARN: Disabled transition "multi_report_alerts"
[2024-08-02 05:24:30] 2024-08-02 05:24:30 WARN: Disabled transition "update_notifications"
[2024-08-02 05:24:30] 2024-08-02 05:24:30 WARN: Disabled transition "update_scheduled_reports"
[2024-08-02 05:24:30] 2024-08-02 05:24:30 WARN: Disabled transition "resolve_pending"
[2024-08-02 05:24:30] 2024-08-02 05:24:30 WARN: Disabled transition "muting"
[2024-08-02 05:24:30] 2024-08-02 05:24:30 WARN: Disabled transition "mark_for_outbound"
[2024-08-02 05:24:30] 2024-08-02 05:24:30 INFO: Configuration loaded successfully
[2024-08-02 05:24:30] 2024-08-02 05:24:30 INFO: Merging translations…

The critical error seems to be:

ERROR: Error loading translations - starting up anyway: { FetchError: invalid json response body at http://haproxy:5984/medic/_design/medic-client/_view/doc_by_type?include_docs=true&key=%5B%22translations%22%2Ctrue%5D reason: Unexpected token < in JSON at position 0

I’ve tried restarting the process multiple times, but the problem persists. Any help would be greatly appreciated.

Many thanks!

2

Hi @Steve_Roy

My guess is that there is some CouchDb view which is indexing.
Do you know how to access CoudhDb from within the medic-os container to check?

3

@diana

I have specifically extracted database and progress fields from couchDB. The output is as below:

[
  {
    "database": "shards/80000000-9fffffff/medic.1716394078",
    "progress": 47
  },
  {
    "database": "shards/20000000-3fffffff/medic.1716394078",
    "progress": 56
  },
  {
    "database": "shards/e0000000-ffffffff/medic.1716394078",
    "progress": 47
  },
  {
    "database": "shards/40000000-5fffffff/medic.1716394078",
    "progress": 56
  },
  {
    "database": "shards/c0000000-dfffffff/medic.1716394078",
    "progress": 47
  },
  {
    "database": "shards/60000000-7fffffff/medic.1716394078",
    "progress": 56
  },
  {
    "database": "shards/a0000000-bfffffff/medic.1716394078",
    "progress": 48
  },
  {
    "database": "shards/00000000-1fffffff/medic.1716394078",
    "progress": 56
  }
]

Below are logs from couchDB.

tail -n100 storage/medic-core/couchdb/logs/startup.log
[notice] 2024-08-03T05:09:44.228471Z couchdb@127.0.0.1 <0.2131.103> 9392b4c62a haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/translations/messages-hi.properties 200 ok 23
[notice] 2024-08-03T05:21:08.956791Z couchdb@127.0.0.1 <0.396.0> -------- couch_replicator_scheduler: Job {"1eca5b976aa55d40bbcd006914265175","+continuous"} started as <0.16154.103>
[notice] 2024-08-03T05:21:08.957028Z couchdb@127.0.0.1 <0.396.0> -------- couch_replicator_scheduler: Job {"e7a18dec27c9f689569bd34179c3b740","+continuous"} started as <0.16112.103>
[error] 2024-08-03T05:21:09.520538Z couchdb@127.0.0.1 <0.16154.103> -------- throw:{invalid_json,{error,{1,invalid_json}}}: Replication 1eca5b976aa55d40bbcd006914265175+continuous failed to start "https://example1.com/_users/" -> "https://example2.com/_users/" doc <<"shards/40000000-5fffffff/_replicator.1716394076">>:<<"8d3b717e1f84654c1799900ec3000652">> stack:[{couch_replicator_api_wrap,db_open,4,[{file,"src/couch_replicator_api_wrap.erl"},{line,114}]},{couch_replicator_scheduler_job,init_state,1,[{file,"src/couch_replicator_scheduler_job.erl"},{line,584}]}]
[error] 2024-08-03T05:21:11.672804Z couchdb@127.0.0.1 <0.16112.103> -------- throw:{invalid_json,{error,{1,invalid_json}}}: Replication e7a18dec27c9f689569bd34179c3b740+continuous failed to start "https://example1.com/medic/" -> "https://example2.com/medic/" doc <<"shards/a0000000-bfffffff/_replicator.1716394076">>:<<"8d3b717e1f84654c1799900ec3002388">> stack:[{couch_replicator_api_wrap,db_open,4,[{file,"src/couch_replicator_api_wrap.erl"},{line,114}]},{couch_replicator_scheduler_job,init_state,1,[{file,"src/couch_replicator_scheduler_job.erl"},{line,584}]}]
[notice] 2024-08-03T06:09:48.073156Z couchdb@127.0.0.1 <0.3644.105> dd3cf45246 haproxy:5984 172.18.0.2 undefined GET /medic 401 ok 0
[notice] 2024-08-03T06:09:48.091707Z couchdb@127.0.0.1 <0.12292.105> ce202caa20 haproxy:5984 172.18.0.2 medic-api GET / 200 ok 11
[notice] 2024-08-03T06:09:51.037474Z couchdb@127.0.0.1 <0.12235.105> e7c9b98ca1 haproxy:5984 172.18.0.2 medic-api GET /medic/ 200 ok 3
[notice] 2024-08-03T06:09:51.037899Z couchdb@127.0.0.1 <0.12299.105> 8f3bf75017 haproxy:5984 172.18.0.2 medic-api GET /medic-sentinel/ 200 ok 4
[notice] 2024-08-03T06:09:51.048365Z couchdb@127.0.0.1 <0.12097.105> 159ab15a57 haproxy:5984 172.18.0.2 medic-api GET /medic-sentinel/ 200 ok 1
[notice] 2024-08-03T06:09:51.048827Z couchdb@127.0.0.1 <0.10906.105> b1d6e0361c haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic? 200 ok 2
[notice] 2024-08-03T06:09:51.066737Z couchdb@127.0.0.1 <0.12194.105> 437aca796a haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/ddocs/sentinel.json 200 ok 6
[notice] 2024-08-03T06:09:51.066788Z couchdb@127.0.0.1 <0.12232.105> 838b27f7fb haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/ddocs/users-meta.json 200 ok 6
[notice] 2024-08-03T06:09:51.083057Z couchdb@127.0.0.1 <0.12267.105> 31c69aaeb1 haproxy:5984 172.18.0.2 medic-api GET /medic-sentinel/_design/sentinel? 200 ok 9
[notice] 2024-08-03T06:09:51.136830Z couchdb@127.0.0.1 <0.12154.105> d355105ee9 haproxy:5984 172.18.0.2 medic-api GET /medic-users-meta/ 200 ok 62
[notice] 2024-08-03T06:09:51.142692Z couchdb@127.0.0.1 <0.12165.105> c69d992bb6 haproxy:5984 172.18.0.2 medic-api GET /medic-users-meta/_design/users-meta? 200 ok 1
[notice] 2024-08-03T06:09:51.163484Z couchdb@127.0.0.1 <0.12184.105> a5812b9f7d haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/ddocs/medic.json 200 ok 103
[notice] 2024-08-03T06:09:51.481475Z couchdb@127.0.0.1 <0.12268.105> ae838dd44f haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic-sms? 200 ok 3
[notice] 2024-08-03T06:09:51.481642Z couchdb@127.0.0.1 <0.12388.105> faa6b136e6 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic-conflicts? 200 ok 3
[notice] 2024-08-03T06:09:51.482680Z couchdb@127.0.0.1 <0.12238.105> 370bb10418 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic-client? 200 ok 4
[notice] 2024-08-03T06:09:51.489418Z couchdb@127.0.0.1 <0.12331.105> b83b61922d haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic-scripts? 200 ok 11
[notice] 2024-08-03T06:09:51.489602Z couchdb@127.0.0.1 <0.8906.105> e5c4d74fb2 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic-admin? 200 ok 12
[notice] 2024-08-03T06:09:51.517902Z couchdb@127.0.0.1 <0.12272.105> a81df8a621 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic-admin/fonts/enketo-icons-v2.svg?rev=5-a305aa6b73dc95fad7e40326e2edda1c 200 ok 18
[notice] 2024-08-03T06:09:51.523630Z couchdb@127.0.0.1 <0.12436.105> 78784bfd21 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic-admin/fonts/enketo-icons-v2.eot?rev=5-a305aa6b73dc95fad7e40326e2edda1c 200 ok 24
[notice] 2024-08-03T06:09:51.523920Z couchdb@127.0.0.1 <0.12138.105> b7464b6e38 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic-admin/fonts/enketo-icons-v2.woff?rev=5-a305aa6b73dc95fad7e40326e2edda1c 200 ok 24
[notice] 2024-08-03T06:09:51.524061Z couchdb@127.0.0.1 <0.12364.105> 5535416e14 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic-admin/fonts/enketo-icons-v2.ttf?rev=5-a305aa6b73dc95fad7e40326e2edda1c 200 ok 24
[notice] 2024-08-03T06:09:51.524244Z couchdb@127.0.0.1 <0.12188.105> 3dbd291f13 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic-admin/css/main.css?rev=5-a305aa6b73dc95fad7e40326e2edda1c 200 ok 25
[notice] 2024-08-03T06:09:51.533522Z couchdb@127.0.0.1 <0.12419.105> 35a92a8de7 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic-admin/fonts/fontawesome-webfont.eot?rev=5-a305aa6b73dc95fad7e40326e2edda1c 200 ok 10
[notice] 2024-08-03T06:09:51.544197Z couchdb@127.0.0.1 <0.12491.105> 378e7334ee haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic-admin/fonts/fontawesome-webfont.svg?rev=5-a305aa6b73dc95fad7e40326e2edda1c 200 ok 12
[notice] 2024-08-03T06:09:51.544827Z couchdb@127.0.0.1 <0.12420.105> 0bdcb7cb1b haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic-admin/fonts/fontawesome-webfont.woff?rev=5-a305aa6b73dc95fad7e40326e2edda1c 200 ok 12
[notice] 2024-08-03T06:09:51.545424Z couchdb@127.0.0.1 <0.12320.105> 604780a514 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic-admin/fonts/fontawesome-webfont.ttf?rev=5-a305aa6b73dc95fad7e40326e2edda1c 200 ok 13
[notice] 2024-08-03T06:09:51.547685Z couchdb@127.0.0.1 <0.12416.105> 36a3f21fc3 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic-admin/fonts/fontawesome-webfont.woff2?rev=5-a305aa6b73dc95fad7e40326e2edda1c 200 ok 11
[notice] 2024-08-03T06:09:51.550918Z couchdb@127.0.0.1 <0.10560.105> 0bbd75ab00 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic-admin/fonts/FontAwesome.otf?rev=5-a305aa6b73dc95fad7e40326e2edda1c 200 ok 14
[notice] 2024-08-03T06:09:51.565549Z couchdb@127.0.0.1 <0.12368.105> 81cc0bdd5d haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic-admin/fonts/NotoSans-Bold.ttf?rev=5-a305aa6b73dc95fad7e40326e2edda1c 200 ok 12
[notice] 2024-08-03T06:09:51.568068Z couchdb@127.0.0.1 <0.12526.105> e5acd115b8 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic-admin/index.html?rev=5-a305aa6b73dc95fad7e40326e2edda1c 200 ok 14
[notice] 2024-08-03T06:09:51.576717Z couchdb@127.0.0.1 <0.12540.105> 41867bdc9c haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic-admin/fonts/NotoSans-Regular.ttf?rev=5-a305aa6b73dc95fad7e40326e2edda1c 200 ok 22
[notice] 2024-08-03T06:09:51.583344Z couchdb@127.0.0.1 <0.12409.105> 44e774b296 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic-admin/fonts/NotoSans-Italic.ttf?rev=5-a305aa6b73dc95fad7e40326e2edda1c 200 ok 29
[notice] 2024-08-03T06:09:51.586309Z couchdb@127.0.0.1 <0.12513.105> 78657fbb5e haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic-admin/fonts/NotoSans-BoldItalic.ttf?rev=5-a305aa6b73dc95fad7e40326e2edda1c 200 ok 32
[notice] 2024-08-03T06:09:51.590433Z couchdb@127.0.0.1 <0.12551.105> 83ff330f8a haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic-admin/js/templates.js?rev=5-a305aa6b73dc95fad7e40326e2edda1c 200 ok 19
[notice] 2024-08-03T06:09:51.632809Z couchdb@127.0.0.1 <0.12581.105> 0742d8610e haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic-admin/js/main.js?rev=5-a305aa6b73dc95fad7e40326e2edda1c 200 ok 61
[notice] 2024-08-03T06:09:51.800233Z couchdb@127.0.0.1 <0.9025.105> 71cd8e496e haproxy:5984 172.18.0.2 medic-api GET /medic/service-worker-meta? 200 ok 13
[notice] 2024-08-03T06:09:51.806048Z couchdb@127.0.0.1 <0.9027.105> f103988223 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic? 200 ok 2
[notice] 2024-08-03T06:09:51.866624Z couchdb@127.0.0.1 <0.10865.105> 0145932838 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/audio/alert.mp3 200 ok 5
[notice] 2024-08-03T06:09:51.871592Z couchdb@127.0.0.1 <0.12352.105> c3494e98ea haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/default-docs/form-contact-health_center-edit.doc.json 200 ok 8
[notice] 2024-08-03T06:09:51.871934Z couchdb@127.0.0.1 <0.12467.105> 29df7ffef6 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/default-docs/form-contact-clinic-edit.doc.json 200 ok 8
[notice] 2024-08-03T06:09:51.871961Z couchdb@127.0.0.1 <0.12429.105> a5da47ca24 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/default-docs/form-contact-district_hospital-edit.doc.json 200 ok 8
[notice] 2024-08-03T06:09:51.874364Z couchdb@127.0.0.1 <0.7810.105> 537512c00d haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/default-docs/form-death_report.doc.json 200 ok 10
[notice] 2024-08-03T06:09:51.874897Z couchdb@127.0.0.1 <0.12557.105> ce9147bea2 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/default-docs/form-contact-person-edit.doc.json 200 ok 11
[notice] 2024-08-03T06:09:51.874925Z couchdb@127.0.0.1 <0.12473.105> 8693c43242 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/default-docs/form-contact-district_hospital-create.doc.json 200 ok 11
[notice] 2024-08-03T06:09:51.874959Z couchdb@127.0.0.1 <0.12485.105> 1059614299 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/default-docs/form-pnc_danger_sign_follow_up_baby.doc.json 200 ok 11
[notice] 2024-08-03T06:09:51.875022Z couchdb@127.0.0.1 <0.12573.105> 8f1ecd4629 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/default-docs/form-contact-person-create.doc.json 200 ok 11
[notice] 2024-08-03T06:09:51.875044Z couchdb@127.0.0.1 <0.12424.105> a75f049702 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/default-docs/form-contact-health_center-create.doc.json 200 ok 11
[notice] 2024-08-03T06:09:51.895273Z couchdb@127.0.0.1 <0.12670.105> fa1b89627d haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/default-docs/messages-en.doc.json 200 ok 23
[notice] 2024-08-03T06:09:51.895584Z couchdb@127.0.0.1 <0.9051.105> 568070e5fe haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/default-docs/form-pregnancy_facility_visit_reminder.doc.json 200 ok 23
[notice] 2024-08-03T06:09:51.901890Z couchdb@127.0.0.1 <0.11413.105> ed98991704 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/default-docs/form-pnc_danger_sign_follow_up_mother.doc.json 200 ok 29
[notice] 2024-08-03T06:09:51.902265Z couchdb@127.0.0.1 <0.12584.105> a1e60032ca haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/img/icon-nurse.svg 200 ok 29
[notice] 2024-08-03T06:09:51.902379Z couchdb@127.0.0.1 <0.12574.105> 66dc794406 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/default-docs/form-delivery.doc.json 200 ok 30
[notice] 2024-08-03T06:09:51.902586Z couchdb@127.0.0.1 <0.12572.105> 182cb4ef45 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/default-docs/form-pregnancy_danger_sign_follow_up.doc.json 200 ok 30
[notice] 2024-08-03T06:09:51.902731Z couchdb@127.0.0.1 <0.12554.105> c52d0b04c1 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/default-docs/form-pregnancy_home_visit.doc.json 200 ok 30
[notice] 2024-08-03T06:09:51.904471Z couchdb@127.0.0.1 <0.9056.105> 78f373db25 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/fonts/fontawesome-webfont.woff2 200 ok 32
[notice] 2024-08-03T06:09:51.909575Z couchdb@127.0.0.1 <0.12620.105> 5312e3ccdf haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/fonts/fontawesome-webfont.svg 200 ok 37
[notice] 2024-08-03T06:09:51.909932Z couchdb@127.0.0.1 <0.9108.105> 6f260ff8cd haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/img/icon-chw-selected.svg 200 ok 37
[notice] 2024-08-03T06:09:51.909990Z couchdb@127.0.0.1 <0.12596.105> 0a60b380b4 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/default-docs/form-pregnancy_danger_sign.doc.json 200 ok 37
[notice] 2024-08-03T06:09:51.910109Z couchdb@127.0.0.1 <0.12509.105> 7ee7cdf184 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/default-docs/form-pregnancy.doc.json 200 ok 37
[notice] 2024-08-03T06:09:51.910269Z couchdb@127.0.0.1 <0.12515.105> f92bd5820d haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/default-docs/form-undo_death_report.doc.json 200 ok 37
[notice] 2024-08-03T06:09:51.910299Z couchdb@127.0.0.1 <0.12593.105> b8ad33ba48 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/default-docs/resources.doc.json 200 ok 37
[notice] 2024-08-03T06:09:51.910323Z couchdb@127.0.0.1 <0.9168.105> 3df4c41c23 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/fonts/enketo-icons-v2.ttf 200 ok 37
[notice] 2024-08-03T06:09:51.914946Z couchdb@127.0.0.1 <0.12529.105> 6d9b247c8f haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/img/icon-chw.svg 200 ok 42
[notice] 2024-08-03T06:09:51.915964Z couchdb@127.0.0.1 <0.12544.105> c84736592e haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/fonts/NotoSans-BoldItalic.ttf 200 ok 43
[notice] 2024-08-03T06:09:51.918762Z couchdb@127.0.0.1 <0.12533.105> b9b4715c66 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/img/simprints.png 200 ok 45
[notice] 2024-08-03T06:09:51.918842Z couchdb@127.0.0.1 <0.12638.105> e167937d9b haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/fonts/enketo-icons-v2.woff 200 ok 45
[notice] 2024-08-03T06:09:51.918875Z couchdb@127.0.0.1 <0.10848.105> 4f270cc5be haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/manifest.json 200 ok 45
[notice] 2024-08-03T06:09:51.919993Z couchdb@127.0.0.1 <0.12649.105> e2f9e700de haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/fonts/NotoSans-Regular.ttf 200 ok 46
[notice] 2024-08-03T06:09:51.922022Z couchdb@127.0.0.1 <0.12606.105> c5ba04c720 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/img/icon-pregnant.svg 200 ok 48
[notice] 2024-08-03T06:09:51.922541Z couchdb@127.0.0.1 <0.12623.105> 5d2b14fd26 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/img/setup-wizard-demo.png 200 ok 49
[notice] 2024-08-03T06:09:51.922835Z couchdb@127.0.0.1 <0.12673.105> 968f2201b1 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/js/service-worker.js 200 ok 49
[notice] 2024-08-03T06:09:51.923188Z couchdb@127.0.0.1 <0.12546.105> 9345285e54 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/fonts/enketo-icons-v2.eot 200 ok 49
[notice] 2024-08-03T06:09:51.923267Z couchdb@127.0.0.1 <0.12654.105> 5782179b97 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/fonts/NotoSans-Bold.ttf 200 ok 49
[notice] 2024-08-03T06:09:51.923325Z couchdb@127.0.0.1 <0.9157.105> 27fbf1d6ba haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/fonts/fontawesome-webfont.eot 200 ok 50
[notice] 2024-08-03T06:09:51.923636Z couchdb@127.0.0.1 <0.12604.105> ba052dcaa3 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/js/templates.js 200 ok 50
[notice] 2024-08-03T06:09:51.923661Z couchdb@127.0.0.1 <0.12518.105> 2fbeb6fb45 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/templates/inbox.html 200 ok 50
[notice] 2024-08-03T06:09:51.923774Z couchdb@127.0.0.1 <0.12625.105> 36625266ec haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/fonts/enketo-icons-v2.svg 200 ok 50
[notice] 2024-08-03T06:09:51.923994Z couchdb@127.0.0.1 <0.12658.105> a5932f79c5 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/fonts/FontAwesome.otf 200 ok 50
[notice] 2024-08-03T06:09:51.924492Z couchdb@127.0.0.1 <0.12644.105> c84e7a01f6 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/img/icon-pregnant-selected.svg 200 ok 50
[notice] 2024-08-03T06:09:51.925229Z couchdb@127.0.0.1 <0.12650.105> 72b5e92a04 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/fonts/NotoSans-Italic.ttf 200 ok 51
[notice] 2024-08-03T06:09:51.925954Z couchdb@127.0.0.1 <0.9237.105> 25a4120a52 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/fonts/fontawesome-webfont.woff 200 ok 51
[notice] 2024-08-03T06:09:51.926018Z couchdb@127.0.0.1 <0.12561.105> 8d4106022c haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/img/icon-nurse-selected.svg 200 ok 52
[notice] 2024-08-03T06:09:51.926293Z couchdb@127.0.0.1 <0.12612.105> 690c1d69b6 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/fonts/fontawesome-webfont.ttf 200 ok 52
[notice] 2024-08-03T06:09:51.927501Z couchdb@127.0.0.1 <0.9032.105> 7d2bdcda2f haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/default-docs/form-contact-clinic-create.doc.json 200 ok 53
[notice] 2024-08-03T06:09:51.927568Z couchdb@127.0.0.1 <0.12523.105> 43d48fe1bb haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/js/inbox.js 200 ok 55
[notice] 2024-08-03T06:09:51.927684Z couchdb@127.0.0.1 <0.12576.105> d39a218576 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/default-docs/settings.doc.json 200 ok 54
[notice] 2024-08-03T06:09:51.927868Z couchdb@127.0.0.1 <0.9073.105> 500c846f54 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic/css/inbox.css 200 ok 54
[notice] 2024-08-03T06:09:51.989440Z couchdb@127.0.0.1 <0.9002.105> 14f6648726 haproxy:5984 172.18.0.2 medic-api GET /medic/settings? 200 ok 32
[notice] 2024-08-03T06:09:51.999705Z couchdb@127.0.0.1 <0.12538.105> bf32994e62 haproxy:5984 172.18.0.2 medic-api GET /medic/_design/medic? 200 ok 2
[notice] 2024-08-03T06:09:52.002960Z couchdb@127.0.0.1 <0.12560.105> 3f8166cf81 haproxy:5984 172.18.0.2 medic-api GET /medic/settings? 200 ok 5
[notice] 2024-08-03T06:09:52.010233Z couchdb@127.0.0.1 <0.12617.105> e067aeb1b7 haproxy:5984 172.18.0.2 medic-api GET /medic/settings? 200 ok 3
[error] 2024-08-03T06:10:06.020450Z couchdb@127.0.0.1 <0.29975.102> a116596e00 req_err(1125717067) unknown_error : normal
    [<<"chttpd:catch_error/3 L356">>,<<"chttpd:handle_req_after_auth/2 L322">>,<<"chttpd:process_request/1 L303">>,<<"chttpd:handle_request_int/1 L243">>,<<"mochiweb_http:headers/6 L128">>,<<"proc_lib:init_p_do_apply/3 L247">>]
[error] 2024-08-03T06:10:06.689369Z couchdb@127.0.0.1 <0.27897.100> d32b7d2680 req_err(1125717067) unknown_error : normal
    [<<"chttpd:catch_error/3 L356">>,<<"chttpd:handle_req_after_auth/2 L322">>,<<"chttpd:process_request/1 L303">>,<<"chttpd:handle_request_int/1 L243">>,<<"mochiweb_http:headers/6 L128">>,<<"proc_lib:init_p_do_apply/3 L247">>]
4

If views are currently being indexed, then this behavior is expected.
Once indexing is finished, your instance should become available.

Normally, we pre-index views when upgrades happen. How did you start / upgrade this instance?

1 Like
5

Hi @diana

The instance became available when the indexing finished. However, it intermittently fails with the below error:

[2024-08-07  18:36:35] COUCH_NODE_NAME couchdb@127.0.0.1
[2024-08-07  18:36:35] CouchDB Version: 2.3.1
[2024-08-07  18:36:36] 2024-08-07 18:36:36 INFO: Extracting ddoc…
[2024-08-07  18:36:36] 2024-08-07  18:36:36 INFO: DDoc extraction completed successfully
[2024-08-07  18:36:36] 2024-08-07  18:36:36 INFO: Extracting resources…
[2024-08-07  18:36:36] 2024-08-07  18:36:36 ERROR: Fatal error initialising medic-api
[2024-08-07  18:36:36] 2024-08-07  18:36:36 ERROR: { TypeError: Cannot convert undefined or null to object
[2024-08-07  18:36:36]     at Function.keys (<anonymous>)
[2024-08-07  18:36:36]     at db.medic.get.then.ddoc (/srv/software/medic-api/md5-O3yLnjz8HcvO7g+wuf0kiw==/src/resource-extraction.js:28:42)
[2024-08-07  18:36:36]     at <anonymous>
[2024-08-07  18:36:36]     at process._tickCallback (internal/process/next_tick.js:188:7)
[2024-08-07  18:36:36]   [stack]: 'TypeError: Cannot convert undefined or null to object\n    at Function.keys (<anonymous>)\n    at db.medic.get.then.ddoc (/srv/software/medic-api/md5-O3yLnjz8HcvO7g+wuf0kiw==/src/resource-extraction.js:28:42)\n    at <anonymous>\n    at process._tickCallback (internal/process/next_tick.js:188:7)',
[2024-08-07  18:36:36]   [message]: 'Cannot convert undefined or null to object' }

When this happens, I always re-attach the volume from a backup.

The instance is on version 3.9, and there’s an ongoing replication to the instance.

6

Hi @Steve_Roy

Ongoing replication?
It seems that your main ddoc, the medic ddoc doesn’t have attachments, while the application expects them.
This is where API is failing: cht-core/api/src/resource-extraction.js at 3.9.x · medic/cht-core · GitHub

Sorry the message is not clearer. I have no idea how your ddoc got corrupted or edited, except during upgrade, this doc should not be touched.
The only way to fix this is to update the ddoc and set the correct attachments. I believe the easiest way is to reinstall the same version of CHT on top.