Error initialising medic-api - badmatch

Our CHT server failed and now we are trying to restore from a backup.

Restoring from a month-old backup works fine.

But when restoring from the most recent backup, the cht-api container has this error when starting:

Info: Starting CHT API
2024-01-24 04:10:37 INFO: Running server checks… 
Node Environment Options: 'undefined'
Node Version: 16.20.1 in development mode
CouchDB Version: 2.3.1
CouchDb Cluster ready
2024-01-24 04:10:37 INFO: Checks passed successfully 
2024-01-24 04:10:37 INFO: Extracting initial documents… 
2024-01-24 04:10:37 ERROR: Fatal error initialising medic-api 
2024-01-24 04:10:37 ERROR: {
  error: 'error',
  reason: "{badmatch,{'EXIT',{{badmatch,eof},\n" +
    '                   [{couch_file,read_raw_iolist_int,3,\n' +
    '                                [{file,"src/couch_file.erl"},{line,627}]},\n' +
    '                    {couch_file,handle_call,3,\n' +
    '                                [{file,"src/couch_file.erl"},{line,449}]},\n' +
    '                    {gen_server,try_handle_call,4,\n' +
    '                                [{file,"gen_server.erl"},{line,636}]},\n' +
    '                    {gen_server,handle_msg,6,\n' +
    '                                [{file,"gen_server.erl"},{line,665}]},\n' +
    '                    {proc_lib,init_p_do_apply,3,\n' +
    '                              [{file,"proc_lib.erl"},{line,247}]}]}}}',
  status: 500,
  name: 'error',
  message: "{badmatch,{'EXIT',{{badmatch,eof},\n" +
    '                   [{couch_file,read_raw_iolist_int,3,\n' +
    '                                [{file,"src/couch_file.erl"},{line,627}]},\n' +
    '                    {couch_file,handle_call,3,\n' +
    '                                [{file,"src/couch_file.erl"},{line,449}]},\n' +
    '                    {gen_server,try_handle_call,4,\n' +
    '                                [{file,"gen_server.erl"},{line,636}]},\n' +
    '                    {gen_server,handle_msg,6,\n' +
    '                                [{file,"gen_server.erl"},{line,665}]},\n' +
    '                    {proc_lib,init_p_do_apply,3,\n' +
    '                              [{file,"proc_lib.erl"},{line,247}]}]}}}',
  stack: 'Error\n' +
    '    at Object.generateErrorFromResponse (/api/node_modules/pouchdb-errors/lib/index.js:104:18)\n' +
    '    at /api/node_modules/pouchdb-adapter-http/lib/index.js:254:33\n' +
    '    at processTicksAndRejections (node:internal/process/task_queues:96:5)',
  docId: 'settings'
} 

Any help would be greatly appreciated!

Hi @jschwab77

It looks like your backup is somehow corrupted, CouchDb has trouble reading it. Unfortunately I don’t know how to guide you further. Have you tried reapplying the same backup?

I’ve tried restoring from the backup several times but got the same error.

Is there a way I could investigate and possibly repair this ‘settings’ doc?

I don’t think it’s the settings doc that you’re having issues here with. I don’t have an avenue to investigate this. Can you perhaps try a different backup? Not the most recent, but the one before?

I’d also be interested in knowing more about your server failure and the reason you need to restore from backup, if that’s something you’re willing to share. Thanks!