CouchDB crashes with error:

Technical Support
1

Hello,
My couchdb creashes with the errors below.

[error] 2024-07-11T07:47:49.418872Z couchdb@127.0.0.1 <0.13560.3> -------- rexi_server: from: couchdb@127.0.0.1(<0.13559.3>) mfa: fabric_rpc:all_docs/3 error:badarg [{erlang,binary_to_term,[<<131,0,104,2,100,0,7,107,112,95,110,111,100,108,0,0,0,3,104,2,109,0,0,0,58,99,114,101,97,116,101,100,58,109,101,100,105,99,45,112,117,114,103,101,100,45,114,111,108,101,45,100,99,54,97,101,102,50,102,53,98,98,97,100,49,55,97,53,49,100,102,51,99,98,102,53,101,101,97,49,48,53,97,104,3,98,72,173,49,215,104,3,97,2,97,0,104,3,100,0,9,115,105,122,101,95,105,110,102,111,97,90,97,6,98,0,0,1,77,104,2,109,0,0,0,34,117,112,100,97,116,101,100,58,109,101,100,105,99,45,117,115,101,114,45,97,109,97,100,111,117,45,107,111,110,45,109,101,116,97,104,3,98,72,194,14,52,104,3,97,48,97,0,104,3,100,0,9,115,105,122,101,95,105,110,102,111,98,0,0,29,91,97,188,98,0,0,147,170,104,2,109,0,0,0,38,117,112,100,97,116,101,100,58,109,101,100,105,99,45,117,115,101,114,45,121,111,117,99,101,102,95,100,97,104,109,97,110,101,45,109,101,116,97,104,3,98,72,191,21,174,104,3,97,23,97,0,104,3,100,0,9,115,105,122,101,95,105,110,102,111,98,0,0,27,63,97,90,98,0,0,93,0,106>>],[{error_info,#{module => erl_erts_errors}}]},{couch_compress,decompress,1,[{file,"src/couch_compress.erl"},{line,65}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,156}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,474}]},{couch_btree,stream_node,8,[{file,"src/couch_btree.erl"},{line,1069}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,242}]},{couch_bt_engine,fold_docs_int,5,[{file,"src/couch_bt_engine.erl"},{line,1129}]},{couch_mrview,get_total_rows,2,[{file,"src/couch_mrview.erl"},{line,704}]}]
[error] 2024-07-11T07:47:49.421107Z couchdb@127.0.0.1 <0.13556.3> -------- could not load validation funs {{badmatch,{error,{badarg,nil,[{erlang,binary_to_term,[<<131,0,104,2,100,0,7,107,112,95,110,111,100,108,0,0,0,3,104,2,109,0,0,0,58,99,114,101,97,116,101,100,58,109,101,100,105,99,45,112,117,114,103,101,100,45,114,111,108,101,45,100,99,54,97,101,102,50,102,53,98,98,97,100,49,55,97,53,49,100,102,51,99,98,102,53,101,101,97,49,48,53,97,104,3,98,72,173,49,215,104,3,97,2,97,0,104,3,100,0,9,115,105,122,101,95,105,110,102,111,97,90,97,6,98,0,0,1,77,104,2,109,0,0,0,34,117,112,100,97,116,101,100,58,109,101,100,105,99,45,117,115,101,114,45,97,109,97,100,111,117,45,107,111,110,45,109,101,116,97,104,3,98,72,194,14,52,104,3,97,48,97,0,104,3,100,0,9,115,105,122,101,95,105,110,102,111,98,0,0,29,91,97,188,98,0,0,147,170,104,2,109,0,0,0,38,117,112,100,97,116,101,100,58,109,101,100,105,99,45,117,115,101,114,45,121,111,117,99,101,102,95,100,97,104,109,97,110,101,45,109,101,116,97,104,3,98,72,191,21,174,104,3,97,23,97,0,104,3,100,0,9,115,105,122,101,95,105,110,102,111,98,0,0,27,63,97,90,98,0,0,93,0,106>>],[{error_info,#{module => erl_erts_errors}}]},{couch_compress,decompress,1,[{file,"src/couch_compress.erl"},{line,65}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,156}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,474}]},{couch_btree,stream_node,8,[{file,"src/couch_btree.erl"},{line,1069}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,242}]},{couch_bt_engine,fold_docs_int,5,[{file,"src/couch_bt_engine.erl"},{line,1129}]},{couch_mrview,get_total_rows,2,[{file,"src/couch_mrview.erl"},{line,704}]}]}}},[{ddoc_cache_entry_validation_funs,recover,1,[{file,"src/ddoc_cache_entry_validation_funs.erl"},{line,29}]},{ddoc_cache_entry,do_open,1,[{file,"src/ddoc_cache_entry.erl"},{line,275}]}]}
[error] 2024-07-11T07:47:49.421587Z couchdb@127.0.0.1 emulator -------- Error in process <0.13557.3> on node 'couchdb@127.0.0.1' with exit value:
{{badmatch,{error,{badarg,nil,[{erlang,binary_to_term,[<<131,0,104,2,100,0,7,107,112,95,110,111,100,108,0,0,0,3,104,2,109,0,0,0,58,99,114,101,97,116,101,100,58,109,101,100,105,99,45,112,117,114,103,101,100,45,114,111,108,101,45,100,99,54,97,101,102,50,102,53,98,98,97,100,49,55,97,53,49,100,102,51,99,98,102,53,101,101,97,49,48,53,97,104,3,98,72,173,49,215,104,3,97,2,97,0,104,3,100,0,9,115,105,122,101,95,105,110,102,111,97,90,97,6,98,0,0,1,77,104,2,109,0,0,0,34,117,112,100,97,116,101,100,58,109,101,100,105,99,45,117,115,101,114,45,97,109,97,100,111,117,45,107,111,110,45,109,101,116,97,104,3,98,72,194,14,52,104,3,97,48,97,0,104,3,100,0,9,115,105,122,101,95,105,110,102,111,98,0,0,29,91,97,188,98,0,0,147,170,104,2,109,0,0,0,38,117,112,100,97,116,101,100,58,109,101,100,105,99,45,117,115,101,114,45,121,111,117,99,101,102,95,100,97,104,109,97,110,101,45,109,101,116,97,104,3,98,72,191,21,174,104,3,97,23,97,0,104,3,100,0,9,115,105,122,101,95,105,110,102,111,98,0,0,27,63,97,90,98,0,0,93,0,106>>],[{error_info,#{module => erl_erts_errors}}]},{couch_compress,decompress,1,[{file,"src/couch_compress.erl"},{line,65}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,156}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,474}]},{couch_btree,stream_node,8,[{file,"src/couch_btree.erl"},{line,1069}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,242}]},{couch_bt_engine,fold_docs_int,5,[{file,"src/couch_bt_engine.erl"},{line,1129}]},{couch_mrview,get_total_rows,2,[{file,"src/couch_mrview.erl"},{line,704}]}]}}},[{ddoc_cache_entry_validation_funs,recover,1,[{file,"src/ddoc_cache_entry_validation_funs.erl"},{line,29}]},{ddoc_cache_entry,do_open,1,[{file,"src/ddoc_cache_entry.erl"},{line,275}]}]}

[error] 2024-07-11T07:47:49.421804Z couchdb@127.0.0.1 emulator -------- Error in process <0.13557.3> on node 'couchdb@127.0.0.1' with exit value:
{{badmatch,{error,{badarg,nil,[{erlang,binary_to_term,[<<131,0,104,2,100,0,7,107,112,95,110,111,100,108,0,0,0,3,104,2,109,0,0,0,58,99,114,101,97,116,101,100,58,109,101,100,105,99,45,112,117,114,103,101,100,45,114,111,108,101,45,100,99,54,97,101,102,50,102,53,98,98,97,100,49,55,97,53,49,100,102,51,99,98,102,53,101,101,97,49,48,53,97,104,3,98,72,173,49,215,104,3,97,2,97,0,104,3,100,0,9,115,105,122,101,95,105,110,102,111,97,90,97,6,98,0,0,1,77,104,2,109,0,0,0,34,117,112,100,97,116,101,100,58,109,101,100,105,99,45,117,115,101,114,45,97,109,97,100,111,117,45,107,111,110,45,109,101,116,97,104,3,98,72,194,14,52,104,3,97,48,97,0,104,3,100,0,9,115,105,122,101,95,105,110,102,111,98,0,0,29,91,97,188,98,0,0,147,170,104,2,109,0,0,0,38,117,112,100,97,116,101,100,58,109,101,100,105,99,45,117,115,101,114,45,121,111,117,99,101,102,95,100,97,104,109,97,110,101,45,109,101,116,97,104,3,98,72,191,21,174,104,3,97,23,97,0,104,3,100,0,9,115,105,122,101,95,105,110,102,111,98,0,0,27,63,97,90,98,0,0,93,0,106>>],[{error_info,#{module => erl_erts_errors}}]},{couch_compress,decompress,1,[{file,"src/couch_compress.erl"},{line,65}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,156}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,474}]},{couch_btree,stream_node,8,[{file,"src/couch_btree.erl"},{line,1069}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,242}]},{couch_bt_engine,fold_docs_int,5,[{file,"src/couch_bt_engine.erl"},{line,1129}]},{couch_mrview,get_total_rows,2,[{file,"src/couch_mrview.erl"},{line,704}]}]}}},[{ddoc_cache_entry_validation_funs,recover,1,[{file,"src/ddoc_cache_entry_validation_funs.erl"},{line,29}]},{ddoc_cache_entry,do_open,1,[{file,"src/ddoc_cache_entry.erl"},{line,275}]}]}
[error] 2024-07-11T07:59:28.760469Z couchdb@127.0.0.1 <0.17943.0> -------- rexi_server: from: couchdb@127.0.0.1(<0.17939.0>) mfa: fabric_rpc:all_docs/3 error:badarg [{erlang,binary_to_term,[<<131,0,104,2,100,0,7,107,112,95,110,111,100,108,0,0,0,3,104,2,109,0,0,0,58,99,114,101,97,116,101,100,58,109,101,100,105,99,45,112,117,114,103,101,100,45,114,111,108,101,45,100,99,54,97,101,102,50,102,53,98,98,97,100,49,55,97,53,49,100,102,51,99,98,102,53,101,101,97,49,48,53,97,104,3,98,72,173,49,215,104,3,97,2,97,0,104,3,100,0,9,115,105,122,101,95,105,110,102,111,97,90,97,6,98,0,0,1,77,104,2,109,0,0,0,34,117,112,100,97,116,101,100,58,109,101,100,105,99,45,117,115,101,114,45,97,109,97,100,111,117,45,107,111,110,45,109,101,116,97,104,3,98,72,194,14,52,104,3,97,48,97,0,104,3,100,0,9,115,105,122,101,95,105,110,102,111,98,0,0,29,91,97,188,98,0,0,147,170,104,2,109,0,0,0,38,117,112,100,97,116,101,100,58,109,101,100,105,99,45,117,115,101,114,45,121,111,117,99,101,102,95,100,97,104,109,97,110,101,45,109,101,116,97,104,3,98,72,191,21,174,104,3,97,23,97,0,104,3,100,0,9,115,105,122,101,95,105,110,102,111,98,0,0,27,63,97,90,98,0,0,93,0,106>>],[{error_info,#{module => erl_erts_errors}}]},{couch_compress,decompress,1,[{file,"src/couch_compress.erl"},{line,65}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,156}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,474}]},{couch_btree,stream_node,8,[{file,"src/couch_btree.erl"},{line,1069}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,242}]},{couch_bt_engine,fold_docs_int,5,[{file,"src/couch_bt_engine.erl"},{line,1129}]},{couch_mrview,get_total_rows,2,[{file,"src/couch_mrview.erl"},{line,704}]}]
[error] 2024-07-11T07:59:28.778328Z couchdb@127.0.0.1 emulator -------- Error in process <0.17937.0> on node 'couchdb@127.0.0.1' with exit value:
{{badmatch,{error,{badarg,nil,[{erlang,binary_to_term,[<<131,0,104,2,100,0,7,107,112,95,110,111,100,108,0,0,0,3,104,2,109,0,0,0,58,99,114,101,97,116,101,100,58,109,101,100,105,99,45,112,117,114,103,101,100,45,114,111,108,101,45,100,99,54,97,101,102,50,102,53,98,98,97,100,49,55,97,53,49,100,102,51,99,98,102,53,101,101,97,49,48,53,97,104,3,98,72,173,49,215,104,3,97,2,97,0,104,3,100,0,9,115,105,122,101,95,105,110,102,111,97,90,97,6,98,0,0,1,77,104,2,109,0,0,0,34,117,112,100,97,116,101,100,58,109,101,100,105,99,45,117,115,101,114,45,97,109,97,100,111,117,45,107,111,110,45,109,101,116,97,104,3,98,72,194,14,52,104,3,97,48,97,0,104,3,100,0,9,115,105,122,101,95,105,110,102,111,98,0,0,29,91,97,188,98,0,0,147,170,104,2,109,0,0,0,38,117,112,100,97,116,101,100,58,109,101,100,105,99,45,117,115,101,114,45,121,111,117,99,101,102,95,100,97,104,109,97,110,101,45,109,101,116,97,104,3,98,72,191,21,174,104,3,97,23,97,0,104,3,100,0,9,115,105,122,101,95,105,110,102,111,98,0,0,27,63,97,90,98,0,0,93,0,106>>],[{error_info,#{module => erl_erts_errors}}]},{couch_compress,decompress,1,[{file,"src/couch_compress.erl"},{line,65}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,156}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,474}]},{couch_btree,stream_node,8,[{file,"src/couch_btree.erl"},{line,1069}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,242}]},{couch_bt_engine,fold_docs_int,5,[{file,"src/couch_bt_engine.erl"},{line,1129}]},{couch_mrview,get_total_rows,2,[{file,"src/couch_mrview.erl"},{line,704}]}]}}},[{ddoc_cache_entry_validation_funs,recover,1,[{file,"src/ddoc_cache_entry_validation_funs.erl"},{line,29}]},{ddoc_cache_entry,do_open,1,[{file,"src/ddoc_cache_entry.erl"},{line,275}]}]}

[error] 2024-07-11T07:59:28.780745Z couchdb@127.0.0.1 emulator -------- Error in process <0.17937.0> on node 'couchdb@127.0.0.1' with exit value:
{{badmatch,{error,{badarg,nil,[{erlang,binary_to_term,[<<131,0,104,2,100,0,7,107,112,95,110,111,100,108,0,0,0,3,104,2,109,0,0,0,58,99,114,101,97,116,101,100,58,109,101,100,105,99,45,112,117,114,103,101,100,45,114,111,108,101,45,100,99,54,97,101,102,50,102,53,98,98,97,100,49,55,97,53,49,100,102,51,99,98,102,53,101,101,97,49,48,53,97,104,3,98,72,173,49,215,104,3,97,2,97,0,104,3,100,0,9,115,105,122,101,95,105,110,102,111,97,90,97,6,98,0,0,1,77,104,2,109,0,0,0,34,117,112,100,97,116,101,100,58,109,101,100,105,99,45,117,115,101,114,45,97,109,97,100,111,117,45,107,111,110,45,109,101,116,97,104,3,98,72,194,14,52,104,3,97,48,97,0,104,3,100,0,9,115,105,122,101,95,105,110,102,111,98,0,0,29,91,97,188,98,0,0,147,170,104,2,109,0,0,0,38,117,112,100,97,116,101,100,58,109,101,100,105,99,45,117,115,101,114,45,121,111,117,99,101,102,95,100,97,104,109,97,110,101,45,109,101,116,97,104,3,98,72,191,21,174,104,3,97,23,97,0,104,3,100,0,9,115,105,122,101,95,105,110,102,111,98,0,0,27,63,97,90,98,0,0,93,0,106>>],[{error_info,#{module => erl_erts_errors}}]},{couch_compress,decompress,1,[{file,"src/couch_compress.erl"},{line,65}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,156}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,474}]},{couch_btree,stream_node,8,[{file,"src/couch_btree.erl"},{line,1069}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,242}]},{couch_bt_engine,fold_docs_int,5,[{file,"src/couch_bt_engine.erl"},{line,1129}]},{couch_mrview,get_total_rows,2,[{file,"src/couch_mrview.erl"},{line,704}]}]}}},[{ddoc_cache_entry_validation_funs,recover,1,[{file,"src/ddoc_cache_entry_validation_funs.erl"},{line,29}]},{ddoc_cache_entry,do_open,1,[{file,"src/ddoc_cache_entry.erl"},{line,275}]}]}

[error] 2024-07-11T07:59:28.797215Z couchdb@127.0.0.1 <0.17935.0> -------- could not load validation funs {{badmatch,{error,{badarg,nil,[{erlang,binary_to_term,[<<131,0,104,2,100,0,7,107,112,95,110,111,100,108,0,0,0,3,104,2,109,0,0,0,58,99,114,101,97,116,101,100,58,109,101,100,105,99,45,112,117,114,103,101,100,45,114,111,108,101,45,100,99,54,97,101,102,50,102,53,98,98,97,100,49,55,97,53,49,100,102,51,99,98,102,53,101,101,97,49,48,53,97,104,3,98,72,173,49,215,104,3,97,2,97,0,104,3,100,0,9,115,105,122,101,95,105,110,102,111,97,90,97,6,98,0,0,1,77,104,2,109,0,0,0,34,117,112,100,97,116,101,100,58,109,101,100,105,99,45,117,115,101,114,45,97,109,97,100,111,117,45,107,111,110,45,109,101,116,97,104,3,98,72,194,14,52,104,3,97,48,97,0,104,3,100,0,9,115,105,122,101,95,105,110,102,111,98,0,0,29,91,97,188,98,0,0,147,170,104,2,109,0,0,0,38,117,112,100,97,116,101,100,58,109,101,100,105,99,45,117,115,101,114,45,121,111,117,99,101,102,95,100,97,104,109,97,110,101,45,109,101,116,97,104,3,98,72,191,21,174,104,3,97,23,97,0,104,3,100,0,9,115,105,122,101,95,105,110,102,111,98,0,0,27,63,97,90,98,0,0,93,0,106>>],[{error_info,#{module => erl_erts_errors}}]},{couch_compress,decompress,1,[{file,"src/couch_compress.erl"},{line,65}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,156}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,474}]},{couch_btree,stream_node,8,[{file,"src/couch_btree.erl"},{line,1069}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,242}]},{couch_bt_engine,fold_docs_int,5,[{file,"src/couch_bt_engine.erl"},{line,1129}]},{couch_mrview,get_total_rows,2,[{file,"src/couch_mrview.erl"},{line,704}]}]}}},[{ddoc_cache_entry_validation_funs,recover,1,[{file,"src/ddoc_cache_entry_validation_funs.erl"},{line,29}]},{ddoc_cache_entry,do_open,1,[{file,"src/ddoc_cache_entry.erl"},{line,275}]}]}

The crash rate has increased thus resulting to sentinel and api containers to restart which interrupts the entire app.

2

Hi @Job_Isabai

I’ve never seen this error before. Can you share which version of the CHT you’re running?

3

I am running on CHT 4.5.0. This error is present on Production Environment.

4

Did this start happening suddenly or can you correlate it with some event?

5

Yes it might be related with the upgrade from 4.2.0 to 4.5.0

6

“Might be” is too loose. When did you upgrade to 4.5.0?

7

I upgraded on June 6th. The reason why I attribute this to the upgrade to 4.5.0, prior to the upgrade, couch_db, sentinel, api containers were stable and barely restarted due to error.

8

When did you first see this error happen?

9

4 Hours after upgrade and indexing is completed.

10

Have you added additional ddocs to your installation or made any changes to the views?

11

I only creased a new view on the database afterwards.
Might this related to this issue: getting badarg in rexi_server (fabric_rpc) · Issue #1163 · apache/couchdb · GitHub

12

This is a closed issue with no information.
Can you share the view you added?

13 
function(doc) {
  if (doc.type === 'data_record' && 
      (doc.form === 'ped_label_form_pause' || 
       doc.form === 'yi_label_form_pause')) {
    emit(doc.form, doc);
  }
}

Reduce = None

14

did you create a new ddoc for this view?

15

Hello Diana,
I deleted the view but the error is still there.
The errors and restarts of couchdb are not that of a blocker to access and use of the application but rather they prevent me from refreshing data on Power BI. I have connected Power BI to CHT which pulls data via the API for exporting reports. Due to the increased number of records and unplanned couchdb restart/failure, it has been challenging for me to refresh data. Please advise on the alternative for Power BI connection, can I set up Couc2pg? or would replication be an option?
Thanks,
Job

16

Hi @Job_Isabai

If you stop your Power BI connection, are you still seeing crashes?