medic-replication is another user that pushes data into the DB not the user we were logging in with.
If you stop that user form pushing data, do you still get the crashes?
Let’s try that and share our observation.
Hi @hareet
According to @melema 's answer.
Disk IOPS is from AWS shows 3000
Is this enough to exclude disk IOPS as the cause of this issue?
Hi @melema
There are some “…” here in the error log. Is that how the error appears in your logs or is that ending up being formatted by something. Could CouchDB logs be accessed directly to get the full error log for this (without the “…”).
Thanks!
Another followup question: did this start happening at a specific time? For example after a restart?
medic-replication
What is medic-replication pushing info from? What is the source? How is replication achieved?
Is it through CouchDb replication? Is it through PouchDb?
Ok, with the help of some CouchDb guys, I was able to replicate this issue.
It’s indeed caused by this request:
[notice] 2022-07-04T10:13:23.994029Z couchdb@127.0.0.1 <0.22169.2508> b56fc793d3 echis-production-go-ke.lg-apps.com 44.199.169.171 medic-replication PUT /medic/6197b8f4-2650-45e6-ab3e-b7e755e59560?new_edits=false 500 ok 5
This request wants to do a PUT with new_edits=false. In this case, CouchDb expects the full rev tree to be attached to the doc body.
Example failing request:
curl -X PUT http://<host>/<database>/newdoc?new_edits=false -H "Content-Type: application/json" -d '{"_id":"newdoc"}'
Example correct request:
curl -X PUT http://<host>/<database>/newdoc?new_edits=false -H "Content-Type: application/json" -d '{"_id":"newdoc","_rev":"1-fdsjfs"}'
As far as I can tell, this is not affecting the operation of CouchDb, and the only improvement (on CouchDb side) would be to anticipate these requests and handle the error more gracefully.
The request itself is malformed, because new_edits=false requires the full rev tree to be attached.
Can you please share how this replication is done?