Server Side Purging, purgelog docs are not saved in medic-sentinel(v3.7.x)

Technical Support
,
1

NOTE: This is a public repository. Images or logs containing protected health information (PHI) must be fully-redacted.

Describe the Bug
When doing server side purging, purgelog docs are not saved in medic-sentinel after every purge. It only creates medic-purged-roles-<roles-hash> purge dbs. Also, we expect user docs to reduce based on the configured purge rules but no change

How To Reproduce The Issue
List steps for someone trying to reproduce the behavior. For example:

  1. create a purge.js file with the purge rules
  2. Save, compile app settings and upload app settings to the server

Expected Behavior
purgelog docs are saved in medic-sentinel after every purge.
user docs to reduce based on the configured purge rules but no change

Environment

  • Instance: medic webapp
  • Browser: Chrome
  • Client platform: Linux
  • App: (eg: webapp, admin, sentinel, api, couch2pg, medic-conf, etc)
  • Version: v3.7.x cht core
1 Like
2

Thank you for reporting.

I have a few followup questions:

  • Do you see any errors in the Sentinel logs? If yes, please share
  • Do you see any errors in the CouchDB logs? If yes, please share
  • Can you please share Sentinel’s memory usage and maximum allowed memory?
  • Can you please share how many documents Sentinel has left to process?

One related issue could be: Batch processing of sentinel changes · Issue #6108 · medic/cht-core · GitHub, where Sentinel ran out of memory when it has a large list of documents to process. This has been fixed in 3.9.

This issue might be related to purging: Server Side Purge does not complete to the end · Issue #7280 · medic/cht-core · GitHub .
In this case, the large data set was caused by a misconfiguration, although resolving it required updating documents in the database.
Fixing this has been scheduled for our next release: 3.14.

Another related issue could be: Call 'db.close' for ephemeral PouchDB objects to avoid memory leaks · Issue #6106 · medic/cht-core · GitHub, where a known memory leak in PouchDB also affected Sentinel. This has been fixed in 3.8.

3

Hi Diana,

how can I get these from medic-os;
“* Can you please share Sentinel’s memory usage and maximum allowed memory?”
“* Can you please share how many documents Sentinel has left to process?”

4

@diana thanks for the quick reply. Here are the medic-sentinel errors

[2021-09-24 06:30:59] 2021-09-24 06:31:00 [33mWARN [39m: Reminder configuration invalid: { form: ‘’,
[2021-09-24 06:30:59] text_expression: ‘’,
[2021-09-24 06:30:59] cron: ‘’,
[2021-09-24 06:30:59] mute_after_form_for: ‘’,
[2021-09-24 06:30:59] message: ‘’ }

[2021-09-24 06:37:46] 2021-09-24 06:37:46 [31mERROR [39m: Error while running Server Side Purge: { FetchError: Could not create Buffer from response body for http://haproxy:5984/medic/_design/medic/_view/docs_by_replication_key?include_docs=true: “size” argument must not be larger than 2147483647
[2021-09-24 06:37:46] at PassThrough. (/srv/software/medic-sentinel/md5-jkODRqiwM6Wpv4ECz2eM9Q==/node_modules/node-fetch/lib/index.js:425:12)

[2021-09-24 06:42:20] [stack]: ‘FetchError: invalid json response body at http://haproxy:5984/medic/_changes?feed=longpoll&heartbeat=10000&since=8768663-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____szKYkxiE2p1zgWLsaUmJqRZpxtj04DEpSQFIJtnDDStsBhtmaWJkbJxmSKphDiDD4uGGxRWBDUs2Tkk2MTEn1bAEkGH1cMOir0JcZpqcnJZoRqJheSxAkqEBSAHNmw8xMGwVJNyABhqlJpNl4AKIgfvBBrK6zIW4MM0gxcKSPAMPQAy8D3Gh7UywgeaJaRZJJqZkGfgAYiA0DB0gXk5NSks1SsYaIVkAfCeoaQ&limit=25 reason: Unexpected end of JSON input\n at /srv/software/medic-sentinel/md5-jkODRqiwM6Wpv4ECz2eM9Q==/node_modules/node-fetch/lib/index.js:275:32\n at \n at process._tickCallback (internal/process/next_tick.js:188:7)’ }

5

@waok2

RE: memory usage

I’m not aware of there being a medic-os “approved” way of getting resource usage.
Can you use:

ps -aux | sed -n '1p;/medic-sentinel.*js$/p

Your output would look something like:

USER        PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
medic-s+  25591  0.0  0.1 1247776 98368 ?       Ssl  Sep23   0:13 /srv/software/medic-core/v2.1.1/x64/bin/node /srv/software/medic-sentinel/current/server.js

You should see the resource usage for your medic-sentinel process. You might need to tweak the regex if you get no results.

There are alternative commands that would list processes by memory usage, like:

ps -o pid,user,vsz,rsz,command ax | sort -b -k3 -r | head -20

RE: how many documents

  • get the SEQ Sentinel is currently on by getting the metadata document. This document contains a property processed_seq which you will use in the next steps
curl "<your_server_url>/medic-sentinel/_local/sentinel-meta-data"
  • check how many changes there are in medic database after your Sentinel’s processed_seq
curl "<your_server_url>/medic/_changes?limit=0&since=<processed_seq>"

The result should look like:

{
    results: [],
    last_seq: <something>,
    pending: <number of pending changes>
}

<number of pending changes> is the number of documents Sentinel has left to process.

6 
{ FetchError: Could not create Buffer from response body for http://haproxy:5984/medic/_design/medic/_view/docs_by_replication_key?include_docs=true: “size” argument must not be larger than 2147483647
[2021-09-24 06:37:46] at PassThrough. (/srv/software/medic-sentinel/md5-jkODRqiwM6Wpv4ECz2eM9Q==/node_modules/node-fetch/lib/index.js:425:12)

This looks more and more like Server Side Purge does not complete to the end · Issue #7280 · medic/cht-core · GitHub, where you get too many results from CouchDB for the server to parse.

7

@sheila_abby
We’ve been working on a fix for the purging issue which I believe you’re seeing.

The fix is not yet released, but I have tested it on the data set we originally saw it on.
Because the bug only affects some instances, depending on the existent data, it would be very helpful for us if you would test it on a clone (please don’t directly test in production) with your data set.
To test, please install the 3.7-dynamic-purge-batches branch on your clone.

Thanks!

2 Likes