While testing this feature to write the upcoming 5.3 release notes, I had some key points I wanted to share!
I started off by creating a local dev instance with docker helper. I then used the easy mode in the test data generator to flood the instance with new data.
Archiving is dependent on the Sentinel process not only running, but more importantly, not having any backlog. Check your Watchdog analytics to see if there’s a backlog. If you’re 100% sure you know the ramifications, you can artificially zero out your backlog - do this with extreme caution!
For me, I had over 100,000 document backlog in my dev instance from repeatedly running the easy mode script. Forcing it down to zero was a safe procedure because this is not a production database and there are no users. Here’s what that looked like in watchdog:
I then used the /api/v2/export/reports API to download many report IDs. Specifically I picked an arbitrary start time and then converted it to seconds since the epoch. So Monday, August 24, 2026 at 2:35:15 PM UTC-07:00 DST became 1787607315000. I then looked the query format to use against the API - this forum post covers the topic well. Since I wanted the raw IDs, with out any other data, I cleansed the output with cut and sed like so:
curl -qs -g \
"https://medic:password@192-168-68-26.local-ip.medicmobile.org:10473/api/v2/export/reports?filters[search]=&filters[date][from]=0&filters[date][to]=1787255664000" | \
cut -f1 -d, | sed "s/\"//g" > test.IDs.to.archive
Be patient! I found this command can take some time to run - for me it took over an hour. Consider a very narrow query (short date period, specific reports etc) to test with before letting it run.
I wanted to slowly test, so i first cut off 10 IDs from the list: head -n10 test.IDs.to.archive > 10.ids.test.archive and ran this against my dev instance with a curl call:
curl -qs -X POST \
-H "Content-Type: text/csv" \
--data-binary @10.ids.test.archive \
https://medic:password@192-168-68-26.local-ip.medicmobile.org:10473/api/v1/archive | jq .
This immediately returned some JSON:
{
"jobs": [
{
"id": "archive:01a035b8-5477-7aa1-9d08-b3e740df9162",
"count": 10
}
]
}
As my dev instance was still busy indexing new documents sentinel only runs every 5 minutes, at first I was confused because the archive docs say:
Each archive job writes a log document to the medic-logs database with the same ID as the job.
However, I didn’t see any entries in the medic-logs database. I realized that this can take a bit, possibly many minutes for a busy instance until sentinel runs on its regular interval. It can be helpful to tail the logs of your Sentinel container. For me, I used this call:
docker logs -f cht-sentinel-1 --since 100m 2>&1 | grep -E "Archiving: processing job archive|skipped"
When you see activity here, you can then go to into Fauxton and look at the medic-logs database and look for documents that start with archive::
I also had a watchdog instance running so I could monitor the size of the medic database both for test data generate calls increasing it (see #1) and the archive feature decreasing (see #2):
Once I gained confidence in how the system worked, I uploaded file with thousands of lines to the archiving API and used the above techniques to monitor its progress. I was able to successfully archive over 100,000 reports!
