Thank you for your detailed response!
I’m hoping our very recent upgrade to 4.3.x, containing the downwards replication rewrite, will help address some of our downward syncing woes with higher hierarchy level users. That being said, as work continues the sheer amount of data these folks will need to download will increase. It does not make sense, at least for our purposes, to replicate signatures down. It also won’t be a one-off thing, with consent being given and taken away at any given time.
This is why the couchDB filter functions would suite our needs to well, as it allows us to omit parts of records we’re no longer interested - rather than the entire record like in purging.
I’m not sure if custom migration scripts will work for us either as our server is quite modestly resourced, especially since our last migration conversation.
And, as you’ve noted, we don’t really want to introduce additional syncing overhead.
Am I correct in thinking, while CHT doesn’t use it, it’s still possible to have couchDB use filter functions?
The above only only pertains to our one use case, where we need to omit a specific property before it’s replicated down.
Our other use case, only pulling specific users & their death reports, I believe can be facilitated via purging.
I’ve yet to do some testing as the form/feature itself is still under development, but this is what I’ve dreamt up so far:
// As of 3.14.0, contacts that have more than 20,000 associated reports + messages will be skipped,
// and none of their associated reports and messages will be purged.
// https://docs.communityhealthtoolkit.org/apps/guides/performance/purging/#considerations
module.exports = {
run_every_days: 7,
fn: function(userCtx = {}, contact = {}, reports = [], messages, chtScriptApi, permissions) {
// The following purging rules only apply to the "Verbal Autopsy Practitioner (VAP)" user.
// We've been experiencing some replication issues on the uppermost levels (DHO, Team Lead).
// We're attempting to proactively guard this role against those same woes.
if((userCtx && userCtx.roles && userCtx.roles.indexOf('vap') !== -1)){
// We're only concerned with the following contact types,
// which includes the "household member (hhm), however, the hhm db entry has some additional checks and is therefore a special entry."
const HHM = 'hhm';
const allowedContactTypes = ["npo", "team_area", "indawo", "dwelling", "household", HHM];
// For the VAP we're only really interested in hhm's that are flagged for death.
// However, we need to replicate the entire hierarchy down in order to get to them.
const DEATH_FLAGGED = 'death_flag';
const shouldPurgeContact = (doc) => {
// Purge if the document does not contain a "contact_type" property
if(!doc.contact_type){
return true;
}
// Purge if the contact_type is not in the allowed list
if(!allowedContactTypes.includes(doc.contact_type)) {
return true;
}
// Purge hhm documents that have not been flagged as deceased
if(doc.contact_type === HHM && (doc[DEATH_FLAGGED] === false || doc[DEATH_FLAGGED] === 'false' || !doc[DEATH_FLAGGED])) {
return true;
}
return false;
};
const purgeContact = (shouldPurgeContact(contact) ? [contact._id] : []).filter((value) => value);
// Purge all the reports, except the death report so that VAPs can see the work they've done
const reportsToPurge = reports.filter((value) => value.form !== 'death_report').map(r => r._id).filter((v) => v);
// Purging of messages is not needed as we don't use the feature
return [
...purgeContact,
...reportsToPurge
];
}
return []; // Do not purge anything
}
};