Doc's replication on Client Device

Hello,
I have been receiving this notification when configuring client devices:
“Warning! You are about to download 10384 docs, which exceeds recommended limit of 10000. Do you wish to continue?”
Purging has been in place for close to 1 year now. It seems it not working correctly.
Kindly help.
Thanks,
Job

Hi @Job_Isabai

The fact that you have over 10k docs (slightly over) does not mean purging isn’t running, it could mean that you need to change your purging rules.
You can check for purgelogs documents to verify whether indeed purging works or not.
There are some instructions about how to check these in our documentation: Purging | Community Health Toolkit

1 Like

Hello Diana,
I have the below code as my purging function:

  "purge": {
    "text_expression": "everday at 11:00am",
    "run_every_days": 1,
    "cron": "0 0 11 * * ?",
    "fn": "function(userCtx, contact, reports, messages, chtScriptApi, permissions) {\r\n    const old = Date.now() - (1000 * 60 * 60 * 24 * 60);\r\n    const oldMessages = Date.now() - (1000 * 60 * 60 * 24 * 90);\r\n\r\n    const reportsToPurge = reports\r\n      .filter(r => r.reported_date < old)\r\n      .map(r => r._id);\r\n    const messagesToPurge = messages\r\n      .filter(m => m.reported_date < oldMessages)\r\n      .map(m => m._id);\r\n\r\n    return [...reportsToPurge, ...messagesToPurge];\r\n  }"
  }

I would like to reduce the number of records, my purge logs have not changed in the past 1 week, hence the feeling that purging is not working correctly.
Please advise me on how I can reduce the number of records during replication.
Thanks,
Job

Hi @Job_Isabai

Without server logs, it’s hard to tell what is wrong. Can you share the last purge log you find? Can you check if Sentinel is running?

Hi Diana,
Thanks for the insights, Looking at the logs, I see sentinel has been restarting:

2024-06-04T09:33:11.161 WARN: Disabled transition "conditional_alerts"
2024-06-04T09:33:11.161 WARN: Disabled transition "multi_report_alerts"
2024-06-04T09:33:11.162 WARN: Disabled transition "update_notifications"
2024-06-04T09:33:11.162 WARN: Disabled transition "update_scheduled_reports"
2024-06-04T09:33:11.162 WARN: Disabled transition "resolve_pending"
2024-06-04T09:33:11.162 WARN: Disabled transition "muting"
2024-06-04T09:33:11.162 INFO: Loading transition "mark_for_outbound"
2024-06-04T09:33:11.166 WARN: Disabled transition "create_user_for_contacts"
2024-06-04T09:33:11.175 DEBUG: Reminder messages allowed between 0:0 and 23:0
2024-06-04T09:33:11.193 DEBUG: Initiating all tasks
2024-06-04T09:33:11.193 INFO: Task dueTasks started
2024-06-04T09:33:11.202 INFO: Task reminders started
2024-06-04T09:33:11.202 INFO: Task replications started
2024-06-04T09:33:11.205 INFO: Task outbound started
2024-06-04T09:33:11.206 INFO: Task purging started
2024-06-04T09:33:11.211 INFO: Task reminders completed
2024-06-04T09:33:11.211 INFO: Task replications completed
2024-06-04T09:33:11.212 ERROR: Fatal error intialising sentinel
2024-06-04T09:33:11.216 ERROR: TypeError: schedule.next(...).getTime is not a function
    at Object.nextScheduleMillis (/service/sentinel/src/lib/scheduling.js:33:32)
    at Object.execute (/service/sentinel/src/schedule/purging.js:22:58)
    at /service/sentinel/src/schedule/index.js:60:8
    at Array.forEach (<anonymous>)
    at runTasks (/service/sentinel/src/schedule/index.js:51:22)
    at Object.exports.init (/service/sentinel/src/schedule/index.js:72:3)
    at /service/sentinel/server.js:47:33
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  [stack]: 'TypeError: schedule.next(...).getTime is not a function\n' +
    '    at Object.nextScheduleMillis (/service/sentinel/src/lib/scheduling.js:33:32)\n' +
    '    at Object.execute (/service/sentinel/src/schedule/purging.js:22:58)\n' +
    '    at /service/sentinel/src/schedule/index.js:60:8\n' +
    '    at Array.forEach (<anonymous>)\n' +
    '    at runTasks (/service/sentinel/src/schedule/index.js:51:22)\n' +
    '    at Object.exports.init (/service/sentinel/src/schedule/index.js:72:3)\n' +
    '    at /service/sentinel/server.js:47:33\n' +
    '    at processTicksAndRejections (node:internal/process/task_queues:96:5)',
  [message]: 'schedule.next(...).getTime is not a function'

Sentinel image: public.ecr.aws/medic/cht-sentinel:4.3.0-8457-api-forms-payload

I have upgraded from 4.3 to 4.5.0 but still getting the error. @diana & @mrjones please help.

I think this might be causing it? “everday”? Can you try changing this to “every day” ?

Here is some info about how to build the text_expression: Parsers

If this is too confusing, you can remove it and just use the cron expression.
The text expression takes precedence over cron, so you will have to delete the entry.

Hi Diana, I have removed purging on my app_settings. But still getting the error. To me it seems like an issue with Sentinel container.

Info: Starting CHT Sentinel
2024-06-04T20:23:10.666 INFO: Running server checks...
Node Environment Options: 'undefined'
Node Version: 16.20.2 in development mode
CouchDB Version: 3.3.2
CouchDb Cluster ready
2024-06-04T20:23:10.847 INFO: API is ready: {"ready":true,"handler":"medic-api","version":"0.1.0","detail":"All required services are running normally"}
2024-06-04T20:23:11.489 WARN: Disabled transition "update_clinics"
2024-06-04T20:23:11.489 WARN: Disabled transition "self_report"
2024-06-04T20:23:11.490 WARN: Disabled transition "registration"
2024-06-04T20:23:11.490 WARN: Disabled transition "accept_patient_reports"
2024-06-04T20:23:11.490 WARN: Disabled transition "accept_case_reports"
2024-06-04T20:23:11.491 WARN: Disabled transition "generate_shortcode_on_contacts"
2024-06-04T20:23:11.491 WARN: Disabled transition "generate_patient_id_on_people"
2024-06-04T20:23:11.491 WARN: Disabled transition "default_responses"
2024-06-04T20:23:11.492 WARN: Disabled transition "update_sent_by"
2024-06-04T20:23:11.492 WARN: Disabled transition "death_reporting"
2024-06-04T20:23:11.492 WARN: Disabled transition "conditional_alerts"
2024-06-04T20:23:11.493 WARN: Disabled transition "multi_report_alerts"
2024-06-04T20:23:11.493 WARN: Disabled transition "update_notifications"
2024-06-04T20:23:11.493 WARN: Disabled transition "update_scheduled_reports"
2024-06-04T20:23:11.493 WARN: Disabled transition "resolve_pending"
2024-06-04T20:23:11.493 WARN: Disabled transition "muting"
2024-06-04T20:23:11.494 INFO: Loading transition "mark_for_outbound"
2024-06-04T20:23:11.499 WARN: Disabled transition "create_user_for_contacts"
2024-06-04T20:23:11.501 DEBUG: Reminder messages allowed between 0:0 and 23:0
2024-06-04T20:23:11.515 DEBUG: Initiating all tasks
2024-06-04T20:23:11.516 INFO: Task dueTasks started
2024-06-04T20:23:11.521 INFO: Task reminders started
2024-06-04T20:23:11.522 INFO: Task replications started
2024-06-04T20:23:11.526 INFO: Task outbound started
2024-06-04T20:23:11.527 INFO: Task purging started
2024-06-04T20:23:11.529 INFO: Task reminders completed
2024-06-04T20:23:11.529 INFO: Task replications completed
2024-06-04T20:23:11.529 ERROR: Fatal error intialising sentinel
2024-06-04T20:23:11.533 ERROR: TypeError: schedule.next(...).getTime is not a function
    at Object.nextScheduleMillis (/service/sentinel/src/lib/scheduling.js:33:32)
    at Object.execute (/service/sentinel/src/schedule/purging.js:22:58)
    at /service/sentinel/src/schedule/index.js:60:8
    at Array.forEach (<anonymous>)
    at runTasks (/service/sentinel/src/schedule/index.js:51:22)
    at Object.exports.init (/service/sentinel/src/schedule/index.js:72:3)
    at /service/sentinel/server.js:47:33
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  [stack]: 'TypeError: schedule.next(...).getTime is not a function\n' +
    '    at Object.nextScheduleMillis (/service/sentinel/src/lib/scheduling.js:33:32)\n' +
    '    at Object.execute (/service/sentinel/src/schedule/purging.js:22:58)\n' +
    '    at /service/sentinel/src/schedule/index.js:60:8\n' +
    '    at Array.forEach (<anonymous>)\n' +
    '    at runTasks (/service/sentinel/src/schedule/index.js:51:22)\n' +
    '    at Object.exports.init (/service/sentinel/src/schedule/index.js:72:3)\n' +
    '    at /service/sentinel/server.js:47:33\n' +
    '    at processTicksAndRejections (node:internal/process/task_queues:96:5)',
  [message]: 'schedule.next(...).getTime is not a function'
}

What does your outbound config look like?

My base_settings.json

{
  "permissions":{
    "can_export_messages": [
      "program_officer",
      "analytics"
    ],
    "can_export_contacts": [
      "program_officer"
    ],
    "can_export_feedback": [
      "program_officer",
      "analytics"
    ],
    "can_view_analytics": [
      "program_officer",
      "chw_supervisor",
      "analytics",
      "chw",
	  "chw_logout",
      "supervisor_online"
    ],
    "can_edit_people": [
      "program_officer",
      "chw_supervisor"
    ],
    "can_edit_verification": [
      "program_officer",
      "chw_supervisor",
      "chw",
	  "chw_logout"
    ],
    "can_view_unallocated_data_records": [
      "chw",
	  "chw_logout",
      "program_officer",
      "gateway",
      "chw_supervisor",
      "supervisor_online"
    ],
    "can_view_users": [
      "program_officer"
    ],
    "can_write_wealth_quintiles": [
      "program_officer"
    ],
    "can_edit": [
      "gateway",
      "program_officer",
      "chw",
	  "chw_logout",
      "chw_supervisor",
      "supervisor_online"
    ],
    "can_log_out_on_android": [
      "program_officer",
      "chw_supervisor",
      "supervisor_online",
	  "chw_logout"
    ],
    "can_update_users": [
      "program_officer"
    ],
    "can_upgrade": [
      "program_officer"
    ],
    "can_export_dhis": [
      "crfo"
    ],
    "can_update_places": [
      "program_officer",
      "chw_supervisor"
    ],
    "can_verify_reports": [
      "program_officer",
      "chw_supervisor",
      "chw",
	  "chw_logout"
    ],
    "can_create_records": [
      "chw",
      "chw_logout",
      "chw_supervisor",
      "supervisor_online",
      "program_officer",
      "gateway",
      "data_entry"
    ],
    "can_access_gateway_api": [
      "gateway"
    ],
    "can_aggregate_targets": [
      "chw_supervisor"
    ],
    "can_create_users": [
      "program_officer"
    ],
    "can_create_people": [
      "program_officer"
    ],
    "can_create_places": [
      "program_officer"
    ],
    "can_view_tasks": [
      "chw",
      "chw_supervisor",
      "supervisor_online",
	  "chw_logout"
    ],
    "can_view_tasks_tab": [
      "chw",
	  "chw_logout",
      "chw_supervisor",
      "supervisor_online"
    ],
    "can_configure": [
      "crfo",
      "program_officer"
    ],
    "can_view_message_action": [
      "program_officer",
      "chw_supervisor",
      "supervisor_online"
    ],
    "can_view_messages": [
      "program_officer"
    ],
    "can_view_messages_tab": [
      "program_officer"
    ],
    "can_view_uhc_stats": [
    ],
    "can_view_reports": [
      "program_officer",
      "chw_supervisor",
      "supervisor_online"
    ],
    "can_view_reports_tab": [
      "program_officer",
      "chw_supervisor",
      "supervisor_online"
    ],
    "can_view_analytics_tab": [
      "program_officer",
      "chw_supervisor",
      "supervisor_online",
      "chw",
	  "chw_logout"
    ],
    "can_view_call_action": [
      "program_officer",
      "chw_supervisor",
      "chw"
    ],
    "can_view_contacts": [
      "program_officer",
      "chw_supervisor",
      "supervisor_online",
      "chw",
	  "chw_logout"
    ],
    "can_view_contacts_tab": [
      "program_officer",
      "chw_supervisor",
      "supervisor_online",
      "chw",
	  "chw_logout"
    ],
    "can_view_last_visited_date": [
    ],
    "can_edit_profile": [
      "program_officer",
      "chw_supervisor",
      "supervisor_online"
    ],
    "can_delete_reports": [
      "program_officer"
    ],
    "can_bulk_delete_reports": [
      "program_officer"
    ],
    "can_update_reports": [
      "program_officer"
    ],
    "can_delete_contacts": [
      "program_officer"
    ],
    "can_delete_messages": [
      "program_officer"
    ],
    "can_delete_users": [
      "program_officer"
    ],
    "can_view_outgoing_messages": [
      "program_officer"
    ],
    "can_export_all": [
      "program_officer",
      "crfo"
    ],
    "place_hierarchy_types": [
    ]
  }
}

These are just permissions

Yes nothing else, I removed the purge script together with the outbound script.

Can you share the settings document you have on the server?


The file is quite long I can’t uploaded it here but this is how it looks

it’s quite essential I see the whole of it in order to figure out which setting crashes your sentinel container.

We have file type and character limitation on the amout of code I can post here. I have put the file here: WeTransfer - Send Large Files & Share Photos Online - Up to 2GB Free

The config looks ok, although not complete, so I’m not sure what exactly you’ve shared with me.
The settings document from your server would have a lot more fields.

But the purging schedule looks ok now. can you confirm that you have uploaded this config and share how your sentinel is behaving?

Yes this is all we have on my app_settings.json. Is there anything am missing?
I have the same config on my server and sentinel is still restarting.