Configure cronjob or Batch Processing for CHT Outbound

Technical Support Development
1

Hello Team,
I would like your guidance on how we can configure cronjob or batch processing on CHT outbound. The aim is to put a schedule on when Outbound should be executed and if possible can we trigger outbound for a batch of reports?
Thanks,
Job

2

Hi @Job_Isabai

I don’t think that selective processing of the outbound queue is possible at this moment. The “cron” job is handled by Sentinel, which will attempt to process all outbound tasks, along with other scheduled jobs, every 5 minutes.
Additionally, sending a batch of reports, instead of a single one, is also not supported at the moment.

It would be very helpful if you could provide more detail about your use case, in case we might find some workaround that would suit you.

Thanks in advance!

3

Hello Diana,
We are using OpenHIM mediator for data interoperability between CHT and DHIS2.
CHT Triggers outbound immediately after form is saved. The pushed report is then received as incoming payloads on OpenHIM which this then processed by a mediator then pushed to DHIS2 as outgoing payload.
This happens once the form is saved, I would like to set a cronjob that initiates this data pipeline at a specific time. Can this be done from CHT?
Thanks,
Job

4

Hi @Job_Isabai

Currently, I don’t believe there is a way to delay or reschedule outbound pushes. What is the reason you would want to delay pushes to DIH2?

5

Hello @diana
To navigate around this issue, can I manually trigger outbound on CHT.
This will help me push data outside CHT on a weekly basis thus giving me 100% control over data push to DHIS2.
Thanks,
Job

6

Hello @Job_Isabai

There is no mechanism to trigger outbound, the service that sends outbound requests doesn’t have an API or an interface that a user can interact with.

Is this a hard requirement for you?
I can think of some hacky workarounds, but I wouldn’t recommend going that route. I think that sending outbound on a specific schedule is a good feature to implement. I’ve created an issue to track this: Send outbound tasks on a schedule · Issue #8229 · medic/cht-core · GitHub

In the meantime I’m passing this along to @njuguna or @imoisili with a question: can a mediator, or some other interoperability layer, be configured to handle this type of scheduling? Thanks!

7

Hello @Job_Isabai,

As @diana has alluded to you can achieve this by having your mediator handle the scheduling. You would need to have a database connected to your mediator where you would store the data coming in from the CHT and then add the logic to process it on a weekly basis. I’m happy to give more details if you want. This approach has an additional benefit of making it easier to send the same data to other systems should that need arise.

4 Likes
8

Hello @njuguna,
Thanks for coming-in on this.
If there is a possibility of scheduling OpenHIM Mediator, I will gladly take that option. Please give me the details on how I can implement this on OpenHIM.

9

Hey @Job_Isabai allow me to chime in with lessons from a recent implementation of CHT to DHIS2 data push via OpenHIM.

Problem

Get past month aggregate data from CHT to DHIS2 every mid of the preceding month.

Solution

  1. Aggregate the data in a Postgres database. We created a function that returns a table with all the required data.
  2. Configure a cron service that queries Postgres for the past month data every mid of the preceding month. For this, we used node-cron for scheduling.
  3. Create a GET mediator endpoint, that will be requested by the cron service to query the database. We used pg to create the database connection pool for querying. Axios will come in handy to call the endpoint from within the mediator. Remember to define the function that queries Postgres and returns your data of interest.
  4. Define a function that POSTs the aggregate data payload to the DHIS2 endpoint that consumes the data. For convenience, you may want to call the function from with the function executed in (3) above so that after the data is pushed after it is obtained from the database.
  5. Register the cron service and endpoints with the mediator.

Let us know how it progresses.

2 Likes
10

Hey @kitsao ,
Thanks for the suggestion.
My use-case is somehow different, I am sending event level data from CHT to OpenHIM. Currently each form entry is pushed to OpenHIM as an incoming payload which is then posted to DHIS2 as an event. This means that I am not sending aggregate data to DHIS2.
Please let me know how I can apply the above suggestion in my context.
Regards,
Job

11

Hello @Job_Isabai,

Did you manage to get this working or do you still need some assistance on it?

12

Hello @njuguna ,
Yes I need your help on this,
The solution proposed by @kitsao was not applicable in my scenario.
My scenario is as per my comments above, do you think we can achieve it?
Regards,
Job

1 Like
13

Hello @Job_Isabai,

Yes, I think we can. Do all the form entries in the CHT need to go to DHIS2 or only a subset of them? We can remove the need for postgres and just have a mediator in OpenHIM that does the filtering of events (if that is required) and calls DHIS2. The mediator can be a simple nodejs app similar to this one. You can find instructions on building a basic mediator here.

14

Hello @njuguna,
Yes we don’t need Postgres at this point.
I filter events at the outbound level, but given a chance I would take the option of putting the filter at mediator level. I will review the above nodejs app together guide. If possible let me know your availability for a quick call, I would like to take you through our setup.
Thanks,
Job

15

Hi @Job_Isabai happy to jump on a call. I’ll get your email address from a colleague and we can set it up.

16

Hello @Job_Isabai,

I have some great news regarding this request! We finally merged the code necessary for this feature and it should be available in the next release of the CHT which is 4.5. If you would like to give it a try before the release we can work together and have you try it on a feature release.

2 Likes
17

Hi @njuguna , @Job_Isabai , We’ve just released v4.5.0.

3 Likes
18

@njuguna I configured my cron as per the instructions on the doc, but still the docs are sent immediately after submission. Please refer to the attached logs:

image
image1682Ă—1700 468 KB

Outbound configuration:
image
image1682Ă—332 25.3 KB

19

Hi @Job_Isabai I wanted to follow up after our call and find out if you were able to get this to work.

20

Hi Njuguna,
After numerous testing, I have not been able to implement this feature.
Outbound config:
image
Logs

image
image1284Ă—67 79.5 KB

Docs:

  "outbound": {
    "first config": {
      "relevant_to": "...",
      "destination": {},
      "mapping": {},
      "cron": "..."
    },
    "second config": { }
  }
}