Bizarre recursive place case

Technical Support Development
, ,
1

We ran into a very strange, and luckily once off, case where a household was somehow saved as a child of another household.
Households are a contact_type on our side, which should only ever have a parent of the dwelling contact type.
This has not changed since the day we deployed our project configuration in last year November.
It was found when a routine SQL report downstream kept producing unexpected results.
We’ve tried asking the affected CHW to take us through the steps that lead up to this entry’s creation, unfortunately they’re unable to recall.
So far we’ve been unable to replicate this behavior ourselves.

Does anyone here have any ideas of how this could happen?

Our hierarchy structure for reference:

{
      "id": "npo",
      "name_key": "contact.type.npo",
      "group_key": "contact.type.npo.plural",
      "create_key": "contact.type.npo.new",
      "edit_key": "contact.type.place.edit",
      "icon": "wcg-npo",
      "create_form": "form:contact:npo:create",
      "edit_form": "form:contact:npo:edit"
    },
    {
      "id": "dho",
      "name_key": "contact.type.dho",
      "group_key": "contact.type.dho.plural",
      "create_key": "contact.type.dho.new",
      "edit_key": "contact.type.dho.edit",
      "primary_contact_key": "contact.type.dho-primary",
      "parents": [
        "npo"
      ],
      "icon": "wcg-dho",
      "create_form": "form:contact:dho:create",
      "edit_form": "form:contact:dho:edit",
      "person": true
    },
    {
      "id": "vap",
      "name_key": "contact.type.vap",
      "group_key": "contact.type.vap.plural",
      "create_key": "contact.type.vap.new",
      "edit_key": "contact.type.vap.edit",
      "primary_contact_key": "contact.type.vap-primary",
      "parents": [
        "npo"
      ],
      "icon": "wcg-vap",
      "create_form": "form:contact:vap:create",
      "edit_form": "form:contact:vap:edit",
      "person": true
    },
    {
      "id": "team_area",
      "name_key": "contact.type.team_area",
      "group_key": "contact.type.team_area.plural",
      "create_key": "contact.type.team_area.new",
      "edit_key": "contact.type.place.edit",
      "parents": [
        "npo"
      ],
      "icon": "wcg-team_area",
      "create_form": "form:contact:team_area:create",
      "edit_form": "form:contact:team_area:edit"
    },
    {
      "id": "team_lead",
      "name_key": "contact.type.team_lead",
      "group_key": "contact.type.team_lead.plural",
      "create_key": "contact.type.team_lead.new",
      "edit_key": "contact.type.team_lead.edit",
      "primary_contact_key": "contact.type.team_lead-primary",
      "parents": [
        "team_area"
      ],
      "icon": "wcg-team_lead",
      "create_form": "form:contact:team_lead:create",
      "edit_form": "form:contact:team_lead:edit",
      "person": true
    },
    {
      "id": "cca",
      "name_key": "contact.type.cca",
      "group_key": "contact.type.cca.plural",
      "create_key": "contact.type.cca.new",
      "edit_key": "contact.type.cca.edit",
      "primary_contact_key": "contact.type.cca-primary",
      "parents": [
        "team_area"
      ],
      "icon": "wcg-cca",
      "create_form": "form:contact:cca:create",
      "edit_form": "form:contact:cca:edit",
      "person": true
    },
    {
      "id": "indawo",
      "name_key": "contact.type.indawo",
      "group_key": "contact.type.indawo.plural",
      "create_key": "contact.type.indawo.new",
      "edit_key": "contact.type.place.edit",
      "parents": [
        "team_area"
      ],
      "icon": "wcg-indawo",
      "create_form": "form:contact:indawo:create",
      "edit_form": "form:contact:indawo:edit"
    },
    {
      "id": "super_chw",
      "name_key": "contact.type.super_chw",
      "group_key": "contact.type.super_chw.plural",
      "create_key": "contact.type.super_chw.new",
      "edit_key": "contact.type.super_chw.edit",
      "primary_contact_key": "contact.type.super_chw-primary",
      "parents": [
        "indawo"
      ],
      "icon": "icon-risk",
      "create_form": "form:contact:super_chw:create",
      "edit_form": "form:contact:super_chw:edit",
      "person": true
    },
    {
      "id": "chw",
      "name_key": "contact.type.chw",
      "group_key": "contact.type.chw.plural",
      "create_key": "contact.type.chw.new",
      "edit_key": "contact.type.chw.edit",
      "primary_contact_key": "contact.type.chw-primary",
      "parents": [
        "indawo"
      ],
      "icon": "wcg-chw",
      "create_form": "form:contact:chw:create",
      "edit_form": "form:contact:chw:edit",
      "person": true
    },
    {
      "id": "dwelling",
      "name_key": "contact.type.dwelling",
      "group_key": "contact.type.dwelling.plural",
      "create_key": "contact.type.dwelling.new",
      "edit_key": "contact.type.place.edit",
      "parents": [
        "indawo"
      ],
      "icon": "wcg-dwelling",
      "create_form": "form:contact:dwelling:create",
      "edit_form": "form:contact:dwelling:edit"
    },
    {
      "id": "household",
      "name_key": "contact.type.household",
      "group_key": "contact.type.household.plural",
      "create_key": "contact.type.household.new",
      "edit_key": "contact.type.household.edit",
      "parents": [
        "dwelling"
      ],
      "icon": "wcg-household",
      "create_form": "form:contact:household:create",
      "edit_form": "form:contact:household:edit",
      "count_visits": true
    },

Our CHT version is currently still 4.3.x.

2

Very weird! I don’t have any solid ideas here, but I can say that the CHT is generally very unopinionated about the hierarchy of contact types. To be clear, the CHT cares a lot about the contact hierarchy (which contact is the parent of what other docs and so on), but the code does not have a lot of logic for checking that the type of a child contact is appropriate based on the type of a given parent contact.

IIRC, the contact_types structure defined in your app_settings does determine which contact forms are shown on the profile page of particular types of contacts. (On a given contact’s profile page, you should only be able to access create forms for contact types that have the current contact’s type defined in the parents property.)

Technically, though, it is possible, via improper form config, to write contacts at the wrong hierarchy level. In both contact and app forms, bugs in the form logic for calculating the parent or the contact_type could result in a contact being created with an unexpected type or at an unexpected place in the hierarchy. I don’t think there is much in the way of validation in the CHT code to prevent this. The same goes for any external scripts that might be adding/modifying contacts in Couch. As long as the contact’s parent hierarchy extends back up to a hierarchy tree that includes the user’s facility_id, then the user would be able to sync the contact and see it.

One interesting investigation point would be to see if you can figure out when the parent data of the household contact was set. If you GET the contact doc directly from Couch (e.g. via curl) you can pass the revs_info=true query param to get back a _revs_info object in the response that will contain an array of revision data. Any entries that do not have "status": "available" have unfortunately been compacted already and we will not be able to see the changes. But if any are available, you can GET that revision by passing the rev in the rev query parameter and then diff it with the current version of the doc to see if the parent data changed.

1 Like
3

Given how CHT has to support multiple varying configs, it makes sense that it’s unopinionated.
I have seen that the conf-tool seems to respect the hierarchy quite closely, restricting the movement of contacts to their intended parents.

That’s the behavior we’ve been witnessing, and that we expect. As I was typing this, I logged in to the system as admin and navigated to the faulty entries and sure enough I get the usual creation menu. Somehow this person is/was able to circumvent that.

You raise a good point, and we have been feverishly changing forms to keep up with our backlog. It’s entirely possible that one of the forms had an unexpected bug.
Seemingly to confirm your theory, we’ve run a couple of queries against our db and it seems each record of concern, which seems to be 6 in total (all under the same CHW), has only one revision. Problematic from the start :smiling_face_with_tear:.

The initial thought was that perhaps we fixed the issue, unknowingly, before there was widespread issues. However, our release cycles to prod is not as quick as I’d like (~ 1 month), so that seems unlikely.
Since this is the only user exhibiting these strange behaviors, is it not possible that the issue is device specific? Although, that too seems unlikely given the nature of the issue.

4

I agree that it seems odd to have only one user presenting this issue if it was actually a bug in the form… Are your users the ones responsible for creating all the households? (Or was some/all of the household hierarchy pre-built?) I just cannot find anything in the save contact form code that would somehow override what is coming from the contact form (and if there are no revisions for the households, then it seems like they could not have been somehow updated later…).

5

Exactly. From a data standpoint we’re glad it’s an isolated problem, but that kind of makes it a little difficult to pinpoint it’s origin. Which we’d rather try to identify than just stick a plaster on the issue.

We’ve only ever hydrated our instance until the Indawo level. From the plot down, including households, are all created by the users.

Could you please provide an example of how one could accidentally change the hierarchy via an app form?
We’re aware of how it can be done via a contact form, after all that’s how one facilitates the moving of a person.

We’re at a bit of a loss on this one too. Perhaps this was one of those perfect storm cases. I’ll discuss with the team whether we’re happy with just fixing the issue, and if this issue ever rears its head again, we’ll do a deep dive.

6

So, modifying a contact via an app form is technically not supported. (Though, since these contacts only have one revision, I guess we would not be looking for a modification scenario…) However, you can add a contact (or other kinds of docs) via an app form. This is how the default delivery form adds new patient contacts for the recorded babies. When you add a contact from an app form, the app form is responsible for setting the parent data for the new contact. If the parent for the new contact points to the wrong contact or wrong level, the CHT will still write the new contact (and, as long as the contact was written somewhere in the hierarchy that is actually accessible to the current user, the data should sync to the server just fine).

7

Ah yes, we have heard of this cool feature a while back!
However, up until now we have not utilized it in our instance.
I believe it’ll provide value and should prove to be a valid argument in rethinking our pregnancy capture that’s currently part of a larger form.
Thank you, I’ve sent the info along.

Unfortunately, this does not seem to apply to our current strange case :sweat:.
We’ll be monitoring this situation. Thank you for brain storming with us!

1 Like
8

@jkuester , unfortunately, it has been brought to my attention that another “household in household” case was found as recently as last Monday (2024-09-02). This time in a different Indawo, for a different CHW. Which means the issue can no longer be seen as isolated. :cry:

We couldn’t find anything wrong with the save method, but perhaps the issue lies with the fast action button service? I noticed the 4.3.x (our current version) release notes mention fixing a “Race condition in loading contacts can cause previously selected contact to load”. Does the fix also affect the fast action button service?

The theory is that when our CHW creates a household within a dwelling and the system automatically navigates there, the fast action service, when the “+” button is pressed, sometimes still retains the previous contact reference, displaying the incorrect menu and allowing the CHW to create a household within another household.
Would be great to hear your thoughts.

9

@Jennifer_Quesada I think this is going to need some of your Angular expertise! Basically we are trying to understand how it is possible for one household to be added as the child of another household (when the contact_type configuration does prevent the household-create form from being available at the household level.

I agree with @Anro’s theory that it seems like a race condition (has been replicated more than once, but cannot be replicated (so far) in house.

The parent value for a new contact gets injected into the contact create form’s data model in the contacts-edit.component. As far as I can tell, this value should not get adjusted downstream (assuming the form logic does not do anything wonky). @Jennifer_Quesada where I got stuck is that I cannot figure out how/where that parent_id value gets populated! (I guess it is some Angular magic happening, but maybe I am missing something… :thinking: )

@Anro could you post the household contact create form (or perhaps email it to me at jkuester@medic.org if you cannot make it public)? For my own peace of mind, I just want to confirm that the parent logic is as expected… :sweat_smile:

10

@jkuester sure thing. This is the file that’s currently on our prod environment: Loading Google Sheets
The parent content is as follows:

type name default
begin group PLACE_TYPE
hidden parent PARENT
11

As expected, I do not see anything unexpected with the household parent config in that form.

I guess that still brings us back to some kind of race condition between launching the form and changing which contact is in context… I have tried to recreate similar behavior by slowing down the CPU/Network and clicking around in the app, but so far I have not been successful. (This is not surprising, though. If it was easy to produce this behavior, I guess it would be much more common.)

The main lingering doubt I have is that if it is a race condition, why has it only been observed in this deployment? With that prompting some more out-of-the-box thinking, there is no chance that you might have some dynamic url linking that could be opening the /add/household path?

12

The parent’s information comes from the URL (parent_id and contact type). This is the execution flow:

Correct, it doesn’t get adjusted downstream. A good improvement could be that Contact Edit checks if the parent_id in the URL is a direct parent of the contact type the CHW wants to create. I’ve opened a ticket for that: Prevent creating contacts under facilities that are not the direct parent · Issue #9426 · medic/cht-core · GitHub

I wasn’t able to reproduce a race condition when testing with the Safaricom Neon Ultra phone (2GB RAM), so I wonder the same about dynamic URL linking. Could it be that this CHW is opening a link from another source?

13

Thank you @jkuester and @Jennifer_Quesada for taking the time to investigate this.

That is a good question, and an avenue we haven’t considered before. Prod does not enjoy that, what we call follow-up, feature just yet. It’s still in development and, as far as I can tell, only on our development branch. Even in dev our dynamic linking only points to a household edit form, so we would have seen records with more than 1 iteration.

From what we can tell the first occurrences started happening around 2024-04-05, which is quite a bit before the work on the follow-up started.

Thank you for opening up the improvement issue!
I’m wondering if, as a stop gap, we could add a db lookup to our forms that checks the type of the parent before allowing the item to be saved.
For example:
In the household create we know the parent should be of type dwelling. If we do a lookup on the parent ID, and it’s not the correct type, a constraint could halt the form from being submitted.
If the above works, it should prohibit any new issue records from being created until we can pinpoint where things go wrong.