High load, slow response when changing password for user with a lot of documents

Technical Support
1

Hi there!

I have a clone of a production instance that I was doing some testing on. When I went into the admin GUI to change the password for a user who I suspect has a high document, it took a very long time. SSHing into the server I can see there’s ~18 instances of beam.smp running and a high load (20 on an 8 core machine).

When I clicked “Submit” the button changed “Submitting…”:

image
image1060×491 22.1 KB

And then after a good long while it said:

Warning! This user would replicate 169378 docs, which exceeds the recommended limit. Edit the Role or Place to make changes as needed, then press Submit to proceed. If there are too many docs for a user consider adjusting the doc purge rules.

image
image1053×693 48.4 KB

I assume it’s hitting the /api/v1/initial-replication/get-ids endpoint to get the 169378 docs count which is why it’s taking so long? I was just surprised why changing a password would take 10+ minutes. I would have expected it take <1second.

Maybe if you use the /api/v1/users/{USERNAME} API you can make the change instantly without incurring the cost of calculating docs?

cc @binod who’s doing the testing with me.

2

Confirming some important findings:

  1. when you experience the Warning! This user would replicate... error, the password change does not take effect.
  2. curl takes just a few seconds and does indeed work using the user API. Be sure to replace MRJONES-NEW-PASSWORD , MEDIC-PASSWORD and CHT-URL with real values:
    curl --header "Content-Type: application/json"  \
 --request POST  \
 --data '{"password":"MRJONES-NEW-PASSWORD"}' \
 https://medic:MEDIC-PASSWORD@CHT-URL/api/v1/users/mrjones

I believe maybe this error was in API logs at the time of the error? It may be unrelated:

2025-10-14T21:35:13.703 RES: 4194e2cce496 75.83.68.217 - POST /medic/_bulk_get?revs=true&attachments=true HTTP/1.0 200 - 14253.895 ms
2025-10-14T21:35:14.154 ERROR: Error on Log Replication Limit Error: Error when persisting log: Log Information missing.
    at persistLog (/service/api/src/services/replication-limit-log.js:14:19)
    at Object.logReplicationLimit [as put] (/service/api/src/services/replication-limit-log.js:87:10)
    at Object.getContext (/service/api/src/services/replication.js:19:29)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async getAllowedDocsCounts (/service/api/src/controllers/users.js:100:47) {
  [stack]: [Getter/Setter],
  [message]: 'Error when persisting log: Log Information missing.'
} 
2025-10-14T21:35:14.155 RES: e11d8895074c 24.253.112.58 - GET /api/v1/users-info?contact_id=f32ec621-b457-49a9-954a-622067ef0154&facility_id=71e75e97-36a2-47c5-950b-38cb55f0d154&role=munic_level HTTP/1.0 200 66 1050865.412 ms

All in all, I think this is a bug and I’ll file a ticket later today or tomorrrow - but I wasn’t sure if it was a bug because I hadn’t seen the error message and I hadn’t tested if the password change took effect!

3

@binod - you were asking which error I meant in step 1 above. I’ve edited my post to clarify it’s the Warning! This user would replicate... error you get when you use the admin GUI to try and change the password.

4

Is it an error or just a warning?

If it’s a warning, isn’t it expected that password is not changed until you confirm by hitting Submit again? Or would you expect this to happen only when adding new users or changing the user’s associated place?

In my tests, it took a long time to show this warning:

  • 26 minutes for 171,197 docs
  • 13 minutes for 440,052 docs on a different day

But after submitting again, the password was changed quickly.

Also, the number of documents shown in this warning was different from what was seen later in the initial replication message “Fetching info (x of y docs) …”:

  • 171,197 docs in warning jumped to 177,554 docs during the initial replication
  • 440,052 docs in the warning jumped to 446,627 docs
5

oh - goodness me - thank you @binod ! I was working so fast and furious I didn’t read the literal text that was right there in the on my screen (emphasis mine):

Edit the Role or Place to make changes as needed, then press Submit to proceed

So…yes, I assume if I had clicked “Submit” again, it would have worked. However, it was so very long to wait and I had 5 users to change their password, I just assumed it was a bug :sweat_smile:

Thanks again for pointing out the error of my ways!

That all said, this is a pretty terrible user experience to just change the password. Maybe we could detect if no place changes have been made we don’t do the potentially very slow API call? I assume the place change will change how many documents you sync. Thus in the case of just a password change, this is not a bug, but an improvement we could make?

Oh, we could change the button from saying “Submitting…” to “Validating data…”? I don’t know?? 20+ min is just too long to wait I say.

2 Likes
6

I noticed some discrepancies in the numbers reported as well! Mainly for numbers in /api/v1/users-info API that I made earlier and then right now using my script.

7

I noticed some discrepancies in the numbers reported as well! Mainly for numbers in /api/v1/users-info API that I made earlier and then right now using my script.

That’s expected. That user-info call doesn’t take the actual user as parameter, it takes their new role and the new facility and calculates based on that. The actual user would also download some user-specific documents (like tasks) that this call would not know about.

1 Like
8

The users-info call does return accurate numbers when requesting a request authenticated with the user’s username and password.

1 Like