Upper limit on number of docs server will sync

I have been discussing an issue with @melema and @Patrick_K where occasionally users on large instances get misconfigured such that the user has an excessive amount of docs associated with them (e.g. the users are added at the wrong level in the hierarchy). In those cases, these users can have a very negative impact on server performance when they try to sync.

I know that when a user is associated with 10000 or more docs that a warning will be presented on the client side asking if they really want to sync that many docs. However, the user can simply continue the sync, negatively affecting the performance for everyone else on the server.

Is there a way to actually prevent a user with too many docs from syncing altogether?

Related:

1 Like

There is a warning shown to administrators when creating users that would replicate too many docs either via the admin app or cht-conf. However I suspect this warning doesn’t show when moving contacts. That would be a useful addition to cht-conf.

After the fact, these misconfigured users can be detected using the APIs created in this issue.

It would be possible to block replication completely when the user is over the limit, but I worry that be jarring for existing users that create one more doc and suddenly are blocked from replication. One compromise would be to have a warning limit as well as a higher limit that blocks the user completely, so there’s time for the admin to fix the replication/purging settings before the user is cut off.

Another issue is the user is only shown the warning the first time they log in, so a user who has been misconfigured will not see the warning at all.

Do you know how far over the limit the users were?

1 Like

I didn’t know about the API resulting from the issue @gareth cited, so I enjoyed exploring it just now. Here’s my findings!

The API was released in CHT 3.11.0, is documented here, and needs a users with _admin role and is called at at /api/v1/users-doc-count.

I have only three users in my local dev instance, all with less than 100 docs. I can call it, pass the results into jq and see the output:

curl -s https://medic:password@192-168-68-108.my.local-ip.co/api/v1/users-doc-count | jq
{
  "limit": 10000,
  "users": [
    {
      "_id": "replication-count-abdul",
      "_rev": "1-0a35a0e096a985510662dc2fd4417eca",
      "user": "abdul",
      "date": 1659111882929,
      "count": 46
    },
    {
      "_id": "replication-count-mrjones",
      "_rev": "1-4de640ce4ef29b812b30b5b8a6c040c7",
      "user": "mrjones",
      "date": 1659101850205,
      "count": 34
    },
    {
      "_id": "replication-count-mrjones-replacement",
      "_rev": "1-7a5593328e882f851ddc80e87763bcf8",
      "user": "mrjones-replacement",
      "date": 1659110993531,
      "count": 40
    }
  ]
}

However, what if I have 10s, 100s or even 1000s of users? How can I easily know which users are over the limit of 10000? Thanks to the power of jq, we can easily filter out any users above 10,000 and show just their count and username. For my case, I’ll filter at or above 40 to show the filter working:

curl -s https://medic:password@192-168-68-108.my.local-ip.co/api/v1/users-doc-count | \
   jq '.users[] | select(.count >= 40) | .count, .user'
46
"abdul"
40
"mrjones-replacement"

Hopefully this helps anyone else reading this thread - cheers!

3 Likes