We are running CHT in a cluster having 1 control plane and 3 worker nodes. Each worker node has 32 vCPUs, 32 GB RAM, and 500GB of storage. We are however having a huge backlog count for Sentinel as in the screenshot below.
Whereas the the backlog count is reducing, it is doing so at a slow pace(roughly 400k docs a day). This is also impacted by the CPUs getting saturated during peak hours.
Is there a way to speed up sentinel?
Is the huge backlog and CPU saturation an indication of an issue with the cluster configuration?
Is there a way to tell what exactly the process beam.smp is doing that causes the CPUs to be saturated?
It looks like there was a spike in doc count for Sentinel, am I right?
Was it misconfigured for a long time and you just started it?
Unfortunately, Sentinel processes one doc at a time, and 400k docs per day is the rhythm of your hardware. Sentinel does query the database for every change it processes, so it does explain the high CPU usage. I’m quite happy all cores are used (old CouchDb versions had trouble with using so many cores!)
There is no way to speed up Sentinel, given the same “config”.
I would advise to inspect your configuration. Which transitions are enabled? Is outbound push configured? Try disabling some transitions and outbound push and check if the speed increases.
As a last resort, and only if you don’t care about Sentinel processing the current queue, you could advance the Sentinel sequence and effectively make it skip the queue.
Let me know what you decide, and I would be curious how this queue was accumulated in the first place. Good luck!
