Not sure where to go with this question, but is it possible at this point to horizontally scale the API?
How would one scale sentinel alongside it?
Hi @fardarter
At this moment, it’s not possible to scale API or Sentinel horizontally. There are some workflows that cannot run in parallel that are baked into both apps.
We considered scaling API horizontally, but in our testing, we have not identified API as a bottleneck in scalability, so we deprioritized it. It’s likely we will reconsider this decision in the future.
Most of API would work fine with horizontal scale as is, but there are some parts that watch the db and make updates based on what’s there, for example, the config watcher listens for form xml being updated and generates the enketo html and model xml needed to render this on the phone. If you had multiple API services doing this then they would fire at the same time and conflict frequently. The solution I had in mind was to split out the code that can only run on a single instance as a separate service, then the remaining API code would be stateless request handling.
Sentinel is more difficult, because its been designed to listen for changes and sequentially process them. I think we need a fundamental rethink of how Sentinel works because I’m sure there’s a better model to achieve the same goal. For example it could publish changes on a queue which is then processed by individual subscribers. Or the sentinel transitions could be run on the phone before the doc is even synced to the server (some transitions need access to the whole db but not all).
To add to what Diana said, in our scalability testing it became clear that scaling CouchDB was the most impactful change and we haven’t yet found any project that is limited by API performance. With the round of scalability improvements that have shipped in 4.0-4.4 it’s a good time to review where the bottlenecks are again and refocus effort to where it’s needed.
Were you thinking about horizontal scalability for server throughput, for high availability, or both?
HA is just one advantage. Horizontally scalable containers can also be auto-scaled, which would let me set a low allocated resource value per container and just let it scale up for use, which is cost saving (as with one container I have to pay consistently for the base allocation), and also provides some safety on the server usage.
At Medic we haven’t had a need to autoscale API because we find the database is the bottleneck. Have you found that API containers are the bottleneck for your use case?
If so, it would be worth looking at automatically scaling the API node vertically to see if that solves the issue without incurring ongoing hosting charges.
Allowing the request handling parts of API to scale horizontally is on the roadmap which will give more flexibility here.
We don’t yet have a need. I’m just looking ahead.