I/O performance optimisation

Technical Support
1

We are working with a hosting provider with slow I/O are are also looking to explore potential optimisation strategies to help mitigate the situation. Below are some options we are considering:

  • Cold storage. Do we know if it has yields significant improvements?
  • RAID setup

Are there others we can consider?

1 Like
2

Thanks for posting your question @derick!

The very best solution would be to use a hosting solution with solid I/O. Medic has been hosting the CHT for many years across hundreds of instances and have not had this issue. The rest of my response assumes you can not improve I/O and are forced to use the current hosting provider.

@diana , @jkuester and @elijah and I have been working on this issue for some time in a private ticket, which Iā€™ll summarize:

  • While the private ticket is about a clone of a VM, the underlying slow storage issue applies to your production VMs as well
  • The issue being faced is that upgrades fail
  • The main issue is that iowait is very high, the exact symptom you would expect with slow I/O (slow reads and writes to disk)
  • Current focus of the ticket has been to serially stage views during upgrades in hopes that this is less I/O intensive so that upgrades succeed
  • Weā€™re perusing using the pre-stage branch out the CHToolbox to test serial upgrades

To your question:

Cold storage. Do we know if it has yields significant improvements?

Any activity you can do to reduce I/O will help, including cold storage. However cold storage is very hard to implement and may cause CHWs to sync heavy amounts of data. We do not provide documentation around this. An easier solutions would be purging.

However, weā€™re seeing errors like this in our tests:

could not load validation funs + corrupted_data

This suggests that when the server is under load, any transaction could fail. That is, with your current number of users writing the current number of new documents, any transaction may fail and need to be retried. Iā€™m doubtful that deleting documents via cold storage or implementing purging will help.

RAID setup

Weā€™re seeing a iowait (from iostat) on your VMs as high as 40. A single SSD should be able to support any given CHT instance. So, yes, RAID might help, but so would having a single, local to the VM SSD. RAID is one solution to ā€œSlow I/Oā€, but maybe not the best given how prescriptive it is. Hypothetically speaking, if your I/O bound VM is on a heavily congested network connection to a SAN that somehow doesnā€™t have RAID, demanding RAID on the SAN wonā€™t solve the heavily congested network connection.

Understanding why the VM has such bad iowait would be the first step. Maybe you can ask your hosting provider for a bare-metal box instead of a VM? I assume the SAN gives you features like disk snapshots, rollbacks and easily increasing the size of the VM disk. However if your CHT instance can server the basic function of supporting CHWs, these SAN features arenā€™t worth it.

3

Hi @derick

So sorry to hear about your disk performance issues!

Unfortunately, cold storage via CouchDb purging turned out to be unfeasible, due to the need to rebuild view indexes after a significant amount of docs were purged. The purging operation is also quite slow (even on performant hardware!) .

Another option for cold storage would be filtered replication - replicate only the needed documents to a new database and start the CHT project using that database.
This would most likely involve some downtime and it will take additional resources to replicate to the new DB.

While itā€™s true that having a smaller database will reduce some number of disk reads, the writes will still be slow. So itā€™s an unknown how much of a impact a smaller database will have with low disk performance.

4

@diana We do that when setting up staging/UAT instances ā€“ basically a slim instance where a user can log in with their prod credentials, view their hierarchy and complete workflows, but they donā€™t have any of their old data (data_records). Essentially we do a filtered replication of only contacts, _users and user-settings. This method is very quick and you can come up with a ā€˜cloneā€™ in only a few hrs.

Applying it to this problem, I think it would work and the users will log in quickly, but then they all will start uploading documents en-masse (which are now not present in the slim instance), leading to the same performance issues. Another challenge is the fastest couch filtered replication method (using views) cannot be applied here since views not building is a big part of the problem. The replications using a selector or a filter were too slow in our tests. So ultimately an external filtered replication might be considered, which is where we ended up.

2 Likes
5

Hi @rmayore

I donā€™t think this will be a problem, because likely the amount of docs on userā€™s devices is much less than what exists on the server due to purging, You do have purging enabled, right
?

6

I checked out the purging configurationā€¦ I think itā€™s well set up (most docs are set to be purged after a 1-3 month window).