Hello,
After migrating a CHT cluster with 47 nodes to a new datacenter, 503 errors have been persistent. The cluster setup was maintained as the previous datacenter with the exception of using 1 control plane instead of two in the old environment. In the previous datacenter, 503 errors were not experienced. However in the new datacenter, they have become very common.
Initially, the errors were attributed to slow disk performance and the disks were upgraded to SSDs. However, even after upgrading disks to SSDs 503 errors are still persistent. Here are logs snippets from the different services:
API
2025-05-18T19:33:47.341 RES: 78bb89b5-58f9-4e98-ad27-ab37bdb8c560 10.42.148.0 - GET /api/v1/replication/get-ids HTTP/1.1 500 - 3.772 ms
2025-05-18T19:33:47.481 REQ: 41ac0c9b-56a1-4339-8439-175af6bbb782 10.42.148.0 - GET /dbinfo HTTP/1.1
2025-05-18T19:33:47.483 ERROR: 503 - {"error":"503 Service Unavailable","reason":"No server is available to handle this request","server":"haproxy"} StatusCodeError: 503 - {"error":"503 Service Unavailable","reason":"No server is available to handle this request","server":"haproxy"}
at new StatusCodeError (/service/api/node_modules/request-promise-core/lib/errors.js:32:15)
at plumbing.callback (/service/api/node_modules/request-promise-core/lib/plumbing.js:104:33)
at Request.RP$callback [as _callback] (/service/api/node_modules/request-promise-core/lib/plumbing.js:46:31)
at self.callback (/service/api/node_modules/request/request.js:185:22)
at Request.emit (node:events:519:28)
at Request.<anonymous> (/service/api/node_modules/request/request.js:1154:10)
at Request.emit (node:events:519:28)
at IncomingMessage.<anonymous> (/service/api/node_modules/request/request.js:1076:12)
at Object.onceWrapper (node:events:633:28)
at IncomingMessage.emit (node:events:531:35)
at endReadableNT (node:internal/streams/readable:1696:12)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
CouchDB
[notice] 2025-05-18T19:34:35.691094Z couchdb@127.0.0.1 <0.26238.2040> 70f5710da4 couchdb.echis-nairobi.svc.cluster.local:5984 10.42.148.13 medic GET /_membership 200 ok 1
[notice] 2025-05-18T19:34:39.874658Z couchdb@127.0.0.1 <0.18675.1995> 0b64d91c7a nairobi-echis.health.go.ke 10.42.148.19 undefined GET /_session 401 ok 0
[notice] 2025-05-18T19:34:39.886852Z couchdb@127.0.0.1 <0.18675.1995> 57ecbc5501 haproxy.echis-nairobi.svc.cluster.local:5984 10.42.148.19 medic GET /medic-sentinel/_local/transitions-seq? 200 ok 1
[info] 2025-05-18T19:34:39.903992Z couchdb@127.0.0.1 <0.8764.1909> -------- Starting index update for db: shards/2aaaaaaa-3ffffffe/medic-sentinel.1682956797 idx: _design/sentinel
[info] 2025-05-18T19:34:39.905250Z couchdb@127.0.0.1 <0.19642.2073> -------- Starting index update for db: shards/7ffffffe-95555552/medic-sentinel.1682956797 idx: _design/sentinel
[info] 2025-05-18T19:34:39.909521Z couchdb@127.0.0.1 <0.8573.1942> -------- Starting index update for db: shards/3fffffff-55555553/medic.1682956796 idx: _design/medic-admin
[info] 2025-05-18T19:34:39.909613Z couchdb@127.0.0.1 <0.20726.2159> -------- Starting index update for db: shards/55555554-6aaaaaa8/medic.1682956796 idx: _design/medic-admin
[info] 2025-05-18T19:34:39.910447Z couchdb@127.0.0.1 <0.23889.2149> -------- Starting index update for db: shards/7ffffffe-95555552/medic.1682956796 idx: _design/medic-admin
[info] 2025-05-18T19:34:39.911227Z couchdb@127.0.0.1 <0.26252.2183> -------- Starting index update for db: shards/00000000-15555554/medic.1682956796 idx: _design/medic-admin
[info] 2025-05-18T19:34:39.911513Z couchdb@127.0.0.1 <0.3768.2077> -------- Starting index update for db: shards/00000000-15555554/medic-sentinel.1682956797 idx: _design/sentinel
[info] 2025-05-18T19:34:39.911700Z couchdb@127.0.0.1 <0.12254.1636> -------- Starting index update for db: shards/95555553-aaaaaaa7/medic-sentinel.1682956797 idx: _design/sentinel
[info] 2025-05-18T19:34:39.911722Z couchdb@127.0.0.1 <0.20671.1875> -------- Starting index update for db: shards/7ffffffe-95555552/medic.1682956796 idx: _design/medic-conflicts
[notice] 2025-05-18T19:34:39.911792Z couchdb@127.0.0.1 <0.4654.1945> 424d973157 haproxy.echis-nairobi.svc.cluster.local:5984 10.42.148.19 medic GET /medic-logs/_design/logs/_view/replication_limit 200 ok 16
[info] 2025-05-18T19:34:39.911878Z couchdb@127.0.0.1 <0.9750.2001> -------- Starting index update for db: shards/15555555-2aaaaaa9/medic.1682956796 idx: _design/medic-admin
[info] 2025-05-18T19:34:39.911969Z couchdb@127.0.0.1 <0.24082.2118> -------- Starting index update for db: shards/00000000-15555554/medic.1682956796 idx: _design/medic-conflicts
[notice] 2025-05-18T19:34:39.911985Z couchdb@127.0.0.1 <0.18675.1995> 77888b1de2 haproxy.echis-nairobi.svc.cluster.local:5984 10.42.148.19 medic GET /medic/_changes?since=53131-g1AAAAO1eJyV071OwzAQB3CLUpXPoQMMjIwMKE4dO2GBnQlajww-J1EUFZiYeQJGHoAFWo-sDEwsSIx9iT4DrTl8jBWS5eEv2fJPpzt7zBjbaTol69vbO9uUcMZTdZzg4mM8WjMM9rQetU0HGDu5usa9nhEqAytX3flH-nU0HDk3JezwLWC5qLhMRSyGjoNT75eE7T8FLFPG5BCNoeMBqxsS1v8KmKyNFCKLxdAZQevchLDdD-oZFHJg61gMnSnce78gbJsqq_kARFrFYugsb9bZpdaPxG11A5dUSWFtGcmhNEQJ4wVnQeDmawAFl1UqYseA0gQljG_v3wnceAigzS2U0XNFaYESxoXWMwJ75wHkYLmyJh7E5s0wnp2b_4EHASxMISoV-yeoeXMMfIGewO4n9RBUVgzUqqvtD1Kc_lg&limit=0 200 ok 17
[info] 2025-05-18T19:34:39.912272Z couchdb@127.0.0.1 <0.10238.1986> -------- Starting index update for db: shards/55555554-6aaaaaa8/medic.1682956796 idx: _design/medic-conflicts
[info] 2025-05-18T19:34:39.912819Z couchdb@127.0.0.1 <0.10474.1819> -------- Starting index update for db: shards/15555555-2aaaaaa9/medic.1682956796 idx: _design/medic-conflicts
[info] 2025-05-18T19:34:39.913919Z couchdb@127.0.0.1 <0.15523.1922> -------- Starting index update for db: shards/d5555552-eaaaaaa6/medic.1682956796 idx: _design/medic-admin
[notice] 2025-05-18T19:34:39.915189Z couchdb@127.0.0.1 <0.23940.1959> bcf17df7de haproxy.echis-nairobi.svc.cluster.local:5984 10.42.148.19 medic GET /medic-logs/_design/logs/_view/connected_users?reduce=true&startkey=1745004879878 200 ok 20
[info] 2025-05-18T19:34:39.917102Z couchdb@127.0.0.1 <0.32078.1997> -------- Starting index update for db: shards/3fffffff-55555553/medic.1682956796 idx: _design/medic-conflicts
[notice] 2025-05-18T19:34:39.918955Z couchdb@127.0.0.1 <0.13471.2141> 9615ada3bd haproxy.echis-nairobi.svc.cluster.local:5984 10.42.148.19 medic GET / 200 ok 8
[info] 2025-05-18T19:34:39.919342Z couchdb@127.0.0.1 <0.24792.2082> -------- Starting index update for db: shards/55555554-6aaaaaa8/medic.1682956796 idx: _design/medic-sms
[info] 2025-05-18T19:34:39.919512Z couchdb@127.0.0.1 <0.19502.1970> -------- Starting index update for db: shards/00000000-15555554/medic.1682956796 idx: _design/medic-sms
[info] 2025-05-18T19:34:39.920305Z couchdb@127.0.0.1 <0.2315.1889> -------- Starting index update for db: shards/3fffffff-55555553/medic.1682956796 idx: _design/medic-sms
[info] 2025-05-18T19:34:39.920512Z couchdb@127.0.0.1 <0.19714.1937> -------- Starting index update for db: shards/d5555552-eaaaaaa6/medic.1682956796 idx: _design/medic-sms
[info] 2025-05-18T19:34:39.920832Z couchdb@127.0.0.1 <0.13597.2043> -------- Starting index update for db: shards/7ffffffe-95555552/medic.1682956796 idx: _design/medic-sms
[info] 2025-05-18T19:34:39.920910Z couchdb@127.0.0.1 <0.28562.1971> -------- Starting index update for db: shards/d5555552-eaaaaaa6/medic.1682956796 idx: _design/medic-conflicts
[info] 2025-05-18T19:34:39.922813Z couchdb@127.0.0.1 <0.23587.2058> -------- Starting index update for db: shards/15555555-2aaaaaa9/medic.1682956796 idx: _design/medic-sms
[notice] 2025-05-18T19:34:39.937160Z couchdb@127.0.0.1 <0.4654.1945> 16558990f8 haproxy.echis-nairobi.svc.cluster.local:5984 10.42.148.19 medic POST //_dbs_info 200 ok 25
[notice] 2025-05-18T19:34:39.938699Z couchdb@127.0.0.1 <0.18675.1995> 30e8a85eec haproxy.echis-nairobi.svc.cluster.local:5984 10.42.148.19 medic GET //medic/_design/medic/_info 200 ok 22
[info] 2025-05-18T19:34:39.970475Z couchdb@127.0.0.1 <0.12254.1636> -------- Index update finished for db: shards/95555553-aaaaaaa7/medic-sentinel.1682956797 idx: _design/sentinel
[info] 2025-05-18T19:34:39.972181Z couchdb@127.0.0.1 <0.20726.2159> -------- Index update finished for db: shards/55555554-6aaaaaa8/medic.1682956796 idx: _design/medic-admin
[info] 2025-05-18T19:34:39.974154Z couchdb@127.0.0.1 <0.32078.1997> -------- Index update finished for db: shards/3fffffff-55555553/medic.1682956796 idx: _design/medic-conflicts
[info] 2025-05-18T19:34:39.974775Z couchdb@127.0.0.1 <0.9750.2001> -------- Index update finished for db: shards/15555555-2aaaaaa9/medic.1682956796 idx: _design/medic-admin
[info] 2025-05-18T19:34:39.975557Z couchdb@127.0.0.1 <0.10474.1819> -------- Index update finished for db: shards/15555555-2aaaaaa9/medic.1682956796 idx: _design/medic-conflicts
[info] 2025-05-18T19:34:39.977362Z couchdb@127.0.0.1 <0.3768.2077> -------- Index update finished for db: shards/00000000-15555554/medic-sentinel.1682956797 idx: _design/sentinel
[notice] 2025-05-18T19:34:39.979318Z couchdb@127.0.0.1 <0.26562.2113> 09a5773e69 haproxy.echis-nairobi.svc.cluster.local:5984 10.42.148.19 medic GET /medic-sentinel/_design/sentinel/_view/outbound_push_tasks 200 ok 84
[info] 2025-05-18T19:34:39.981162Z couchdb@127.0.0.1 <0.24792.2082> -------- Index update finished for db: shards/55555554-6aaaaaa8/medic.1682956796 idx: _design/medic-sms
[info] 2025-05-18T19:34:39.981605Z couchdb@127.0.0.1 <0.15523.1922> -------- Index update finished for db: shards/d5555552-eaaaaaa6/medic.1682956796 idx: _design/medic-admin
[info] 2025-05-18T19:34:39.983674Z couchdb@127.0.0.1 <0.24082.2118> -------- Index update finished for db: shards/00000000-15555554/medic.1682956796 idx: _design/medic-conflicts
[notice] 2025-05-18T19:34:39.987020Z couchdb@127.0.0.1 <0.19901.1929> 8b001574ae haproxy.echis-nairobi.svc.cluster.local:5984 10.42.148.19 medic GET /medic/_design/medic-conflicts/_view/conflicts?reduce=true 200 ok 92
[info] 2025-05-18T19:34:39.987303Z couchdb@127.0.0.1 <0.2315.1889> -------- Index update finished for db: shards/3fffffff-55555553/medic.1682956796 idx: _design/medic-sms
[info] 2025-05-18T19:34:39.987311Z couchdb@127.0.0.1 <0.23587.2058> -------- Index update finished for db: shards/15555555-2aaaaaa9/medic.1682956796 idx: _design/medic-sms
[info] 2025-05-18T19:34:39.990310Z couchdb@127.0.0.1 <0.19502.1970> -------- Index update finished for db: shards/00000000-15555554/medic.1682956796 idx: _design/medic-sms
[info] 2025-05-18T19:34:39.992727Z couchdb@127.0.0.1 <0.19714.1937> -------- Index update finished for db: shards/d5555552-eaaaaaa6/medic.1682956796 idx: _design/medic-sms
[info] 2025-05-18T19:34:39.998245Z couchdb@127.0.0.1 <0.23889.2149> -------- Index update finished for db: shards/7ffffffe-95555552/medic.1682956796 idx: _design/medic-admin
[info] 2025-05-18T19:34:40.001244Z couchdb@127.0.0.1 <0.13597.2043> -------- Index update finished for db: shards/7ffffffe-95555552/medic.1682956796 idx: _design/medic-sms
[notice] 2025-05-18T19:34:40.002261Z couchdb@127.0.0.1 <0.31606.1889> 5e345f4cd4 haproxy.echis-nairobi.svc.cluster.local:5984 10.42.148.19 medic GET /medic/_design/medic-sms/_view/messages_by_last_updated_state?limit=100&descending=true&start_key=%5B%22final%22%2C1747596879878%5D&endkey=%5B%22final%22%2C0%5D 200 ok 97
[notice] 2025-05-18T19:34:40.003853Z couchdb@127.0.0.1 <0.2546.2001> fdc1d9c604 haproxy.echis-nairobi.svc.cluster.local:5984 10.42.148.19 medic GET /medic/_design/medic-sms/_view/messages_by_last_updated_state?limit=100&descending=true&start_key=%5B%22pending%22%2C1747596879878%5D&endkey=%5B%22pending%22%2C0%5D 200 ok 99
[notice] 2025-05-18T19:34:40.003905Z couchdb@127.0.0.1 <0.29642.2111> 6ede751ced haproxy.echis-nairobi.svc.cluster.local:5984 10.42.148.19 medic GET /medic/_design/medic-sms/_view/messages_by_last_updated_state?limit=100&descending=true&start_key=%5B%22muted%22%2C1747596879878%5D&endkey=%5B%22muted%22%2C0%5D 200 ok 94
[info] 2025-05-18T19:34:40.009924Z couchdb@127.0.0.1 <0.8573.1942> -------- Index update finished for db: shards/3fffffff-55555553/medic.1682956796 idx: _design/medic-admin
[notice] 2025-05-18T19:34:40.010623Z couchdb@127.0.0.1 <0.27045.1989> a677409a72 haproxy.echis-nairobi.svc.cluster.local:5984 10.42.148.19 medic GET /medic/_design/medic-admin/_view/message_queue?reduce=true&start_key=%5B%22scheduled%22%2C1746921600000%5D&end_key=%5B%22scheduled%22%2C1747596879878%5D 200 ok 114
Sentinel
/0.821 DEBUG: change event on doc a86fc252-cf20-588e-947d-bda26eb5d051 seq 59906-g1AAAAO1eJyV00FKAzEUBuBoxap0oYgLF3oAQZ3MZCYzq7p2p212usjLzDAMVUFw7SnEG2ibZS_gHYReomewjc-8LosQsvgh4X2E95IRY6zXdEp2YJ5eTFPCFY_lRYSLj_BoUzM4UmrYNh1grL_7gHtdLWQKJltX84_05yg4s3ZC2OWpx3JR8SwWoRg6FvrOLQk7fvZYKrXOIRhDxwHebkDY4bnHslpnQqShGDpDaK0dE7b_Tj2DIktMHYqhM4FX5xaE9a49VvMERFyFYugsH7fYrVJvxO3dey6qosKYMpBDaYASxifOYgWeeFDwrIpF6BhQGqOE8ePcF4E73x40uYEyeK4oLVDCuFFqRmB36kEOhkujw0Fs3gzjw9r5CrzzYKELUcnQP0HNm2PgC3QEbm9QD0GmRSLXlba_RSj77A
2025-05-18T19:38:50.848 DEBUG: isRevSame tested false on transition registration for doc a86fc252-cf20-588e-947d-bda26eb5d051 seq 59906-g1AAAAO1eJyV00FKAzEUBuBoxap0oYgLF3oAQZ3MZCYzq7p2p212usjLzDAMVUFw7SnEG2ibZS_gHYReomewjc-8LosQsvgh4X2E95IRY6zXdEp2YJ5eTFPCFY_lRYSLj_BoUzM4UmrYNh1grL_7gHtdLWQKJltX84_05yg4s3ZC2OWpx3JR8SwWoRg6FvrOLQk7fvZYKrXOIRhDxwHebkDY4bnHslpnQqShGDpDaK0dE7b_Tj2DIktMHYqhM4FX5xaE9a49VvMERFyFYugsH7fYrVJvxO3dey6qosKYMpBDaYASxifOYgWeeFDwrIpF6BhQGqOE8ePcF4E73x40uYEyeK4oLVDCuFFqRmB36kEOhkujw0Fs3gzjw9r5CrzzYKELUcnQP0HNm2PgC3QEbm9QD0GmRSLXlba_RSj77A
2025-05-18T19:38:50.848 DEBUG: canRun test failed on transition registration for doc a86fc252-cf20-588e-947d-bda26eb5d051 seq 59906-g1AAAAO1eJyV00FKAzEUBuBoxap0oYgLF3oAQZ3MZCYzq7p2p212usjLzDAMVUFw7SnEG2ibZS_gHYReomewjc-8LosQsvgh4X2E95IRY6zXdEp2YJ5eTFPCFY_lRYSLj_BoUzM4UmrYNh1grL_7gHtdLWQKJltX84_05yg4s3ZC2OWpx3JR8SwWoRg6FvrOLQk7fvZYKrXOIRhDxwHebkDY4bnHslpnQqShGDpDaK0dE7b_Tj2DIktMHYqhM4FX5xaE9a49VvMERFyFYugsH7fYrVJvxO3dey6qosKYMpBDaYASxifOYgWeeFDwrIpF6BhQGqOE8ePcF4E73x40uYEyeK4oLVDCuFFqRmB36kEOhkujw0Fs3gzjw9r5CrzzYKELUcnQP0HNm2PgC3QEbm9QD0GmRSLXlba_RSj77A
2025-05-18T19:38:50.848 DEBUG: isRevSame tested false on transition accept_patient_reports for doc a86fc252-cf20-588e-947d-bda26eb5d051 seq 59906-g1AAAAO1eJyV00FKAzEUBuBoxap0oYgLF3oAQZ3MZCYzq7p2p212usjLzDAMVUFw7SnEG2ibZS_gHYReomewjc-8LosQsvgh4X2E95IRY6zXdEp2YJ5eTFPCFY_lRYSLj_BoUzM4UmrYNh1grL_7gHtdLWQKJltX84_05yg4s3ZC2OWpx3JR8SwWoRg6FvrOLQk7fvZYKrXOIRhDxwHebkDY4bnHslpnQqShGDpDaK0dE7b_Tj2DIktMHYqhM4FX5xaE9a49VvMERFyFYugsH7fYrVJvxO3dey6qosKYMpBDaYASxifOYgWeeFDwrIpF6BhQGqOE8ePcF4E73x40uYEyeK4oLVDCuFFqRmB36kEOhkujw0Fs3gzjw9r5CrzzYKELUcnQP0HNm2PgC3QEbm9QD0GmRSLXlba_RSj77A
2025-05-18T19:38:50.848 DEBUG: canRun test failed on transition accept_patient_reports for doc a86fc252-cf20-588e-947d-bda26eb5d051 seq 59906-g1AAAAO1eJyV00FKAzEUBuBoxap0oYgLF3oAQZ3MZCYzq7p2p212usjLzDAMVUFw7SnEG2ibZS_gHYReomewjc-8LosQsvgh4X2E95IRY6zXdEp2YJ5eTFPCFY_lRYSLj_BoUzM4UmrYNh1grL_7gHtdLWQKJltX84_05yg4s3ZC2OWpx3JR8SwWoRg6FvrOLQk7fvZYKrXOIRhDxwHebkDY4bnHslpnQqShGDpDaK0dE7b_Tj2DIktMHYqhM4FX5xaE9a49VvMERFyFYugsH7fYrVJvxO3dey6qosKYMpBDaYASxifOYgWeeFDwrIpF6BhQGqOE8ePcF4E73x40uYEyeK4oLVDCuFFqRmB36kEOhkujw0Fs3gzjw9r5CrzzYKELUcnQP0HNm2PgC3QEbm9QD0GmRSLXlba_RSj77A
2025-05-18T19:38:50.848 DEBUG: isRevSame tested false on transition accept_case_reports for doc a86fc252-cf20-588e-947d-bda26eb5d051 seq 59906-g1AAAAO1eJyV00FKAzEUBuBoxap0oYgLF3oAQZ3MZCYzq7p2p212usjLzDAMVUFw7SnEG2ibZS_gHYReomewjc-8LosQsvgh4X2E95IRY6zXdEp2YJ5eTFPCFY_lRYSLj_BoUzM4UmrYNh1grL_7gHtdLWQKJltX84_05yg4s3ZC2OWpx3JR8SwWoRg6FvrOLQk7fvZYKrXOIRhDxwHebkDY4bnHslpnQqShGDpDaK0dE7b_Tj2DIktMHYqhM4FX5xaE9a49VvMERFyFYugsH7fYrVJvxO3dey6qosKYMpBDaYASxifOYgWeeFDwrIpF6BhQGqOE8ePcF4E73x40uYEyeK4oLVDCuFFqRmB36kEOhkujw0Fs3gzjw9r5CrzzYKELUcnQP0HNm2PgC3QEbm9QD0GmRSLXlba_RSj77A
2025-05-18T19:38:50.848 DEBUG: canRun test failed on transition accept_case_reports for doc a86fc252-cf20-588e-947d-bda26eb5d051 seq 59906-g1AAAAO1eJyV00FKAzEUBuBoxap0oYgLF3oAQZ3MZCYzq7p2p212usjLzDAMVUFw7SnEG2ibZS_gHYReomewjc-8LosQsvgh4X2E95IRY6zXdEp2YJ5eTFPCFY_lRYSLj_BoUzM4UmrYNh1grL_7gHtdLWQKJltX84_05yg4s3ZC2OWpx3JR8SwWoRg6FvrOLQk7fvZYKrXOIRhDxwHebkDY4bnHslpnQqShGDpDaK0dE7b_Tj2DIktMHYqhM4FX5xaE9a49VvMERFyFYugsH7fYrVJvxO3dey6qosKYMpBDaYASxifOYgWeeFDwrIpF6BhQGqOE8ePcF4E73x40uYEyeK4oLVDCuFFqRmB36kEOhkujw0Fs3gzjw9r5CrzzYKELUcnQP0HNm2PgC3QEbm9QD0GmRSLXlba_RSj77A
2025-05-18T19:38:50.848 DEBUG: isRevSame tested false on transition death_reporting for doc a86fc252-cf20-588e-947d-bda26eb5d051 seq 59906-g1AAAAO1eJyV00FKAzEUBuBoxap0oYgLF3oAQZ3MZCYzq7p2p212usjLzDAMVUFw7SnEG2ibZS_gHYReomewjc-8LosQsvgh4X2E95IRY6zXdEp2YJ5eTFPCFY_lRYSLj_BoUzM4UmrYNh1grL_7gHtdLWQKJltX84_05yg4s3ZC2OWpx3JR8SwWoRg6FvrOLQk7fvZYKrXOIRhDxwHebkDY4bnHslpnQqShGDpDaK0dE7b_Tj2DIktMHYqhM4FX5xaE9a49VvMERFyFYugsH7fYrVJvxO3dey6qosKYMpBDaYASxifOYgWeeFDwrIpF6BhQGqOE8ePcF4E73x40uYEyeK4oLVDCuFFqRmB36kEOhkujw0Fs3gzjw9r5CrzzYKELUcnQP0HNm2PgC3QEbm9QD0GmRSLXlba_RSj77A
2025-05-18T19:38:50.848 DEBUG: canRun test failed on transition death_reporting for doc a86fc252-cf20-588e-947d-bda26eb5d051 seq 59906-g1AAAAO1eJyV00FKAzEUBuBoxap0oYgLF3oAQZ3MZCYzq7p2p212usjLzDAMVUFw7SnEG2ibZS_gHYReomewjc-8LosQsvgh4X2E95IRY6zXdEp2YJ5eTFPCFY_lRYSLj_BoUzM4UmrYNh1grL_7gHtdLWQKJltX84_05yg4s3ZC2OWpx3JR8SwWoRg6FvrOLQk7fvZYKrXOIRhDxwHebkDY4bnHslpnQqShGDpDaK0dE7b_Tj2DIktMHYqhM4FX5xaE9a49VvMERFyFYugsH7fYrVJvxO3dey6qosKYMpBDaYASxifOYgWeeFDwrIpF6BhQGqOE8ePcF4E73x40uYEyeK4oLVDCuFFqRmB36kEOhkujw0Fs3gzjw9r5CrzzYKELUcnQP0HNm2PgC3QEbm9QD0GmRSLXlba_RSj77A
2025-05-18T19:38:50.848 DEBUG: isRevSame tested false on transition muting for doc a86fc252-cf20-588e-947d-bda26eb5d051 seq 59906-g1AAAAO1eJyV00FKAzEUBuBoxap0oYgLF3oAQZ3MZCYzq7p2p212usjLzDAMVUFw7SnEG2ibZS_gHYReomewjc-8LosQsvgh4X2E95IRY6zXdEp2YJ5eTFPCFY_lRYSLj_BoUzM4UmrYNh1grL_7gHtdLWQKJltX84_05yg4s3ZC2OWpx3JR8SwWoRg6FvrOLQk7fvZYKrXOIRhDxwHebkDY4bnHslpnQqShGDpDaK0dE7b_Tj2DIktMHYqhM4FX5xaE9a49VvMERFyFYugsH7fYrVJvxO3dey6qosKYMpBDaYASxifOYgWeeFDwrIpF6BhQGqOE8ePcF4E73x40uYEyeK4oLVDCuFFqRmB36kEOhkujw0Fs3gzjw9r5CrzzYKELUcnQP0HNm2PgC3QEbm9QD0GmRSLXlba_RSj77A
2025-05-18T19:38:50.848 DEBUG: canRun test failed on transition muting for doc a86fc252-cf20-588e-947d-bda26eb5d051 seq 59906-g1AAAAO1eJyV00FKAzEUBuBoxap0oYgLF3oAQZ3MZCYzq7p2p212usjLzDAMVUFw7SnEG2ibZS_gHYReomewjc-8LosQsvgh4X2E95IRY6zXdEp2YJ5eTFPCFY_lRYSLj_BoUzM4UmrYNh1grL_7gHtdLWQKJltX84_05yg4s3ZC2OWpx3JR8SwWoRg6FvrOLQk7fvZYKrXOIRhDxwHebkDY4bnHslpnQqShGDpDaK0dE7b_Tj2DIktMHYqhM4FX5xaE9a49VvMERFyFYugsH7fYrVJvxO3dey6qosKYMpBDaYASxifOYgWeeFDwrIpF6BhQGqOE8ePcF4E73x40uYEyeK4oLVDCuFFqRmB36kEOhkujw0Fs3gzjw9r5CrzzYKELUcnQP0HNm2PgC3QEbm9QD0GmRSLXlba_RSj77A
2025-05-18T19:38:50.848 DEBUG: isRevSame tested false on transition mark_for_outbound for doc a86fc252-cf20-588e-947d-bda26eb5d051 seq 59906-g1AAAAO1eJyV00FKAzEUBuBoxap0oYgLF3oAQZ3MZCYzq7p2p212usjLzDAMVUFw7SnEG2ibZS_gHYReomewjc-8LosQsvgh4X2E95IRY6zXdEp2YJ5eTFPCFY_lRYSLj_BoUzM4UmrYNh1grL_7gHtdLWQKJltX84_05yg4s3ZC2OWpx3JR8SwWoRg6FvrOLQk7fvZYKrXOIRhDxwHebkDY4bnHslpnQqShGDpDaK0dE7b_Tj2DIktMHYqhM4FX5xaE9a49VvMERFyFYugsH7fYrVJvxO3dey6qosKYMpBDaYASxifOYgWeeFDwrIpF6BhQGqOE8ePcF4E73x40uYEyeK4oLVDCuFFqRmB36kEOhkujw0Fs3gzjw9r5CrzzYKELUcnQP0HNm2PgC3QEbm9QD0GmRSLXlba_RSj77A
2025-05-18T19:38:50.848 DEBUG: calling transition.onMatch for doc a86fc252-cf20-588e-947d-bda26eb5d051 and transition mark_for_outbound seq 59906-g1AAAAO1eJyV00FKAzEUBuBoxap0oYgLF3oAQZ3MZCYzq7p2p212usjLzDAMVUFw7SnEG2ibZS_gHYReomewjc-8LosQsvgh4X2E95IRY6zXdEp2YJ5eTFPCFY_lRYSLj_BoUzM4UmrYNh1grL_7gHtdLWQKJltX84_05yg4s3ZC2OWpx3JR8SwWoRg6FvrOLQk7fvZYKrXOIRhDxwHebkDY4bnHslpnQqShGDpDaK0dE7b_Tj2DIktMHYqhM4FX5xaE9a49VvMERFyFYugsH7fYrVJvxO3dey6qosKYMpBDaYASxifOYgWeeFDwrIpF6BhQGqOE8ePcF4E73x40uYEyeK4oLVDCuFFqRmB36kEOhkujw0Fs3gzjw9r5CrzzYKELUcnQP0HNm2PgC3QEbm9QD0GmRSLXlba_RSj77A
2025-05-18T19:38:50.850 DEBUG: finished transition mark_for_outbound doc a86fc252-cf20-588e-947d-bda26eb5d051 is unchanged seq 59906-g1AAAAO1eJyV00FKAzEUBuBoxap0oYgLF3oAQZ3MZCYzq7p2p212usjLzDAMVUFw7SnEG2ibZS_gHYReomewjc-8LosQsvgh4X2E95IRY6zXdEp2YJ5eTFPCFY_lRYSLj_BoUzM4UmrYNh1grL_7gHtdLWQKJltX84_05yg4s3ZC2OWpx3JR8SwWoRg6FvrOLQk7fvZYKrXOIRhDxwHebkDY4bnHslpnQqShGDpDaK0dE7b_Tj2DIktMHYqhM4FX5xaE9a49VvMERFyFYugsH7fYrVJvxO3dey6qosKYMpBDaYASxifOYgWeeFDwrIpF6BhQGqOE8ePcF4E73x40uYEyeK4oLVDCuFFqRmB36kEOhkujw0Fs3gzjw9r5CrzzYKELUcnQP0HNm2PgC3QEbm9QD0GmRSLXlba_RSj77A
2025-05-18T19:38:50.850 DEBUG: transition results: [null,null,null,null,null,false]
2025-05-18T19:38:50.850 DEBUG: nothing changed skipping saveDoc for doc a86fc252-cf20-588e-947d-bda26eb5d051 seq 59906-g1AAAAO1eJyV00FKAzEUBuBoxap0oYgLF3oAQZ3MZCYzq7p2p212usjLzDAMVUFw7SnEG2ibZS_gHYReomewjc-8LosQsvgh4X2E95IRY6zXdEp2YJ5eTFPCFY_lRYSLj_BoUzM4UmrYNh1grL_7gHtdLWQKJltX84_05yg4s3ZC2OWpx3JR8SwWoRg6FvrOLQk7fvZYKrXOIRhDxwHebkDY4bnHslpnQqShGDpDaK0dE7b_Tj2DIktMHYqhM4FX5xaE9a49VvMERFyFYugsH7fYrVJvxO3dey6qosKYMpBDaYASxifOYgWeeFDwrIpF6BhQGqOE8ePcF4E73x40uYEyeK4oLVDCuFFqRmB36kEOhkujw0Fs3gzjw9r5CrzzYKELUcnQP0HNm2PgC3QEbm9QD0GmRSLXlba_RSj77A
2025-05-18T19:38:50.856 DEBUG: change event on doc dd2ad05b-bb1c-4871-b40e-cb83e560ec63 seq 59919-g1AAAAO1eJyV00FKAzEUBuBoxYp0oYiCCz2AoE5mMpOZVV270zY7XeRlZhiGqiC49hTiDbTNshfwDkIv0TPYxmdel0UIWfyQ8D7Ce8mIMdZrOiXbN08vpinhisfyIsLFR3i0qRkcKjVsmw4w1j96wL2uFjIFk62r-Uf6cxScWTsh7PLUY7moeBaLUAwdC33nloQdP3sslVrnEIyh4wBvNyDs4NxjWa0zIdJQDJ0htNaOCdt7p55BkSWmDsXQmcCrcwvCetceq3kCIq5CMXSWj1vsVqk34nbvPRdVUWFMGcihNEAJ4xNnsQJPPCh4VsUidAwojVHC-HHui8Cdbw-a3EAZPFeUFihh3Cg1I7A79SAHw6XR4SA2b4bxYe18Bd55sNCFqGTon6DmzTHwBToCtzeohyDTIpHrSttfcwP7-Q
2025-05-18T19:38:50.889 DEBUG: isRevSame tested false on transition registration for doc dd2ad05b-bb1c-4871-b40e-cb83e560ec63 seq 59919-g1AAAAO1eJyV00FKAzEUBuBoxYp0oYiCCz2AoE5mMpOZVV270zY7XeRlZhiGqiC49hTiDbTNshfwDkIv0TPYxmdel0UIWfyQ8D7Ce8mIMdZrOiXbN08vpinhisfyIsLFR3i0qRkcKjVsmw4w1j96wL2uFjIFk62r-Uf6cxScWTsh7PLUY7moeBaLUAwdC33nloQdP3sslVrnEIyh4wBvNyDs4NxjWa0zIdJQDJ0htNaOCdt7p55BkSWmDsXQmcCrcwvCetceq3kCIq5CMXSWj1vsVqk34nbvPRdVUWFMGcihNEAJ4xNnsQJPPCh4VsUidAwojVHC-HHui8Cdbw-a3EAZPFeUFihh3Cg1I7A79SAHw6XR4SA2b4bxYe18Bd55sNCFqGTon6DmzTHwBToCtzeohyDTIpHrSttfcwP7-Q
2025-05-18T19:38:50.891 DEBUG: nothing changed skipping saveDoc for doc dd2ad05b-bb1c-4871-b40e-cb83e560ec63 seq 59919-g1AAAAO1eJyV00FKAzEUBuBoxYp0oYiCCz2AoE5mMpOZVV270zY7XeRlZhiGqiC49hTiDbTNshfwDkIv0TPYxmdel0UIWfyQ8D7Ce8mIMdZrOiXbN08vpinhisfyIsLFR3i0qRkcKjVsmw4w1j96wL2uFjIFk62r-Uf6cxScWTsh7PLUY7moeBaLUAwdC33nloQdP3sslVrnEIyh4wBvNyDs4NxjWa0zIdJQDJ0htNaOCdt7p55BkSWmDsXQmcCrcwvCetceq3kCIq5CMXSWj1vsVqk34nbvPRdVUWFMGcihNEAJ4xNnsQJPPCh4VsUidAwojVHC-HHui8Cdbw-a3EAZPFeUFihh3Cg1I7A79SAHw6XR4SA2b4bxYe18Bd55sNCFqGTon6DmzTHwBToCtzeohyDTIpHrSttfcwP7-Q
2025-05-18T19:38:50.898 DEBUG: change event on doc e1c7f98c-fc21-5dd9-97d5-cc49f5297057 seq 59920-g1AAAAO1eJyV00FKAzEUBuBoxYp0oYiCCz2AoE5mMklnVdfutM3SRV5mhmGoCoJrTyHeQNssvYB3EHqJnsE2PvO6LELI4ocM-Qj_m4wZY72mU7J9-_hsmxKueKouElx8jJ82DYNDrUdt0wHGBkf3uNc1QuVg5boz_0h_joYz56aEXZ4GrC8qLlMRi6HjYOD9krDjp4Dlypg-RGPoeMDbDQk7OA-YrI0UIo_F0BlB69yEsL036gwKmdk6FkNnCi_eLwjrXQes5hmItIrF0Fk-bLFbrV-J270LXFIlhbVlJIfSECWMD5zFCjwJoOCySkXsGFCaoITx4_0XgTvfAbR9C2X0XFFaoIRxo_WMwO5nADlYrqyJB7G8Gca7c_MVSB0WphCVin0TVN4cA_9AT-D2BnUIKi8yte5o-wtzdvv6
2025-05-18T19:38:50.922 DEBUG: isRevSame tested false on transition registration for doc e1c7f98c-fc21-5dd9-97d5-cc49f5297057 seq 59920-g1AAAAO1eJyV00FKAzEUBuBoxYp0oYiCCz2AoE5mMklnVdfutM3SRV5mhmGoCoJrTyHeQNssvYB3EHqJnsE2PvO6LELI4ocM-Qj_m4wZY72mU7J9-_hsmxKueKouElx8jJ82DYNDrUdt0wHGBkf3uNc1QuVg5boz_0h_joYz56aEXZ4GrC8qLlMRi6HjYOD9krDjp4Dlypg-RGPoeMDbDQk7OA-YrI0UIo_F0BlB69yEsL036gwKmdk6FkNnCi_eLwjrXQes5hmItIrF0Fk-bLFbrV-J270LXFIlhbVlJIfSECWMD5zFCjwJoOCySkXsGFCaoITx4_0XgTvfAbR9C2X0XFFaoIRxo_WMwO5nADlYrqyJB7G8Gca7c_MVSB0WphCVin0TVN4cA_9AT-D2BnUIKi8yte5o-wtzdvv6
2025-05-18T19:38:50.922 DEBUG: isRevSame tested false on transition accept_case_reports for doc e1c7f98c-fc21-5dd9-97d5-cc49f5297057 seq 59920-g1AAAAO1eJyV00FKAzEUBuBoxYp0oYiCCz2AoE5mMklnVdfutM3SRV5mhmGoCoJrTyHeQNssvYB3EHqJnsE2PvO6LELI4ocM-Qj_m4wZY72mU7J9-_hsmxKueKouElx8jJ82DYNDrUdt0wHGBkf3uNc1QuVg5boz_0h_joYz56aEXZ4GrC8qLlMRi6HjYOD9krDjp4Dlypg-RGPoeMDbDQk7OA-YrI0UIo_F0BlB69yEsL036gwKmdk6FkNnCi_eLwjrXQes5hmItIrF0Fk-bLFbrV-J270LXFIlhbVlJIfSECWMD5zFCjwJoOCySkXsGFCaoITx4_0XgTvfAbR9C2X0XFFaoIRxo_WMwO5nADlYrqyJB7G8Gca7c_MVSB0WphCVin0TVN4cA_9AT-D2BnUIKi8yte5o-wtzdvv6
HAproxy
<150>May 18 19:39:50 haproxy[12]: 10.42.148.19,couchdb.echis-nairobi.svc.cluster.local,200,3,2,0,POST,/medic/_bulk_get?revs=true&attachments=true,api,0,'{"docs":[{"id":"service-worker-meta","rev":"1467-ba579ec52cd0d9543555ba3b753b0b18"}]}',35476,3,35251,'node-fetch/1.0 (+https://github.com/bitinn/node-fetch)'
[WARNING] (12) : Server couchdb-servers/couchdb.echis-nairobi.svc.cluster.local is DOWN, reason: Layer7 wrong status, code: 0, info: "via agent : down", check duration: 1030ms. 0 active and 0 backup servers left. 4 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] (12) : backend 'couchdb-servers' has no server available!
<145>May 18 19:39:51 haproxy[12]: Server couchdb-servers/couchdb.echis-nairobi.svc.cluster.local is DOWN, reason: Layer7 wrong status, code: 0, info: "via agent : down", check duration: 1030ms. 0 active and 0 backup servers left. 4 sessions active, 0 requeued, 0 remaining in queue.
<144>May 18 19:39:51 haproxy[12]: backend couchdb-servers has no server available!
<150>May 18 19:39:51 haproxy[12]: 10.42.148.19,<NOSRV>,503,0,1007,0,GET,/_session,-,tjosephinelayau,'-',241,-1,-,'like Gecko) Version/4.0 Chrome/136.0.7103.60 Mobile Safari/537.36 org.medicmobile.webapp.mobile.moh_kenya_echis/v1.4.0'
<150>May 18 19:39:51 haproxy[12]: 10.42.148.19,<NOSRV>,503,0,0,0,GET,/,-,tjosephinelayau,'-',241,-1,-,'like Gecko) Version/4.0 Chrome/136.0.7103.60 Mobile Safari/537.36 org.medicmobile.webapp.mobile.moh_kenya_echis/v1.4.0'
<150>May 18 19:39:51 haproxy[12]: 10.42.148.19,<NOSRV>,503,0,219,0,GET,/_session,-,tjosephinelayau,'-',241,-1,-,'like Gecko) Version/4.0 Chrome/136.0.7103.60 Mobile Safari/537.36 org.medicmobile.webapp.mobile.moh_kenya_echis/v1.4.0'
<150>May 18 19:39:51 haproxy[12]: 10.42.148.19,<NOSRV>,503,0,0,0,GET,/medic/_local/ItQ_z_3AHY3vok2siYT4hg%3D%3D?,-,tjosephinelayau,'-',241,-1,-,'like Gecko) Version/4.0 Chrome/136.0.7103.60 Mobile Safari/537.36 org.medicmobile.webapp.mobile.moh_kenya_echis/v1.4.0'
<150>May 18 19:39:52 haproxy[12]: 10.42.148.19,<NOSRV>,503,0,221,0,GET,/_session,-,tjosephinelayau,'-',241,-1,-,'like Gecko) Version/4.0 Chrome/136.0.7103.60 Mobile Safari/537.36 org.medicmobile.webapp.mobile.moh_kenya_echis/v1.4.0'
<150>May 18 19:39:52 haproxy[12]: 10.42.148.19,<NOSRV>,503,0,724,0,GET,/_session,-,tjosephinelayau,'-',241,-1,-,'like Gecko) Version/4.0 Chrome/136.0.7103.60 Mobile Safari/537.36 org.medicmobile.webapp.mobile.moh_kenya_echis/v1.4.0'
<150>May 18 19:39:52 haproxy[12]: 10.42.148.19,<NOSRV>,503,0,0,0,GET,/,-,tjosephinelayau,'-',241,-1,-,'like Gecko) Version/4.0 Chrome/136.0.7103.60 Mobile Safari/537.36 org.medicmobile.webapp.mobile.moh_kenya_echis/v1.4.0'
<150>May 18 19:39:52 haproxy[12]: 10.42.148.19,<NOSRV>,503,0,177,0,GET,/_session,-,tjosephinelayau,'-',241,-1,-,'like Gecko) Version/4.0 Chrome/136.0.7103.60 Mobile Safari/537.36 org.medicmobile.webapp.mobile.moh_kenya_echis/v1.4.0'
<150>May 18 19:40:01 haproxy[12]: 10.42.148.19,<NOSRV>,503,0,209,0,GET,/_session,-,qireneungub,'-',241,-1,-,'like Gecko) Version/4.0 Chrome/136.0.7103.60 Mobile Safari/537.36 org.medicmobile.webapp.mobile.moh_kenya_echis/v1.4.0'
<150>May 18 19:40:01 haproxy[12]: 10.42.148.19,<NOSRV>,503,0,0,0,GET,/,-,qireneungub,'-',241,-1,-,'like Gecko) Version/4.0 Chrome/136.0.7103.60 Mobile Safari/537.36 org.medicmobile.webapp.mobile.moh_kenya_echis/v1.4.0'
<150>May 18 19:40:02 haproxy[12]: 10.42.148.19,<NOSRV>,503,0,335,0,GET,/_session,-,qireneungub,'-',241,-1,-,'like Gecko) Version/4.0 Chrome/136.0.7103.60 Mobile Safari/537.36 org.medicmobile.webapp.mobile.moh_kenya_echis/v1.4.0'
<150>May 18 19:40:02 haproxy[12]: 10.42.148.19,<NOSRV>,503,0,0,0,GET,/medic/_local/.uimYkVPqljWRKwcsLAkPw%3D%3D?,-,qireneungub,'-',241,-1,-,'like Gecko) Version/4.0 Chrome/136.0.7103.60 Mobile Safari/537.36 org.medicmobile.webapp.mobile.moh_kenya_echis/v1.4.0'
[WARNING] (12) : Server couchdb-servers/couchdb.echis-nairobi.svc.cluster.local is UP, reason: Layer7 check passed, code: 0, info: "via agent : up", check duration: 92ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
For users the 503 manifests as shown in this screenshot after navigating across tabs.
Deployment files used can be found in this gist: echis-release.yaml · GitHub