Hundreds of CHT users seeing all white screen after datacenter move

Technical Support
1

image
image576×1280 65 KB

I’m a little confused about this - I’m seeing 100s of CHT users with this white screen.

  1. Things are working fine, users are engaged with CHT
  2. The CHT app is moved from one data-center to a new data-center (specific details unknown)
  3. Users get this white screen

The instance appears to work fine when I test it. We have tried restarting the phone and reinstalling the app, but the issue persists. I have requested somebody onsite to debug. In the meantime, any pointers or thoughts on what might cause this and how best to resolve it?

2

We have seen this behavior when there was an SSL cert error.
Can you try getting ADB logs?

3

That’s sounds super frustrating - sorry to hear about the issue @kenn !

Is this the nairobi-echis.health.go.ke instance? Given there was a server move (@elijah said in the past week, so between Jan 21-Jan 28), I thought maybe this was the prior invalid TLS cert chain issue you discovered. However, when I check the domain on SSL Labs I don’t see any chain issues flagged.

It is CHT Android 1.4, correct? Do you know what version of Android this is on?

1 Like
4

UPDATE! Please see the next comment - the text below is now out of date

@elijah shared a log from one of the affected devices - I believe this device is Android 9 device on SDK 28 from the provided log:

{
  "metadata": {
    "device": {
      "name": "OPPO CPH1923",
      "isOnline": true,
      "release": "9",
      "sdk": 28,
      "featureLevel": 28,
      "model": "CPH1923",
      "type": "HANDHELD",
      "isEmulator": false
    }
  }
}

I believe Android 9 with SDK 28 only supports TLS 1.2 or lower per Android docs.

When you compare eCHIS TLS support vs a Medic hosted project, you see they only support 1.3, but we support 1.2 and 1.3. (cc @Hareet who might have more details):

$ curl https://waling.app.medicmobile.org -v  2>&1 | cut -d" " -f 2|egrep -i "tls" |uniq
TLSv1.3
TLSv1.2

$ curl https://nairobi-echis.health.go.ke -v  2>&1 | cut -d" " -f 2|egrep -i "tls" |uniq
TLSv1.3

The curl header suggests eCHIS is running nginx (server: nginx), so you’d need to enable TLS 1.2 in the config

5

After spinning up Android 9, 10 and 11 instances with SDK versions 28, 29 and 30 in Android Studio , and checking the result of Chrome in those same versions of Android, we’re fairly confident this is an instance of a TLS misconfiguration per the ticket cited above.

To confirm this we see Failed to validate the certificate chain in the log files of the APK:

X509Util   org...webapp.mobile.moh_kenya_echis  I  Failed to validate the certificate chain, error: java.security.cert.CertPathValidatorException: Trust anchor for certification path not found.
9082-9151  chromium  org...webapp.mobile.moh_kenya_echis  E  [ERROR:ssl_client_socket_impl.cc(946)] handshake failed; returned -1, SSL error code 1, net_error -202

Further confirmation is from running Chrome on the same Android device which shows:

NET::ERR_CERT_AUTHORITY_INVALID
This server could not prove that it is nairobi-echis.health.go.ke; its security certificate is not trusted by your device's operating system. This may be caused by a misconfiguration or an attacker intercepting your connection.

Finally, using sslchecker.com and having it scan nairobi-echis.health.go.ke, shows a number of chain certificates missing.

As this is an emSign cert, to fix this:

  1. Download the certificate
  2. Upload the cert to the nginx server
  3. configure nginx to use the cert

Best of luck and post back any further questions!

4 Likes
6

Hi again! I was chatting with @elijah today - the original white screen issue has still not been resolved. We found two solutions to two different issues.

Pure white screen, nothing else

This is the original issue in from above. Working off the above theory that this was an nginx/TLS chain error, we were able to fix this by creating a Let’s Encrypt cert. We then editing the nginx configuration to change the ssl _certificate and ssl _certificate_key values in nginx config to point to the Let’s Encrypt cert and key.

After checking for a valid nginx config and then reloading nginx the pure white screen was fixed.

Two notes about this fix:

  • If you migrate to Let’s Encrypt it is critical you set up automated certificate renewal. See their docs on how to do this.
  • An equally valid fix would be to re-download the certificate files from emSign, per the 3 steps above.

White screen with spinner

After fixing the certificate error, we found another error caused by an out of date WebView APK on Android 10. This is a known issue, and the fix is to upgrade to WebView 90 or greater, per the CHT Core requirements. It looks like a white page with a spinner, either in the upper left corner or in the middle, but no amount of waiting will make it go away:

image
image1077×1911 43.3 KB

To check what version of WebView you’re running, go to Settings → search → search “WebView” → click “Android System WebView” → scroll down and click “Advanced” → see version very bottom of page:

image
image911×1618 110 KB

To update WebView, the best way is to use the Play Store. You can be logged or not logged, but checking for an update should download a new version of the APK. Here’s a video showing how to download via the Play Store while not logged in and then check that the version is updated:

Alternately, you can try going directly to the APK URL which should open in the Play Store and then you can update it - you need to be logged in for this workflow and I was unable to test it.

Finally, as a last resort, you can side load the APK - this is not the recommend this approach as side loading is much less secure than using the Play Store. You can read more about how to do this on our docs.

3 Likes
7

For scenarios like this, CHT should fail predictably and provide usable feedback for the user to fix their setup.

1 Like