Audit trail/Log for admin activities verified on a block chain

Product Feature Requests
1

Living goods
NGO
What Other Organizations Would Benefit From This Feature:
National level implementations with Ministries of Health and public sector partners.

**Describe the Feature: **
As a system administrator I want to be able to view, publish and extract logs that can be validated for integrity on a public block chain so that there is a trail that can be used to track activities by actors on the system at the OS level medic OS and admin level.

**What “Pain Point” Does the Proposed Feature Address: ** [Is there an unmet need impacting patients, providers, managers, developers, etc?]
No Independently verifiable and accessible logs for activities on the national scale deployments

Proposed Solution: [How you would like to solve the “pain point”? This is a starting point for conversation, and can be high-leveled / general.]

Have a URL accessible by authentication with the audit data from various levels with reference to the commits on the chain.

**Do You Have Funding Available: **
No

Do You Have Resources (Designers, Developers, PMs) Available:
PM
Links To Supporting Information / Uploads: [feature specs or designs, examples of similar features etc]

2

Hi @Patrick_K !

While not documented, the best place to look for audit logs is in the HAProxy container. This is explicitly situated between CouchDB and the CHT API such that every single call is always logged. Further, any PUT or POST JSON payloads, which would normally not be logged, are expanded out and written in raw format in the log. The log file is better than querying CouchDB directly (or the downstream Postrges DB populated via couch2pg) as those records are mutable.

For example, you can see my user log in and then some 20 minutes later POST some documents:

Apr  6 22:02:05 3c9e67d5b8be haproxy[26]: 172.18.0.3,200,POST,/_session,-,mrjones,'{"name":"mrjones","password":"***"}',405,2,45,'-'
Apr  6 22:24:04 3c9e67d5b8be haproxy[26]: 172.18.0.3,201,POST,/medic/_bulk_docs,-,mrjones,'{"docs":[{"form":"pregnancy_facility_visit_reminder","type":"data_record","content_type":"xml", [DATA-TRUNCATED]

To find these logs, you would first find the container name (docker ps) and then run logs on it. For me, my container is called haproxy_1 so I would call:

docker logs haproxy_1

However, this is a very chatty log and it is hard to parse. You could search within them by running grep. Further, you could look within the container to ensure you’re looking across all log data. Here is a search to find all logins:

docker exec -it haproxy_1 grep POST /srv/storage/audit/haproxy.log|grep "200,POST,/_session,-"

To make this a bit more friendly for non-system administrators, you could import the log data into Kibana and then use something like Logtrail to offer a web based solution so users would have to deal with ssh and docker commands.

Always use care with the data in these logs: they contain very sensitive personally identifiable information (PII) as well as protected health information (PHI).

I’ll look into clarifying the exact HAProxy log format as well as some more explicit Kibana bases solutions and report back here. In the meanwhile, feel free to ask any questions if you have them!

2 Likes
3

@Patrick_K ,

Before we getting to more Kibana info, I wanted to mention the technical way to correlate public IPs of user logins to the data being sent to the CHT. With our Docker set up, we deploy an instance of ngnix in the medic-os container which acts as a reverse proxy to terminate TLS connections (one of two reverse proxies when you count HAProxy). This means nginix is the only server to log the public IP of the user, while HAProxy logs all the raw data of activities themselves.

Here’s how you would find all the successful logins from ngnix’s perspective, again assuming your container name is medic-os_1. We’ll grep against the /srv/storage/medic-core/nginx/logs/access.log in the medic-os container and check for a redirect 302 response:

docker exec -it medic-os_1 grep 'POST /medic/login' /srv/storage/medic-core/nginx/logs/access.log|grep 302

Going back to HAProxy’s log, if you found this event in the haproxy container that you wanted to find the public IP for:

Apr 22 18:03:07 98dc12562bd1 haproxy[25]: 172.18.0.3,401,POST,/_session,-,admin,'{"name":"admin","password":"***"}',390,3,67,'-'

You would take the exact time Apr 22 18:03:07 and convert it to ngninx’s format of 22/Apr/2022:18:03:07 and search in the medic-os conatiner like so:

docker exec -it medic-os_1 grep 'POST /medic/login' /srv/storage/medic-core/nginx/logs/access.log|grep '22/Apr/2022:18:03:07'

192.168.108.8 - - [22/Apr/2022:18:03:07 +0000] "POST /medic/login HTTP/1.1" 401 25 "https://192-168-108-17.my.local-ip.co:8443/medic/login" "Mozilla/5.0 (Linux; Android 9; Pixel 2 Build/SP2A.012; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/493.0.78 Mobile Safari/36 org.medicmobile.webapp.mobile/v0.11.0-webview"

In this case we see only an unroutable RFC1918 IP of 192.168.108.8, as this isn’t on the public internet, but on my test LAN.

Getting back to deploying Kibana (which in turn uses Elesticsearch as a storage/search engine). you have to set it up and ensure it is indexing all your logs, including HAProxy and ngnix. Then, continuing our example of auditing for successful logins, you would search across BOTH HAProxy and ngnix with a search of ("POST /_session 200") OR ("POST /medic/login" and 302). You get a nice histogram of events correlating to diurnal patterns which is great for seeing anomalous event spikes. The square on the bottom highlights the same login event across HAProxy and ngnix (username and IPs redacted in red):

image
image1463×834 104 KB

Note the arrow at the top of this screenshot showing Kibana’s “Share” feature. This allows a system administrator to build up a series of complex searches and then simply share deep links. Each link could show what ever you want like, from “successful logins for past 10 days” to “Forms submitted outside of business hours”.

The huge win of course is that within Kibana, it’s easy to drill down to verify what and when a given user did after logging in.

Two final notes:

  • I mentioned Logtrail in my prior post. This is best watching live events in real time (like tail -f ) on all your logs via a web GUI. Though it does have search feature. Kibana’s “Discover” feature, as highlighted above, is better for ad hoc log auditing.
  • This information is applicable to CHT 3.x and will likely change with CHT 4.x

Best of luck and let me know if you have any questions!

2 Likes
4

@Patrick_K - I hadn’t heard back if the suggestions I made met your needs. Please keep ups posted in case there’s anything else we can do to help out!

5

Thanks jones,
the information you have provided is excellent for the logging scenario. However, for the Audit use case the information is too extensive in Kibana / elastic search unless filtered to the relevant details.
The last question is on how verifiable / tamper proof is the recommendation.

6

@Patrick_K - Thanks for getting back to me!

Good to hear that information is helpful - and very much agreed that the audit log is quite verbose and for sure filtering is key to a successful audit.

The last question is on how verifiable / tamper proof is the recommendation.

Great question! Let’s assume we have four actors in an audit scenario:

  1. Investigator - external to the operation of the CHT and needs to verify what happened in specific security incident with a mobile device.
  2. System Administrator - the in house IT expert who is responsible for back end administration of the CHT, including setting up docker and the CHT containers.
  3. Admin - Provisions user accounts, hands out login/passwords, sets up devices to hand off from one CHW to another.
  4. CHW - Community health worker who gave their phone to the Admin, but kept a copy of the login and password. At a later date, after they gave their device up, they tried to login and steal information.

In this case the only actor that could tamper with the log is the System Administrator. Specifically, neither the Admin nor the CHW (or any CHT supervisors) could change the logs of what they did on the CHT. Even if any new documents are deleted so that you can’t find them, the audit log will have a record of this.

Please let me know if there’s a different audit scenario you’re concerned about that this doesn’t address!