Understanding the history a document through HaProxy audit log

Technical Support
1

I’m trying to understand the history of a document using the HaProxy audit log. I’m looking to understand all the times the document was written.

I started by just looking at the docker logs via docker logs cht-haproxy-1 2>&1 | grep uuid.
But there is no history older than 8 hours.

What is the best way to look at HaProxy logs for a container? Is there a preferred method?

2

Hey Kenn!

Thanks so much for posting your questions to the forum - it’s great to help you and, in turn, all others who might have the same question at a later date.

All CHT Core containers keep 50MB of logs per file for 20 files worth. Over time, the oldest is deleted to ensure there’s only 20 files.

To find the logs on disk on your docker server where CHT Core is running:

  1. become root sudo su -
  2. Make a temp directory mkdir ~/ha-proxy-logs
  3. Find the ID of your container with docker ps --format "table {{.ID}}\t{{.Names}}" --filter "name=ha" --no-trunc. This will show an output of something like:
    CONTAINER ID                                                       NAMES
a5ea8b692838c84273b6e2891d98f028a1cfe103e95a87d3ebe8e98a3eb444be   4_9_delete-haproxy-1
  4. using the ID of your HA proxy container, copy the contents of that containers log directory. For me this is cp /var/lib/docker/containers/a5ea8b692838c84273b6e2891d98f028a1cfe103e95a87d3ebe8e98a3eb444be/local-logs/* ~/ha-proxy-logs/.
  5. Un-gzip all files: cd ~/ha-proxy-logs ; gunzip *gz

You can now grep or otherwise search across all logs available.

If you would like to increase the number of files or size of files, you can change either LOG_MAX_SIZE or LOG_MAX_FILES in your .env file and restart your containers.

3

The files in that folder seem to contain the same as docker logs

# docker logs 1d4b3393f975 | head -2
Dec 17 14:00:26 haproxy[12]....

/var/lib/docker/containers/1d4b3393f97583ae58f603e980ddd77e2c8d8f8914a2f3c69a6cd473b1be34e5/local-logs# zcat container.log.19.gz | head -2
Dec 17 14:00:26 haproxy[12]...

Seems that’s just all there is and docker logs is the best interface to it?

4

@kenn you have 20 log archived files which cover less than one day of logs?

as @mrjones said, the number and size of files is controlled through environment variables and docker container settings. If your log rotation settings prove to have insufficient coverage (honestly, one day of logs is VERY little, especially if it doesn’t get consumed and collected by some archiving service), then these need to be changed as soon as possible.

5

We also have this problem with haproxy logging: Haproxy logs - serving as audit trail - truncate logged body when encoding is chunked · Issue #8182 · medic/cht-core · GitHub

Even if by some chance you would have access to the log entry corresponding to your desired doc change, you might still not have access to the doc body (you would not even know it’s the correct entry).

6

Ah - yes - agreed! Sorry to have led you on the wild goose chase there. Indeed the manual even states:

The local logging driver uses file-based storage. These files are designed to be exclusively accessed by the Docker daemon. Interacting with these files with external tools may interfere with Docker’s logging system and result in unexpected behavior, and should be avoided.

While it won’t help you solve the issue for your current problem, dramatically increasing the LOG_MAX_SIZE or LOG_MAX_FILES will help out next time.