Hello community
I did set up the cht-watchdog instance locally and we do also have already a separate grafana instance ie not this one on a remote machine .Wanted to inquire on how to best to deploy :-
How can we best migrate what is in the local cht-watchdog instance into the remote grafana instance? , is it worth the effort?
Or we focus on turning the local watch-dog instance into a remote one by following these steps
Hi @cliff 
If you want to migrate just the dashboards and the alerts to your other grafana instance, you can grab watchdog’s grafana configuration from the repository. You might have to tweak grafana’s configuratios to make sure your grafana instance can still reach prometheus properly.
And if you want to migrate your data points you will need to migrate the data stored by prometheus. Since you set up watchdog with docker & docker compose, you can get prometheus’ data from the related docker volume.
According to this github comment you should be able to archive your prometheus data volume with something like this (double check the volume name is correct before running it):
docker run --rm -v cht-monitoring_prometheus-data:/volume busybox sh -c 'tar -cOzf - /volume' > prom-data.tgz
This command will archive your prometheus data volume to prom-data.tgz. You can send this file to the server that will host the new prometheus instance and extract it there with:
tar -xzf prom-data.tgz
And then point to this directory when setting up prometheus. If you’re going to reuse cht-watchdog, then you will do this in docker-compose.yml by configuring the path to the prometheus-data volume.
+1 to everything Mokhtar said! We have other partners that have integrated our Grafana configuration with their existing Grafana instance. In that case, you just need a stable Prometheus instance deployed somewhere with the Watchdog Prometheus config to scrape your CHT instance and store the metrics.
One additional note is that while it is technically possible to migrate Prometheus data from one instance to another (as Mokhtar described), the general sense that I have gotten from the broader Prometheus community is that often it is not worth the effort. Instead, many folks seem to prefer just deploying a fresh Prometheus instance. This is because, depending on your use case, the metric data in Prometheus tends to rapidly loose value as it gets older. So, if you tend to only look at the last week of data in Watchdog, it might not be worth the developer hours needed to migrate the data when you can just wait a week for your new instance to collect the metrics you want… (That being said, if you do find it useful to review Watchdog data across several months of history, then perhaps the time spent doing the migration would be worthwhile.)
+2 to what Josh and Mokhtar have said!
When approaching an already deployed Grafana/Prometheus Docker Compose based instance where you want to overlay Watchdog on top of it, you might consider adding an additional compose file. With this approach you add on extra mounts into your existing prometheus config which add more scrapes. The same is done for watchdog to add existing dashboards and alerts.
You can look at examples of how this is done by referencing the docs for adding a postgres import on top of watchdog.
thanks @mokhtar @jkuester @mrjones for the insightful feedback on this
Just have a few questions;
No Data ?@jkuester For our use case , we just need to always keep tabs on what’s happen in the CHT instance , getting the alert notifications and act accordingly since we won’t be having direct access to the CHT instance(we will hand it over to the ministry with all the login credentials), that way we always know if the set up is performing as expected
Yes, you are correct. Continuing to host Prometheus on your local system would not be an ideal setup. It would be better to deploy Prometheus to a stable remote environment. This could be the same host as your remote Grafana instance, but it would not have to be.
@mrjones @jkuester @mokhtar we resolved to just do a fresh set up of the watch dog and follow these