Hi
Am currently trying host our production instance in another online server. All the containers are up and the connected to the couchdb. But it has been here for about 17hours
What could be the problem
Hi Edwin,
This is normal as CHT is bootstrapping services during initial startup. It should complete in a couple of minutes and redirect to the login page. Please confirm if it completed successfully.
Hi @elijah
Thankyou for the immediate reply. It has been like this since yesterday. Could this be a problem.
That’s a very long time to be in this state. Confirm that all containers are running and check logs for clues on where the process is blocked.
Hi @elijah
The Logs and containers are okay. The new cht application started properly
not until i joined the couchdb to point to the couchdb data i had. Then the delay started
Please elaborate on how couchdb was updated to point to existing data
in the .env file .
COUCHDB_DATA=/home/ubuntu/cht/couchdb/data
route to the old couchdb data
Please follow this process when updating volume mount location:
- Stop all containers
- Update volume mount location on couchdb container
- Restart containers with new volume mount
Please share an example . or a screenshot
Hi Team,
I am requesting for some help or clarification if this delay is normal or there could be something am missing?
Your timely assistance is much appreciated. Thankyou
Edwin,
Yikes - 10s of hours is way too long to wait for the CHT to start up - something is wrong! Let’s help you figure out what’s going on.
It sounds like you’ve migrated from one server to another. To narrow down some troubleshooting steps, I would:
- Check your
.envfile are the same in the old server and new server. Super important is to make sureCOUCHDB_SECRET,COUCHDB_DATA,COUCHDB_PASSWORDandCOUCHDB_UUIDare the same. If you’ve followed the production docker install docs, the.envfile will be in/home/ubuntu/cht/upgrade-service/.env - Before copying over any files , make sure the CHT starts up on the new server correctly with an empty data set. This will ensure your install of docker,
curling to download the compose files and your.envfile are all working correctly. - Now that your new system is working, let’s delete all the newly created files on it. Start by getting a list of containers with
docker ps --format="{{.Names}}\t\t{{.Status}}" - stop the API and CouchDB servers on the new server. For me this was
docker stop cht-api-1 cht-couchdb-1 - Double check you run this on the new server which has empty data, then delete all the data with
rm -rf /home/ubuntu/cht/couchdb/.delete /home/ubuntu/cht/couchdb/.shards /home/ubuntu/cht/couchdb/* - on your old server, stop couch and copy ALL the files in it’s
COUCHDB_DATAdirectory. Be sure to get files starting with.like.shards. Copy these to the new server into it’sCOUCHDB_DATAdirectory. By default this is/home/ubuntu/cht/couchdb/ - Start API and CouchDB on the new server:
docker start cht-api-1 cht-couchdb-1
Be patient and see if the new server starts up correctly.
If it does not start after a few minutes, check for errors in the containers by checking the logs. Likely API and CouchDB will have errors that are going to be the most helpful:
- To get logs from API call:
docker logs cht-api-1 - And from CouchDB:
docker logs cht-couchdb-1
Be sure not to mix new files from a test install with old files - that’s why it’s important to delete all the files on the new server! (you can move them too, but no need to keep an empty data set of CouchDB around!)
Report back and errors you find along the way - best of luck!
2 Likes
Hi @mrjones
THankyou for the reply. Following the steps provided, they are the exact steps i took . Which led to the continuous cht starting phase.
I started the instance normally. Made sure it was connecting, deleted the couchdb files inside. Brought in the backedup couchdb files, the .env file directing to the couchdb. then restarted the containers. This led it to be stack in the migrating stage .
Note::: the couchdb data was collected for 3 years now
I see this number of docs when i run
docker logs -f cht-api-1 would this also be one of the reasons it is delaying?
Thanks for the update @Edwin! You mentioned:
the couchdb data was collected for 3 years now
Which makes me wonder: are you trying to install couchdb data from a different version than what you setup on the new server? I suspect this will not work, especially if this is a CHT 3.x data that you’re trying to install on a CHT 4.x server.
Instead, what you should do is upgrade the old server to the same 4.x version on the new server. If this is not possible, set up a new server with the same version as you’re running on the old server and upgrade it in place.
We have a number of guides for 3.x to 4.x migrations.
Hi @mrjones
The version was already upgraded to 4.10.0 .
Even the containers that are currently up we also went for 4.10.0 to match the couchdb data version before.
Hi @mrjones
Just a small Update on the processing of docs. It is on 340700 of 679084 docs
From 3000 docs processed from the last time we had a conversation.
Do you think after the processing it will be able to work?
Hi @Edwin
It seems that your system is stuck running migrations. this should not happen!
This happening implies that your data transfer, from one server to another, was incomplete - because migration progress is saved in a document in CouchDb, and migrations running implies that this document is inaccessible (not present).
This means that your CouchDb data is incomplete, and you should not wait for this migration to process, you will end up with a partial data set (the missing migration log means multiple documents are affected).
I suggest you retry moving the data from your backup and spin up another instance. If you see this behavior, it means data transfer was incomplete again.