Core Framework: Failed to create initial CouchDB administrative account

Describe the Bug
During CHT installation I am getting Fatal: Failed to create initial CouchDB administrative account error
Moving conversation from Github for the benefit of other Community members: CouchDB not starting · Issue #5 · medic/cht-infrastructure · GitHub

How To Reproduce The Issue

  1. Clone GitHub - medic/cht-infrastructure: Documentation and scripts needed to host the community health toolkit.
  2. Follow instructions to run CHT
  3. Check the CouchDB logs for the error message

Expected Behavior
CouchDB admin user should be created and CHT app should run
Logs
root@cf3c91e7cff4:/srv/storage/medic-core# tail -f /srv/storage/medic-core/couchdb/logs/startup.log

[notice] 2020-07-26T10:18:52.333172Z couchdb@127.0.0.1 <0.350.0> -------- chttpd_auth_cache changes listener died database_does_not_exist at mem3_shards:load_shards_from_db/6(line:395) <= mem3_shards:load_shards_from_disk/1(line:370) <= mem3_shards:load_shards_from_disk/2(line:399) <= mem3_shards:for_docid/3(line:86) <= fabric_doc_open:go/3(line:39) <= chttpd_auth_cache:ensure_auth_ddoc_exists/2(line:195) <= chttpd_auth_cache:listen_for_changes/1(line:142)
[error] 2020-07-26T10:18:57.335931Z couchdb@127.0.0.1 emulator -------- Error in process <0.19943.1> on node ‘couchdb@127.0.0.1’ with exit value:
{database_does_not_exist,[{mem3_shards,load_shards_from_db,“_users”,[{file,“src/mem3_shards.erl”},{line,395}]},{mem3_shards,load_shards_from_disk,1,[{file,“src/mem3_shards.erl”},{line,370}]},{mem3_shards,load_shards_from_disk,2,[{file,“src/mem3_shards.erl”},{line,399}]},{mem3_shards,for_docid,3,[{file,“src/mem3_shards.erl”},{line,86}]},{fabric_doc_open,go,3,[{file,“src/fabric_doc_open.erl”},{line,39}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,“src/chttpd_auth_cache.erl”},{line,195}]},{chttpd_auth_cache,listen_for_changes,1,[{file,“src/chttpd_auth_cache.erl”},{line,142}]}]}

Environment

  • OS
    root@osboxes:~# lsb_release -a
    No LSB modules are available.
    Distributor ID: Ubuntu
    Description: Ubuntu 18.04.4 LTS
    Release: 18.04
    Codename: bionic

  • Instance: Local instance

  • Version:
    Docker: Docker version 19.03.6, build 369ce74a3c
    Docker Compose: version 1.26.2, build unknown

Additional Context
The problem started after the upgrade of medic-os docker to: medic-os:cht-3.9.0-rc.1

@munjoma please confirm that you are running docker-compose up as root and also that you exported the couchdb admin password in your shell before running docker-compose up?

export DOCKER_COUCHDB_ADMIN_PASSWORD=<random_pw>

You may need to prune your system of past installs because any second install attempt will assume that you have already installed

@nyika

Yes I am running docker-compose up as root. I am running the prepare.sh script as root.

I did not run that step. With the prepare.sh script how do I do that? Do I just follow instructions in Docker Setup?

Yeah the instructions here detail how you should get the CHT Medic-Os up and running through docker. Make sure you use our later docker image though. The process is a simple 3 step

  1. Install all docker prerequisites
  2. Export the CouchDB admin password.
  3. Run docker-compose up (run in detached mode if you are running it in production mode.)

Hi @nyika

I pruned my system of past installs.

the did run

from /cht-infrastructure/self-hosting/main$

confirm I have to replace <random_pw> with a password of my choice? (which I did)

Still no luck.
Please find the log file here (I can not upload the file, file format not supported)

From what I see in this logfile there’s a hint that CouchDB was not set up properly and you are trying over a failed install. The _users DB is missing. To do a fresh install follow these steps.

  1. Prune your system (assuming you only have the CHT docker images )
    docker system prune -a --volumes

  2. Get the latest docker-compose file here.

  3. In your shell as root Export the CouchDB password. export DOCKER_COUCHDB_ADMIN_PASSWORD=myAwesomePassword

  4. In the same shell install the medic-os and ha-proxy containers in detached mode.
    # docker-compose -f /path/to/docker-compose.yml up -d

2 Likes

I have the same issue but on windows 10,
medic-os | [2021/10/04 16:01:56] Info: Creating system databases
haproxy | Oct 4 16:01:56 e9424fa6a828 haproxy[25]: 172.22.0.3,503,GET,/,-,-,‘-’,212,-1,-,‘curl/7.47.0’
medic-os | [2021/10/04 16:01:59] Warning: Failed to created system database ‘_users’
medic-os | [2021/10/04 16:01:59] Info: Setting up CouchDB administrative account
haproxy | Oct 4 16:01:59 e9424fa6a828 haproxy[25]: 172.22.0.3,503,PUT,/_users,-,-,‘-’,212,-1,-,‘curl/7.47.0’
haproxy | Oct 4 16:02:00 e9424fa6a828 haproxy[25]: Connect from 172.22.0.3:42262 to 172.22.0.2:5986 (http-in2/HTTP)
medic-os | [2021/10/04 16:02:01] Info: Setting up software (42% complete)…
haproxy | Oct 4 16:02:07 e9424fa6a828 haproxy[25]: 172.22.0.3,503,PUT,/_users/org.couchdb.user:admin,-,admin,‘{#012 “id”: “org.couchdb.user:admin”, “roles”: [],#012 “type”: “user”, “name”: “admin”#012 }’,212,-1,-,‘curl/7.47.0’
medic-os | Fatal: Failed to create initial CouchDB administrative account

Hi @bamatic, welcome to the CHT community.

A more recent way of starting CHT is by using the docker-helper script.

I’ve been using it in Windows 10 (with WSL2) on a daily basis without problems. With it, we can also run multiple instances quickly. Docker Desktop also integrates nicely with WSL2.

Please give it a try if you want, and please let us know how it goes.

1 Like

When running the script in docker-helper first I get
Port 8443 is not open on 172.25.174.79
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 172-25-174-79.my.local-ip.co:8443
and after
CHT is returning 404 instead of 200
This is the same issue, curls get an error contacting couhdb, it is unable to create _users database and finally curl gets 404 when accessing the non created _users database
I’ve tryed to clone cht-core with github desktop and installl doing docket-compose up on windows power shelll, the same when doing it from ubuntu on windows, i’ve clone a again cht-core on ubuntu in windows wsl2 and execute docker-helper script but the problem is the same, open_ssl and closed port , any ideas ?

Hi,
I was trying to setup cht on my Ubuntu local machine and I ran into this same problem. I any guidance on how this issue is resolved. Thank you.
Erick

Hi @ewafula

Could you please include a description or logs of the error that you are seeing?
It looks like a fresh install resolved this problem for the original poster: Core Framework: Failed to create initial CouchDB administrative account - #7 by nyika

Thanks!

Thank you @diana.
As you can see from the logs below. It has failed to create CouchDB admin user
Creating network “cht_demo_medic-net” with the default driver
Creating volume “cht_demo_medic-data” with default driver
Creating haproxy … done
Creating medic-os … done
Attaching to haproxy, medic-os
haproxy | Starting enhanced syslogd: rsyslogd.
medic-os | mesg: ttyname failed: Inappropriate ioctl for device
haproxy | # Setting log here with the address of 127.0.0.1 will have the effect
haproxy | # of haproxy sending the udp log messages to its own rsyslog instance
haproxy | # (which sits at 127.0.0.1) at the local0 facility including all
haproxy | # logs that have a priority greater or equal to the specified log level
haproxy | # log 127.0.0.1 local0 warning
haproxy | global
haproxy | maxconn 4096
haproxy | lua-load /usr/local/etc/haproxy/parse_basic.lua
haproxy | lua-load /usr/local/etc/haproxy/parse_cookie.lua
haproxy | lua-load /usr/local/etc/haproxy/replace_password.lua
haproxy | log /dev/log len 65535 local2 info
haproxy |
haproxy | defaults
haproxy | mode http
haproxy | log global
haproxy | option dontlognull
haproxy | option http-ignore-probes
haproxy | timeout client 150000
haproxy | timeout server 3600000
haproxy | timeout connect 15000
haproxy | stats enable
haproxy | stats refresh 30s
haproxy | stats auth admin:“test123”
haproxy | stats uri /haproxy?stats
haproxy |
haproxy | frontend http-in
haproxy | bind *:5984
haproxy | acl has_user req.hdr(x-medic-user) -m found
haproxy | acl has_cookie req.hdr(cookie) -m found
haproxy | acl has_basic_auth req.hdr(authorization) -m found
haproxy | declare capture request len 400000
haproxy | http-request set-header x-medic-user %[lua.parseBasic] if has_basic_auth
haproxy | http-request set-header x-medic-user %[lua.parseCookie] if !has_basic_auth !has_user has_cookie
haproxy | http-request capture req.body id 0 # capture.req.hdr(0)
haproxy | http-request capture req.hdr(x-medic-service) len 200 # capture.req.hdr(1)
haproxy | http-request capture req.hdr(x-medic-user) len 200 # capture.req.hdr(2)
haproxy | http-request capture req.hdr(user-agent) len 600 # capture.req.hdr(3)
haproxy | capture response header Content-Length len 10 # capture.res.hdr(0)
haproxy | log-format “%ci,%ST,%[capture.req.method],%[capture.req.uri],%[capture.req.hdr(1)],%[capture.req.hdr(2)],‘%[capture.req.hdr(0),lua.replacePassword]’,%B,%Tr,%[capture.res.hdr(0)],‘%[capture.req.hdr(3)]’”
haproxy | default_backend couch-backend
haproxy |
haproxy | frontend http-in2
haproxy | bind *:5986
haproxy | default_backend couch-backend2
haproxy |
haproxy | backend couch-backend
haproxy | balance roundrobin
haproxy | server couchdb1 medic-os:5985
haproxy |
haproxy | backend couch-backend2
haproxy | balance roundrobin
haproxy | server couchdb1 medic-os:5987
medic-os | [2022/11/11 12:56:32] Info: Setting up software…
haproxy | [alert] 314/125633 (1) : parseBasic loaded
haproxy | [alert] 314/125633 (1) : parseCookie loaded
haproxy | [alert] 314/125633 (1) : replacePassword loaded
haproxy | Nov 11 12:56:33 3cd21125b106 haproxy[1]: Proxy http-in started.
haproxy | Nov 11 12:56:33 3cd21125b106 haproxy[1]: Proxy http-in2 started.
haproxy | Nov 11 12:56:33 3cd21125b106 haproxy[1]: Proxy couch-backend started.
haproxy | Nov 11 12:56:33 3cd21125b106 haproxy[1]: Proxy couch-backend2 started.
medic-os | [2022/11/11 12:56:34] Info: Running setup task ‘horticulturalist/sudoers’
medic-os | [2022/11/11 12:56:34] Info: Running setup task ‘horticulturalist/users’
medic-os | [2022/11/11 12:56:34] Info: Service ‘horticulturalist/horticulturalist’ started successfully
medic-os | [2022/11/11 12:56:36] Info: Setting up software (14% complete)…
medic-os | [2022/11/11 12:56:36] Info: Running setup task ‘medic-api/link-logs’
medic-os | [2022/11/11 12:56:36] Info: Running setup task ‘medic-api/logrotate’
medic-os | [2022/11/11 12:56:36] Info: Running setup task ‘medic-api/users’
medic-os | [2022/11/11 12:56:36] Info: Service ‘medic-api/medic-api’ started successfully
medic-os | [2022/11/11 12:56:37] Info: Setting up software (28% complete)…
medic-os | [2022/11/11 12:56:44] Info: Running setup task ‘medic-core/ldconfig’
medic-os | [2022/11/11 12:56:44] Info: Running setup task ‘medic-core/link-logs’
medic-os | [2022/11/11 12:56:44] Info: Running setup task ‘medic-core/logrotate’
medic-os | [2022/11/11 12:56:44] Info: Running setup task ‘medic-core/nginx’
medic-os | [2022/11/11 12:56:44] Info: Running setup task ‘medic-core/nginx-ssl’
medic-os | [2022/11/11 12:56:49] Info: Running setup task ‘medic-core/profile’
medic-os | [2022/11/11 12:56:49] Info: Running setup task ‘medic-core/ssh-authorized-keys’
medic-os | [2022/11/11 12:56:49] Info: Running setup task ‘medic-core/ssh-keygen’
medic-os | [2022/11/11 12:56:50] Info: Running setup task ‘medic-core/usb-modeswitch’
medic-os | [2022/11/11 12:56:50] Info: Service ‘medic-core/couchdb’ started successfully
medic-os | [2022/11/11 12:56:50] Info: Service ‘medic-core/nginx’ started successfully
medic-os | [2022/11/11 12:56:50] Info: Setting up CouchDB for the first time
medic-os | [2022/11/11 12:56:50] Info: Service ‘medic-core/openssh’ started successfully
haproxy | Nov 11 12:56:53 3cd21125b106 haproxy[25]: 172.19.0.3,503,GET,/,-,-,‘-’,212,-1,-,‘curl/7.47.0’
medic-os | [2022/11/11 12:56:53] Info: Creating system databases
medic-os | [2022/11/11 12:56:55] Info: Setting up software (42% complete)…
medic-os | [2022/11/11 12:56:56] Info: Running setup task ‘medic-couch2pg/link-logs’
medic-os | [2022/11/11 12:56:56] Info: Running setup task ‘medic-couch2pg/logrotate’
medic-os | [2022/11/11 12:56:56] Info: Service ‘medic-couch2pg/medic-couch2pg’ started successfully
haproxy | Nov 11 12:56:56 3cd21125b106 haproxy[25]: 172.19.0.3,503,PUT,/_users,-,-,‘-’,212,-1,-,‘curl/7.47.0’
medic-os | [2022/11/11 12:56:56] Warning: Failed to created system database ‘_users’
medic-os | [2022/11/11 12:56:56] Info: Setting up CouchDB administrative account
haproxy | Nov 11 12:56:57 3cd21125b106 haproxy[25]: Connect from 172.19.0.3:55176 to 172.19.0.2:5986 (http-in2/HTTP)
medic-os | [2022/11/11 12:56:58] Info: Setting up software (57% complete)…
haproxy | Nov 11 12:56:58 3cd21125b106 haproxy[25]: 172.19.0.3,404,PUT,/_users/org.couchdb.user:admin,-,admin,‘{#012 “id”: “org.couchdb.user:admin”, “roles”: [],#012 “type”: “user”, “name”: “admin”#012 }’,326,32,58,‘curl/7.47.0’
medic-os | Fatal: Failed to create initial CouchDB administrative account
medic-os | [2022/11/11 12:57:05] Info: Running setup task ‘medic-rdbms/ldconfig’
medic-os | [2022/11/11 12:57:05] Info: Running setup task ‘medic-rdbms/link-logs’
medic-os | [2022/11/11 12:57:05] Info: Running setup task ‘medic-rdbms/reconfigure’
medic-os | [2022/11/11 12:57:05] Info: Service ‘medic-rdbms/postgresql’ started successfully
medic-os | [2022/11/11 12:57:06] Info: Setting up software (71% complete)…
medic-os | [2022/11/11 12:57:07] Info: Running setup task ‘medic-sentinel/link-logs’
medic-os | [2022/11/11 12:57:07] Info: Running setup task ‘medic-sentinel/logrotate’
medic-os | [2022/11/11 12:57:07] Info: Running setup task ‘medic-sentinel/users’
medic-os | [2022/11/11 12:57:08] Info: Service ‘medic-sentinel/medic-sentinel’ started successfully
medic-os | [2022/11/11 12:57:08] Info: Setting up software (85% complete)…
medic-os | [2022/11/11 12:57:08] Info: Running setup task ‘system-services/home-directories’
medic-os | [2022/11/11 12:57:08] Info: Running setup task ‘system-services/link-logs’
medic-os | [2022/11/11 12:57:08] Info: Running setup task ‘system-services/logrotate’
medic-os | [2022/11/11 12:57:08] Info: Service ‘system-services/cron’ started successfully
medic-os | [2022/11/11 12:57:08] Info: Service ‘system-services/syslog’ started successfully
medic-os | [2022/11/11 12:57:08] Info: Setting up software (100% complete)…
medic-os | [2022/11/11 12:57:08] Info: Starting services…
medic-os | [2022/11/11 12:57:09] Info: Synchronizing disks…
medic-os | [2022/11/11 12:57:20] Info: System started successfully
medic-os | [2022/11/11 12:57:20] Info: Starting log streaming

Thanks for sharing logs, @ewafula .
It would be helpful if you could also share the CouchDb logs, the original post includes instructions on how to get those, but as a shortcut, run this inside the container:

tail -f /srv/storage/medic-core/couchdb/logs/startup.log

Have you tried following the steps in this post: Core Framework: Failed to create initial CouchDB administrative account - #7 by nyika to re-create all services?

Thanks!

Hi @diana,
I followed the steps specified above by @nyika and this issue was resolved.
Thank you