`ulimit` error in couchdb docker container on fedora linux

Development
1

Hello, I am trying to setup cht-core as per give instruction in the documentation.

I am stuck on this command

docker compose -f docker-compose.yml -f couchdb-override.yml up -d

Output

10:45 ~/cht-docker $ docker ps
CONTAINER ID   IMAGE                                           COMMAND                  CREATED         STATUS                         PORTS     NAMES
a123d7f42e52   public.ecr.aws/medic/cht-couchdb:4.18.0-alpha   "tini -- /docker-ent…"   9 seconds ago   Restarting (2) 2 seconds ago             cht-docker-couchdb-1
10:45 ~/cht-docker $ docker logs cht-docker-couchdb-1

sh: 1: ulimit: error setting limit (Operation not permitted)
sh: 1: ulimit: error setting limit (Operation not permitted)
sh: 1: ulimit: error setting limit (Operation not permitted)
sh: 1: ulimit: error setting limit (Operation not permitted)
sh: 1: ulimit: error setting limit (Operation not permitted)
sh: 1: ulimit: error setting limit (Operation not permitted)
sh: 1: ulimit: error setting limit (Operation not permitted)

Solutions I have tried:

  1. Adding ulimits in docker-compose.yml
    I added the following snippet in docker-compose-yml
    ulimits:
      nofile:
        soft: 65536
        hard: 65536

It didn’t fix anyhing.

  1. Adding privileged: true in docker-compose.yml
    Still same problem.
2

Hello @rajeevtapadia

I’m aware of this issue, and unfortunately I found no workaround.
I stumbled upon this trying to run the CHT on a EC2 AWS instance, and after failing to find a workaround, I ended up making a no-ulimit couchdb image.

The internet says:

A regular (non‑root) process can lower its own soft limit, but it can’t raise the soft limit above the hard limit, nor can it raise the hard limit at all. Only root—or a process with the CAP_SYS_RESOURCE capability—can do that.

I suppose that you would have to somehow add CAP_SYS_RESOURCE to the docker process. I also tried to add the capability to the docker service in the docker-compose file, and that did not work either.

I’ve created an issue to track this: CouchDb fails to start when system does not allow setting `ulimit` · Issue #9923 · medic/cht-core · GitHub

@hareet I would appreciate your input here. Thanks!

1 Like
3

Thanks for the response @diana
As there is no work around for this, I tried to setup development environment on WSL2. It worked as expected.

3 Likes
4

Thanks for the reply @rajeevtapadia , and thank you for joining the Dev Hour call and asking this question. I believe this is a really tricky issue to bump into with seemingly no solution, just highly technical rabbit holes and futile tries with no results. It was very frustrating for me when I encountered this.

It’s unfortunate there is no workaround, except “try on another machine”. I believe that we should document this limitation in a way that makes sense for people who are not intimately acquainted with OS ↔ VM ↔ Docker deep permissions logic.

1 Like
5

I have found a workaround.

So the problem is caused by the following line in couchdb/docker-entrypoint.sh

su -c "ulimit -n 100000 && exec $@" couchdb

Remove the ulimit part of this line

su -c "exec $@" couchdb

and then run ulimit -n 100000 on the host OS.

After this rebuild the couchdb docker image by running

cd couchdb
docker build . -t custom-couchdb-image

Then change the image name in docker-componse.yaml which is located in ~/cht-docker (if you followed documentation)
After the change docker-compose.yaml should look like

services:
  couchdb:
    image: custom-couchdb-image
    volumes:
      - ${COUCHDB_DATA:-./srv}:/opt/couchdb/data
      - cht-credentials:/opt/couchdb/etc/local.d/
    environment:
      - "COUCHDB_USER=${COUCHDB_USER:-admin}"
      - "COUCHDB_PASSWORD=${COUCHDB_PASSWORD:?COUCHDB_PASSWORD must be set}"
      - "COUCHDB_SECRET=${COUCHDB_SECRET}"
      - "COUCHDB_UUID=${COUCHDB_UUID}"
      - "SVC_NAME=${SVC_NAME:-couchdb}"
      - "COUCHDB_LOG_LEVEL=${COUCHDB_LOG_LEVEL:-info}"
    restart: always
    logging:
      driver: "local"
      options:
        max-size: "${LOG_MAX_SIZE:-50m}"
        max-file: "${LOG_MAX_FILES:-20}"
    networks:
      cht-net:

volumes:
  cht-credentials:

networks:
  cht-net:
    name: ${CHT_NETWORK:-cht-net}

Now just follow the usual process to start development server.

cd ~/cht-docker 
COUCHDB_USER=medic COUCHDB_PASSWORD=password docker compose -f docker-compose.yml -f couchdb-override.yml up -d

And these three npm commands in three different terminals

cd ~/cht-core
npm run build-dev-watch
npm run dev-api
npm run dev-sentinel

PS:
I am not sure when side effect might be there because of removing ulimit command from docker-entrypoint.sh but I think we are setting the limit directly in host OS so it should not cause any problems.
Also thanks to @apoorvapendse for helping me to debug this problem.

1 Like
6

Thanks @rajeevtapadia

The idea was to find a workaround without rebuilding the image, and having the image as-is, with the set ulimit, to work on the client machine.

2 Likes
7

@diana - what do you think about making an obscure .env/compose flag that skips the ulimit call inside the container? This is a workaround that will still require rebuilding the image, for sure. However, it would be used only in development (documented it clearly) and would unblock future users who are stuck like @rajeevtapadia is and prevent every affected local dev from needing to rebuild the image.

8

Right now I’m thinking whether to:

  1. Add this environment variable
  2. Add a try-catch in the image and rerun the startup command without setting ulimit, but instead log loudly that Couch started without setting ulimit.
  3. Continue to fail hard but add a more descriptive message.
9

I would vote a combo of 1 and 3! Continue to fail hard but add a more description method telling them now to not have the call be made via the env var.

1 Like