Upgrade of couchdb from version 2.3.1 to 3.3.1

robinmurphy · July 21, 2023, 8:19am

Hi

We want to upgrade couchDB version from 2.3.1 to a later version 3.3.1. I see a PR was created and the work done to do the upgrade a couple of years ago. Is there a reason why this has not been merged into the main branch. I have tested with this version ( feat(#6289): Bumps CouchDb version to 3.2.1) and everything appears to work ok

regards

jkuester · July 21, 2023, 1:34pm

There was some substantial challenges in the CHT 3.x architecture that prevented a clean upgrade of Couch to a new major version. One of the main goals of the new CHT 4.x architecture was to enable seamless Couch upgrades even for major versions. So, that is why we ended up so behind on our Couch version (and why we are now working to get it upgraded).

@diana might be able to share some insights into why we are targeting 3.2.1 instead of 3.3.1!

diana · July 22, 2023, 4:12pm

We are targetting 3.3.2, actually. There was a typo in the PR title that I have now fixed.

I’m really happy to hear that the branch is working well for you, @robinmurphy , it’s awesome to have this kind of deep interest from our community.
We are still assessing expected performance from CouchDb 3.x and validating upgrade paths, this is the reason this has not been merged yet.

fardarter · July 25, 2023, 9:14am

Hi all.

I’m working with robin on this but he’s been unavailable recently. I’m having a hard time validating the code (I’m an ops person and I don’t know the codebase particularly well).

I’m trying to find out how you would locally run the tests from the PR to validate that it is working. At the moment I’m running “wdio-local”, but I’m having a hard time deciding if I’ve just misconfigured the tests or not. There’s a lot going on here.

(Sidenote: difficult to run the e2e tests in WSL – have had to move to mac to get them running at all.)

fardarter · July 25, 2023, 9:17am

To be clear, we’re happy to patch with a working config and remove the patch when the feature lands.

diana · July 25, 2023, 9:31am

Hi @fardarter

I would not advise trying to run e2e tests on WSL, it’s not something we support and you might end up wasting precious time. The learning curve for understanding our e2e tests is pretty steep.
I would suggest you run the integration suite rather than wdio suite at first, which requires less setup (no wdio, no browser, just API requests).
Without seeing your code changes, it’s basically impossible to offer you support.

fardarter · July 25, 2023, 10:03am

@diana I’m looking at some recent unincorporated changes, including setting the UNIT_TEST_ENV to 1.

Going to try that.

Changes were, I thought, as per PR, but I’m catching up.

I have wdio running on macos but appreciate the suggestion to run the integration suite.

Is there some behaviour we could use as a proxy for a valid upgrade?

fardarter · July 25, 2023, 12:03pm

Changed the file name to include wdio so that it ran. Should this test be succeeding?

------------------------------------------------------------------
[chrome 115.0.5790.102 darwin #0-1] Running: chrome (v115.0.5790.102) on darwin
[chrome 115.0.5790.102 darwin #0-1] Session ID: 7307b71b-2881-4e18-b3a1-e7894123294f
[chrome 115.0.5790.102 darwin #0-1]
[chrome 115.0.5790.102 darwin #0-1] » /tests/e2e/default/db/db-access-for-custom-roles.wdio-spec.js
[chrome 115.0.5790.102 darwin #0-1] Database access for new roles
[chrome 115.0.5790.102 darwin #0-1]    ✖ "before all" hook for Database access for new roles
[chrome 115.0.5790.102 darwin #0-1]
[chrome 115.0.5790.102 darwin #0-1] 1 failing (2.6s)
[chrome 115.0.5790.102 darwin #0-1]
[chrome 115.0.5790.102 darwin #0-1] 1) Database access for new roles "before all" hook for Database access for new roles
[chrome 115.0.5790.102 darwin #0-1] 400 - {"code":400,"error":{"message":"Missing required fields: username, place, contact","translationKey":"fields.required","translationParams":{"fields":"username, place, contact"}}}
[chrome 115.0.5790.102 darwin #0-1] StatusCodeError: 400 - {"code":400,"error":{"message":"Missing required fields: username, place, contact","translationKey":"fields.required","translationParams":{"fields":"username, place, contact"}}}
[chrome 115.0.5790.102 darwin #0-1]     at new StatusCodeError (node_modules/request-promise-core/lib/errors.js:32:15)
[chrome 115.0.5790.102 darwin #0-1]     at node_modules/request-promise-core/lib/plumbing.js:97:41
[chrome 115.0.5790.102 darwin #0-1]     at processTicksAndRejections (node:internal/process/task_queues:95:5)

Anro · July 26, 2023, 1:00pm

Hi @diana and @jkuester

With the help of @fardarter we’ve built an 3.2 couch db image that seemed to run fine during our smoke tests.
We did take note of the following issue popping up when running the cht --local command:

ERROR Error trying to get couchdb config: TypeError: Cannot read properties of undefined (reading 'split')

Which, after cloning the cht-conf repo, seems to occur in the api.js file’s getCompressibleTypes method. I’m unsure why, would be great to hear your ideas.

While looking through the cht-conf repo I noticed the tests that could be run against couch db.
And, as discussed earlier in the thread, we wanted to be a bit more sure regarding the stability of the upgrade I wanted to run those tests against the new image.

Since simply upgrading the couchdb version in the script did not work, due to the .ini lookup, I’ve done some of the following in hopes of getting a service up to run the tests against - but have been unsuccessful until now.

As mentioned above running the following yields an error and the container stops:

docker run -p 6984:5984 --rm --name cht-conf-couchdb couchdb:3.2

Waiting for cht couchdb
[info] 2023-07-25T14:34:27.372397Z couchdb@127.0.0.1 <0.254.0> -------- Preflight check: Checking For Monsters

[info] 2023-07-25T14:34:27.372457Z couchdb@127.0.0.1 <0.254.0> -------- Preflight check: Asserting Admin Account

[info] 2023-07-25T14:34:27.372488Z couchdb@127.0.0.1 <0.254.0> --------
*************************************************************
ERROR: CouchDB 3.0+ will no longer run in "Admin Party"
       mode. You *MUST* specify an admin user and
       password, either via your own .ini file mapped
       into the container at /opt/couchdb/etc/local.ini
       or inside /opt/couchdb/etc/local.d, or with
       "-e COUCHDB_USER=admin -e COUCHDB_PASSWORD=password"
       to set it via "docker run".
*************************************************************

Changing the the query to something like the following managed to start up the service, but with an periodic error noting that the “_users” db is missing:

docker run -p 6984:5984 -e 'COUCHDB_USER=admin' -e 'COUCHDB_PASSWORD=password' --rm --name cht-conf-couchdb couchdb:3.2

[error] 2023-07-26T06:41:52.918740Z nonode@nohost emulator -------- Error in process <0.390.0> with exit value:
{database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,400}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,375}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,404}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,97}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,39}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,198}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,145}]}]}

I did eventually manage to get the service started without the “_users” error by doing the following:

docker rm cht-conf-couchdb
docker volume rm couchdb_data
docker system prune -a
docker build -t my-couchdb-image .
docker run -p 6984:5984 -e 'COUCHDB_USER=admin' -e 'COUCHDB_PASSWORD=password' -v couchdb_data:/opt/couchdb/data --name cht-conf-couchdb my-couchdb-image

This was paired with an edit to the docker file:

...

FROM couchdb:latest
EXPOSE 5984

# Copy the setup script into the container
COPY setup.sh /setup.sh

# Make the setup script executable
RUN chmod +x /setup.sh

# Run the setup script during the container startup
CMD ["/bin/bash", "-c", "/setup.sh"]

And a setup script that added the missing “_user” database:

#!/bin/sh -xe

# Set the CouchDB admin credentials
COUCHDB_ADMIN_USERNAME="admin"
COUCHDB_ADMIN_PASSWORD="password"

cat >/opt/couchdb/etc/local.ini <<EOF
[couchdb]
single_node=true
require_valid_user = false

[admins]
$COUCHDB_ADMIN_USERNAME = $COUCHDB_ADMIN_PASSWORD

COUCHDB_USER=admin
COUCHDB_PASSWORD=password
EOF

nohup bash -c "/docker-entrypoint.sh /opt/couchdb/bin/couchdb &"
sleep 15

# Wait for CouchDB to start and recognize the admin credentials
while true; do
  curl http://127.0.0.1:5984/_up
  curl http://$COUCHDB_ADMIN_USERNAME:$COUCHDB_ADMIN_PASSWORD@127.0.0.1:5984/_up
#   curl -s http://127.0.0.1:5984/_up | grep -q 'true' && break
  curl -s http://127.0.0.1:5984/_up | grep -q 'ok' && break
  sleep 1
done

echo the site is up

# Create the admin user using the _users database endpoint with basic authentication
curl -X PUT http://127.0.0.1:5984/_users/org.couchdb.user:$COUCHDB_ADMIN_USERNAME \
     -H "Content-Type: application/json" \
     -u "$COUCHDB_ADMIN_USERNAME:$COUCHDB_ADMIN_PASSWORD" \
     -d '{"name": "'"$COUCHDB_ADMIN_USERNAME"'", "password": "'"$COUCHDB_ADMIN_PASSWORD"'", "roles": [], "type": "user"}'

# Make the new admin user an admin
curl -X PUT http://127.0.0.1:5984/_config/admins/$COUCHDB_ADMIN_USERNAME \
     -u "$COUCHDB_ADMIN_USERNAME:$COUCHDB_ADMIN_PASSWORD" \
     -d "\"$COUCHDB_ADMIN_PASSWORD\""

# Create _replicator database
curl -X PUT http://127.0.0.1:5984/_replicator \
     -u "$COUCHDB_ADMIN_USERNAME:$COUCHDB_ADMIN_PASSWORD"

Appending && sh test/scripts/wait_for_response_code.sh 6984 200 CouchDB to the docker run command, in order to tell if the service was up, got stuck in a continuous loop.
Important to note that I could not access Fauxton at any time while testing.

Checking the docker container’s /opt/couchdb/etc/local.ini file, nothing immediately jumped out to me as being missing and overwriting it with the cat command in the setup.sh had no effect to Fauxton’s availability.
The couchdb v3.2 default /opt/couchdb/etc/local.ini content looks as follows:

; CouchDB Configuration Settings

; Custom settings should be made in this file. They will override settings
; in default.ini, but unlike changes made to default.ini, this file won't be
; overwritten on server upgrade.

[couchdb]
;max_document_size = 4294967296 ; bytes
;os_process_timeout = 5000

[couch_peruser]
; If enabled, couch_peruser ensures that a private per-user database
; exists for each document in _users. These databases are writable only
; by the corresponding user. Databases are in the following form:
; userdb-{hex encoded username}
;enable = true
; If set to true and a user is deleted, the respective database gets
; deleted as well.
;delete_dbs = true
; Set a default q value for peruser-created databases that is different from
; cluster / q
;q = 1

[chttpd]
;port = 5984
;bind_address = 127.0.0.1
; Options for the MochiWeb HTTP server.
;server_options = [{backlog, 128}, {acceptor_pool_size, 16}]
; For more socket options, consult Erlang's module 'inet' man page.
;socket_options = [{sndbuf, 262144}, {nodelay, true}]

[httpd]
; NOTE that this only configures the "backend" node-local port, not the
; "frontend" clustered port. You probably don't want to change anything in
; this section.
; Uncomment next line to trigger basic-auth popup on unauthorized requests.
;WWW-Authenticate = Basic realm="administrator"

; Uncomment next line to set the configuration modification whitelist. Only
; whitelisted values may be changed via the /_config URLs. To allow the admin
; to change this value over HTTP, remember to include {httpd,config_whitelist}
; itself. Excluding it from the list would require editing this file to update
; the whitelist.
;config_whitelist = [{httpd,config_whitelist}, {log,level}, {etc,etc}]

[chttpd_auth]
; If you set this to true, you should also uncomment the WWW-Authenticate line
; above. If you don't configure a WWW-Authenticate header, CouchDB will send
; Basic realm="server" in order to prevent you getting logged out.
; require_valid_user = false

[ssl]
;enable = true
;cert_file = /full/path/to/server_cert.pem
;key_file = /full/path/to/server_key.pem
;password = somepassword
; set to true to validate peer certificates
;verify_ssl_certificates = false
; Set to true to fail if the client does not send a certificate. Only used if verify_ssl_certificates is true.
;fail_if_no_peer_cert = false
; Path to file containing PEM encoded CA certificates (trusted
; certificates used for verifying a peer certificate). May be omitted if
; you do not want to verify the peer.
;cacert_file = /full/path/to/cacertf
; The verification fun (optional) if not specified, the default
; verification fun will be used.
;verify_fun = {Module, VerifyFun}
; maximum peer certificate depth
;ssl_certificate_max_depth = 1
;
; Reject renegotiations that do not live up to RFC 5746.
;secure_renegotiate = true
; The cipher suites that should be supported.
; Can be specified in erlang format "{ecdhe_ecdsa,aes_128_cbc,sha256}"
; or in OpenSSL format "ECDHE-ECDSA-AES128-SHA256".
;ciphers = ["ECDHE-ECDSA-AES128-SHA256", "ECDHE-ECDSA-AES128-SHA"]
; The SSL/TLS versions to support
;tls_versions = [tlsv1, 'tlsv1.1', 'tlsv1.2']

; To enable Virtual Hosts in CouchDB, add a vhost = path directive. All requests to
; the Virual Host will be redirected to the path. In the example below all requests
; to http://example.com/ are redirected to /database.
; If you run CouchDB on a specific port, include the port number in the vhost:
; example.com:5984 = /database
[vhosts]
;example.com = /database/

; To create an admin account uncomment the '[admins]' section below and add a
; line in the format 'username = password'. When you next start CouchDB, it
; will change the password to a hash (so that your passwords don't linger
; around in plain-text files). You can add more admin accounts with more
; 'username = password' lines. Don't forget to restart CouchDB after
; changing this.
[admins]
;admin = mysecretpassword

Would you perhaps be able to provide some guidance on next steps?

diana · July 26, 2023, 1:08pm

Thanks for noticing this! I’ve updated the test on the branch.

diana · July 26, 2023, 1:11pm

Hi @Anro

Please have a look at how we build our CouchDb image to get a list of steps that are required: cht-core/couchdb at master · medic/cht-core · GitHub
However, I think you should just use the image that results from our build.

Which, after cloning the cht-conf repo, seems to occur in the api.js file’s getCompressibleTypes method. I’m unsure why, would be great to hear your ideas.

Thanks for highlighting this!
It looks like CouchDb 3 doesn’t have a setting to set compressible types, this means that we will need to update cht-conf to gracefully handle empty responses from CouchDb.

fardarter · July 26, 2023, 1:35pm

Hi @diana

Is this a relatively easy fix?

We need to make a call by tomorrow morning (not your deadline so please no pressure) about whether to build a 2.2 or 3.2 deployment. Behaviorally things seem to be working but we’re seeing these test failures and errors being thrown and it’s making us nervous.

I realise that you haven’t actually done a release, so it’s asking a lot for you to say “yes, this is fine”, but I think we’d be alright taking the risk with passing test suites given have much testing you have going on. I’d very much like to avoid a db upgrade with live data with the 3.2 work seemingly so close to completion.

Again, gratitude only here for fast responses, no expectations. We’ll figure it out.

diana · July 26, 2023, 2:08pm

Hi @fardarter

I’d like to know which tests are failing and which errors.
The cht-conf issue is very easy to fix, I’m adding that additional config to CouchDb 3 and testing at the moment, instead of patching cht-conf. I’m not sure if you’re referring to another issue.

fardarter · July 26, 2023, 2:17pm

I’ll run the local integration tests and send you an output. I’ll do the same with wdio on my mac, but it’ll take some time.

@Anro is running the cht-conf tests (I’ll ask him to post):

against the couchdb:3.3.2 image
against the image we’ve built with our patches

We have a very, very light patch set at the moment (branch v4.2.x) outside of stuff we’ve added for the PR and a project folder in the config.

fardarter · July 26, 2023, 2:51pm

This is my integration test output

Test done. Signing off ...

  455 passing (22m)
  1 pending
  8 failing

  1) Outbound
       should find existing outbound tasks and execute them, leaving them if the send was unsuccessful:
     Error: Timeout of 200000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/tests/integration/sentinel/schedules/outbound.spec.js)
      at createTimeoutError (/node_modules/mocha/lib/errors.js:498:15)
      at Runnable._timeoutError (/node_modules/mocha/lib/runnable.js:431:10)
      at Timeout.<anonymous> (/node_modules/mocha/lib/runnable.js:246:24)
      at listOnTimeout (node:internal/timers:569:17)
      at process.processTimers (node:internal/timers:512:7)

  2) mark_for_outbound
       when external server is up
         correctly creates and sends an outbound request immediately:
     AssertionError: expected [ { …(6) } ] to be empty
      at /tests/integration/sentinel/transitions/mark-for-outbound.spec.js:112:30
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

  3) mark_for_outbound
       when external server is up
         correctly skips sending the same outbound multiple times:
     AssertionError: expected [ { …(6) } ] to be empty
      at /tests/integration/sentinel/transitions/mark-for-outbound.spec.js:162:30
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

  4) mark_for_outbound
       when external server is up
         correctly sends changed doc outbound multiple times:
     AssertionError: expected [ { …(6) } ] to be empty
      at /tests/integration/sentinel/transitions/mark-for-outbound.spec.js:227:30
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

  5) mark_for_outbound
       when external server is up
         correctly creates a task if the immediate server request fails:

      AssertionError: expected [] to have a length of 1 but got +0
      + expected - actual

      -0
      +1
      
      at /tests/integration/sentinel/transitions/mark-for-outbound.spec.js:308:50
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

  6) mark_for_outbound
       error logging
         logs an error if the server returns !2xx:

      AssertionError: expected [ Array(1) ] to have a length of 2 but got 1
      + expected - actual

      -1
      +2
      
      at /tests/integration/sentinel/transitions/mark-for-outbound.spec.js:496:32
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

  7) cht-conf actions tests
       should execute compile-app-settings:
     Error: the string "\u001b[32mINFO Processing config in default. \u001b[0m\n\u001b[32mINFO Actions:\n     - compile-app-settings \u001b[0m\n\u001b[32mINFO Starting action: compile-app-settings… \u001b[0m\n\u001b[33mWARN app_settings.json file should not be edited directly.\n    Please create a base_settings.json file in app_settings folder and move any manually defined configurations there. \u001b[0m\n\u001b[32mINFO Packaging contact-summary \u001b[0m\n" was thrown, throw an Error :)
      at thrown2Error (/node_modules/mocha/lib/runner.js:1241:10)
      at Runner.fail (/node_modules/mocha/lib/runner.js:443:11)
      at /node_modules/mocha/lib/runner.js:814:18
      at done (/node_modules/mocha/lib/runnable.js:310:5)
      at /node_modules/mocha/lib/runnable.js:377:11
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

  8) cht-conf actions tests
       should execute convert-contact-forms:
     Error: the string "\u001b[32mINFO Processing config in default. \u001b[0m\n\u001b[32mINFO Actions:\n     - convert-contact-forms \u001b[0m\n\u001b[32mINFO Starting action: convert-contact-forms… \u001b[0m\n\u001b[32mINFO Error: ENOENT: no such file or directory, copyfile '/config/default/forms/contact/PLACE_TYPE-create.xlsx' -> '/config/default/forms/contact/clinic-create.xlsx'\n    at Object.copyFileSync (node:fs:2894:3)\n    at Object.copy (/node_modules/cht-conf/src/lib/sync-fs.js:79:8)\n    at /node_modules/cht-conf/src/fn/convert-contact-forms.js:15:12\n    at Array.forEach (<anonymous>)\n    at convertContactForm (/node_modules/cht-conf/src/fn/convert-contact-forms.js:14:8)\n    at Object.execute (/node_modules/cht-conf/src/fn/convert-contact-forms.js:94:18)\n    at executeAction (/node_modules/cht-conf/src/lib/main.js:248:40)\n    at module.exports (/node_modules/cht-conf/src/lib/main.js:238:11)\n    at /node_modules/cht-conf/src/bin/index.js:16:11\n    at Object.<anonymous> (/node_modules/cht-conf/src/bin/index.js:22:3) \u001b[0m\n\u001b[31mERROR ENOENT: no such file or directory, copyfile '/config/default/forms/contact/PLACE_TYPE-create.xlsx' -> '/config/default/forms/contact/clinic-create.xlsx' \u001b[0m\n" was thrown, throw an Error :)
      at thrown2Error (/node_modules/mocha/lib/runner.js:1241:10)
      at Runner.fail (/node_modules/mocha/lib/runner.js:443:11)
      at /node_modules/mocha/lib/runner.js:814:18
      at done (/node_modules/mocha/lib/runnable.js:310:5)
      at /node_modules/mocha/lib/runnable.js:377:11
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Anro · July 26, 2023, 3:31pm

Hi @diana

Thank you for taking the time to point us in the right direction.

I’ve built an image with the linked build process and tested it against cht-conf - which yielded the following result:

[error] 2023-07-26T15:26:59.274509Z couchdb@127.0.0.1 emulator -------- Error in process <0.6866.0> on node 'couchdb@127.0.0.1' with exit value:
{database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,430}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,405}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,434}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,100}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,39}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,214}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,160}]}]}

The updated command to reference said image is as follows:

"docker-start-couchdb": "npm run docker-stop-couchdb && docker run -p 6984:5984 -e 'COUCHDB_USER=admin' -e 'COUCHDB_PASSWORD=password' --rm --name cht-conf-couchdb cht-couchdb:dev && sh test/scripts/wait_for_response_code.sh 6984 200 CouchDB",

fardarter · July 26, 2023, 3:33pm

Running the wdio process with the following config:

specs: [
    'db/*.wdio-spec.js',
  ],
  // Patterns to exclude.
  exclude: [
    'sms/*'
  ],

Getting:

[chrome 115.0.5790.114 darwin #0-1] » /tests/e2e/default/db/db-access-for-custom-roles.wdio-spec.js
[chrome 115.0.5790.114 darwin #0-1] Database access for new roles
[chrome 115.0.5790.114 darwin #0-1]    ✓ user with custom role should be able to log in
[chrome 115.0.5790.114 darwin #0-1]    :heavy_multiplication_x: should be able to sync documents up
[chrome 115.0.5790.114 darwin #0-1]    :heavy_multiplication_x: should be able to sync documents up
[chrome 115.0.5790.114 darwin #0-1]    :heavy_multiplication_x: should be able to sync documents up
[chrome 115.0.5790.114 darwin #0-1]    :heavy_multiplication_x: should be able to sync documents up
[chrome 115.0.5790.114 darwin #0-1]    :heavy_multiplication_x: should be able to sync documents up
[chrome 115.0.5790.114 darwin #0-1]    :heavy_multiplication_x: should be able to sync documents up
[chrome 115.0.5790.114 darwin #0-1]    :heavy_multiplication_x: should be able to sync documents down
[chrome 115.0.5790.114 darwin #0-1]    :heavy_multiplication_x: should be able to sync documents down
[chrome 115.0.5790.114 darwin #0-1]    :heavy_multiplication_x: should be able to sync documents down
[chrome 115.0.5790.114 darwin #0-1]    :heavy_multiplication_x: should be able to sync documents down
[chrome 115.0.5790.114 darwin #0-1]    :heavy_multiplication_x: should be able to sync documents down
[chrome 115.0.5790.114 darwin #0-1]    :heavy_multiplication_x: should be able to sync documents down

[chrome 115.0.5790.114 darwin #0-0] » /tests/e2e/default/db/checkpointer-migration.wdio-spec.js
[chrome 115.0.5790.114 darwin #0-0] Storing checkpointer on target migration
[chrome 115.0.5790.114 darwin #0-0]    :heavy_multiplication_x: should store replication checkpointers on target and source
[chrome 115.0.5790.114 darwin #0-0]    :heavy_multiplication_x: should store replication checkpointers on target and source
[chrome 115.0.5790.114 darwin #0-0]    :heavy_multiplication_x: should store replication checkpointers on target and source
[chrome 115.0.5790.114 darwin #0-0]    :heavy_multiplication_x: should store replication checkpointers on target and source
[chrome 115.0.5790.114 darwin #0-0]    ✓ should store replication checkpointers on target and source
[chrome 115.0.5790.114 darwin #0-0]    ✓ should copy existent browser checkpointer when db is not migrated
[chrome 115.0.5790.114 darwin #0-0]    ✓ should not flag migration as ran if the browser was offline
[chrome 115.0.5790.114 darwin #0-0]    ✓ should re-upload docs when the server rolls back

[chrome 115.0.5790.114 darwin #0-3]
[chrome 115.0.5790.114 darwin #0-3] » /tests/e2e/default/db/initial-replication.wdio-spec.js
[chrome 115.0.5790.114 darwin #0-3] initial-replication
[chrome 115.0.5790.114 darwin #0-3]    :heavy_multiplication_x: should log user in
[chrome 115.0.5790.114 darwin #0-3]    ✓ should log user in
[chrome 115.0.5790.114 darwin #0-3]    ✓ should support "disconnects"

diana · July 27, 2023, 2:58pm

Hi @fardarter

From the looks of it, it seems that some of your tests fail the first time and succeed in the next. I wonder if this is a timeout issue.
The only tests that I see fail repeatedly are:

Database access for new roles -> should be able to sync documents up and
Database access for new roles -> should be able to sync documents down

These are passing in our CI: feat(#6289): Bumps CouchDb version to 3.3.2 · medic/cht-core@3a19675 · GitHub

Have you made any changes to the CouchDb image?

fardarter · July 27, 2023, 3:08pm

Hi @diana

Only the changes in the PR, unless I’m doing something quite silly and not noticing.

Timeout issues are possible for the other items.

The integration tests are more stable in the errors.

diana · July 27, 2023, 3:17pm

I pushed some changes to the PR recently, are you sure you’re running the latest code?
When a test fails, a folder containing all logs is saved, in tests/logs. Additionally, you should find check the allure report to see screenshots of the failures.
Please find details about how to debug e2e tests in our Testing doc: https://github.com/medic/cht-core/blob/master/TESTING.md#view-the-ci-report