CouchDB Index ddoc

[Anro]

Is there a way to include a CouchDB index JSON configuration in the ddocs for CHT to process?

We’ve experimented with creating an index via Fauxton and using a curl command, but we believe these configurations should be tracked in the app repository rather than managed separately.

Since CHT views are already tracked within the ddocs folder, we were wondering if the same approach could be used for index configurations.

For reference, here is the index we want to create:

{
  "index": {
    "fields": ["_id", "type"],
    "partial_filter_selector": {
      "type": { "$nin": ["form", "translations", "meta"] },
      "_id": { "$nin": ["branding", "extension-libs", "resources"] }
    }
  },
  "ddoc": "testing_by_id_and_type",
  "name": "testing_by_id_and_type",
  "type": "json",
}

Relevant documentation:

• CouchDB Index API
• Partial Indexes

Curl command:

curl -X POST http://admin:password@localhost:5984/medic/_index \
  -H "Content-Type: application/json" \
  -d '{
    "index": {
      "fields": ["_id", "type"],
      "partial_filter_selector": {
        "type": { "$nin": ["form", "translations", "meta"] },
        "_id": { "$nin": ["branding", "extension-libs", "resources"] }
      }
    },
    "ddoc": "testing_by_id_and_type",
    "name": "testing_by_id_and_type",
    "type": "json"
  }'

Output:

{"result":"created","id":"_design/testing_by_id_and_type","name":"testing_by_id_and_type"}

[twier]

Hello @Anro although views (and soon Nouveau indexes) are stored in the ddocs directory in the cht-core repository, I’m not sure this would work for this use case. It would be relatively straightforward to add Mango indexes to the same process, but the larger issue is that because they are stored in the cht-core repository, those indexes would be expected to be relevant to all cht apps, not just to one particular application. Indexes have a significant performance footprint in Couchdb, so we would need to be careful about adding new indexes that affect all CHT deployments.

So to add indexes that are specific to a particular CHT application I think would require a change to cht-conf?
To allow modifying the design docs and adding new indexes which are stored in an application configuration repository, rather than cht-core.

[diana]

Hi @Anro

I we were to add indexes to core, then yes, we would include them in the ddocs section, because these indexes would need to be tracked by version and be updated with CHT-Core upgrades.

The CHT-Core does not use mango indexes now. Due to historical poor performance, lack of implementation for PouchDb and other reasons.

If you wish to add your own indexes, these will probably need to be:

1. part of your configuration and uploaded when you upload config
2. be stored in a separate ddoc, rather in a CHT-Core ddoc - as these ddocs will be overwritten with upgrades.

I think it would be a really cool addition to cht-conf to have custom ddocs / indexes for deployments to use.

[Anro]

​Thank you, @twier and @diana, for your prompt and insightful responses. We really appreciate the information.​

To offer some context on our project structure: we incorporate the CHT as a submodule within our project. During the build process, we apply our custom patches to the CHT code, subsequently creating Docker images or hosting the components via Node.js (on Azure it works a little different, but I’ll get into that on the appropriate thread). We do not perform upgrades through the site; in fact, I believe this is restricted on Nginx. When updating our submodule, we undertake a meticulous patch review process. In theory, the views and indexes we create should persist. For instance, we’ve modified the “contacts_by_freetext” and “contacts_by_type_freetext” views to address issues related to duplicate data capture, as discussed in this thread:

Mitigate duplicate data capture

I think (have not tested) it is just a matter of changing this check in the CouchDB view and then this value in the app code. Like Gareth said, this will have a significant impact on the size of this Couch view and the subsequent query performance. (And, this view will have to be re-indexed. This will take a long time for any non-trivial data set.) It is hard to be sure (since it seems like the challenges for stuff like this are always the little things you don’t anticipate), but I briefly …

That said, we aim to maintain alignment with the CHT as much as possible and are intrigued by the prospect of uploading indexes through cht-conf.​

Due to other pressing commitments, we won’t have the time to delve into this work just yet. As an interim measure, I attempted to create a view to facilitate backend requests with a similar outcome. However, the view did not upload as expected. I added ‘doc_by_id_and_type’ to medicDB/medic/views/, but upon checking Fauxton locally, it appeared that this view was not present in the database. Am I overlooking something in this process?

[diana]

Hi @Anro

I believe CouchDb has some documentation on how to upload indexes: 1.3.6. /{db}/_find — Apache CouchDB® 3.4 Documentation
The index is not a view, where in Fauxton did you check?

[Anro]

Hi @diana,

My apologies for the confusion earlier. I was able to upload indexes just fine using the curl command mentioned in my original post. However, since that has to be done manually for now, I decided to try creating a view that mimics the functionality of the index (since, after all, an index uses a view-like mechanism under the hood).

ddocs/medic-db/medic/views/doc_by_id_and_type:

function(doc) {
  const type = doc.type === 'contact' ? doc.contact_type : doc.type;

  if (
    type === 'form' || type === 'translations' || type === 'meta' ||
    doc._id === 'branding' || doc._id === 'extension-libs' || doc._id === 'resources'
  ) {
    return;
  }

  emit([doc._id, type]);
}

Unfortunately, this additional view didn’t upload along with the others. I might be missing something - any suggestions would be appreciated.

We hoped to query _changes on this view directly, but that does not seem to be possible. We then thought about using it as a view-based filter. Still targeting the medic database, but querying via _changes?filter=_view&view=my_view. I don’t believe this will improve performance. As the name suggests, it’s just a filter rather than a pre-built index used to optimize query speed. So, while functional, it doesn’t provide the efficiency benefits we were aiming for.

Still, it’s good to know how to properly add views, in case we need that flexibility in the future.

Some additional info:

1.3.12. /{db}/_changes — Apache CouchDB® 3.4 Documentation

Change Notifications

---

For reference, here’s how I verified the presence of indexes and views in Fauxton:

To view Mango indexes:

1. Navigate to the database in question.
2. Click the “+” next to Design Documents.
3. Then click on Mango Indexes.

To check views:

1. Navigate to the desired database (e.g., medic, medic-admin, etc.).
2. Expand the relevant design document.
3. Click on Views.
4. Select the view you want to inspect.

[diana]

@Anro

Unfortunately, this additional view didn’t upload along with the others. I might be missing something - any suggestions would be appreciated.

I’m not sure which others
From your post, it looks like you are using the medic ddoc to keep your views. I would advise against that, as API will potentially overwrite these ddocs at startup and will definitely overwrite these ddocs on upgrade, which will effectively remove your view.
Please upload your views into a different ddoc (create your own).

[Anro]

The functionality to consume index config as part of the project setup is ready for review .

[Anro]

Index config consumption is now supported with the cht-conf tool .