Announcing the release of 2.0.0 of CHT Sync

Product Releases
1

We’re excited to announce the release of 2.0.0 of CHT Sync

Highlights

Improved dbt Performance Tuning

CHT Sync V2 introduces several configuration options to improve dbt performance and control for large-scale deployments:

  • Multithreading: dbt can run independent models concurrently using threads.
  • Batch Processing: Large inserts can be broken up into smaller batches to improve perfromance and observability.
  • Multi-container dbt Support: Instead of running all models in one container, V2 allows splitting dbt workloads across multiple containers. Each container can be configured with a DBT_SELECTOR (e.g., package:cht_pipeline_base or tag:custom-models) to run only a subset of models.

Read the full tuning guide »

Support for multiple CHT instances and multiple CouchDB databases

  • CHT Sync V2 adds support for syncing multiple CHT instances, multiple CouchDB databases, to one PostgreSQL database.
  • This was possible in V1, but the source, host, and database were not separated into their own columns.

Changes to docker compose and local developemnt

  • V2 makes it easier to develop dbt models locally, without requiring a separate dbt configuration, or a remote github repository.
  • The docker compose files in V2 were changed to use profiles instead of separate docker compose files to handle different deployments environments.

Upgrade Notes

See upgrading notes for detailed upgrade instructions.

Breaking changes

V2 added the source column to the couchdb table, and several other columns to the metadata table. To add these columns log in to the database and run this sql.

Configuration Changes

  • Add DBT_THREADS, DBT_BATCH_SIZE, and DBT_SELECTOR to tune performance and customize dbt execution.
  • When using Kubernetes, update your values.yaml to define multiple dbt containers with dbt_selectors.
6 Likes