Skip to content

feat: add embedded chDB driver (driver: chdb) - #693

Open
wudidapaopao wants to merge 7 commits into
ClickHouse:mainfrom
wudidapaopao:chdb-driver
Open

feat: add embedded chDB driver (driver: chdb)#693
wudidapaopao wants to merge 7 commits into
ClickHouse:mainfrom
wudidapaopao:chdb-driver

Conversation

@wudidapaopao

@wudidapaopao wudidapaopao commented Jul 27, 2026

Copy link
Copy Markdown

Summary

Adds an experimental driver: chdb that runs dbt against chDB — ClickHouse as an in-process engine — with no server, Docker.

Continues the work started in #369 and closes #297. It builds on clickhouse-connect's pluggable backend (ClickHouse/clickhouse-connect#872, released in 1.6.0), which exposes the chDB engine behind get_client(interface="chdb") — so the driver reuses the existing HTTP client surface and only changes how the connection is created.

Also fixes clickhouse__drop_relation so it drops temporary tables. This affects all drivers.

Usage

pip install "dbt-clickhouse[chdb]"
my_project:
  target: ci
  outputs:
    ci:
      type: clickhouse
      driver: chdb
      chdb_path: .chdb   # data directory; ':memory:' (default) for ephemeral
      schema: my_db
    prod:                # production still points at a real server
      type: clickhouse
      driver: http
      host: ...

The embedded engine is single-node, so cluster, cluster_mode, Replicated, and the distributed materializations are rejected. The full integration suite runs against the driver in a new no-Docker CI job (DBT_CH_TEST_DRIVER=chdb).

Checklist

  • Unit and integration tests covering the common scenarios were added
  • A human-readable description of the changes was provided to include in CHANGELOG

The macro rendered a schema-qualified DROP, which never matches a TEMPORARY
table (they live outside databases), so temp tables were never dropped. Add
an is_temporary branch emitting DROP TEMPORARY TABLE.
@wudidapaopao wudidapaopao changed the title feat: Add embedded chDB driver (driver: chdb) feat: add embedded chDB driver (driver: chdb) Jul 27, 2026
Comment thread dbt/adapters/clickhouse/chdbclient.py Outdated
Comment thread dbt/adapters/clickhouse/chdbclient.py
Run dbt against in-process chDB via clickhouse-connect 1.6.0's built-in chdb
backend — no server or Docker, for CI and local development. Adds a chdb_path
profile field and a dbt-clickhouse[chdb] extra; all dbt threads share one
engine client (chDB is one engine per process). Cluster/replicated/distributed
configs are rejected. Includes a chdb test profile (DBT_CH_TEST_DRIVER=chdb)
and a no-Docker CI workflow.
Comment thread dbt/adapters/clickhouse/chdbclient.py
Comment thread dbt/adapters/clickhouse/chdbclient.py
When the shared chdb engine is replaced by a different path, clear the
process-wide server-state caches (ensured databases, capability probes) so the
new engine is not judged by the previous one's state. Keeps the logic in the
chdb module instead of dbclient.
Comment thread dbt/include/clickhouse/macros/adapters.sql
…se on drop

chDB is one engine per process. Instead of switching the shared engine (which
left stale handles/caches), reject a second path with a clear error. Override
database_dropped so a schema drop on one thread doesn't clear the shared
client's default database for the others.
Comment thread dbt/adapters/clickhouse/chdbclient.py
@wudidapaopao

wudidapaopao commented Jul 29, 2026

Copy link
Copy Markdown
Author

Hi @BentsiLeviav @koletzilla, this revives #369. With the built-in chDB backend in clickhouse-connect 1.6.0 (ClickHouse/clickhouse-connect#872), wiring chDB into dbt is now much simpler — the driver reuses the existing client surface and only swaps how the connection is created.

Would appreciate a review when you have a chance.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit 04c7679. Configure here.

Comment thread dbt/adapters/clickhouse/credentials.py
The file merged in from main was not ruff-formatted (implicit string
concatenation); ruff format joins it. Fixes the check-ruff-format CI step.
Replicated engines are written with arguments, e.g. Replicated('/path', ...),
so an exact '== Replicated' check missed them. Use a case-insensitive
substring match, matching relation.py's existing convention.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support chdb

1 participant