feat(clickhouse): Add clickhouse-connect (HTTP) driver behind a runtime config [EAP-497]

[phacops]

Linear: EAP-497 — Evaluate clickhouse-driver upgrade

Summary

Introduces an HTTP-based ClickHouse client backed by clickhouse-connect as a drop-in alternative to the native clickhouse-driver protocol, selectable at runtime.

The goal is to migrate off the native TCP protocol and onto HTTP via clickhouse-connect, reusing the connection pooling and retry machinery that library gives us for free, and to do it incrementally so it can be rolled out (and rolled back) without a deploy.

Architecture

The two pools derive from an abstract base, so the native and HTTP drivers are siblings. There is a single, driver-agnostic reader.

Pools (ClickhousePool is the abstract base — the name is kept as the abstract type so existing annotations are unchanged):

• ClickhousePool — abstract base (ABC) declaring execute / execute_robust / close and the common attributes.
• ClickhouseNativePool (native.py) — native protocol implementation (the renamed former ClickhousePool).
• ClickhouseConnectPool (connect.py) — HTTP via clickhouse-connect; connects on the node's HTTP port (default 8123, env CLICKHOUSE_HTTP_PORT) — the same port HTTPBatchWriter uses for inserts.

Reader:

• ClickhouseReader (native.py) — the one reader. It adapts ClickhouseResult → Result and is driver-agnostic: it wraps the abstract ClickhousePool, so the same reader serves both drivers (both return the same ClickhouseResult). No native/HTTP reader split is needed.

Node: ClickhouseNode carries both native_port and an optional http_port, so a node is self-describing for either driver. Cluster-produced nodes (the query node and those discovered via system.clusters, which only reports the native port) are stamped with the cluster's HTTP port.

Acquisition flow: ConnectionCache is the single place pools are instantiated and the single place the driver is selected. Every caller goes through it and gets one shared, cached, runtime-selected pool behind the abstract ClickhousePool type:

• ClickhouseCluster holds the ConnectionCache; get_node_connection() delegates to ConnectionCache.get_node_connection(), and get_reader()/get_deleter() wrap the returned pool in ClickhouseReader. The reader is built per query (get_reader() is called per execution), so it picks up the current driver each time and just holds the abstract pool — it does not touch the cache itself.
• The admin by-host helpers (admin/clickhouse/common.py) and the cleanup / optimize / querylog_to_csv CLIs also acquire their pools through ConnectionCache.get_node_connection() rather than constructing one directly, so they get the same runtime-selected driver and the same shared singleton.

ConnectionCache.get_node_connection() reads the use_clickhouse_connect_driver runtime config itself and constructs the concrete ClickhouseNativePool / ClickhouseConnectPool accordingly (connecting on the node's native_port or http_port), caching both variants side by side (the driver is part of the cache key). Because every connection (reads via get_query_connection/get_reader, and migrations, replacer, optimize, admin, CLIs …) goes through get_node_connection, the choice applies everywhere, and ConnectionCache only ever exposes the abstract ClickhousePool.

Class hierarchy

classDiagram
    class Reader {
        <<abstract>>
    }
    class ClickhouseReader {
        driver-agnostic
        ClickhouseResult to Result
    }
    Reader <|-- ClickhouseReader

    class ClickhousePool {
        <<abstract>>
        execute / execute_robust / close
    }
    class ClickhouseNativePool {
        native protocol (clickhouse-driver)
    }
    class ClickhouseConnectPool {
        HTTP protocol (clickhouse-connect)
    }
    ClickhousePool <|-- ClickhouseNativePool
    ClickhousePool <|-- ClickhouseConnectPool

    ClickhouseReader o-- ClickhousePool : client (abstract)

    class ConnectionCache {
        single place pools are instantiated
        selects driver from runtime config
        get_node_connection returns abstract ClickhousePool
    }
    class ClickhouseCluster {
        get_node_connection delegates to cache
        get_reader wraps pool in ClickhouseReader
    }
    class AdminAndCliHelpers {
        admin common.py
        cleanup / optimize / querylog CLIs
    }
    ClickhouseCluster --> ConnectionCache : holds / gets pool via
    AdminAndCliHelpers ..> ConnectionCache : gets pool via
    ConnectionCache ..> ClickhousePool : creates and caches
    ClickhouseCluster ..> ClickhouseReader : wraps pool in

Loading

ClickhouseCluster (and the admin/CLI helpers) only ever touch the abstract ClickhousePool and the single ClickhouseReader; the native-vs-connect choice lives entirely inside ConnectionCache.

Other changes:

• snuba/settings/__init__.py — new USE_CLICKHOUSE_CONNECT_DRIVER default (off).
• pyproject.toml / uv.lock — add the clickhouse-connect dependency (+ lz4, zstandard) and mypy overrides.
• Pool acquisition funneled through ConnectionCache — the cluster, the admin common.py helpers, and the cleanup/optimize/querylog CLIs all get pools via ConnectionCache.get_node_connection, so the driver is runtime-selected behind the abstract type everywhere; only the cache constructs ClickhouseNativePool / ClickhouseConnectPool. The admin credential-leak AST regression guard now treats both direct pool construction and cache acquisition (get_node_connection) as "acquiring a pool", so _validate_node must still run first, and the admin connection caches are driver-aware so runtime switching reaches admin too.

Timeouts

Timeouts are driven by the per-profile ClickhouseClientSettings timeout, applied identically by both drivers (the connect pool honors it as-is — no blanket cap):

• Read queries (QUERY profile): 30s — set in the shared profile, so it applies to both drivers.
• Migrations (MIGRATE), OPTIMIZE, and other long-running operations keep their existing (default or longer) timeouts.
• Profiles without a timeout fall back to a 300s default on the HTTP path.

Note: setting the read timeout in the QUERY profile also gives the native read path a 30s socket timeout (previously unset) — intended, but a behavior change for native worth calling out.

Exception handling

Every clickhouse-connect error inherits from ClickHouseError (the analog of clickhouse_driver's errors.Error); the connect pool catches that base — DatabaseError, ProgrammingError, DataError, etc. — and re-raises a snuba ClickhouseError preserving the server code, exactly as the native pool wraps the errors.Error family. Connection/transport failures (OperationalError) additionally emit the connection_error metric.

Connection pool sizing

The HTTP (urllib3) pool size is always taken from the clickhouse_connect_pool_size runtime config, falling back to CLICKHOUSE_MAX_POOL_SIZE (same default as native). It is not a per-caller construction parameter: there is a single shared pool per connection (instantiated by ConnectionCache), sized from the config and read once when the cached client is first created — so the size can be tuned at runtime.

Retry behavior

The connect pool does not reimplement the native pool's retry logic — all retries are delegated to clickhouse-connect (stale keep-alive sockets, transport errors, HTTP 429/503/504). Deliberately dropped: the native TOO_MANY_SIMULTANEOUS_QUERIES (code 202) backoff — over HTTP ClickHouse returns it as a 500 + X-ClickHouse-Exception-Code: 202 header, which clickhouse-connect doesn't retry, so it's surfaced directly as a ClickhouseError preserving the code. execute_robust is therefore a thin alias for execute; retryable is kept only for interface parity.

How to roll out

Controlled by the use_clickhouse_connect_driver runtime config (default USE_CLICKHOUSE_CONNECT_DRIVER, off). get_node_connection() reads it per call (≤10s config memoization), so flipping it takes effect without a restart and applies to every connection.

Runtime configs added

• use_clickhouse_connect_driver — route connections through clickhouse-connect (HTTP). Default off.
• clickhouse_connect_pool_size — the HTTP connection pool size. Default CLICKHOUSE_MAX_POOL_SIZE.

Dependency / lockfile

clickhouse-connect (1.3.0, plus lz4 and zstandard) is now in the internal PyPI mirror (getsentry/pypi#2273), and uv.lock is regenerated to include it — so the dependency-install CI jobs can resolve it.

Known follow-ups (from automated review)

• WITH TOTALS over HTTP is covered by a unit test (test_with_totals_handled_over_http): clickhouse-connect's Native-format reader concatenates the trailing totals block into result_set, so the trailing row is the totals row and the driver-agnostic reader's "totals = last row" assumption holds on the HTTP path too. Worth a final confirmation against a live server before relying on it. (If HTTP totals ever turn out to differ, that's the point at which a connect-specific reader subclass would be reintroduced.)
• Tracing / trace_output: the admin API is driver-agnostic for query execution, but the admin trace view and its profile-events parsing depend on trace_output (server send_logs_level logs). clickhouse-connect doesn't surface those (it only reads the X-ClickHouse-Summary header for the profile), so trace_output is empty on the HTTP path and those two admin features return nothing when the connect driver is enabled. Accepted for now (documented in connect.py); reconstructing traces over HTTP would mean querying system.text_log by query_id — a separate feature.
• No fail-fast pool-get timeout equivalent (CLICKHOUSE_POOL_GET_TIMEOUT_SECONDS) on the HTTP path.

Testing

• New tests/clickhouse/test_connect.py (incl. result/profile mapping, error wrapping, timeout passthrough, pool sizing, the totals-over-HTTP test, and the type-name→reader-transform test), a driver-selection test in tests/clusters/test_cluster.py, the admin credential-leak AST guard, and existing native/reader tests pass.
• mypy (strict) and ruff pass on the changed files (and the CI pre-commit check).
• A live-ClickHouse-over-HTTP end-to-end test was not possible in the dev sandbox; the clickhouse-connect API usage was validated against the installed library.

🤖 Generated with Claude Code

[cursor]

Debug trace logs always empty

Medium Severity

On the HTTP path, ClickhouseResult.trace_output is always set to an empty string even when per-query settings include send_logs_level trace. The native pool captures driver trace logs in that case and returns them in trace_output, which RPC debug metadata relies on.

 

^{Reviewed by Cursor Bugbot for commit 2cfaab9. Configure here.}

[cursor]

HTTP pool lacks get timeout

Medium Severity

ClickhouseConnectPool.execute sends every query through a shared HTTP client with no equivalent of the native pool’s bounded queue and CLICKHOUSE_POOL_GET_TIMEOUT_SECONDS fail-fast. Under load or a slow ClickHouse, workers can block waiting for urllib3 pool slots instead of failing after the configured pool-get timeout like ClickhousePool does.

 

^{Reviewed by Cursor Bugbot for commit 1e8affa. Configure here.}

[cursor]

Zero error code becomes negative

Low Severity

HTTP error wrapping uses getattr(e, "code", None) or -1, so a legitimate ClickHouse error code of 0 is stored as -1 instead of 0, unlike the native pool which passes through e.code directly.

 

^{Reviewed by Cursor Bugbot for commit 1e8affa. Configure here.}

[cursor]

None timeout breaks pool default

Medium Severity

ConnectionCache now passes send_receive_timeout=timeout even when the profile timeout is None. That overrides ClickhousePool's intentional 35s default with None (native driver default), changing socket timeouts for profiles like CLEANUP, INSERT, and REPLACE on every deploy—not only when the HTTP driver is enabled.

Additional Locations (1)

• snuba/clickhouse/connect.py#L130-L135

 

^{Reviewed by Cursor Bugbot for commit a87be31. Configure here.}

[cursor]

HTTP params placeholder mismatch

High Severity

The HTTP pool forwards parameters to clickhouse-connect while Snuba SQL still uses %(name)s placeholders written for clickhouse-driver. clickhouse-connect expects {name:Type} in the query text, so parameterized statements can fail or run with wrong bindings once use_clickhouse_connect_driver is enabled (e.g. system.clusters discovery on multi-node clusters).

Additional Locations (1)

• snuba/clusters/cluster.py#L536-L541

 

^{Reviewed by Cursor Bugbot for commit fac0556. Configure here.}

[linear-code]

EAP-497

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 6 total unresolved issues (including 5 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 9a71608. Configure here.}

[cursor]

Admin pool size limit removed

Medium Severity

Admin ClickHouse connections no longer cap the native pool at two connections. _build_validated_pool now uses the shared connection_cache, which builds pools with the default CLICKHOUSE_MAX_POOL_SIZE (25) or the HTTP runtime pool size, so targeted admin nodes can see far more simultaneous connections than before.

 

^{Reviewed by Cursor Bugbot for commit 9a71608. Configure here.}

[phacops]

This is intentional, not an oversight. Earlier in the PR the admin helper built its own ClickhouseNativePool(..., max_pool_size=2); we deliberately removed that per-caller cap so admin acquires the one shared, runtime-config-sized pool from ConnectionCache like every other caller (the connect pool always sizes from clickhouse_connect_pool_size, the native pool from CLICKHOUSE_MAX_POOL_SIZE). Keeping a hardcoded 2 would mean reintroducing the per-caller max_pool_size plumbing through the cache that we just took out, and would give admin a different-sized pool than everything else hitting the same node.

Admin is low-traffic (human operators running system queries / tracing), so sharing the standard sizing is acceptable, and pool size is now runtime-tunable via clickhouse_connect_pool_size if it ever needs limiting. Leaving as-is.

---

Generated by Claude Code