feat(typelayer)!: validate, coerce and filter through chtypes

[EricAndrechek]

Reference PR — not for merge. This branch is based on main at e462135 (2026-09-17), before the multi-tenant work landed. It is kept open as a draft so the full migration and its test evidence stay readable in one place. The migration has been redone on current main in #712; review that one.

Summary

WaveHouse stops modelling ClickHouse itself. Validation, coercion, DEFAULT substitution, timestamp parsing, row-level security and insert checks run through chtypes (github.com/wave-rf/chtypes/go v0.4.0, ABI revision 6): a cgo dlopen of a per-ClickHouse-version shared library that runs the server's own parser and analyzer in-process. The hand-written Go that used to approximate ClickHouse is deleted.

Breaking, by design: cgo is unconditional (no pure-Go build), supported platforms narrow to linux/amd64, linux/arm64 and darwin/arm64, the Linux binaries are dynamically linked (glibc ≥ 2.34), and several documented behaviours move to "whatever ClickHouse does" (listed below).

What changed

Ingest — the request body goes to chtypes as-is. Content-Type declares the format: the JSON family (object, array, NDJSON are one format), text/csv and text/tab-separated-values. The RFC 4180 header parameter maps three ways: header=present reads CSVWithNames/TSVWithNames, header=absent reads positionally with ClickHouse's header detection off, and no parameter keeps ClickHouse's own default (it detects a header line); any other value is 415. One parse per body: RowsExportWith with a row filter returns each record's parse outcome and its check verdict together, so there is no Go-side decode, re-encode or second parse. Column policy and _eq auto-inject are per-role compiled schemas; check clauses and row filters are chtypes filters over the parsed rows with every claim bound as a String parameter. Per-record rejections carry ClickHouse's code. The published row is the exact bytes ClickHouse's own writer produced, so SSE, the table and /v1/query agree by construction (#372).

Integer claims — a claim bound as a plain {p:String} and compared against an integer column wraps modulo 2^64 (and at their own width on [U]Int128/[U]Int256), on the server and in chtypes alike, so a claim of 18446744073709551621 read and wrote tenant 5. Row filters, insert checks and the /v1/query policy predicate now compare integer columns (Nullable/LowCardinality included) against if(toString(accurateCastOrNull({p:String}, 'T')) = {p:String}, accurateCastOrNull({p:String}, 'T'), NULL): an in-range canonical claim answers exactly as before and keeps the primary key in use; an out-of-range or non-canonical claim (007, +5, 1.0) matches no row and fails an insert check with 403. Ingested data is unaffected ("007" still stores 7).

Query path — /v1/query and pipes ask ClickHouse for FORMAT JSONEachRow over HTTP and pass the bytes through; the 424-line result transformer is gone.

Identifiers and literals — chsql.QuoteIdent is byte-identical to ClickHouse's backQuote (a test compares it with the SDK's QuoteIdentifier over every byte); DDL literals use the SDK's QuoteLiteral.

Build/CI/release — CGO_ENABLED=1 everywhere, go 1.27, glibc builder + distroless/cc runtime that bakes the pinned artifact (chtypes.lock, scripts/fetch-chtypes.sh --frozen; the lock pins the revision-6 26.6.8.7 build), golangci-lint v2.13.2, audit-cgo removed. Releases build each target natively on ubuntu-latest, ubuntu-24.04-arm and macos-latest (cgo darwin cannot be cross-built from Linux: the Prometheus client's darwin collector needs a Mach header); one job assembles archives, checksums, provenance, the GHCR image and the Release.

Deleted — internal/discovery/{validation,timestamp}.go, internal/ingest/compact.go, internal/policy/{rowfilter,numeric}.go, internal/api/{record_reader,ingest_seams,clickhouse_exec}.go, policy.LiteralValue/CanonicalNumericLiteral, the two differential-oracle integration tests, every CGO_ENABLED=0 path.

Size, measured — production Go is 17,114 lines at the base commit and 18,007 here (non-test .go outside tests/): the migration adds about 890 lines (695 at this PR's first push). An earlier version of this description said "about 800 lines fewer than main"; that counted only some files and was wrong. The migration trades hand-written ClickHouse modelling for a wrapper around the real thing; it does not shrink the codebase.

Documented contract changes (please review these first)

surface	before	after
denied insert column	403 column "x" not allowed for insert	400 {"code":117,"error":"Unknown field …"} (does not reveal whether the column exists)
unknown field / EPHEMERAL / ALIAS / MATERIALIZED supplied	WaveHouse wording	ClickHouse code 117
_in check on an absent column	403	the table default is tested
parse error + check failure on one record	403	the 400 with ClickHouse's code
out-of-range or non-canonical integer claim (007, +5, 1.0, ≥ 2^64) on an integer column	wrapped, or accepted	matches no row on a read; 403 on an insert check
explicit null on a checked column	403	same as omitting it (takes the injected claim)
text/csv / text/tab-separated-values ingest	not accepted	positional; header=present / header=absent / no parameter (ClickHouse detects a header)
wire / SSE / query timestamps	RFC 3339 Z	ClickHouse's rendering, e.g. 2026-06-21 04:00:00.123
/v1/query Decimal	JSON string	JSON number (the SDK docs already said so)
/v1/query key order	alphabetical	SELECT order
/v1/query null filter value	silent empty result	400
/v1/query in list	unbounded	one Array(String) param, ~6,000-element ceiling
unparseable timestamp on ingest	passed through, failed at the worker	400 with ClickHouse's code at the edge
"missing required column"	400	ClickHouse accepts and stores the type zero
SSE withheld-reason label for an unreadable numeric filter constant	decline	filter (the strict cast makes it a non-match)
compact JSON array with one bad record	whole batch lost (ClickHouse's answer)	the other records still ingest

Verification

• make ci green on this exact tree: Go unit 1,774 (-race), integration 95, e2e 87 (1 skipped), TS unit 229, Go total coverage 92.6%, no threshold lowered.
• Differential tests against ClickHouse 26.6: the SSE-replayed row equals SELECT … FORMAT JSONCompactEachRow of the stored row; stream row-filter verdicts equal the production /v1/query handler's answer on every column shape × operator × claim, including hostile spellings, escaped values and integer claims at and beyond every width (8,838 cells, zero disagreements); a /v1/query type-rendering pin test.
• CSV/TSV header handling matches a live ClickHouse server on every combination tried (header present, absent, bare; with and without a header-like first line).
• Pre-push reviewers were skipped for this reference push (logged in tmp/review-skips-<sha>.log); they run on the redo PR.

First-run risks on GitHub

• e2e on a cold cache is the long pole (timeout raised to 35 min); if CI is red, read that job's log first and check for a timeout rather than a failure.
• CodeQL default setup will try to build go1.27 with cgo (not a required check).
• First use of ubuntu-24.04-arm and macos-latest (advisory validation job).
• artifacts.wavehouse.dev is a CI dependency on a cache miss.

Known limits (carried into the redo)

• chtypes.Timezone is process-global and fixed when a ClickHouse version's library is first opened; a per-call session_timezone is accepted and ignored. One process serves one server time zone per ClickHouse version.
• Filters over a LowCardinality(<integer>) column are declined by chtypes (fail closed).
• Base-table handle pool is min(GOMAXPROCS, 8); re-measure on deployment hardware.

🤖 Generated with Claude Code

https://claude.ai/code/session_018uEnYtmudjD1nn3T44zuhB

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 7d4e59af-a790-470e-9c9d-0ebd5065f8d7

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[EricAndrechek]

Superseded by #712, which redoes this migration on current main (multi-tenant) with chtypes SDK go/v0.5.2 and ClickHouse 26.8.