feat(cache): shared redis cache, snapshot keys, uncached write pipes

[EricAndrechek]

Summary

This PR makes the query cache correct under concurrent writes and shareable across instances. It folds in #621, #634, #626 and #630, which were reviewed separately against this branch.

• Version snapshot at lookup (fixes Cache: result can be keyed to a newer version than the data it reflects (write-during-query staleness) #382). The cache.Cache interface is now Lookup(ctx, tenant, sha, deps) (Entry, Snapshot, error) and Set(ctx, Snapshot, value, ttl). A result's versions are read once, before its query runs and before the handler takes the tenant's ClickHouse pool, and the fill is filed under what was read. Previously POST /v1/query and pipe execution rebuilt the version-folded key after the query, so an insert that landed mid-query filed pre-insert rows under post-insert versions and they were served as fresh until their TTL. A reload that moves a tenant to another address or database now orphans a fill taken from the old pool the same way. The singleflight key and coalescing are unchanged.
• A tenant token on every key. Every entry key folds the tenant's version, a pipe's dependency-free key included, so InvalidateTenant now drops a returning or moved tenant's cached pipe results too. A Lookup whose deps name another tenant is refused (ErrForeignDependency). cache.Namespace carries raw table and scope names and the cache escapes them itself with internal/keyenc, so query.SafeEncodeToken is gone and names that would run together under an unescaped join can no longer share a key.
• Flat local version index (part of perf(cache): cache version-manager map grows unbounded with tables × scopes #262). VersionManager holds one version per tenant, per (tenant, table) and per (tenant, table, scope), bumped in place, so the index no longer grows with every bump. A tenant's version is a process-unique generation: InvalidateTenant drops the tenant's index and the next key gets a fresh generation. After every settings reload, LocalCache.Prune drops the index of each tenant no longer served.
• Write pipes run uncached (fixes bug(pipes): mutation pipe results are cached and coalesced — the write silently drops on repeat calls #386). A pipe whose bound SQL IsMutation classifies as a write skips the cache lookup, the fill and singleflight, and runs on every call. Before, a repeat within the TTL answered 200 without writing, and N concurrent identical calls became one write. The classifier now reads the leading keyword the way ClickHouse's lexer does (comments, quoted text, heredocs, the whitespace ClickHouse accepts), classifies a WITH-led statement by INSERT INTO alone, and looks through EXECUTE AS. An integration test checks every case, and every keyword in system.keywords in 12 WITH shapes, against ClickHouse's own parser.
• A Redis-compatible shared cache backend. cache.RedisCache runs against Redis, Valkey, Dragonfly, ElastiCache and MemoryDB, standalone or cluster, using only GET, SET and MGET. Versions are random 8-byte tokens under the tenant's hash tag, and a lookup is one round trip. A lost token can only cause a miss. Values of 1 KiB or more are zstd-compressed, and stored values are capped at 1 MiB. Every operation has a 100 ms timeout, and a failure is a miss, a skipped fill or a deferred invalidation, never a failed query. A circuit breaker opens after 5 consecutive failures, or at once on a reply that refuses writes (READONLY, OOM, …) or the credentials, and only a successful probe write closes it. Deferred invalidations are retried until they land, and while a process owes one it bypasses the lookups that invalidation would orphan. Eight wavehouse_cache_* metrics, all labeled backend="redis", report hits, round-trip time, breaker state, owed invalidations, value size and failed fills.
• cache.backend: redis. A new cache.redis boot-config block (WH_CACHE_REDIS_*): addrs, mode (standalone or cluster), credentials, db, TLS files, key_prefix, timeout and dial_timeout (each capped at 1s), max_value_bytes, compress_min_bytes and version_ttl. wireCache builds the backend from it. It is the shared cache that splitting the api and ingest roles into separate processes needs, and the boot error for such a split now names it; every split is still refused while the queue is embedded. The e2e suite now runs on Redis.

Behaviour and compatibility notes

• Write pipes answer X-Cache: BYPASS with Cache-Control: no-store and are never coalesced. Each call executes, so identical concurrent calls are that many writes. A failed write keeps the read path's status and code but always answers retryable: false with no Retry-After, since the statement may have run. Read pipes are unchanged.
• A write pipe does not invalidate cached reads of the table it writes (bug(query): admin mutations via /v1/admin/query don't invalidate the structured-query cache — stale reads for up to an hour #394, and feat(pipes): resolve table dependencies for cache invalidation #343 for read pipes), and its rows do not reach /v1/stream subscribers (SSE/live-query yields no events for tables populated by a ClickHouse MATERIALIZED VIEW (only ingest-API writes republish to NATS) #362).
• InvalidateTenant drops more than before: a tenant's cached pipe results as well as its query results. Inserts still do not reach pipe results (feat(pipes): resolve table dependencies for cache invalidation #343).
• In-process cache keys changed (the caller's query key is now escaped inside the entry key). They are in-process only, so a restart is the whole migration.
• Shared-cache token keys are a protocol between builds. Every process sharing a server reads and bumps them for itself, so a later change to that layout needs a rolling-upgrade plan. A change to value keys only orphans entries and is safe to roll.
• Boot with Redis down or refusing the password succeeds, degraded. The cache starts bypassed and keeps reconnecting. When a closed breaker opens, it logs one WARN, or one ERROR for rejected credentials. A failed probe reopening it logs at DEBUG, unless it failed for another cause than the one last logged (rejected credentials after a restart, say), which is logged at its own level. A long outage is one line.
• mode: sentinel refuses boot until bug(cache): a demoted Redis primary reads as healthy; Sentinel mode is unconfigured #656. A URL-style address is refused without echoing it, and a standalone server takes exactly one address.
• The ingest worker logs an invalidation that did not land at WARN, not ERROR: the shared backend defers and retries it. wavehouse_cache_invalidations_pending is the signal to alert on.
• Run the server with an evicting maxmemory-policy and without persistence. Under noeviction a full server refuses the token writes. Restoring a snapshot, or a crash-restart that reloads the last save, is a rollback that serves previously invalidated entries until their TTL. The deployment guide covers both.
• Known follow-ups:
  • security(pipes): a quoted placeholder lets a value break out of its literal #662: a quoted placeholder lets a bound value break out of its literal.
  • bug(pipes): a write pipe answers GET, which proxies and clients replay #663: a write pipe answers GET, which proxies and clients may replay.
  • bug(pipes): a BACKUP/RESTORE/UNDROP pipe runs as a read and is cached #666: BACKUP, RESTORE, UNDROP and MOVE pipes are not classified as writes.
  • security(pipes): SET/USE/EXECUTE AS in a pipe leak into the pooled session #671: SET, USE and EXECUTE AS in a pipe leak into the pooled session.
  • Also still open: per-table scope cardinality (perf(cache): cache version-manager map grows unbounded with tables × scopes #262, until Clarify/implement scope (currently always empty) #235 populates scope), the rest of bug(cache): Redis recovery probe must fit a fresh dial in the op timeout #664 (the probe reconnects one connection of several), Sentinel (bug(cache): a demoted Redis primary reads as healthy; Sentinel mode is unconfigured #656), and the near-cache.

Tests

• Conformance suite internal/testutil/cachetest.Run: miss, hit and TTL, dependency order, tenant isolation, foreign deps, the scope lattice, raw names that would run together, Invalidate/InvalidateTenant, a bump during the query (Cache: result can be keyed to a newer version than the data it reflects (write-during-query staleness) #382), oversize values, zero snapshots, and concurrent use under -race. Shared backends also get two-instances-over-one-store cases. LocalCache runs it, and so does RedisCache against pinned Redis, Valkey, Dragonfly and a Redis Cluster node.
• Cache: result can be keyed to a newer version than the data it reflects (write-during-query staleness) #382: on both cached routes, a bump from inside the ClickHouse call, and one from inside the pool lookup, each give MISS, MISS, HIT. The read's namespace and the ingest worker's bump are pinned to meet for raw table names.
• Flat index: 10,000 rounds of interleaved bumps and key reads leave the index at its settled size. Generations never repeat, a table bump drops its scopes, and a bump against a tenant with no index records nothing. A reload prunes the index to the tenants still served.
• Write pipes: INSERT, WITH … INSERT and ALTER … DELETE pipes each run on every call. Three identical concurrent calls are three writes in flight. A read pipe over a table named like a write verb stays cached. Every row of the ClickHouse error table on a write pipe answers retryable: false. Over the Redis-backed e2e stack, a write pipe called twice leaves both rows, and a failed one answers 400 clickhouse.rejected with retryable: false. There are 152 table-driven classifier cases, each also checked against EXPLAIN AST on the pinned ClickHouse, and 17,928 keyword-named WITH statements where the classifier must agree with the parser.
• Redis backend (integration): lost tokens miss, and so does a flushed server. On a paused server, lookups fail within the bound, the breaker opens, invalidations are deferred, and all of it recovers. An owed bump holds its lookups. Refused writes (READONLY, OOM) open the breaker at once. A failover behind a stable address delivers the owed bump. A cluster topology read is bounded. A slow reconnect still closes the breaker, and a slow server stays bypassed. Rotated credentials open the breaker. A restored snapshot behaves as a rollback. Unit tests cover the key schema, the codec and its zip-bomb refusal, the breaker state machine, pending coalescing and collapse, and the breaker logging each opening once and a changed cause at its own level.
• Config and wiring: defaults, env, YAML, the validation table, URL-style addresses (neither the address nor the secret is echoed), boot against a closed port (bypassed, not failed), and an unreadable TLS file refusing boot. Two app.New instances over one Redis: an ingest on one invalidates the other, and with Redis paused, queries bypass and still succeed.
• Most behavioural tests are mutation-checked: each fails with the fix removed.
• make ci passes: static checks, unit, integration, e2e on Redis, and coverage.

Fixes #382. Fixes #386. Part of #262. Part of #613.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FyrXjhR7iDg33paioLQHFq

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: e9d137d1-29f6-4933-bd93-f93d5b264f96

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}