feat(cache): shared cache backend on Redis, Valkey and Dragonfly

[EricAndrechek]

Part of #613. Stacked on #614 (feat/cache-snapshot). #614 no longer stacks on #655: refactor/keyenc is merged into main, which #614 includes. The key schema escapes names with internal/keyenc.

Addresses #664 in part: the breaker's probe now fits a reconnect slower than the per-operation timeout, but the client's other connections must still reconnect within it (see "Failure means bypass" and "Deliberately left").

What this adds

cache.RedisCache is a cache.Cache shared by every process pointed at one Redis-compatible server: Redis, Valkey, Dragonfly, ElastiCache or MemoryDB. It sends only GET, SET and MGET. There are no scripts and no client tracking, so ElastiCache Serverless and cluster mode work unchanged.

It is built and tested, but nothing selects it. cache.backend accepts only local until #630 adds redis, the cache.redis.* boot config and the wiring. Every deployment still runs LocalCache.

• Version tokens (redis_codec.go): each tenant, each table and each scope has a random 8-byte token under the tenant's hash tag (<prefix>:{t}:T, :B:<table>, :S:<table>:<scope>). A bump sets a fresh token. A Namespace carries raw names (feat(cache): shared redis cache, snapshot keys, uncached write pipes #614), and the codec escapes the table and scope after the fixed <prefix>:{t}:B: / :S: pieces with keyenc.AppendJoin, so a : in a name never reads as the separator. The {t} hash tag goes in verbatim, as a tenant id is its own escaped form. The subsumption lattice is the same as VersionManager's:
  • a dependency folds its tenant, table and scope tokens;
  • a scopeless write bumps B:;
  • a scoped write bumps S:<scope> and S: (the whole-table view).
• One round trip per lookup: an MGET of the tokens is pipelined with a GET of the value. A value carries the tokens it was filed under, and it is a hit only while all of them are still current. Value keys (<prefix>:q:<t>:<hash of sha and deps>) have no hash tag, so a tenant's values spread across shards. The hash is over the escaped sha and each dependency's escaped, joined table and scope, each ended by a NUL, which escaping never writes. Hashing the raw names let a name holding a NUL share a value key with another set of names (table a\x00b against table a with scope b\x00).
• Lost tokens can only cause misses. A missing token is created with SET NX and read back; it is never read as a value. Eviction, expiry, FLUSHALL or a restart without persistence therefore orphan entries and never revive them, so allkeys-lru is safe. A token or value key that holds something else (a string of another length, a list, a hash) is replaced.
  • Restoring an RDB or AOF snapshot, or a backup, is not a loss but a rollback, and so is a restart after a crash that reloads the server's last save, which stock Redis and Valkey make by default: the old tokens come back with the values filed under them, so what was invalidated since is served again until its TTL, as after a failover to a replica that missed the bumps.
• Values: payloads of 1 KiB or more are zstd-compressed when that makes them smaller, with a 1 MiB window. The stored-size cap is 1 MiB, and decoding is capped at 8× that to refuse a zip bomb. An unknown format is a miss, so a rolling upgrade is safe.
• Failure means bypass. Each operation has a 100 ms timeout. A failure or timeout makes the lookup a miss, the fill a no-op and the invalidation deferred. Queries never fail on the cache.
  • The breaker (breaker.go) opens after 5 consecutive transport failures or timeouts. It opens at once on an error reply saying the server takes no writes: READONLY (a demoted primary), MASTERDOWN, OOM (full under noeviction), NOREPLICAS, MISCONF, LOADING, BUSY or CLUSTERDOWN. It also opens at once on a reply refusing the credentials, WRONGPASS or NOAUTH, which a new connection's handshake meets after a password rotation; that is logged at ERROR. Any other reply counts as up, including one about a single key or command (WRONGTYPE, NOPERM, TRYAGAIN) and a malformed one.
  • After 5 s, one background write (SET <prefix>:probe) decides whether the breaker closes, and only that probe closes it, when a write is answered within Timeout. A server that answers but refuses writes therefore stays bypassed, and so does one that answers slower than Timeout.
  • Every connection is replaced after a minute; rueidis retries what was in flight. After a failover behind a stable address (a DNS name, a proxy), the connections a process holds stay on the demoted node, which answers but refuses writes, and rueidis does not redial on READONLY. The replacement re-resolves the address, so a bypassed process reaches the new primary, and delivers the bumps it owes, within about a minute.
  • rueidis dials a replacement lazily, under the context of the operation that lands on it, bounding the dial (TLS included) by DialTimeout and then the handshake by DialTimeout again, so a reconnect slower than Timeout fails that operation. The probe's first write gets twice DialTimeout for a reconnect on top of Timeout, so such a reconnect still closes an open breaker. The allowance is for a reconnect only: a first write slower than Timeout is repeated under Timeout, and the repeat decides. But rueidis spreads commands over several connections (up to four to one server, by GOMAXPROCS, and one per cluster node), and the probe reconnects only the one it lands on. Timeout should therefore still exceed a reconnect, or operations that land on the other connections keep failing.
  • NewRedis never fails on an unreachable server. The cache starts bypassed and keeps dialing, with backoff up to 30 s. Each dial is bounded by DialTimeout, and a cluster client's topology read by the larger of that and Timeout (rueidis leaves it at 10 s otherwise).
• Deferred invalidations (pending.go): repeats coalesce per key.
  • A background drain retries them: the first one owed at once, then with backoff from 100 ms to 10 s. While the breaker is open, it retries when the probe is due, so a process with no lookups recovers as promptly as one that has them.
  • Invalidate and the drain send 1,000 bumps per round trip.
  • Past 100,000 keys the set collapses to one tenant bump per affected tenant.
  • Close makes one last attempt past the breaker.
• An owed bump holds its lookups. While this process owes a bump on any of a lookup's token keys, that lookup is a bypass with a zero snapshot: no hit, no fill. The tenant token that a collapsed set bumps is one of every lookup's keys, so a collapse holds the whole tenant. Other processes cannot know what this one owes, and serve those entries until the bump lands (the pending gauge shows the window).
• Metrics (metrics.go): the eight series listed below.

Decisions beyond the design doc

• NewRedis(cfg) takes no ctx. It dials once, bounded by the dial timeout, and then retries in the background; nothing at boot needs a ctx.
• Breaker accounting: transport failures and timeouts count against the server, and so does a reply that refuses writes, which opens the breaker at once: no bump can land, and a threshold would let successful reads keep resetting the count. A reply about one key, or one the backend cannot use, counts as up, so a foreign value under a token key does not open the breaker for every tenant. The probe writes for the same reason.
• Connection lifetime, not a re-dial on refusal: replacing connections is rueidis's own mechanism, with in-flight retries. Swapping a whole client from the probe would race Close. The cost is one reconnect per connection a minute. The lifetime is not configurable: an unexported field lets tests shorten it.
• Foreign keys are replaced: a token or value key holding a non-token string, a list or a hash is overwritten.
• Raw-size cap: a raw value over 8× MaxValueBytes is refused before compression. Otherwise the cache could store a value it could never decode.
• VersionTTL under 2 s is refused: EX has one-second resolution (rueidis truncates to whole seconds) and the jitter spans [0.9, 1.1) of the TTL. Below about 1.1 s, a token could be set with EX 0; Redis rejects that, so the bump is deferred and retried. The 2 s floor leaves a margin.
• zstd window 1 MiB: the encoder keeps one window-sized history per concurrent caller for the life of the process. With 240 KB values at 14 procs it kept about 121 MiB at SpeedFastest's default 4 MiB window, and about 36 MiB at 1 MiB, with the same speed and ratio on row payloads. WithLowerEncoderMem was rejected: it was 6× slower on 4 MiB values. The decoder accepts windows up to the decoded-size cap, so values written before this change still decode.
• Where the tests live: the integration tests sit beside the package (internal/cache/redis_integration_test.go, //go:build integration), and make test-integration now also runs ./internal/cache/....
• Coverage: the new files are excluded from the e2e coverage suite only. The e2e stack cannot reach an unwired backend, and they had pulled that suite to 56.4% against its 60% floor. The merged total still counts them.
• Redis Cluster: tested as a one-node cluster owning every slot. That exercises CROSSSLOT enforcement and the cluster client's topology read, not routing across nodes (a node owning every slot never answers MOVED); nodes announce 127.0.0.1 on a same-numbered host port. A 3-master cluster in containers cannot gossip across a typical desktop container network's port mapping.
• Probe budget: twice DialTimeout for a reconnect, then Timeout for the write. rueidis (v1.0.78) bounds a reconnect's dial, TLS included, by DialTimeout, then gives the HELLO/AUTH handshake a fresh DialTimeout, so a reconnect can take about twice that. The allowance is for a reconnect only: a first probe write slower than Timeout is repeated under Timeout alone, and the repeat decides. Without that, a server answering slower than Timeout but within the allowance passed every probe: the breaker closed, the next operations each waited out Timeout and failed, and it reopened, every cycle. A repeat that lands on another connection still needing a reconnect fails, which costs one more BreakerOpenFor cycle.
• Credential replies open the breaker; NOPERM does not. NOPERM names one key or command, which the rest of the work may not touch. Counting it as a refusal made TestRedis_OwedBumpHoldsItsLookups fail: an ACL denying one key's write bypassed the cache for every tenant.
• Not implemented: pool_size and TLS file loading. TLS is a *tls.Config that feat(config): cache.backend=redis selects the shared cache #630 builds from files.

Tests

• Conformance: cachetest.Run (from feat(cache): shared redis cache, snapshot keys, uncached write pipes #614's suite) against pinned redis:8.10.2-alpine, valkey/valkey:8.1.10-alpine, dragonflydb/dragonfly:v2.0.0 and the Redis Cluster node:
  • NewPair drives the cross-instance cases.
  • Entries counts the values under the case's key prefix plus the empty key, so the zero-snapshot case runs on all four. Mutation-checked: dropping Set's zero-snapshot guard fails it.
  • Each server also runs a cross-tenant invalidation spanning 20 slots, a compression round trip, and an incompressible value between the stored-size and raw-size limits, which must not be stored. Mutation-checked: without the stored-size guard it is stored on all four.
• TestRedis_LostTokensAreMisses: deletes the T, B, S or all token keys (the values survive), then FLUSHALL. Every case must miss, twice. Mutation-checked: recreating tokens as a constant, as a counter restarted at 0 would, fails all four.
• TestRedis_ServerStopsAnswering: the server is paused, then:
  • each failed lookup returns in under 10× the 100 ms timeout (the bound the test asserts), and three failures open the breaker;
  • once open, lookups skip the server and cost no round trip;
  • an invalidation is deferred;
  • a process that boots meanwhile starts bypassed;
  • after the pause lifts, the deferred bump lands, the late process connects, and a second instance misses the pre-pause fill.
• TestRedis_OwedBumpHoldsItsLookups: an ACL lets the process read tokens but not replace them, so a bump stays owed with the breaker closed.
  • The lookup that bump would orphan is a bypass with a zero snapshot, while other tables' and pipe lookups are served.
  • After the grant, no lookup returns the pre-write value before the bump lands.
  • Mutation-checked: without the hold, it fails 3/3.
• TestRedis_RefusedWrites, for a primary demoted with REPLICAOF to an unreachable host (READONLY) and for maxmemory 1 under noeviction (OOM):
  • the first refusal opens the breaker despite a threshold of 1000, and the bump stays owed;
  • a process tripped by a refused fill, which owes nothing, stays bypassed across 7 s of refused probes;
  • after recovery, the owed bump lands before anything it would orphan is served;
  • an ingest-only process with 1,501 owed bumps (two batches) recovers within the probe period plus 3 s, where a drain left at its backoff would take about 5.7 s;
  • a fresh process misses.
  • Mutation-checked: a PING probe, refusals counted as up, and a drain that ignores the probe's due time each fail it.
• TestRedis_FailoverBehindAStableAddress: a primary and its replica sit behind an in-process forwarder that moves new connections only. The test promotes the replica, demotes the primary, and invalidates (READONLY, deferred), then switches the forwarder. The owed bump lands on the new primary with no pre-write hit in between, and a fresh process misses. Mutation-checked: without the connection lifetime, the bump never lands.
• TestRedis_ClusterTopologyReadIsBounded: a node that answers the handshake but not CLUSTER SHARDS held rueidis.NewClient for 10.0 s. It now fails within the bound. Mutation-checked.
• TestRedis_CloseDeliversPastAnOpenBreaker: mutation-checked.
• TestRedis_ForeignTokenIsReplaced: a short string under B:, a list under T, a hash under the value key. Mutation-checked.
• TestRedis_RestoredSnapshotIsARollback: fill, SAVE, invalidate (a miss confirmed), DEBUG RELOAD NOSAVE: a fresh process gets the pre-write rows back. It pins the documented exception to "lost tokens can only cause misses".
• TestRedis_ProbeFitsASlowReconnect: an in-process proxy terminates TLS, drops the connections, and delays both the TLS handshake and the forward of each new one by 700 ms, with Timeout 100 ms and DialTimeout 1 s: the reconnect, about 1.4 s, outlasts DialTimeout plus Timeout. The lookup that must reconnect fails and opens the breaker, and the probe reconnects and closes it. Mutation-checked: a probe bounded by Timeout, or by DialTimeout plus Timeout, never closes it.
• TestRedis_ProbeKeepsASlowServerBypassed: the proxy delays every reply by 300 ms (Timeout 100 ms, DialTimeout 1 s, threshold 1, BreakerOpenFor 200 ms). The breaker stays open through 4 s of probes, and closes once replies are prompt again. Mutation-checked: without the repeat under Timeout, a probe closes it within the 4 s.
• TestRedis_RejectedCredentialsOpenTheBreaker: the ACL user's password is rotated and its connections killed. The reconnect is refused with WRONGPASS, the breaker opens despite a threshold of 1000 and stays open across refused probes, and it closes once the old password works again. Mutation-checked: with credential replies counted as up, the breaker never opens.
• Names that would run together: feat(cache): shared redis cache, snapshot keys, uncached write pipes #614's conformance cases for raw names (raw names read and bump alike, names never run together: pairs that collide under an unescaped join at a ., :, | or NUL) pass on all four servers. Mutation-checked: the codec before escaping failed names never run together on all four.
• Unit tests (no server):
  • the key schema goldens: escaped table and scope; a : collision in token keys; :, NUL and cross-dependency collisions in value keys;
  • the codec round trip, its corruption and zip-bomb refusals, and the encoder window, including a wider-window value still decoding;
  • the breaker state machine on a fake clock, including the trip and the probe schedule;
  • pending coalescing, collapse, and which lookups a pending set holds;
  • the refusal and credential classification, both size limits counted in wavehouse_cache_oversize_total (mutation-checked), the first deferral waking the drain, config validation (including ConnWriteTimeout), bypass on an unreachable server, and the metrics.
• make ci (static checks, unit, integration — including TestRedis_Conformance on Redis, Valkey, Dragonfly and a one-node Redis Cluster — e2e and coverage) is green locally, and go test -race -tags integration ./internal/cache/... passes.

Metrics (new series)

Meter wavehouse-cache, every series carrying backend="redis", no tenant label. None is emitted until #630 wires the backend.

series	meaning
wavehouse_cache_lookups_total{result}	lookups by hit / miss / stale (filed under since-bumped tokens) / bypass (server skipped, or held by a bump this process owes) / error
wavehouse_cache_op_duration_seconds{op}	round-trip time of lookup / set / invalidate
wavehouse_cache_breaker_open	1 while the cache is bypassed (breaker open, or never connected)
wavehouse_cache_invalidations_total{result}	ok counts every bump that lands, retried ones included; deferred counts each bump an invalidation could not deliver when made, a repeat of one already owed included; a failed retry is not counted again. They overlap; the pending gauge is what is still owed
wavehouse_cache_invalidations_pending	bumps still owed to the server; other processes may serve the entries they would orphan meanwhile
wavehouse_cache_value_bytes	stored value size (after compression)
wavehouse_cache_oversize_total	results not cached for size
wavehouse_cache_set_failures_total{reason}	failed fills by oom / timeout / other

Deliberately left to later PRs

• feat(config): cache.backend=redis selects the shared cache #630: cache.backend / cache.redis.* config, the wiring and its Close budget, the configuration and deployment docs (including maxmemory-policy, the consistency statement, and no replica reads), and the two-App end-to-end test.
  • The consistency statement also owns the token-key layout, which is now a protocol between builds: every process sharing a server reads and bumps those keys for itself.
  • So a change to the layout, or to what keyenc keeps, needs one of two things. Either the new build reads and bumps both layouts (folding the old tokens into what it files) until no old build is left, and a later build drops the old; or the upgrade never runs two builds against the server at once.
  • Otherwise one build's bumps miss the entries the other filed, and those are served until their TTL for the whole rolling deploy; a flush afterwards only trims the tail.
  • A valueKey change only orphans values and is safe to roll.
• A later PR: the near-cache with client tracking.
• perf(cache): flat version index, pruned per tenant #621: flattening VersionManager. This PR does not touch it.
• bug(pipes): mutation pipe results are cached and coalesced — the write silently drops on repeat calls #386 / bug(query): admin mutations via /v1/admin/query don't invalidate the structured-query cache — stale reads for up to an hour #394: a shared cache widens both from one pod to the fleet (design §8). They should be fixed before feat(config): cache.backend=redis selects the shared cache #630 flips a real deployment.
• The rest of bug(cache): Redis recovery probe must fit a fresh dial in the op timeout #664: rueidis spreads commands over up to four connections per server (one per cluster node) and dials each lazily under the calling operation's context, and the probe reconnects only the one it lands on. The options are one connection per server (PipelineMultiplex) or a probe that reaches every connection. Until then, Timeout should exceed a reconnect.
• Standalone mode against a cluster-enabled server is not detected. A MOVED reply counts as up, so keys in other slots always miss and their bumps stay owed. A check at connect (cluster_enabled:1) could also refuse single-endpoint cluster services that work in standalone mode today, so it waits for feat(config): cache.backend=redis selects the shared cache #630's configuration.
• bug(cache): a demoted Redis primary reads as healthy; Sentinel mode is unconfigured #656 (second half): Sentinel mode: sentinel credentials, a topology refresh interval, and a Sentinel integration case. Must land before feat(config): cache.backend=redis selects the shared cache #630 makes Sentinel selectable.

🤖 Generated with Claude Code

https://claude.ai/code/session_017aS7rLrH1RKkUMem7X4ckd

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: ca8af015-cbbe-4f7b-98ae-aa41376e0908

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}