From 3022b926392f9cdfa16931ada447607b94fe09b7 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Thu, 24 Sep 2026 23:27:32 -0400 Subject: [PATCH 01/45] feat(config): choose each layer's implementation at boot mq.backend, cache.backend, dedupe.backend and coord.backend select each layer's implementation; only today's in-process one exists per layer and it is the default. Validate refuses an unknown value, internal/app picks the implementation in one switch per layer, data_dir is probed only when a selected backend keeps state there, and boot logs Config.Warnings. Part of #613. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01EJr5tY4WQUy2sc4MbW67vL --- CHANGELOG.md | 1 + cmd/wavehouse/main.go | 19 ++- config.yaml | 10 ++ docs/src/content/docs/architecture.md | 2 +- docs/src/content/docs/configuration.mdx | 31 +++- docs/src/content/docs/settings-directory.mdx | 2 +- internal/app/app.go | 4 + internal/app/app_test.go | 27 +++- internal/app/wire.go | 63 ++++++-- internal/config/backends.go | 143 ++++++++++++++++ internal/config/backends_test.go | 161 +++++++++++++++++++ internal/config/config.go | 12 +- internal/config/config_test.go | 10 +- tests/integration/setup_test.go | 5 +- tests/integration/tenants_test.go | 5 +- 15 files changed, 453 insertions(+), 42 deletions(-) create mode 100644 internal/config/backends.go create mode 100644 internal/config/backends_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 23c0c7152..9f2e821bc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added +- **Each layer's implementation is chosen at boot** (`internal/config/{backends,config}.go` (+ tests), `internal/app/{app,wire}.go` (+ tests), `cmd/wavehouse/main.go`, `tests/integration/{setup,tenants}_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,settings-directory.mdx,architecture.md}`): the first step of running WaveHouse as more than one process ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config gains `mq.backend` (`WH_MQ_BACKEND`, default `embedded`), `cache.backend` (`WH_CACHE_BACKEND`, `local`), `dedupe.backend` (`WH_DEDUPE_BACKEND`, `pebble`) and `coord.backend` (`WH_COORD_BACKEND`, `local`). Each layer has only its in-process backend so far, and it is the default, so nothing changes for a config that sets none of them; a value with no backend refuses boot and names the valid ones. A backend's own settings will go in a `.` sub-block, which until that backend lands is an unknown key and refuses boot. `internal/app` picks each implementation in one `switch` per layer (`wireMQ`, `wireCache`, `wireDedupe`), `data_dir` is probed only when a selected backend keeps state there (`Config.NeedsDataDir`), and boot logs at `WARN` each line of `Config.Warnings`, the combinations that are correct for one replica only once a shared queue exists. A `config.Config` built without `config.Load` must now name every backend: the zero value is not the default, and `app.New` refuses it. - **A tenant removed or rejected at runtime has its open streams ended** (`internal/stream/{hub,subscriber,bucket}.go` (+ tests), `internal/api/stream.go` (+ tests), `internal/api/router.go`, `internal/settings/{registry,tree}.go` (+ tests), `internal/ingest/worker.go` (+ tests), `internal/app/wire.go` (+ tests), `clients/ts/src/stream/sse.ts`, `docs/src/content/docs/{api,deployment,architecture,ingest-pipeline}.md`, `docs/src/content/docs/settings-directory.mdx`, `AGENTS.md`): story 3 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)). A `GET /v1/stream` used to outlive its tenant: the hub read no policy for it and withheld every row while the keepalive wheel held the connection open, so the client could not tell it from a quiet table. After every reload the hub now evicts the subscribers of each tenant no longer served, removed or rejected alike (`Hub.Prune`, through the close-once `Subscriber.Evict`), and the handler ends the stream: a gap-fill in progress included, and a stream `TenantMW` admitted just before the reload but registered just after it, which the handler checks for as it registers. The client's reconnect gets `404` (removed) or `503` (rejected); the SDK stops on the first and retries the second, resuming from `Last-Event-ID` once the folder is back — except in a browser going cross-origin where tenant `0` is not served or its CORS list does not admit the page, which cannot read either refusal (both are decorated from tenant `0`'s list) and re-dials as after a dropped connection. A flat directory never stops serving tenant `0`, so nothing changes there. Removing a tenant is deleting its folder, then reloading the whole directory, the last folder included: a server started with tenant folders reads the emptied directory as no folder left, where it read as a change of shape and the reload was rejected whole, leaving that tenant served; at boot an empty directory still reads as the four files, missing. Reloading the deleted folder by name leaves its tenant rejected. `GET /v1/health` keeps resolving a tenant, deliberately: its `404` tells a caller with no token no more than every tenant route's does, since they all answer before authenticating, and resolving answers a served tenant's ping from that tenant's own CORS list. A batch queued for a tenant with no ClickHouse connection — one no longer served, or one no pool could be opened for (such as by the connection ceiling) — skips the row-by-row retry, which no row of it could pass, and meets its DLQ switch once, whole, logged once per batch rather than twice per row; a tenant no longer served reads the switch as on, so its queued rows are parked under its own subject rather than dropped, inserted into another tenant's ClickHouse, or left unacked, redelivered for as long as the tenant is away and holding its queue's ack floor, which the purge waits on. Over a nested directory `/livez` no longer keeps naming a tenant that stopped being served before any tenant completed a first discovery: the diagnostic goes back to `no tenant has completed a first discovery yet`. - **One ClickHouse pool per tuple and one schema registry per tenant** (`internal/chconn/chconn.go` (+ tests), `internal/discovery/discovery.go` (+ tests), `internal/app/discoveries.go` (new), `internal/app/{app,wire}.go` (+ tests), `internal/api/{schema,ingest,structured_query,pipes,query,health,errors,clickhouse_exec}.go` (+ tests), `internal/stream/hub.go`, `internal/ingest/worker.go`, `internal/cache/{cache,local,version_manager}.go` (+ tests), `internal/testutil/{testutil,mocks}.go`, `tests/integration/{setup,tenants,boot_resilience,query_limits}_test.go`, `clients/ts/src/{schema,table,sql,client,types}.ts` (+ tests), `tests/e2e/sdk/admin.test.ts`, `docs/src/content/docs/{api,deployment,architecture,ingest-pipeline}.md`, `docs/src/content/docs/{settings-directory,configuration,access-control,reverse-proxy}.mdx`, `docs/src/content/docs/sdk/{admin,reference,queries}.md`, `AGENTS.md`): the second slice of story 6 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)), with no behavior change for a settings directory that holds the four files beyond the three noted at the end. The process opens one native pool per distinct `clickhouse.addr` / `database` / `username` / password / `tls` tuple among the served tenants (`chconn.Identity`, `chconn.Pools`), shared by the tenants naming it and sized to their largest `max_open_conns` and `max_idle_conns`; `http_port`, `http_scheme`, `headers` and `query_timeout` stay each tenant's own. Every reload reconciles the pools: a new tuple opens (never dials), a tenant whose tuple changed is repointed, a tuple no tenant names closes after the longest `query_timeout` among the tenants it had, and a pool whose largest ask changed is resized with the same grace. The boot config's `clickhouse.max_total_conns` now bounds the open pools together: boot is refused naming the sum and the ceiling; at a reload a resize above it is refused with the pool kept at its size, and a tuple that cannot be opened — the ceiling, a certificate file that cannot be read, or options the driver refuses — leaves its tenants on the pool they had (the keep-previous-wiring rule of the connection ceiling) or on none when they had none; both are logged and the next reload retries. A tenant on no pool fails closed: `503` with `Retry-After: 30` on `POST /v1/query`, `GET/POST /v1/pipes/{name}`, `POST /v1/ops/query` and `POST /v1/ops/schema/refresh`, ahead of the cache. Each served tenant gets a `discovery.SchemaRegistry` of its own over its pool, kept fresh by its own loop — the boot retry until the first success, then `schema.refresh_interval` with the first refresh at a random point within the interval so tenants adopted together do not refresh together — created and stopped from the settings reload and stopped under `App.Close`; `wavehouse_schema_refresh_failures_total{tenant}` counts a loop's failed attempts. `GET /v1/ops/schema`, `POST /v1/ops/schema/refresh` and `POST /v1/ops/query` take the strict `?tenant=` the pipe reads take (absent is tenant `0`, `400` malformed, `404` unknown, `503` rejected); the SDK sends it as the `tenant` option of `wh.schema.list()`, `wh.schema.refresh()`, `wh.from(t).schema()` and `wh.sql()`. Over a nested directory `/livez` (and `/v1/health`) is `503` with the latest discovery failure, naming its tenant, while no tenant has completed a first discovery, then `200` for the rest of the process lifetime; `/readyz` pings every open pool at once, is ready at the first answer, and names every pool that did not answer when none does — one tenant's ClickHouse outage is its log line and counter, never a probe failure. The ingest worker inserts each batch into its own tenant's ClickHouse — the tenant the message's topic names, through that tenant's HTTP wiring (`chconn.Pools.Target`); a tenant on no pool takes the failure path an unreachable ClickHouse takes — and its cache invalidation fans out to the tenants on the same address and database as the batch's tenant, whatever their user or `tls` block (they read the same tables), rather than to every known tenant, and a tenant adopted after an absence — rejected or removed, so out of that fan-out — or moved to another address or database has its cached structured-query results orphaned in one step (`Cache.InvalidateTenant`, a tenant generation in every version key), so a repaired folder never serves query rows cached before the inserts it missed (a pipe result names no table, so neither this nor any insert invalidates it: it stays until its TTL expires). Three changes reach the single-tenant directory too: a table lookup before the first discovery is a `503` with `Retry-After: 5` (`schema not loaded yet`) rather than a `404`, on `POST /v1/ingest`, `POST /v1/query` and `GET /v1/ops/schema` (the list included, where `[]` would read as no tables); the first periodic refresh fires at a random point within the interval rather than a full interval after boot; and a reload that moves `clickhouse.addr` or `clickhouse.database` now orphans the structured-query results cached before it, where they were served until their TTL. Until [#529](https://github.com/Wave-RF/WaveHouse/issues/529) every tenant's user authenticates with `WH_CH_PASSWORD`, so the tuple is in effect the address, database, username and `tls` block. - **One token verifier per tenant, built off the boot and reload paths** (`internal/auth/auth.go` (+ tests), `internal/api/router.go` (+ tests), `internal/app/{app,wire}.go` (+ tests), `go.mod`, `docs/src/content/docs/sdk/{reference,streaming}.md`, `docs/src/content/docs/{architecture,deployment,api}.md`, `docs/src/content/docs/{settings-directory,configuration}.mdx`, `SECURITY.md`): story 9 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)). Over a nested settings directory each tenant's folder now wires that tenant's verifier (`auth.jwks_url`, `auth.role_claim`), so a JWKS-issued token verifies only under the tenants whose `jwks_url` names its identity provider's key set — under any other tenant's header it is refused as invalid — where before one verifier, tenant `0`'s, accepted a token under any header. `auth.Config` is now the boot-config half alone (the HMAC secret and the operator key, shared by every tenant) and the new `auth.Wiring` a tenant's half; `NewAuthenticator` builds no verifier, `Reconfigure(id, wiring)` gives a tenant one — swapped atomically when its wiring changed, kept when it did not — `Prune` drops the verifiers of the tenants a reload stopped serving, rejected or removed alike (no work runs for a tenant that is not served; a folder adopted again gets a fresh verifier), and `Close` stops every JWKS refresh as a component of `App.Close`. This holds on every reload because the registry's `AfterAdopt` hooks run after every reload it applied, empty list included, so a per-tenant reload that rejects a folder drops its verifier then rather than at the next adoption. The middleware reads the request tenant's verifier through an injected `auth.TenantSource` — the store `api.TenantMW` resolved names its tenant with `settings.Store.Tenant()`, one read of the context — a tenant-exempt route (the ops tree) verifies as tenant `0`, and a tenant with no verifier fails closed. The operator key stamps the request tenant's `admin_role`, read from that tenant's policy, rather than tenant `0`'s. A JWKS key set is fetched on its own goroutine: the verifier is in place at once and *pending* until a set has been stored — the first fetch retried with backoff from one second to a minute, a stored set then kept fresh by the library hourly and, rate-limited, on an unknown key id — so neither boot nor a reload (which holds the registry's lock) waits on the endpoint, and **an unreachable JWKS no longer refuses boot** — it logs (`jwks refresh failed; no token validates until it succeeds`) and that tenant alone is affected. A token checked against a pending verifier is a new outcome, `auth.ErrVerifierPending`: every `/v1` route answers it `503 {"error": "token verifier not ready: …"}` with `Retry-After: 30` (`api.refuseUnverifiable`) rather than evaluate the request under the `default_role`, which could accept its data under a lesser role while another pod holding the keys would have served it as its own; a request without a token, and the operator key, are unaffected. Every fetch goes through one client that caps the response at 1 MiB, refusing a larger one as unreachable with `jwks response exceeds 1048576 bytes` as the logged cause. Nothing changes for a flat directory beyond that boot rule: its one tenant gets exactly the verifier it had. The boot warning for the secretless posture (no `auth.jwt_secret` and no `jwks_url`, whose token refusal landed in [#607](https://github.com/Wave-RF/WaveHouse/pull/607)) is now per tenant — each served tenant with no `jwks_url` while the boot secret is unset — where one line that any JWKS tenant silenced used to stand for the whole directory. diff --git a/cmd/wavehouse/main.go b/cmd/wavehouse/main.go index 042257e2b..3a1304f0b 100644 --- a/cmd/wavehouse/main.go +++ b/cmd/wavehouse/main.go @@ -171,14 +171,17 @@ func run(ctx context.Context) int { return 1 } - // data_dir must be writable before anything dials out, so the refusal - // (and, for the typical cause — a bind mount owned by root rather than - // UID 65532 — the remediation) lands at the top of the log rather than - // after ClickHouse discovery. NATS and Pebble still fail loud on their - // own if the directory changes underneath us. - if err := config.CheckDataDir(cfg.DataDir); err != nil { - logger.Error("check data_dir", "error", err) - return 1 + // data_dir, when a selected backend keeps state there, must be writable + // before anything dials out, so the refusal (and, for the typical cause — + // a bind mount owned by root rather than UID 65532 — the remediation) + // lands at the top of the log rather than after ClickHouse discovery. + // NATS and Pebble still fail loud on their own if the directory changes + // underneath us. + if cfg.NeedsDataDir() { + if err := config.CheckDataDir(cfg.DataDir); err != nil { + logger.Error("check data_dir", "error", err) + return 1 + } } a, err := app.New(ctx, app.Options{ diff --git a/config.yaml b/config.yaml index 53a435029..65c384eaa 100644 --- a/config.yaml +++ b/config.yaml @@ -43,9 +43,19 @@ clickhouse: password: "" max_total_conns: 0 # ceiling on open native connections across pools; 0 = none +# Each layer's implementation, chosen at boot. Only the in-process backend +# exists for each today, and it is the default. +mq: + backend: embedded # NATS JetStream under /nats +dedupe: + backend: pebble # Pebble under /pebble +coord: + backend: local + # In-process L1 cache size. The query time-bucket # (query.timestamp_bucket_seconds) is a settings key. cache: + backend: local l1_max_cost: 67108864 # Auth has no on/off switch — the JWT middleware always runs. A request with no diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 6eaf3d54e..938751ff8 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -90,7 +90,7 @@ The API layer uses [Chi](https://github.com/go-chi/chi) for routing with Request ### `app/` — Process wiring - **app.go** — `New(ctx, Options)` builds every component from the boot config (`Options.Config`) and the settings directory it names, in dependency order: settings registry, observability, ClickHouse pools, schema discovery, the dedupe stores, embedded NATS (ingest + DLQ streams), cache, sweeper, streaming (hub, MQ→hub bridge, keepalive wheel), ingest worker, auth, reload triggers, HTTP. Each is one `component` value — what it opens, what it loops, what it releases — so a failure part-way releases what was already opened and returns the error. `Run(ctx)` drives every loop under one `errgroup` until `ctx` is canceled (a clean stop: every loop drains, the API server and the ingest worker within `server.shutdown_timeout`; open SSE streams are ended as the drain begins rather than waited on) or a component fails, which stops the rest and returns that error. `Close(ctx)` releases what `New` opened, newest first, under the caller's release budget (`ReleaseTimeout`, 5s), a real bound: a remote implementation's close gives up at the deadline itself, and a close that ignores the context (the local stores) is abandoned at it, with the components below it left unreleased rather than overlapping it, both named in the error — and then flushes telemetry under its own 3s budget, so the flush that reports on the stop is never handed a deadline a slow close already spent. The SIGHUP registration is released last of all. `Handler`, `Registry`, and `MQ` expose the pieces a harness needs; `Options.Listener` lets one serve the API on its own listener instead of `server.port`. -- **wire.go** — one `wire*` function per component, each handed the settings registry whole and deriving the per-call getters the internal packages take (`DLQFor`, `DedupeFor`, `GapWindow`, …) and registering its `AfterAdopt` hook there where it has one. Those wiring functions are where the per-tenant registry of [#583](https://github.com/Wave-RF/WaveHouse/issues/583) is injected, not `main`: `wireSettings` opens the `settings.Registry`, the HTTP handlers get store-keyed getters (method expressions such as `(*settings.Store).Policy`), and `perTenant` adapts a store accessor into the `func(tenant.ID) T` getter the async packages take, with the tenant each message's `mq.Topic` names for the stream hub and the ingest worker — a tenant the registry is not serving is logged and read as the zero value, except in `dlqFor`, the ingest worker's DLQ switch, where it reads as on so a message the worker cannot read is parked rather than dropped, and a removed or rejected tenant's queued rows are parked rather than left unacked, where they would hold the ack floor and stop the sweeper. The ClickHouse pools (`chconn.Pools`) and the per-tenant schema registries (`discoveries`, in `discoveries.go`) are reconciled from `AfterAdopt` after every reload ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 6): `wireClickHouse` builds each served tenant's `chconn.Member` from its store and logs what the reconcile refused; `wireDiscovery` builds a registry over `pools.For` for each newly served tenant — a flat directory's tenant `0` refreshed synchronously first, as before — runs its loop under the App's stop context, stops the loop of a tenant no longer served, and drives the `BootState` from the first tenant's first discovery, sticky from there; before that, a diagnostic naming a tenant a reload stopped serving goes back to the no-tenant one. The handlers resolve both per request through store-keyed getters (`chConnFor`, `registryFor`, `chTargetFor`, `queryTimeout`), the hub and the ingest worker through tenant-keyed ones (`discoveries.For`, `pools.Target`) called with the tenant the message's topic names; a tenant on no pool is an untyped nil connection, the handlers' `503`. The ingest worker is handed the cache through `sharedTables`, which bumps each namespace the worker invalidates under every tenant on the same ClickHouse address and database (`pools.SharingTables`), and the pools hook orphans the table-keyed cache — the structured-query results — of a tenant back on a pool after an absence (`Cache.InvalidateTenant`), since it was out of that fan-out while away, and of a tenant moved to another address or database, since it now reads other tables (both returned by `Pools.Reconcile`). The one setting that still follows the default tenant is read per request, the admin role of a flat directory's ops gate: `defaultSetting` reads the store tenant `0` last adopted (`App.defaultStore`, tracked by an `onDefaultAdopt` hook that runs only after a reload that adopted it), so a `0` folder that a reload rejects or removes leaves it as it was. The auth verifiers are per tenant: `wireAuth` builds one for each tenant being served, its `AfterAdopt` hook reconfigures the adopted tenants' (rebuilt only when their wiring changed) and prunes the ones no longer served, and the operator key's admin role is read from the request tenant's policy. `wireStreaming`'s hook prunes the stream hub the same way (`Hub.Prune`, with the one `served` predicate the auth and dedupe hooks use too), ending the open streams of a tenant no longer served. One setting is shared by folding over the tenants being served rather than by following tenant `0`: the keepalive wheel runs at the shortest `stream.keepalive_interval` among them (`shortestKeepalive`), re-derived after every reload the registry applies — an adoption, a rejection, or a removal — so a dropped tenant's interval leaves the wheel at once ([#597](https://github.com/Wave-RF/WaveHouse/issues/597)). The sweeper is handed each served tenant's own `stream.gap_window_minutes` (`gapWindows`, read every sweep), since each tenant's events have a queue of their own. The dedupe stores are per tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 7): `wireDedupe` builds a `dedupe.Stores` over the `Tenant` factory of the embedded Pebble implementation (`dedupe.NewEmbedded`), handing it `data_dir` once; the implementation decides where every tenant's store lives — one instance, each key led by its tenant (story 3) — and one reconcile closure, the boot apply and the `AfterAdopt` hook alike, sets every store to what the registry says: open exactly when its tenant is served with `dedupe.enabled` on, closed with its seen ids kept when the tenant is switched off, rejected, or removed. An instance that cannot open follows the registry's rule for the shape: fatal at boot over a flat directory, fail-closed for every tenant with dedupe on over a nested one. The system gauges report that one instance's figures (`Embedded.Stats`), not a sum over tenants. The ingest handler picks the tenant's store off the request's `settings.Store` (`Store.Tenant()`). The reload triggers only start in `Run`, after `New` has registered every hook, so the watcher's first reload already drives all of them: SIGHUP in both shapes, the directory watcher for a flat directory only. `wireMQ` hands each served tenant's `mq.max_bytes_gb` to `mq.Broker.SetMaxBytes` at boot and again after every reload, under the App's stop context, which opens that tenant's queue the first time; a queue that cannot be opened or resized follows the registry's rule for the shape — fatal at boot over a flat directory, logged over a nested one — and is retried by the next reload. How the budget is split across the tenant's streams, the time bounds, the rollback, and the dead-letter shrink guard are `internal/mq`'s. +- **wire.go** — one `wire*` function per component, each handed the settings registry whole and deriving the per-call getters the internal packages take (`DLQFor`, `DedupeFor`, `GapWindow`, …) and registering its `AfterAdopt` hook there where it has one. A layer with a choice of implementation — `wireMQ`, `wireCache`, `wireDedupe` — picks it there and nowhere else, in a `switch` on the boot config's `.backend` with one case per backend; the default case refuses boot, which only a `config.Config` built without `config.Load` reaches, since `Validate` refuses a value no case handles. Those wiring functions are where the per-tenant registry of [#583](https://github.com/Wave-RF/WaveHouse/issues/583) is injected, not `main`: `wireSettings` opens the `settings.Registry`, the HTTP handlers get store-keyed getters (method expressions such as `(*settings.Store).Policy`), and `perTenant` adapts a store accessor into the `func(tenant.ID) T` getter the async packages take, with the tenant each message's `mq.Topic` names for the stream hub and the ingest worker — a tenant the registry is not serving is logged and read as the zero value, except in `dlqFor`, the ingest worker's DLQ switch, where it reads as on so a message the worker cannot read is parked rather than dropped, and a removed or rejected tenant's queued rows are parked rather than left unacked, where they would hold the ack floor and stop the sweeper. The ClickHouse pools (`chconn.Pools`) and the per-tenant schema registries (`discoveries`, in `discoveries.go`) are reconciled from `AfterAdopt` after every reload ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 6): `wireClickHouse` builds each served tenant's `chconn.Member` from its store and logs what the reconcile refused; `wireDiscovery` builds a registry over `pools.For` for each newly served tenant — a flat directory's tenant `0` refreshed synchronously first, as before — runs its loop under the App's stop context, stops the loop of a tenant no longer served, and drives the `BootState` from the first tenant's first discovery, sticky from there; before that, a diagnostic naming a tenant a reload stopped serving goes back to the no-tenant one. The handlers resolve both per request through store-keyed getters (`chConnFor`, `registryFor`, `chTargetFor`, `queryTimeout`), the hub and the ingest worker through tenant-keyed ones (`discoveries.For`, `pools.Target`) called with the tenant the message's topic names; a tenant on no pool is an untyped nil connection, the handlers' `503`. The ingest worker is handed the cache through `sharedTables`, which bumps each namespace the worker invalidates under every tenant on the same ClickHouse address and database (`pools.SharingTables`), and the pools hook orphans the table-keyed cache — the structured-query results — of a tenant back on a pool after an absence (`Cache.InvalidateTenant`), since it was out of that fan-out while away, and of a tenant moved to another address or database, since it now reads other tables (both returned by `Pools.Reconcile`). The one setting that still follows the default tenant is read per request, the admin role of a flat directory's ops gate: `defaultSetting` reads the store tenant `0` last adopted (`App.defaultStore`, tracked by an `onDefaultAdopt` hook that runs only after a reload that adopted it), so a `0` folder that a reload rejects or removes leaves it as it was. The auth verifiers are per tenant: `wireAuth` builds one for each tenant being served, its `AfterAdopt` hook reconfigures the adopted tenants' (rebuilt only when their wiring changed) and prunes the ones no longer served, and the operator key's admin role is read from the request tenant's policy. `wireStreaming`'s hook prunes the stream hub the same way (`Hub.Prune`, with the one `served` predicate the auth and dedupe hooks use too), ending the open streams of a tenant no longer served. One setting is shared by folding over the tenants being served rather than by following tenant `0`: the keepalive wheel runs at the shortest `stream.keepalive_interval` among them (`shortestKeepalive`), re-derived after every reload the registry applies — an adoption, a rejection, or a removal — so a dropped tenant's interval leaves the wheel at once ([#597](https://github.com/Wave-RF/WaveHouse/issues/597)). The sweeper is handed each served tenant's own `stream.gap_window_minutes` (`gapWindows`, read every sweep), since each tenant's events have a queue of their own. The dedupe stores are per tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 7): `wireDedupe`'s `pebble` case builds a `dedupe.Stores` over the `Tenant` factory of the embedded Pebble implementation (`dedupe.NewEmbedded`), handing it `data_dir` once; the implementation decides where every tenant's store lives — one instance, each key led by its tenant (story 3) — and one reconcile closure, the boot apply and the `AfterAdopt` hook alike, sets every store to what the registry says: open exactly when its tenant is served with `dedupe.enabled` on, closed with its seen ids kept when the tenant is switched off, rejected, or removed. An instance that cannot open follows the registry's rule for the shape: fatal at boot over a flat directory, fail-closed for every tenant with dedupe on over a nested one. The system gauges report that one instance's figures (`Embedded.Stats`), not a sum over tenants. The ingest handler picks the tenant's store off the request's `settings.Store` (`Store.Tenant()`). The reload triggers only start in `Run`, after `New` has registered every hook, so the watcher's first reload already drives all of them: SIGHUP in both shapes, the directory watcher for a flat directory only. `wireMQ`'s `embedded` case hands each served tenant's `mq.max_bytes_gb` to `mq.Broker.SetMaxBytes` at boot and again after every reload, under the App's stop context, which opens that tenant's queue the first time; a queue that cannot be opened or resized follows the registry's rule for the shape — fatal at boot over a flat directory, logged over a nested one — and is retried by the next reload. How the budget is split across the tenant's streams, the time bounds, the rollback, and the dead-letter shrink guard are `internal/mq`'s. ### `stream/` — SSE keepalive & fan-out diff --git a/docs/src/content/docs/configuration.mdx b/docs/src/content/docs/configuration.mdx index 8a709426f..8d311adc5 100644 --- a/docs/src/content/docs/configuration.mdx +++ b/docs/src/content/docs/configuration.mdx @@ -17,7 +17,7 @@ WaveHouse is configured via a YAML file with environment variable overrides. All 2. Environment variables override any values from the YAML file. 3. If no config file exists, all values are read from environment variables. Every key has a default except `settings.dir` (`WH_SETTINGS_DIR`), which must be set either way. 4. Both sources are **strict**. A YAML key this page doesn't list — a typo, or a tunable that has moved to the settings directory (`dlq.enabled`, `clickhouse.addr`, `stream.*`, a leftover `policy:` or `pipes:` block, …) — refuses to boot and names every offending key, so nothing is read, ignored, and believed. A `WH_*` environment variable that binds to no key on this page (`WH_DEDUPE_ENABLED`, `WH_CH_ADDR`, a misspelling) refuses to boot the same way. Two variables have no YAML key and are exempt because they are not config keys at all but process-level settings `main` reads directly: `WH_CONFIG` (below), which locates the file, and `WH_LOG_LEVEL`. Only the `WH_` prefix is checked, since the environment always carries names that aren't WaveHouse's. One outside source does share the prefix. Kubernetes injects `{SERVICE}_SERVICE_HOST`, `{SERVICE}_PORT`, and similar link variables into every pod in a Service's own namespace, for each Service with a cluster IP that existed before the pod started (a headless Service injects nothing, and a Service in another namespace is harmless). The name is uppercased with `-` mapped to `_`, so a Service named `wh` produces `WH_SERVICE_HOST` and `WH_PORT`, one named `wh-foo` produces `WH_FOO_SERVICE_HOST` and `WH_FOO_PORT`, and either way the pod refuses to boot on its next restart. Set `enableServiceLinks: false` on the pod spec, or name the Service something else. The error says so. -5. Before anything dials out, `data_dir` is probed, and boot refuses on any of these: the value is empty; the path exists but is not a directory; the path, or any component above it, is a dangling symlink (a mount that never came up); the directory exists but the process cannot write to it; the directory is absent and its nearest existing ancestor is not writable, so it could not be created. The probe runs before ClickHouse discovery, so the refusal lands at the top of the log, and a permission denial — on the write probe, or on reaching the path at all through a parent without search permission — carries the UID-65532 remediation, since a bind mount owned by root is the typical cause. +5. Before anything dials out, `data_dir` is probed — when a selected [backend](#backends) keeps state there, as the in-process `mq` and `dedupe` backends do — and boot refuses on any of these: the value is empty; the path exists but is not a directory; the path, or any component above it, is a dangling symlink (a mount that never came up); the directory exists but the process cannot write to it; the directory is absent and its nearest existing ancestor is not writable, so it could not be created. The probe runs before ClickHouse discovery, so the refusal lands at the top of the log, and a permission denial — on the write probe, or on reaching the path at all through a parent without search permission — carries the UID-65532 remediation, since a bind mount owned by root is the typical cause. Boot is the validator for this half of configuration: there is no dry run, and a refused boot with the offending key, variable, or path named in the error is the loud signal. The hot-reloadable half has a dry run — `wavehouse validate` — because it is edited under a running server; boot config only ever takes effect through a restart, so the restart is where it is checked. @@ -37,6 +37,19 @@ This page is boot config only — what the platform operator owns (wiring, lifec | --- | --- | ------- | ----------- | | `data_dir` | `WH_DATA_DIR` | `./data` | Root directory for embedded state. NATS JetStream lives at `/nats`; Pebble, holding every tenant's dedupe store while any tenant has dedupe enabled, at `/pebble`. Subdirectory names are conventions, not config — one knob, one mount. **In a container this MUST resolve to a host-backed volume**; the relative default is for local binary use. WaveHouse logs a startup `WARN` when the directory is missing or empty (no prior state). See [Persistent Storage](/deployment#persistent-storage-required-for-containers). | +### Backends + +Each layer's implementation is chosen once, at boot. Today every layer has one backend, the in-process one, and it is the default, so a config that sets none of these keys runs as it always has. A value this build has no backend for refuses boot and names the valid ones. + +| YAML Key | Env Var | Default | Description | +| --- | --- | ------- | ----------- | +| `mq.backend` | `WH_MQ_BACKEND` | `embedded` | The message queue. `embedded`: NATS JetStream inside this process, under `/nats`. It listens on no port, so no other process can reach its queue. | +| `cache.backend` | `WH_CACHE_BACKEND` | `local` | The query-result cache. `local`: in this process, sized by `cache.l1_max_cost`. | +| `dedupe.backend` | `WH_DEDUPE_BACKEND` | `pebble` | Where ingest dedupe keeps the event ids it has seen. `pebble`: in this process, under `/pebble`, open while any tenant has dedupe on. | +| `coord.backend` | `WH_COORD_BACKEND` | `local` | Where the leases for work only one process may do at a time are held. `local`: in this process. | + +Settings for one backend go in a sub-block named after it, `.`, read only when that backend is selected. A sub-block for a backend this build does not have is an unknown key and refuses boot. `mq` and `dedupe` also appear in the settings directory's `config.json`, with different keys (`mq.max_bytes_gb`, `dedupe.enabled`, …): those are per-tenant tunables and stay there, and one written in `config.yaml` refuses boot as an unknown key. + ### Server | YAML Key | Env Var | Default | Description | @@ -97,6 +110,8 @@ WaveHouse's per-role caps are sent as per-query `SETTINGS` on its connection, so ### Message Queue (NATS) +This section describes the `embedded` [backend](#backends), the only one today. + Each tenant's queue has its own disk budget, `mq.max_bytes_gb`, a hot-reloadable key in the [Settings Directory](/settings-directory#message-queue) — there is no boot-config knob for it. **Durability.** The embedded server runs with JetStream `SyncAlways`, so every event is `fsync`'d to disk before `POST /v1/ingest` returns `200`. This makes your storage's `fsync` latency your ingest latency floor — see [Durability & Storage](/durability) to check whether your substrate can sustain it. There is no knob to relax this today ([#139](https://github.com/Wave-RF/WaveHouse/issues/139) tracks a configurable group-commit interval). @@ -191,9 +206,19 @@ clickhouse: # headers and pool sizes are settings (config.json) max_total_conns: 0 # ceiling on open native connections; 0 = none +mq: + backend: embedded # in-process NATS JetStream under /nats + cache: + backend: local l1_max_cost: 67108864 +dedupe: + backend: pebble # in-process Pebble under /pebble + +coord: + backend: local + auth: jwt_secret: change-me-in-production # jwks_url and role_claim are settings (config.json) operator_key: "" # non-JWT full-access operator credential (Authorization: Operator , or X-Operator-Key); empty disables @@ -239,7 +264,11 @@ WH_SERVER_SHUTDOWN_TIMEOUT=10 WH_CH_PASSWORD= WH_CH_MAX_TOTAL_CONNS=0 +WH_MQ_BACKEND=embedded +WH_CACHE_BACKEND=local WH_CACHE_L1_MAX_COST=67108864 +WH_DEDUPE_BACKEND=pebble +WH_COORD_BACKEND=local WH_AUTH_JWT_SECRET=change-me-in-production WH_AUTH_OPERATOR_KEY= diff --git a/docs/src/content/docs/settings-directory.mdx b/docs/src/content/docs/settings-directory.mdx index c0e7a19b7..561354627 100644 --- a/docs/src/content/docs/settings-directory.mdx +++ b/docs/src/content/docs/settings-directory.mdx @@ -179,7 +179,7 @@ The tenant tunables. Every key is required (a missing one is a validation error) } ``` -What stays in boot config is only what cannot change under a running process — resource sizing (`data_dir`, `cache.l1_max_cost`, `clickhouse.max_total_conns`), the listeners, the observability exporters — and the **secrets**: `clickhouse.password`, `auth.jwt_secret`, `auth.operator_key`. Secrets never belong in a tracked JSON file, so they stay in the environment and are combined with the wiring here on every (re)connect; rotating one is a restart. See [Configuration](/configuration). Everything else lives here and reloads. +What stays in boot config is only what cannot change under a running process — the implementation each layer runs on (`mq.backend`, `cache.backend`, `dedupe.backend`, `coord.backend`), resource sizing (`data_dir`, `cache.l1_max_cost`, `clickhouse.max_total_conns`), the listeners, the observability exporters — and the **secrets**: `clickhouse.password`, `auth.jwt_secret`, `auth.operator_key`. Secrets never belong in a tracked JSON file, so they stay in the environment and are combined with the wiring here on every (re)connect; rotating one is a restart. See [Configuration](/configuration). Everything else lives here and reloads. ## Deduplication diff --git a/internal/app/app.go b/internal/app/app.go index dc15bf5d7..726dad19a 100644 --- a/internal/app/app.go +++ b/internal/app/app.go @@ -170,6 +170,10 @@ func New(ctx context.Context, opts Options) (app *App, err error) { return nil, err } a.wireObservability(ctx) + // After observability, so an OTLP log pipeline carries them too. + for _, w := range a.cfg.Warnings() { + slog.Warn(w) + } if err := a.wireClickHouse(); err != nil { return nil, err } diff --git a/internal/app/app_test.go b/internal/app/app_test.go index 2ecd77d98..10658ddc3 100644 --- a/internal/app/app_test.go +++ b/internal/app/app_test.go @@ -95,7 +95,10 @@ func testConfig(t *testing.T, settingsDir string) *config.Config { return &config.Config{ DataDir: t.TempDir(), Server: config.Server{Port: closedPort(t), ShutdownTimeout: 2}, - Cache: config.Cache{L1MaxCost: 1 << 20}, + MQ: config.MQ{Backend: config.MQEmbedded}, + Cache: config.Cache{Backend: config.CacheLocal, L1MaxCost: 1 << 20}, + Dedupe: config.Dedupe{Backend: config.DedupePebble}, + Coord: config.Coord{Backend: config.CoordLocal}, Auth: config.Auth{JWTSecret: "unit-test-secret"}, Settings: config.Settings{Dir: settingsDir}, } @@ -535,6 +538,28 @@ func TestNew_NestedDedupeStoreFollowsEachTenant(t *testing.T) { assert.False(t, restored.Open(), "Close releases every open store") } +// Validate refuses a backend no layer has a case for, so the switch's default +// is reached only by a Config built by hand; it must refuse boot, not wire +// nothing. +func TestNew_RefusesALayerWithoutABackend(t *testing.T) { + for _, tc := range []struct { + key string + unset func(*config.Config) + }{ + {"dedupe.backend", func(c *config.Config) { c.Dedupe.Backend = "" }}, + {"mq.backend", func(c *config.Config) { c.MQ.Backend = "" }}, + {"cache.backend", func(c *config.Config) { c.Cache.Backend = "" }}, + } { + t.Run(tc.key, func(t *testing.T) { + guardGlobals(t) + cfg := testConfig(t, writeSettings(t, nil)) + tc.unset(cfg) + _, err := New(t.Context(), Options{Config: cfg}) + require.ErrorContains(t, err, tc.key+` "" has no wiring`) + }) + } +} + // A Pebble instance that cannot open follows the registry's own rule for the // shape: a flat directory refuses boot, like every other store, and a nested // one fails closed for every tenant with dedupe on, since they share the diff --git a/internal/app/wire.go b/internal/app/wire.go index 60cbdcd82..d53720253 100644 --- a/internal/app/wire.go +++ b/internal/app/wire.go @@ -471,7 +471,18 @@ func (a *App) wireDiscovery(ctx context.Context) { a.add(component{name: "schema discovery", close: d.close}) } -// wireDedupe builds the dedupe stores: one per tenant (#583 story 7), each +// wireDedupe builds the dedupe stores — the one place the implementation is +// chosen. +func (a *App) wireDedupe() error { + switch b := a.cfg.Dedupe.Backend; b { + case config.DedupePebble: + return a.wirePebbleDedupe() + default: + return unreachableBackend("dedupe.backend", b) + } +} + +// wirePebbleDedupe builds the dedupe stores: one per tenant (#583 story 7), each // following its own tenant's hot-reloadable dedupe.enabled, over the // embedded Pebble implementation, which is handed data_dir and decides the // rest: every tenant's seen ids in one instance there, open while any @@ -488,7 +499,7 @@ func (a *App) wireDiscovery(ctx context.Context) { // than silently publishing un-deduped, since the files asked for dedupe; // nested fails closed the same way at boot too, for every tenant with // dedupe on, the next reload retrying, so it never costs the process. -func (a *App) wireDedupe() error { +func (a *App) wirePebbleDedupe() error { nested := a.tenants.Nested() embedded := dedupe.NewEmbedded(a.cfg.DataDir) stores := dedupe.NewStores(embedded.Tenant) @@ -530,10 +541,20 @@ func (a *App) wireDedupe() error { return nil } -// wireMQ starts the MQ — the embedded NATS under data_dir/nats, the one -// place the implementation is chosen; everything after it sees mq.Broker — -// and hands it each served tenant's mq.max_bytes_gb, which opens that -// tenant's queue the first time. The budget is hot-reloadable: after every +// wireMQ starts the MQ — the one place the implementation is chosen; +// everything after it sees mq.Broker. +func (a *App) wireMQ() error { + switch b := a.cfg.MQ.Backend; b { + case config.MQEmbedded: + return a.wireEmbeddedMQ() + default: + return unreachableBackend("mq.backend", b) + } +} + +// wireEmbeddedMQ starts the embedded NATS under data_dir/nats and hands it +// each served tenant's mq.max_bytes_gb, which opens that tenant's queue the +// first time. The budget is hot-reloadable: after every // reload the registry applies, each served tenant's is handed over again, // and the MQ owns how it is split across the tenant's queues and keeps them // consistent (see mq.Broker.SetMaxBytes). A tenant no longer served keeps @@ -543,7 +564,7 @@ func (a *App) wireDedupe() error { // previous budget; a nested directory logs it at boot too, so it never costs // the process — the tenant's ingest answers 503 until a reload opens its // queue. The hook is registered before the boot apply, as the dedupe one is. -func (a *App) wireMQ() error { +func (a *App) wireEmbeddedMQ() error { dir := filepath.Join(a.cfg.DataDir, "nats") config.WarnIfFreshDataDir("nats", dir) var broker mq.Broker @@ -591,16 +612,28 @@ func (a *App) wireMQ() error { return nil } -// wireCache opens the L1 cache — the only tier in standalone mode. +// wireCache opens the query-result cache — the one place the implementation +// is chosen. func (a *App) wireCache() error { - l1, err := cache.NewLocal(a.cfg.Cache.L1MaxCost) - if err != nil { - return fmt.Errorf("cache init: %w", err) + switch b := a.cfg.Cache.Backend; b { + case config.CacheLocal: + l1, err := cache.NewLocal(a.cfg.Cache.L1MaxCost) + if err != nil { + return fmt.Errorf("cache init: %w", err) + } + a.cache = l1 + a.add(component{name: "cache", close: withoutContext(l1.Close)}) + return nil + default: + return unreachableBackend("cache.backend", b) } - // TODO: eventually this is where we can switch between ristretto, redis, tiered (both), etc - a.cache = l1 - a.add(component{name: "cache", close: withoutContext(l1.Close)}) - return nil +} + +// unreachableBackend is each layer switch's default case. config.Validate +// refuses a backend with no case, so reaching it means a Config built by hand +// without one (the zero value is not the default), or a case missing here. +func unreachableBackend[T ~string](key string, got T) error { + return fmt.Errorf("%s %q has no wiring: a Config built without config.Load must name every backend", key, got) } // wireSweeper adds the active sweeper — purges messages that are both diff --git a/internal/config/backends.go b/internal/config/backends.go new file mode 100644 index 000000000..8328cea61 --- /dev/null +++ b/internal/config/backends.go @@ -0,0 +1,143 @@ +package config + +import ( + "fmt" + "slices" + "strings" +) + +// Each layer's implementation is chosen here, once, at boot: `.backend` +// names it, and the default is today's in-process one. Settings for one +// backend go in `.`, a sub-block read only when that backend +// is selected. Adding a backend is its constant in the layer's list, a case +// in the layer's validate for its sub-block, and a case in the layer's +// wire function in internal/app — nothing else in Validate changes. + +// MQBackend names the message queue implementation. +type MQBackend string + +// MQEmbedded is the NATS JetStream server inside this process, under +// /nats. +const MQEmbedded MQBackend = "embedded" + +var mqBackends = []MQBackend{MQEmbedded} + +// MQ selects the message queue. The per-tenant byte budget, mq.max_bytes_gb, +// is a settings-directory key, not this block's. +type MQ struct { + Backend MQBackend `yaml:"backend" env:"WH_MQ_BACKEND" env-default:"embedded"` +} + +func (m MQ) validate() error { + return checkBackend("mq.backend", "WH_MQ_BACKEND", m.Backend, mqBackends) +} + +// CacheBackend names the query-result cache implementation. +type CacheBackend string + +// CacheLocal is the in-process Ristretto cache, sized by cache.l1_max_cost. +const CacheLocal CacheBackend = "local" + +var cacheBackends = []CacheBackend{CacheLocal} + +// Cache selects and sizes the query-result cache. The time-range bucket +// structured queries normalize to is a settings-directory key +// (query.timestamp_bucket_seconds) — query shaping, not process memory. +type Cache struct { + Backend CacheBackend `yaml:"backend" env:"WH_CACHE_BACKEND" env-default:"local"` + L1MaxCost int64 `yaml:"l1_max_cost" env:"WH_CACHE_L1_MAX_COST" env-default:"67108864"` +} + +func (c Cache) validate() error { + return checkBackend("cache.backend", "WH_CACHE_BACKEND", c.Backend, cacheBackends) +} + +// DedupeBackend names where ingest dedupe keeps the ids it has seen. +type DedupeBackend string + +// DedupePebble is the Pebble instance inside this process, under +// /pebble, opened while any tenant has dedupe on. +const DedupePebble DedupeBackend = "pebble" + +var dedupeBackends = []DedupeBackend{DedupePebble} + +// Dedupe selects the dedupe store. Whether a tenant dedupes, and on which +// field, are settings-directory keys, not this block's. +type Dedupe struct { + Backend DedupeBackend `yaml:"backend" env:"WH_DEDUPE_BACKEND" env-default:"pebble"` +} + +func (d Dedupe) validate() error { + return checkBackend("dedupe.backend", "WH_DEDUPE_BACKEND", d.Backend, dedupeBackends) +} + +// CoordBackend names the lease implementation singleton work (the sweeper) +// is elected through. +type CoordBackend string + +// CoordLocal holds leases in this process: correct while no other process +// shares its queue. +const CoordLocal CoordBackend = "local" + +var coordBackends = []CoordBackend{CoordLocal} + +// Coord selects the coordination layer. +type Coord struct { + Backend CoordBackend `yaml:"backend" env:"WH_COORD_BACKEND" env-default:"local"` +} + +func (c Coord) validate() error { + return checkBackend("coord.backend", "WH_COORD_BACKEND", c.Backend, coordBackends) +} + +// checkBackend refuses a backend this build has no implementation for, +// listing the ones it has. env repeats the struct tag's literal: a tag can't +// reference a constant. +func checkBackend[T ~string](key, env string, got T, valid []T) error { + if slices.Contains(valid, got) { + return nil + } + names := make([]string, len(valid)) + for i, v := range valid { + names[i] = string(v) + } + return fmt.Errorf("%s (%s) %q is not a backend this build has; valid: %s", key, env, got, strings.Join(names, ", ")) +} + +// validateBackends checks every layer's backend and its sub-block. +func (c *Config) validateBackends() error { + for _, check := range []func() error{c.MQ.validate, c.Cache.validate, c.Dedupe.validate, c.Coord.validate} { + if err := check(); err != nil { + return err + } + } + return nil +} + +// Distributed reports whether the message queue is shared with other +// processes. The embedded one listens on no port, so while it is selected +// every process is an island: nothing else can reach its queue. +func (c *Config) Distributed() bool { return c.MQ.Backend != MQEmbedded } + +// NeedsDataDir reports whether a selected backend keeps state under data_dir, +// and so whether boot must probe it (CheckDataDir). +func (c *Config) NeedsDataDir() bool { + return c.MQ.Backend == MQEmbedded || c.Dedupe.Backend == DedupePebble +} + +// Warnings returns what a valid configuration is still likely to get wrong, +// one line each, for boot to log at WARN. They are not errors because each is +// correct for a single replica, and one process cannot count its replicas. +func (c *Config) Warnings() []string { + if !c.Distributed() { + return nil + } + var out []string + if c.Cache.Backend == CacheLocal { + out = append(out, "cache.backend=local with a shared mq.backend is correct for one replica only: an event ingested on another replica never invalidates this one's cache, so its reads stay stale until the cached entry expires") + } + if c.Dedupe.Backend == DedupePebble { + out = append(out, "dedupe.backend=pebble with a shared mq.backend dedupes per replica only: an id seen by another replica is not seen by this one") + } + return out +} diff --git a/internal/config/backends_test.go b/internal/config/backends_test.go new file mode 100644 index 000000000..0e70dc7a3 --- /dev/null +++ b/internal/config/backends_test.go @@ -0,0 +1,161 @@ +package config + +import ( + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// withDefaultBackends sets what Load's env-defaults would: a literal Config +// names no backend, and Validate refuses that. +func withDefaultBackends(c Config) *Config { + c.MQ.Backend, c.Cache.Backend = MQEmbedded, CacheLocal + c.Dedupe.Backend, c.Coord.Backend = DedupePebble, CoordLocal + return &c +} + +func defaultBackends() Config { + return *withDefaultBackends(Config{Server: Server{Port: 8080}, Settings: Settings{Dir: "./settings"}}) +} + +func TestLoad_BackendDefaults(t *testing.T) { + t.Parallel() + cfg, err := Load("nonexistent.yaml") + require.NoError(t, err) + assert.Equal(t, MQEmbedded, cfg.MQ.Backend) + assert.Equal(t, CacheLocal, cfg.Cache.Backend) + assert.Equal(t, DedupePebble, cfg.Dedupe.Backend) + assert.Equal(t, CoordLocal, cfg.Coord.Backend) + assert.False(t, cfg.Distributed()) + assert.True(t, cfg.NeedsDataDir()) + assert.Empty(t, cfg.Warnings()) +} + +func TestLoad_BackendsFromEnv(t *testing.T) { + t.Setenv("WH_MQ_BACKEND", "embedded") + t.Setenv("WH_CACHE_BACKEND", "local") + t.Setenv("WH_DEDUPE_BACKEND", "pebble") + t.Setenv("WH_COORD_BACKEND", "local") + cfg, err := Load("nonexistent.yaml") + require.NoError(t, err) + assert.Equal(t, MQEmbedded, cfg.MQ.Backend) + assert.Equal(t, CoordLocal, cfg.Coord.Backend) +} + +func TestLoad_BackendFromEnvRefusesAnUnknownValue(t *testing.T) { + t.Setenv("WH_MQ_BACKEND", "nats") + _, err := Load("nonexistent.yaml") + require.Error(t, err) + assert.Contains(t, err.Error(), `mq.backend (WH_MQ_BACKEND) "nats" is not a backend this build has; valid: embedded`) +} + +func TestLoad_BackendsFromYAML(t *testing.T) { + t.Parallel() + path := filepath.Join(t.TempDir(), "config.yaml") + require.NoError(t, os.WriteFile(path, []byte(` +mq: + backend: embedded +cache: + backend: local + l1_max_cost: 1024 +dedupe: + backend: pebble +coord: + backend: local +`), 0o600)) + cfg, err := Load(path) + require.NoError(t, err) + assert.Equal(t, MQEmbedded, cfg.MQ.Backend) + assert.Equal(t, CacheLocal, cfg.Cache.Backend) + assert.Equal(t, int64(1024), cfg.Cache.L1MaxCost) + assert.Equal(t, DedupePebble, cfg.Dedupe.Backend) + assert.Equal(t, CoordLocal, cfg.Coord.Backend) +} + +// A sub-block written before its backend exists, and a settings-directory +// key under a block both files share, are unknown keys — not read and ignored. +func TestLoad_BackendBlocksRefuseUnknownKeys(t *testing.T) { + t.Parallel() + path := filepath.Join(t.TempDir(), "config.yaml") + require.NoError(t, os.WriteFile(path, []byte(` +mq: + backend: embedded + max_bytes_gb: 5 + nats: + urls: nats://localhost:4222 +dedupe: + enabled: true +`), 0o600)) + _, err := Load(path) + require.Error(t, err) + assert.Contains(t, err.Error(), "dedupe.enabled, mq.max_bytes_gb, mq.nats") + assert.Contains(t, err.Error(), EnvSettingsDir) +} + +func TestUnboundEnv_KnowsTheBackendVariables(t *testing.T) { + t.Parallel() + assert.Empty(t, unboundEnv([]string{ + "WH_MQ_BACKEND=embedded", "WH_CACHE_BACKEND=local", + "WH_DEDUPE_BACKEND=pebble", "WH_COORD_BACKEND=local", + })) +} + +func TestValidate_UnknownBackend(t *testing.T) { + t.Parallel() + cases := []struct { + name string + set func(*Config) + want string + }{ + {"mq", func(c *Config) { c.MQ.Backend = "kafka" }, `mq.backend (WH_MQ_BACKEND) "kafka" is not a backend this build has; valid: embedded`}, + {"cache", func(c *Config) { c.Cache.Backend = "redis" }, `cache.backend (WH_CACHE_BACKEND) "redis" is not a backend this build has; valid: local`}, + {"dedupe", func(c *Config) { c.Dedupe.Backend = "dynamodb" }, `dedupe.backend (WH_DEDUPE_BACKEND) "dynamodb" is not a backend this build has; valid: pebble`}, + {"coord", func(c *Config) { c.Coord.Backend = "nats" }, `coord.backend (WH_COORD_BACKEND) "nats" is not a backend this build has; valid: local`}, + // The zero value, which a Config built without Load carries. + {"empty", func(c *Config) { c.MQ.Backend = "" }, `mq.backend (WH_MQ_BACKEND) "" is not a backend`}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + cfg := defaultBackends() + require.NoError(t, cfg.Validate()) + tc.set(&cfg) + err := cfg.Validate() + require.Error(t, err) + assert.Contains(t, err.Error(), tc.want) + }) + } +} + +// Every warning keys on a shared queue, which no backend offers yet, so the +// value is set directly: Warnings reads the choice, it doesn't validate it. +func TestWarnings_SharedQueue(t *testing.T) { + t.Parallel() + cfg := defaultBackends() + assert.Empty(t, cfg.Warnings()) + + cfg.MQ.Backend = "shared" + require.True(t, cfg.Distributed()) + got := cfg.Warnings() + require.Len(t, got, 2) + assert.Contains(t, got[0], "cache.backend=local") + assert.Contains(t, got[1], "dedupe.backend=pebble") + + cfg.Cache.Backend, cfg.Dedupe.Backend = "shared", "shared" + assert.Empty(t, cfg.Warnings()) +} + +func TestNeedsDataDir(t *testing.T) { + t.Parallel() + cfg := defaultBackends() + assert.True(t, cfg.NeedsDataDir()) + cfg.MQ.Backend = "shared" + assert.True(t, cfg.NeedsDataDir(), "pebble dedupe still keeps state under data_dir") + cfg.Dedupe.Backend = "shared" + assert.False(t, cfg.NeedsDataDir()) + cfg.MQ.Backend = MQEmbedded + assert.True(t, cfg.NeedsDataDir(), "the embedded mq keeps state under data_dir") +} diff --git a/internal/config/config.go b/internal/config/config.go index 68b0314b6..cc7566965 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -19,7 +19,10 @@ type Config struct { DataDir string `yaml:"data_dir" env:"WH_DATA_DIR" env-default:"./data"` Server Server `yaml:"server"` ClickHouse ClickHouse `yaml:"clickhouse"` + MQ MQ `yaml:"mq"` Cache Cache `yaml:"cache"` + Dedupe Dedupe `yaml:"dedupe"` + Coord Coord `yaml:"coord"` Auth Auth `yaml:"auth"` OTel OTel `yaml:"otel"` Prometheus Prometheus `yaml:"prometheus"` @@ -132,13 +135,6 @@ type ClickHouse struct { MaxTotalConns int `yaml:"max_total_conns" env:"WH_CH_MAX_TOTAL_CONNS" env-default:"0"` } -// Cache sizes the in-process L1 cache. The time-range bucket structured -// queries normalize to is a settings-directory key -// (query.timestamp_bucket_seconds) — query shaping, not process memory. -type Cache struct { - L1MaxCost int64 `yaml:"l1_max_cost" env:"WH_CACHE_L1_MAX_COST" env-default:"67108864"` -} - // Auth holds the authentication secrets. The verifier wiring — `jwks_url`, // `role_claim` — is the settings directory's `auth` block (hot-reloadable: // a change rebuilds the verifier). There is no on/off switch: the middleware always runs. A request @@ -224,7 +220,7 @@ func (c *Config) Validate() error { } } - return nil + return c.validateBackends() } // Load reads config from a YAML file (if it exists) with env var overrides. diff --git a/internal/config/config_test.go b/internal/config/config_test.go index 822d639d1..ee8b07cc5 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -203,7 +203,7 @@ func TestValidate_SampleRatesIgnoredWhenObservabilityDisabled(t *testing.T) { Logs: OTelLogs{SampleRate: -1}, }, } - assert.NoError(t, cfg.Validate()) + assert.NoError(t, withDefaultBackends(cfg).Validate()) } func TestValidate_SampleRatesIgnoredWhenSignalDisabled(t *testing.T) { @@ -219,7 +219,7 @@ func TestValidate_SampleRatesIgnoredWhenSignalDisabled(t *testing.T) { Logs: OTelLogs{Enabled: false, SampleRate: -1}, }, } - assert.NoError(t, cfg.Validate()) + assert.NoError(t, withDefaultBackends(cfg).Validate()) } func TestLoad_Defaults_PrometheusDisabled(t *testing.T) { @@ -336,7 +336,7 @@ func TestValidate_PrometheusV1PathAllowedOnSidecarPort(t *testing.T) { Settings: Settings{Dir: "./settings"}, Prometheus: Prometheus{Enabled: true, Path: "/v1/metrics", Port: 9091}, } - assert.NoError(t, cfg.Validate()) + assert.NoError(t, withDefaultBackends(cfg).Validate()) } func TestValidate_PrometheusOnly_NoOTel(t *testing.T) { @@ -348,7 +348,7 @@ func TestValidate_PrometheusOnly_NoOTel(t *testing.T) { Settings: Settings{Dir: "./settings"}, Prometheus: Prometheus{Enabled: true, Path: "/metrics", Port: 0}, } - assert.NoError(t, cfg.Validate()) + assert.NoError(t, withDefaultBackends(cfg).Validate()) } func TestValidate_PrometheusIgnoredWhenDisabled(t *testing.T) { @@ -365,7 +365,7 @@ func TestValidate_PrometheusIgnoredWhenDisabled(t *testing.T) { Port: 8080, }, } - assert.NoError(t, cfg.Validate()) + assert.NoError(t, withDefaultBackends(cfg).Validate()) } // TestEnvSettingsDir_MatchesStructTag pins the exported constant to the diff --git a/tests/integration/setup_test.go b/tests/integration/setup_test.go index a01064a29..ade560f7c 100644 --- a/tests/integration/setup_test.go +++ b/tests/integration/setup_test.go @@ -161,7 +161,10 @@ func setup() (int, func()) { DataDir: dataDir, Server: config.Server{ShutdownTimeout: 10}, ClickHouse: config.ClickHouse{Password: testCHPassword}, - Cache: config.Cache{L1MaxCost: 1 << 30}, // 1 GB + MQ: config.MQ{Backend: config.MQEmbedded}, + Cache: config.Cache{Backend: config.CacheLocal, L1MaxCost: 1 << 30}, // 1 GB + Dedupe: config.Dedupe{Backend: config.DedupePebble}, + Coord: config.Coord{Backend: config.CoordLocal}, Settings: config.Settings{Dir: settingsDir}, } a, err := app.New(ctx, app.Options{Config: cfg, Listener: ln}) diff --git a/tests/integration/tenants_test.go b/tests/integration/tenants_test.go index 40c5a700c..ca00f42f3 100644 --- a/tests/integration/tenants_test.go +++ b/tests/integration/tenants_test.go @@ -62,7 +62,10 @@ func TestNestedDirectory_PerTenantPoolsAndDiscovery(t *testing.T) { Server: config.Server{ShutdownTimeout: 10}, ClickHouse: config.ClickHouse{Password: testCHPassword}, Auth: config.Auth{OperatorKey: operatorKey}, - Cache: config.Cache{L1MaxCost: 1 << 20}, + MQ: config.MQ{Backend: config.MQEmbedded}, + Cache: config.Cache{Backend: config.CacheLocal, L1MaxCost: 1 << 20}, + Dedupe: config.Dedupe{Backend: config.DedupePebble}, + Coord: config.Coord{Backend: config.CoordLocal}, Settings: config.Settings{Dir: root}, } a, err := app.New(ctx, app.Options{Config: cfg, Listener: ln}) From fde17ba4578a2a03521e085a77fd2d3326a75c7d Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Thu, 24 Sep 2026 23:30:43 -0400 Subject: [PATCH 02/45] docs(config): say coord.backend is reserved; sync the boot-config lists Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01EJr5tY4WQUy2sc4MbW67vL --- CHANGELOG.md | 2 +- config.yaml | 2 +- docs/src/content/docs/architecture.md | 7 ++++--- docs/src/content/docs/configuration.mdx | 4 ++-- internal/app/wire.go | 2 +- internal/config/backends.go | 8 ++++---- internal/settings/settings.go | 8 +++++--- 7 files changed, 18 insertions(+), 15 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9f2e821bc..6564775a8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **Each layer's implementation is chosen at boot** (`internal/config/{backends,config}.go` (+ tests), `internal/app/{app,wire}.go` (+ tests), `cmd/wavehouse/main.go`, `tests/integration/{setup,tenants}_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,settings-directory.mdx,architecture.md}`): the first step of running WaveHouse as more than one process ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config gains `mq.backend` (`WH_MQ_BACKEND`, default `embedded`), `cache.backend` (`WH_CACHE_BACKEND`, `local`), `dedupe.backend` (`WH_DEDUPE_BACKEND`, `pebble`) and `coord.backend` (`WH_COORD_BACKEND`, `local`). Each layer has only its in-process backend so far, and it is the default, so nothing changes for a config that sets none of them; a value with no backend refuses boot and names the valid ones. A backend's own settings will go in a `.` sub-block, which until that backend lands is an unknown key and refuses boot. `internal/app` picks each implementation in one `switch` per layer (`wireMQ`, `wireCache`, `wireDedupe`), `data_dir` is probed only when a selected backend keeps state there (`Config.NeedsDataDir`), and boot logs at `WARN` each line of `Config.Warnings`, the combinations that are correct for one replica only once a shared queue exists. A `config.Config` built without `config.Load` must now name every backend: the zero value is not the default, and `app.New` refuses it. +- **Each layer's implementation is chosen at boot** (`internal/config/{backends,config}.go` (+ tests), `internal/app/{app,wire}.go` (+ tests), `cmd/wavehouse/main.go`, `tests/integration/{setup,tenants}_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,settings-directory.mdx,architecture.md}`): the first step of running WaveHouse as more than one process ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config gains `mq.backend` (`WH_MQ_BACKEND`, default `embedded`), `cache.backend` (`WH_CACHE_BACKEND`, `local`), `dedupe.backend` (`WH_DEDUPE_BACKEND`, `pebble`) and `coord.backend` (`WH_COORD_BACKEND`, `local`). Each layer has only its in-process backend so far, and it is the default, so nothing changes for a config that sets none of them; a value with no backend refuses boot and names the valid ones. A backend's own settings will go in a `.` sub-block, which until that backend lands is an unknown key and refuses boot. `internal/app` picks each implementation in one `switch` per layer (`wireMQ`, `wireCache`, `wireDedupe`), `data_dir` is probed only when a selected backend keeps state there (`Config.NeedsDataDir`), and boot logs at `WARN` each line of `Config.Warnings`, the combinations that are correct for one replica only once a shared queue exists. A `config.Config` built without `config.Load` must now name the `mq`, `cache` and `dedupe` backends: the zero value is not the default, and `app.New` refuses it. `coord.backend` is reserved: nothing reads it until the lease layer lands, and the sweeper still runs in every process. - **A tenant removed or rejected at runtime has its open streams ended** (`internal/stream/{hub,subscriber,bucket}.go` (+ tests), `internal/api/stream.go` (+ tests), `internal/api/router.go`, `internal/settings/{registry,tree}.go` (+ tests), `internal/ingest/worker.go` (+ tests), `internal/app/wire.go` (+ tests), `clients/ts/src/stream/sse.ts`, `docs/src/content/docs/{api,deployment,architecture,ingest-pipeline}.md`, `docs/src/content/docs/settings-directory.mdx`, `AGENTS.md`): story 3 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)). A `GET /v1/stream` used to outlive its tenant: the hub read no policy for it and withheld every row while the keepalive wheel held the connection open, so the client could not tell it from a quiet table. After every reload the hub now evicts the subscribers of each tenant no longer served, removed or rejected alike (`Hub.Prune`, through the close-once `Subscriber.Evict`), and the handler ends the stream: a gap-fill in progress included, and a stream `TenantMW` admitted just before the reload but registered just after it, which the handler checks for as it registers. The client's reconnect gets `404` (removed) or `503` (rejected); the SDK stops on the first and retries the second, resuming from `Last-Event-ID` once the folder is back — except in a browser going cross-origin where tenant `0` is not served or its CORS list does not admit the page, which cannot read either refusal (both are decorated from tenant `0`'s list) and re-dials as after a dropped connection. A flat directory never stops serving tenant `0`, so nothing changes there. Removing a tenant is deleting its folder, then reloading the whole directory, the last folder included: a server started with tenant folders reads the emptied directory as no folder left, where it read as a change of shape and the reload was rejected whole, leaving that tenant served; at boot an empty directory still reads as the four files, missing. Reloading the deleted folder by name leaves its tenant rejected. `GET /v1/health` keeps resolving a tenant, deliberately: its `404` tells a caller with no token no more than every tenant route's does, since they all answer before authenticating, and resolving answers a served tenant's ping from that tenant's own CORS list. A batch queued for a tenant with no ClickHouse connection — one no longer served, or one no pool could be opened for (such as by the connection ceiling) — skips the row-by-row retry, which no row of it could pass, and meets its DLQ switch once, whole, logged once per batch rather than twice per row; a tenant no longer served reads the switch as on, so its queued rows are parked under its own subject rather than dropped, inserted into another tenant's ClickHouse, or left unacked, redelivered for as long as the tenant is away and holding its queue's ack floor, which the purge waits on. Over a nested directory `/livez` no longer keeps naming a tenant that stopped being served before any tenant completed a first discovery: the diagnostic goes back to `no tenant has completed a first discovery yet`. - **One ClickHouse pool per tuple and one schema registry per tenant** (`internal/chconn/chconn.go` (+ tests), `internal/discovery/discovery.go` (+ tests), `internal/app/discoveries.go` (new), `internal/app/{app,wire}.go` (+ tests), `internal/api/{schema,ingest,structured_query,pipes,query,health,errors,clickhouse_exec}.go` (+ tests), `internal/stream/hub.go`, `internal/ingest/worker.go`, `internal/cache/{cache,local,version_manager}.go` (+ tests), `internal/testutil/{testutil,mocks}.go`, `tests/integration/{setup,tenants,boot_resilience,query_limits}_test.go`, `clients/ts/src/{schema,table,sql,client,types}.ts` (+ tests), `tests/e2e/sdk/admin.test.ts`, `docs/src/content/docs/{api,deployment,architecture,ingest-pipeline}.md`, `docs/src/content/docs/{settings-directory,configuration,access-control,reverse-proxy}.mdx`, `docs/src/content/docs/sdk/{admin,reference,queries}.md`, `AGENTS.md`): the second slice of story 6 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)), with no behavior change for a settings directory that holds the four files beyond the three noted at the end. The process opens one native pool per distinct `clickhouse.addr` / `database` / `username` / password / `tls` tuple among the served tenants (`chconn.Identity`, `chconn.Pools`), shared by the tenants naming it and sized to their largest `max_open_conns` and `max_idle_conns`; `http_port`, `http_scheme`, `headers` and `query_timeout` stay each tenant's own. Every reload reconciles the pools: a new tuple opens (never dials), a tenant whose tuple changed is repointed, a tuple no tenant names closes after the longest `query_timeout` among the tenants it had, and a pool whose largest ask changed is resized with the same grace. The boot config's `clickhouse.max_total_conns` now bounds the open pools together: boot is refused naming the sum and the ceiling; at a reload a resize above it is refused with the pool kept at its size, and a tuple that cannot be opened — the ceiling, a certificate file that cannot be read, or options the driver refuses — leaves its tenants on the pool they had (the keep-previous-wiring rule of the connection ceiling) or on none when they had none; both are logged and the next reload retries. A tenant on no pool fails closed: `503` with `Retry-After: 30` on `POST /v1/query`, `GET/POST /v1/pipes/{name}`, `POST /v1/ops/query` and `POST /v1/ops/schema/refresh`, ahead of the cache. Each served tenant gets a `discovery.SchemaRegistry` of its own over its pool, kept fresh by its own loop — the boot retry until the first success, then `schema.refresh_interval` with the first refresh at a random point within the interval so tenants adopted together do not refresh together — created and stopped from the settings reload and stopped under `App.Close`; `wavehouse_schema_refresh_failures_total{tenant}` counts a loop's failed attempts. `GET /v1/ops/schema`, `POST /v1/ops/schema/refresh` and `POST /v1/ops/query` take the strict `?tenant=` the pipe reads take (absent is tenant `0`, `400` malformed, `404` unknown, `503` rejected); the SDK sends it as the `tenant` option of `wh.schema.list()`, `wh.schema.refresh()`, `wh.from(t).schema()` and `wh.sql()`. Over a nested directory `/livez` (and `/v1/health`) is `503` with the latest discovery failure, naming its tenant, while no tenant has completed a first discovery, then `200` for the rest of the process lifetime; `/readyz` pings every open pool at once, is ready at the first answer, and names every pool that did not answer when none does — one tenant's ClickHouse outage is its log line and counter, never a probe failure. The ingest worker inserts each batch into its own tenant's ClickHouse — the tenant the message's topic names, through that tenant's HTTP wiring (`chconn.Pools.Target`); a tenant on no pool takes the failure path an unreachable ClickHouse takes — and its cache invalidation fans out to the tenants on the same address and database as the batch's tenant, whatever their user or `tls` block (they read the same tables), rather than to every known tenant, and a tenant adopted after an absence — rejected or removed, so out of that fan-out — or moved to another address or database has its cached structured-query results orphaned in one step (`Cache.InvalidateTenant`, a tenant generation in every version key), so a repaired folder never serves query rows cached before the inserts it missed (a pipe result names no table, so neither this nor any insert invalidates it: it stays until its TTL expires). Three changes reach the single-tenant directory too: a table lookup before the first discovery is a `503` with `Retry-After: 5` (`schema not loaded yet`) rather than a `404`, on `POST /v1/ingest`, `POST /v1/query` and `GET /v1/ops/schema` (the list included, where `[]` would read as no tables); the first periodic refresh fires at a random point within the interval rather than a full interval after boot; and a reload that moves `clickhouse.addr` or `clickhouse.database` now orphans the structured-query results cached before it, where they were served until their TTL. Until [#529](https://github.com/Wave-RF/WaveHouse/issues/529) every tenant's user authenticates with `WH_CH_PASSWORD`, so the tuple is in effect the address, database, username and `tls` block. - **One token verifier per tenant, built off the boot and reload paths** (`internal/auth/auth.go` (+ tests), `internal/api/router.go` (+ tests), `internal/app/{app,wire}.go` (+ tests), `go.mod`, `docs/src/content/docs/sdk/{reference,streaming}.md`, `docs/src/content/docs/{architecture,deployment,api}.md`, `docs/src/content/docs/{settings-directory,configuration}.mdx`, `SECURITY.md`): story 9 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)). Over a nested settings directory each tenant's folder now wires that tenant's verifier (`auth.jwks_url`, `auth.role_claim`), so a JWKS-issued token verifies only under the tenants whose `jwks_url` names its identity provider's key set — under any other tenant's header it is refused as invalid — where before one verifier, tenant `0`'s, accepted a token under any header. `auth.Config` is now the boot-config half alone (the HMAC secret and the operator key, shared by every tenant) and the new `auth.Wiring` a tenant's half; `NewAuthenticator` builds no verifier, `Reconfigure(id, wiring)` gives a tenant one — swapped atomically when its wiring changed, kept when it did not — `Prune` drops the verifiers of the tenants a reload stopped serving, rejected or removed alike (no work runs for a tenant that is not served; a folder adopted again gets a fresh verifier), and `Close` stops every JWKS refresh as a component of `App.Close`. This holds on every reload because the registry's `AfterAdopt` hooks run after every reload it applied, empty list included, so a per-tenant reload that rejects a folder drops its verifier then rather than at the next adoption. The middleware reads the request tenant's verifier through an injected `auth.TenantSource` — the store `api.TenantMW` resolved names its tenant with `settings.Store.Tenant()`, one read of the context — a tenant-exempt route (the ops tree) verifies as tenant `0`, and a tenant with no verifier fails closed. The operator key stamps the request tenant's `admin_role`, read from that tenant's policy, rather than tenant `0`'s. A JWKS key set is fetched on its own goroutine: the verifier is in place at once and *pending* until a set has been stored — the first fetch retried with backoff from one second to a minute, a stored set then kept fresh by the library hourly and, rate-limited, on an unknown key id — so neither boot nor a reload (which holds the registry's lock) waits on the endpoint, and **an unreachable JWKS no longer refuses boot** — it logs (`jwks refresh failed; no token validates until it succeeds`) and that tenant alone is affected. A token checked against a pending verifier is a new outcome, `auth.ErrVerifierPending`: every `/v1` route answers it `503 {"error": "token verifier not ready: …"}` with `Retry-After: 30` (`api.refuseUnverifiable`) rather than evaluate the request under the `default_role`, which could accept its data under a lesser role while another pod holding the keys would have served it as its own; a request without a token, and the operator key, are unaffected. Every fetch goes through one client that caps the response at 1 MiB, refusing a larger one as unreachable with `jwks response exceeds 1048576 bytes` as the logged cause. Nothing changes for a flat directory beyond that boot rule: its one tenant gets exactly the verifier it had. The boot warning for the secretless posture (no `auth.jwt_secret` and no `jwks_url`, whose token refusal landed in [#607](https://github.com/Wave-RF/WaveHouse/pull/607)) is now per tenant — each served tenant with no `jwks_url` while the boot secret is unset — where one line that any JWKS tenant silenced used to stand for the whole directory. diff --git a/config.yaml b/config.yaml index 65c384eaa..5519b78cd 100644 --- a/config.yaml +++ b/config.yaml @@ -50,7 +50,7 @@ mq: dedupe: backend: pebble # Pebble under /pebble coord: - backend: local + backend: local # reserved: nothing is elected yet # In-process L1 cache size. The query time-bucket # (query.timestamp_bucket_seconds) is a settings key. diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 938751ff8..425918076 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -89,7 +89,7 @@ The API layer uses [Chi](https://github.com/go-chi/chi) for routing with Request ### `app/` — Process wiring -- **app.go** — `New(ctx, Options)` builds every component from the boot config (`Options.Config`) and the settings directory it names, in dependency order: settings registry, observability, ClickHouse pools, schema discovery, the dedupe stores, embedded NATS (ingest + DLQ streams), cache, sweeper, streaming (hub, MQ→hub bridge, keepalive wheel), ingest worker, auth, reload triggers, HTTP. Each is one `component` value — what it opens, what it loops, what it releases — so a failure part-way releases what was already opened and returns the error. `Run(ctx)` drives every loop under one `errgroup` until `ctx` is canceled (a clean stop: every loop drains, the API server and the ingest worker within `server.shutdown_timeout`; open SSE streams are ended as the drain begins rather than waited on) or a component fails, which stops the rest and returns that error. `Close(ctx)` releases what `New` opened, newest first, under the caller's release budget (`ReleaseTimeout`, 5s), a real bound: a remote implementation's close gives up at the deadline itself, and a close that ignores the context (the local stores) is abandoned at it, with the components below it left unreleased rather than overlapping it, both named in the error — and then flushes telemetry under its own 3s budget, so the flush that reports on the stop is never handed a deadline a slow close already spent. The SIGHUP registration is released last of all. `Handler`, `Registry`, and `MQ` expose the pieces a harness needs; `Options.Listener` lets one serve the API on its own listener instead of `server.port`. +- **app.go** — `New(ctx, Options)` builds every component from the boot config (`Options.Config`) and the settings directory it names, in dependency order: settings registry, observability (after which each `Config.Warnings` line is logged at `WARN`), ClickHouse pools, schema discovery, the dedupe stores, embedded NATS (ingest + DLQ streams), cache, sweeper, streaming (hub, MQ→hub bridge, keepalive wheel), ingest worker, auth, reload triggers, HTTP. Each is one `component` value — what it opens, what it loops, what it releases — so a failure part-way releases what was already opened and returns the error. `Run(ctx)` drives every loop under one `errgroup` until `ctx` is canceled (a clean stop: every loop drains, the API server and the ingest worker within `server.shutdown_timeout`; open SSE streams are ended as the drain begins rather than waited on) or a component fails, which stops the rest and returns that error. `Close(ctx)` releases what `New` opened, newest first, under the caller's release budget (`ReleaseTimeout`, 5s), a real bound: a remote implementation's close gives up at the deadline itself, and a close that ignores the context (the local stores) is abandoned at it, with the components below it left unreleased rather than overlapping it, both named in the error — and then flushes telemetry under its own 3s budget, so the flush that reports on the stop is never handed a deadline a slow close already spent. The SIGHUP registration is released last of all. `Handler`, `Registry`, and `MQ` expose the pieces a harness needs; `Options.Listener` lets one serve the API on its own listener instead of `server.port`. - **wire.go** — one `wire*` function per component, each handed the settings registry whole and deriving the per-call getters the internal packages take (`DLQFor`, `DedupeFor`, `GapWindow`, …) and registering its `AfterAdopt` hook there where it has one. A layer with a choice of implementation — `wireMQ`, `wireCache`, `wireDedupe` — picks it there and nowhere else, in a `switch` on the boot config's `.backend` with one case per backend; the default case refuses boot, which only a `config.Config` built without `config.Load` reaches, since `Validate` refuses a value no case handles. Those wiring functions are where the per-tenant registry of [#583](https://github.com/Wave-RF/WaveHouse/issues/583) is injected, not `main`: `wireSettings` opens the `settings.Registry`, the HTTP handlers get store-keyed getters (method expressions such as `(*settings.Store).Policy`), and `perTenant` adapts a store accessor into the `func(tenant.ID) T` getter the async packages take, with the tenant each message's `mq.Topic` names for the stream hub and the ingest worker — a tenant the registry is not serving is logged and read as the zero value, except in `dlqFor`, the ingest worker's DLQ switch, where it reads as on so a message the worker cannot read is parked rather than dropped, and a removed or rejected tenant's queued rows are parked rather than left unacked, where they would hold the ack floor and stop the sweeper. The ClickHouse pools (`chconn.Pools`) and the per-tenant schema registries (`discoveries`, in `discoveries.go`) are reconciled from `AfterAdopt` after every reload ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 6): `wireClickHouse` builds each served tenant's `chconn.Member` from its store and logs what the reconcile refused; `wireDiscovery` builds a registry over `pools.For` for each newly served tenant — a flat directory's tenant `0` refreshed synchronously first, as before — runs its loop under the App's stop context, stops the loop of a tenant no longer served, and drives the `BootState` from the first tenant's first discovery, sticky from there; before that, a diagnostic naming a tenant a reload stopped serving goes back to the no-tenant one. The handlers resolve both per request through store-keyed getters (`chConnFor`, `registryFor`, `chTargetFor`, `queryTimeout`), the hub and the ingest worker through tenant-keyed ones (`discoveries.For`, `pools.Target`) called with the tenant the message's topic names; a tenant on no pool is an untyped nil connection, the handlers' `503`. The ingest worker is handed the cache through `sharedTables`, which bumps each namespace the worker invalidates under every tenant on the same ClickHouse address and database (`pools.SharingTables`), and the pools hook orphans the table-keyed cache — the structured-query results — of a tenant back on a pool after an absence (`Cache.InvalidateTenant`), since it was out of that fan-out while away, and of a tenant moved to another address or database, since it now reads other tables (both returned by `Pools.Reconcile`). The one setting that still follows the default tenant is read per request, the admin role of a flat directory's ops gate: `defaultSetting` reads the store tenant `0` last adopted (`App.defaultStore`, tracked by an `onDefaultAdopt` hook that runs only after a reload that adopted it), so a `0` folder that a reload rejects or removes leaves it as it was. The auth verifiers are per tenant: `wireAuth` builds one for each tenant being served, its `AfterAdopt` hook reconfigures the adopted tenants' (rebuilt only when their wiring changed) and prunes the ones no longer served, and the operator key's admin role is read from the request tenant's policy. `wireStreaming`'s hook prunes the stream hub the same way (`Hub.Prune`, with the one `served` predicate the auth and dedupe hooks use too), ending the open streams of a tenant no longer served. One setting is shared by folding over the tenants being served rather than by following tenant `0`: the keepalive wheel runs at the shortest `stream.keepalive_interval` among them (`shortestKeepalive`), re-derived after every reload the registry applies — an adoption, a rejection, or a removal — so a dropped tenant's interval leaves the wheel at once ([#597](https://github.com/Wave-RF/WaveHouse/issues/597)). The sweeper is handed each served tenant's own `stream.gap_window_minutes` (`gapWindows`, read every sweep), since each tenant's events have a queue of their own. The dedupe stores are per tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 7): `wireDedupe`'s `pebble` case builds a `dedupe.Stores` over the `Tenant` factory of the embedded Pebble implementation (`dedupe.NewEmbedded`), handing it `data_dir` once; the implementation decides where every tenant's store lives — one instance, each key led by its tenant (story 3) — and one reconcile closure, the boot apply and the `AfterAdopt` hook alike, sets every store to what the registry says: open exactly when its tenant is served with `dedupe.enabled` on, closed with its seen ids kept when the tenant is switched off, rejected, or removed. An instance that cannot open follows the registry's rule for the shape: fatal at boot over a flat directory, fail-closed for every tenant with dedupe on over a nested one. The system gauges report that one instance's figures (`Embedded.Stats`), not a sum over tenants. The ingest handler picks the tenant's store off the request's `settings.Store` (`Store.Tenant()`). The reload triggers only start in `Run`, after `New` has registered every hook, so the watcher's first reload already drives all of them: SIGHUP in both shapes, the directory watcher for a flat directory only. `wireMQ`'s `embedded` case hands each served tenant's `mq.max_bytes_gb` to `mq.Broker.SetMaxBytes` at boot and again after every reload, under the App's stop context, which opens that tenant's queue the first time; a queue that cannot be opened or resized follows the registry's rule for the shape — fatal at boot over a flat directory, logged over a nested one — and is retried by the next reload. How the budget is split across the tenant's streams, the time bounds, the rollback, and the dead-letter shrink guard are `internal/mq`'s. ### `stream/` — SSE keepalive & fan-out @@ -115,8 +115,9 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ ### `config/` — Configuration -- **config.go** — Loads *boot* configuration from a YAML file with environment variable overrides (using [cleanenv](https://github.com/ilyakaznacheev/cleanenv)); every key has a `WH_`-prefixed env var. Boot config is only what can't change under a running process — resource sizing, listeners, observability exporters, the settings-directory path, and the secrets (`clickhouse.password`, `auth.jwt_secret`, `auth.operator_key`). Everything tenant-tunable lives in the settings directory (`settings/`). Both sources are strict: `Load` refuses to boot naming every YAML key the struct doesn't declare (`strict.go`) and every `WH_*` environment variable no field binds (`check.go`), so a tunable that moved to the settings directory can't be read, ignored, and believed. Boot is the validator for this half — there is no dry-run command. See [Configuration Reference](/configuration). -- **check.go** — `rejectUnboundEnv` is the environment half of the strict loader: `unboundEnv` walks the struct's `env` tags (plus the two process-level names, `WH_CONFIG` and `WH_LOG_LEVEL`) against the environment; `CheckDataDir` probes `data_dir` — run by `main` right after `Load`, so an unusable `data_dir` refuses boot before anything dials out. It refuses an empty value (reachable through `WH_DATA_DIR=`) outright rather than probing the working directory; a path that exists and is not a directory; a dangling symlink at `data_dir` or any component above it (the walk to the nearest existing ancestor uses `Lstat`, so a failed mount is not skipped over as "does not exist"); and a directory the process cannot write to — or, when it does not exist, an unwritable nearest ancestor — probed by creating and removing one temp file. A permission denial, on the probe or on reaching the path through a parent without search permission, carries the UID-65532 hint, since a bind mount owned by root is the typical cause. +- **config.go** — Loads *boot* configuration from a YAML file with environment variable overrides (using [cleanenv](https://github.com/ilyakaznacheev/cleanenv)); every key has a `WH_`-prefixed env var. Boot config is only what can't change under a running process — the implementation each layer runs on, resource sizing, listeners, observability exporters, the settings-directory path, and the secrets (`clickhouse.password`, `auth.jwt_secret`, `auth.operator_key`). Everything tenant-tunable lives in the settings directory (`settings/`). Both sources are strict: `Load` refuses to boot naming every YAML key the struct doesn't declare (`strict.go`) and every `WH_*` environment variable no field binds (`check.go`), so a tunable that moved to the settings directory can't be read, ignored, and believed. Boot is the validator for this half — there is no dry-run command. See [Configuration Reference](/configuration). +- **check.go** — `rejectUnboundEnv` is the environment half of the strict loader: `unboundEnv` walks the struct's `env` tags (plus the two process-level names, `WH_CONFIG` and `WH_LOG_LEVEL`) against the environment; `CheckDataDir` probes `data_dir` — run by `main` right after `Load` when `Config.NeedsDataDir` says a selected backend keeps state there, so an unusable `data_dir` refuses boot before anything dials out. It refuses an empty value (reachable through `WH_DATA_DIR=`) outright rather than probing the working directory; a path that exists and is not a directory; a dangling symlink at `data_dir` or any component above it (the walk to the nearest existing ancestor uses `Lstat`, so a failed mount is not skipped over as "does not exist"); and a directory the process cannot write to — or, when it does not exist, an unwritable nearest ancestor — probed by creating and removing one temp file. A permission denial, on the probe or on reaching the path through a parent without search permission, carries the UID-65532 hint, since a bind mount owned by root is the typical cause. +- **backends.go** — the `.backend` keys: one string type per layer (`MQBackend`, `CacheBackend`, `DedupeBackend`, `CoordBackend`), each with its list of the backends this build has, and a `validate` per layer block (`checkBackend`, run by `validateBackends` at the end of `Validate`), which refuses a value not on the list and names the ones that are. A backend's own settings go in a `.` sub-block that is its `validate`'s case to check. `Distributed` reports whether the MQ is shared with other processes, `NeedsDataDir` whether a selected backend keeps state under `data_dir`, and `Warnings` returns the valid combinations that are correct for one replica only (a shared MQ over a local cache or Pebble dedupe), which `app.New` logs at `WARN`. - **strict.go** — `rejectUnknownKeys`, the YAML half: re-reads the file as a generic tree and walks it against the struct's `yaml` tags, listing every key the struct doesn't declare. cleanenv itself is lenient by design, which is exactly wrong for boot config once keys have moved to the settings directory. - **persistence.go** — `WarnIfFreshDataDir` logs the startup `WARN` when `data_dir` is missing or empty (on a redeploy, the sign that the volume didn't persist); `LogStorageInitError` attaches the UID-65532 `permissionHint` to a NATS or Pebble open failure that looks like a permission denial — the same hint string `CheckDataDir` uses. diff --git a/docs/src/content/docs/configuration.mdx b/docs/src/content/docs/configuration.mdx index 8d311adc5..ee264bf45 100644 --- a/docs/src/content/docs/configuration.mdx +++ b/docs/src/content/docs/configuration.mdx @@ -46,7 +46,7 @@ Each layer's implementation is chosen once, at boot. Today every layer has one b | `mq.backend` | `WH_MQ_BACKEND` | `embedded` | The message queue. `embedded`: NATS JetStream inside this process, under `/nats`. It listens on no port, so no other process can reach its queue. | | `cache.backend` | `WH_CACHE_BACKEND` | `local` | The query-result cache. `local`: in this process, sized by `cache.l1_max_cost`. | | `dedupe.backend` | `WH_DEDUPE_BACKEND` | `pebble` | Where ingest dedupe keeps the event ids it has seen. `pebble`: in this process, under `/pebble`, open while any tenant has dedupe on. | -| `coord.backend` | `WH_COORD_BACKEND` | `local` | Where the leases for work only one process may do at a time are held. `local`: in this process. | +| `coord.backend` | `WH_COORD_BACKEND` | `local` | Reserved for the leases that will elect work only one process may do at a time, such as the sweeper. Nothing is elected yet: every process runs its own sweeper, and `local`, the only value, changes nothing. | Settings for one backend go in a sub-block named after it, `.`, read only when that backend is selected. A sub-block for a backend this build does not have is an unknown key and refuses boot. `mq` and `dedupe` also appear in the settings directory's `config.json`, with different keys (`mq.max_bytes_gb`, `dedupe.enabled`, …): those are per-tenant tunables and stay there, and one written in `config.yaml` refuses boot as an unknown key. @@ -217,7 +217,7 @@ dedupe: backend: pebble # in-process Pebble under /pebble coord: - backend: local + backend: local # reserved: nothing is elected yet auth: jwt_secret: change-me-in-production # jwks_url and role_claim are settings (config.json) diff --git a/internal/app/wire.go b/internal/app/wire.go index d53720253..3cfcad4b5 100644 --- a/internal/app/wire.go +++ b/internal/app/wire.go @@ -633,7 +633,7 @@ func (a *App) wireCache() error { // refuses a backend with no case, so reaching it means a Config built by hand // without one (the zero value is not the default), or a case missing here. func unreachableBackend[T ~string](key string, got T) error { - return fmt.Errorf("%s %q has no wiring: a Config built without config.Load must name every backend", key, got) + return fmt.Errorf("%s %q has no wiring: a Config built without config.Load must name the backend of every layer it wires", key, got) } // wireSweeper adds the active sweeper — purges messages that are both diff --git a/internal/config/backends.go b/internal/config/backends.go index 8328cea61..f2ab93308 100644 --- a/internal/config/backends.go +++ b/internal/config/backends.go @@ -71,12 +71,12 @@ func (d Dedupe) validate() error { return checkBackend("dedupe.backend", "WH_DEDUPE_BACKEND", d.Backend, dedupeBackends) } -// CoordBackend names the lease implementation singleton work (the sweeper) -// is elected through. +// CoordBackend names where leases for singleton work (the sweeper) are held. +// Nothing reads it yet: the lease layer (#613) wires it. type CoordBackend string -// CoordLocal holds leases in this process: correct while no other process -// shares its queue. +// CoordLocal holds leases in this process, which is enough while no other +// process shares its queue. const CoordLocal CoordBackend = "local" var coordBackends = []CoordBackend{CoordLocal} diff --git a/internal/settings/settings.go b/internal/settings/settings.go index 55ec089d3..8db981b3a 100644 --- a/internal/settings/settings.go +++ b/internal/settings/settings.go @@ -59,9 +59,11 @@ type PipesFile struct { // TenantConfig is the shape of config.json: the behavioral tunables that // migrate out of boot config. Boot config (config.yaml/env) keeps only what -// cannot change under a running process — resource sizing (`data_dir`, -// `cache.l1_max_cost`), listeners, the observability -// exporters — and the secrets (`clickhouse.password`, `auth.jwt_secret`, +// cannot change under a running process — the implementation each layer +// runs on (`mq.backend`, `cache.backend`, `dedupe.backend`, +// `coord.backend`), resource sizing (`data_dir`, `cache.l1_max_cost`, +// `clickhouse.max_total_conns`), listeners, the observability exporters — +// and the secrets (`clickhouse.password`, `auth.jwt_secret`, // `auth.operator_key`), which never belong in a tracked JSON file. Every // block and every top-level key inside it is REQUIRED: the binary carries no // compiled defaults, so the adopted snapshot is exactly what the files say. From f129d5775f2ba700c4997d37e3999e5b7f062849 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Thu, 24 Sep 2026 23:36:56 -0400 Subject: [PATCH 03/45] docs(config): no backend has a sub-block yet; index backends.go in AGENTS.md Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01EJr5tY4WQUy2sc4MbW67vL --- AGENTS.md | 2 +- CHANGELOG.md | 2 +- docs/src/content/docs/configuration.mdx | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 165957219..a7e4a28e9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -34,7 +34,7 @@ Eighteen internal packages under `internal/` (plus `internal/testutil/` for shar - **`cache/`** — `Cache` interface → `LocalCache` (Ristretto: one pool for every tenant) + `VersionManager` (the invalidation index). Every key leads with the tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8) — `:query:` for a result and its singleflight, `...
.` for a namespace — so no cached read or coalesced flight crosses tenants, a bump through `Invalidate` names one tenant's namespaces and no other's, and `InvalidateTenant` advances the tenant version that leads every namespace key of one tenant, orphaning every cached query keyed by its tables in one step (a pipe result names no table and keeps its TTL, [#343](https://github.com/Wave-RF/WaveHouse/pull/343)); the one crossing is the wiring's, above the package: `internal/app` hands the ingest worker the cache through `sharedTables`, which repeats each of the worker's bumps under every tenant on the same ClickHouse address and database (`chconn.Pools.SharingTables`, whatever their user or tls block — they read the same tables), and orphans the table-keyed cache (the structured-query results) of a tenant back on a pool after an absence, since it was out of that fan-out while away, or moved to another address or database, since it now reads other tables (story 6) - **`chconn/`** — `Pools`, one `Manager` (a `driver.Conn`) per distinct `Identity{Addr, Database, Username, Password, TLS}` tuple among the served tenants, reconciled from the settings registry's `AfterAdopt` after every reload ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 6): tenants naming one tuple share its pool, sized to their largest `max_open_conns`/`max_idle_conns`; a tenant whose tuple changed is repointed; a tuple no tenant names is released after the longest `query_timeout` among the tenants it had (never dials; a resize swaps the connection with the same grace). The boot config's `clickhouse.max_total_conns` bounds the open pools' `max_open_conns` together: boot refuses naming sum and ceiling; at a reload a resize above it keeps the pool's size, and a tuple that cannot be opened (the ceiling, an unreadable certificate, or options the driver refuses) leaves its tenants on the pool they had or on none — logged, retried by the next reload. Every consumer resolves its tenant's pool per call: `For` (nil for a tenant on no pool, a `503`), `Target` (the tenant's own HTTP wiring over its pool's TLS config), `SharingTables`, `Ping` (every pool at once, ready at the first answer). `HTTPClients` keeps one `http.Client` per TLS config - **`chsql/`** — dependency-free ClickHouse SQL helpers shared by `query`/`policy` (avoids an import cycle): `QuoteIdent` (backtick-quote every identifier) + `BindUnsafe` (reject names with a literal `?`) -- **`config/`** — YAML + env var config loading (cleanenv); strict on both sides (undeclared YAML key, unbound `WH_*` variable) and probes `data_dir` writability — boot is the validator, there is no dry run +- **`config/`** — YAML + env var config loading (cleanenv); strict on both sides (undeclared YAML key, unbound `WH_*` variable) and probes `data_dir` writability when a selected backend keeps state there (`NeedsDataDir`); `backends.go` holds each layer's `.backend` (only the in-process value today; `coord.backend` reserved) — boot is the validator, there is no dry run - **`dedupe/`** — `Deduplicator` interface → `Embedded` (Pebble: every tenant's seen ids in one instance at `data_dir/pebble`, each key led by its tenant, open while any tenant's store is — the layout is the implementation's call, and the wiring hands it `data_dir` once; its `Stats` feed the system gauges), wrapped by `Managed` whose open/closed state follows the hot-reloadable `dedupe.enabled` in the settings directory's `config.json`; `Stores` holds one `Managed` per tenant, built through a `Factory` (`func(tenant.ID) *Managed`, `Embedded.Tenant` in production; `Managed` opens its store through a function, so every backend gets the same switch), and reconciled from the registry's `AfterAdopt` hook — open exactly when the tenant is served with its switch on, closed with its seen ids kept otherwise ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) stories 7 and 3) - **`discovery/`** — `SchemaRegistry`, one per served tenant over a `Source` read once per refresh — the tenant's pool's connection and the database that pool was opened for, one snapshot, so a refused move keeps discovering the database the tenant's queries still use (`internal/app`'s `discoveries` builds, runs and stops them from `AfterAdopt` and `App.Close`: `RetryRefresh` until the first success, then `StartAutoRefresh` with a random first tick; `Lookup` answers `ErrNotLoaded` before the first success — the handlers' `503` with `Retry-After` — and `ErrUnknownTable` after; a failed loop attempt counts in `wavehouse_schema_refresh_failures_total{tenant}`), that introspects ClickHouse `system.columns` (name/type/nullability plus `default_expression` and 1-based `position`) and `system.tables` (each table's `create_table_query`, kept in-process and never serialized — an external-engine table renders its wiring there unconditionally — endpoint, bucket/host, database, username, S3 access key id; ClickHouse masks the password as `[HIDDEN]` from ~23.9, so the exposure is the topology, not the secret), records the server version, + `Validate()` for ingest payloads + `CanonicalizeTimestamps()` rewriting top-level `DateTime`/`DateTime64` column values to the canonical RFC 3339 UTC wire form pre-publish (Key Design Decision #19) - **`ingest/`** — Ingest worker pipeline (`worker.go`: JetStream input → per-table batch INSERT with DLQ output). The pipeline is **insert-only**. The wire format `EventMessage` (`types.go`) carries `{table_name, scope, received_timestamp, format, columns, row}` and nothing else — `row` is one positional `JSONCompactEachRow` line and `columns` names its slots, the table's **insertable** columns (a `MATERIALIZED`/`ALIAS` column cannot be named in an `INSERT`); the worker batches per (tenant, table, column list), the tenant read off each message's `mq.Topic`, and inserts each batch into its tenant's own ClickHouse (`chconn.Pools.Target`); the worker accepts whatever table name the envelope carries (table existence was already checked by the HTTP ingest handler, which `404`s an unknown table before publish; the worker doesn't re-validate), then bulk-INSERTs. In the embedded-NATS deployment (the default), the server runs with `DontListen: true` (`internal/mq/embedded.go`), so the only Publishers reachable on the `ingest.>` subjects are in-process Go code — today, only the HTTP `/v1/ingest?table={table}` handler. Non-insert mutations (`DELETE`/`UPDATE`/`TRUNCATE`/…) must go through `POST /v1/ops/query` under the admin role (the same `RequireAdmin` gate as the rest of `/v1/ops/*`), so non-admin callers never reach the proxy. A request with no token (or an invalid one) resolves to the `default_role`, which in a production config is not the admin role (setting them equal is a loudly-warned dev-only setting), so it can't reach this endpoint. Plus `Sweeper` (Active Sweeper for NATS message lifecycle) + `EventMessage`/`BufferConsumerName` types (`types.go`) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6564775a8..0aac12595 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **Each layer's implementation is chosen at boot** (`internal/config/{backends,config}.go` (+ tests), `internal/app/{app,wire}.go` (+ tests), `cmd/wavehouse/main.go`, `tests/integration/{setup,tenants}_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,settings-directory.mdx,architecture.md}`): the first step of running WaveHouse as more than one process ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config gains `mq.backend` (`WH_MQ_BACKEND`, default `embedded`), `cache.backend` (`WH_CACHE_BACKEND`, `local`), `dedupe.backend` (`WH_DEDUPE_BACKEND`, `pebble`) and `coord.backend` (`WH_COORD_BACKEND`, `local`). Each layer has only its in-process backend so far, and it is the default, so nothing changes for a config that sets none of them; a value with no backend refuses boot and names the valid ones. A backend's own settings will go in a `.` sub-block, which until that backend lands is an unknown key and refuses boot. `internal/app` picks each implementation in one `switch` per layer (`wireMQ`, `wireCache`, `wireDedupe`), `data_dir` is probed only when a selected backend keeps state there (`Config.NeedsDataDir`), and boot logs at `WARN` each line of `Config.Warnings`, the combinations that are correct for one replica only once a shared queue exists. A `config.Config` built without `config.Load` must now name the `mq`, `cache` and `dedupe` backends: the zero value is not the default, and `app.New` refuses it. `coord.backend` is reserved: nothing reads it until the lease layer lands, and the sweeper still runs in every process. +- **Each layer's implementation is chosen at boot** (`internal/config/{backends,config}.go` (+ tests), `internal/app/{app,wire}.go` (+ tests), `cmd/wavehouse/main.go`, `tests/integration/{setup,tenants}_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,settings-directory.mdx,architecture.md}`): the first step of running WaveHouse as more than one process ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config gains `mq.backend` (`WH_MQ_BACKEND`, default `embedded`), `cache.backend` (`WH_CACHE_BACKEND`, `local`), `dedupe.backend` (`WH_DEDUPE_BACKEND`, `pebble`) and `coord.backend` (`WH_COORD_BACKEND`, `local`). Each layer has only its in-process backend so far, and it is the default, so nothing changes for a config that sets none of them; a value with no backend refuses boot and names the valid ones. A backend's own settings will go in a `.` sub-block; no backend has settings yet, so every such sub-block is an unknown key for now and refuses boot. `internal/app` picks each implementation in one `switch` per layer (`wireMQ`, `wireCache`, `wireDedupe`), `data_dir` is probed only when a selected backend keeps state there (`Config.NeedsDataDir`), and boot logs at `WARN` each line of `Config.Warnings`, the combinations that are correct for one replica only once a shared queue exists. A `config.Config` built without `config.Load` must now name the `mq`, `cache` and `dedupe` backends: the zero value is not the default, and `app.New` refuses it. `coord.backend` is reserved: nothing reads it until the lease layer lands, and the sweeper still runs in every process. - **A tenant removed or rejected at runtime has its open streams ended** (`internal/stream/{hub,subscriber,bucket}.go` (+ tests), `internal/api/stream.go` (+ tests), `internal/api/router.go`, `internal/settings/{registry,tree}.go` (+ tests), `internal/ingest/worker.go` (+ tests), `internal/app/wire.go` (+ tests), `clients/ts/src/stream/sse.ts`, `docs/src/content/docs/{api,deployment,architecture,ingest-pipeline}.md`, `docs/src/content/docs/settings-directory.mdx`, `AGENTS.md`): story 3 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)). A `GET /v1/stream` used to outlive its tenant: the hub read no policy for it and withheld every row while the keepalive wheel held the connection open, so the client could not tell it from a quiet table. After every reload the hub now evicts the subscribers of each tenant no longer served, removed or rejected alike (`Hub.Prune`, through the close-once `Subscriber.Evict`), and the handler ends the stream: a gap-fill in progress included, and a stream `TenantMW` admitted just before the reload but registered just after it, which the handler checks for as it registers. The client's reconnect gets `404` (removed) or `503` (rejected); the SDK stops on the first and retries the second, resuming from `Last-Event-ID` once the folder is back — except in a browser going cross-origin where tenant `0` is not served or its CORS list does not admit the page, which cannot read either refusal (both are decorated from tenant `0`'s list) and re-dials as after a dropped connection. A flat directory never stops serving tenant `0`, so nothing changes there. Removing a tenant is deleting its folder, then reloading the whole directory, the last folder included: a server started with tenant folders reads the emptied directory as no folder left, where it read as a change of shape and the reload was rejected whole, leaving that tenant served; at boot an empty directory still reads as the four files, missing. Reloading the deleted folder by name leaves its tenant rejected. `GET /v1/health` keeps resolving a tenant, deliberately: its `404` tells a caller with no token no more than every tenant route's does, since they all answer before authenticating, and resolving answers a served tenant's ping from that tenant's own CORS list. A batch queued for a tenant with no ClickHouse connection — one no longer served, or one no pool could be opened for (such as by the connection ceiling) — skips the row-by-row retry, which no row of it could pass, and meets its DLQ switch once, whole, logged once per batch rather than twice per row; a tenant no longer served reads the switch as on, so its queued rows are parked under its own subject rather than dropped, inserted into another tenant's ClickHouse, or left unacked, redelivered for as long as the tenant is away and holding its queue's ack floor, which the purge waits on. Over a nested directory `/livez` no longer keeps naming a tenant that stopped being served before any tenant completed a first discovery: the diagnostic goes back to `no tenant has completed a first discovery yet`. - **One ClickHouse pool per tuple and one schema registry per tenant** (`internal/chconn/chconn.go` (+ tests), `internal/discovery/discovery.go` (+ tests), `internal/app/discoveries.go` (new), `internal/app/{app,wire}.go` (+ tests), `internal/api/{schema,ingest,structured_query,pipes,query,health,errors,clickhouse_exec}.go` (+ tests), `internal/stream/hub.go`, `internal/ingest/worker.go`, `internal/cache/{cache,local,version_manager}.go` (+ tests), `internal/testutil/{testutil,mocks}.go`, `tests/integration/{setup,tenants,boot_resilience,query_limits}_test.go`, `clients/ts/src/{schema,table,sql,client,types}.ts` (+ tests), `tests/e2e/sdk/admin.test.ts`, `docs/src/content/docs/{api,deployment,architecture,ingest-pipeline}.md`, `docs/src/content/docs/{settings-directory,configuration,access-control,reverse-proxy}.mdx`, `docs/src/content/docs/sdk/{admin,reference,queries}.md`, `AGENTS.md`): the second slice of story 6 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)), with no behavior change for a settings directory that holds the four files beyond the three noted at the end. The process opens one native pool per distinct `clickhouse.addr` / `database` / `username` / password / `tls` tuple among the served tenants (`chconn.Identity`, `chconn.Pools`), shared by the tenants naming it and sized to their largest `max_open_conns` and `max_idle_conns`; `http_port`, `http_scheme`, `headers` and `query_timeout` stay each tenant's own. Every reload reconciles the pools: a new tuple opens (never dials), a tenant whose tuple changed is repointed, a tuple no tenant names closes after the longest `query_timeout` among the tenants it had, and a pool whose largest ask changed is resized with the same grace. The boot config's `clickhouse.max_total_conns` now bounds the open pools together: boot is refused naming the sum and the ceiling; at a reload a resize above it is refused with the pool kept at its size, and a tuple that cannot be opened — the ceiling, a certificate file that cannot be read, or options the driver refuses — leaves its tenants on the pool they had (the keep-previous-wiring rule of the connection ceiling) or on none when they had none; both are logged and the next reload retries. A tenant on no pool fails closed: `503` with `Retry-After: 30` on `POST /v1/query`, `GET/POST /v1/pipes/{name}`, `POST /v1/ops/query` and `POST /v1/ops/schema/refresh`, ahead of the cache. Each served tenant gets a `discovery.SchemaRegistry` of its own over its pool, kept fresh by its own loop — the boot retry until the first success, then `schema.refresh_interval` with the first refresh at a random point within the interval so tenants adopted together do not refresh together — created and stopped from the settings reload and stopped under `App.Close`; `wavehouse_schema_refresh_failures_total{tenant}` counts a loop's failed attempts. `GET /v1/ops/schema`, `POST /v1/ops/schema/refresh` and `POST /v1/ops/query` take the strict `?tenant=` the pipe reads take (absent is tenant `0`, `400` malformed, `404` unknown, `503` rejected); the SDK sends it as the `tenant` option of `wh.schema.list()`, `wh.schema.refresh()`, `wh.from(t).schema()` and `wh.sql()`. Over a nested directory `/livez` (and `/v1/health`) is `503` with the latest discovery failure, naming its tenant, while no tenant has completed a first discovery, then `200` for the rest of the process lifetime; `/readyz` pings every open pool at once, is ready at the first answer, and names every pool that did not answer when none does — one tenant's ClickHouse outage is its log line and counter, never a probe failure. The ingest worker inserts each batch into its own tenant's ClickHouse — the tenant the message's topic names, through that tenant's HTTP wiring (`chconn.Pools.Target`); a tenant on no pool takes the failure path an unreachable ClickHouse takes — and its cache invalidation fans out to the tenants on the same address and database as the batch's tenant, whatever their user or `tls` block (they read the same tables), rather than to every known tenant, and a tenant adopted after an absence — rejected or removed, so out of that fan-out — or moved to another address or database has its cached structured-query results orphaned in one step (`Cache.InvalidateTenant`, a tenant generation in every version key), so a repaired folder never serves query rows cached before the inserts it missed (a pipe result names no table, so neither this nor any insert invalidates it: it stays until its TTL expires). Three changes reach the single-tenant directory too: a table lookup before the first discovery is a `503` with `Retry-After: 5` (`schema not loaded yet`) rather than a `404`, on `POST /v1/ingest`, `POST /v1/query` and `GET /v1/ops/schema` (the list included, where `[]` would read as no tables); the first periodic refresh fires at a random point within the interval rather than a full interval after boot; and a reload that moves `clickhouse.addr` or `clickhouse.database` now orphans the structured-query results cached before it, where they were served until their TTL. Until [#529](https://github.com/Wave-RF/WaveHouse/issues/529) every tenant's user authenticates with `WH_CH_PASSWORD`, so the tuple is in effect the address, database, username and `tls` block. - **One token verifier per tenant, built off the boot and reload paths** (`internal/auth/auth.go` (+ tests), `internal/api/router.go` (+ tests), `internal/app/{app,wire}.go` (+ tests), `go.mod`, `docs/src/content/docs/sdk/{reference,streaming}.md`, `docs/src/content/docs/{architecture,deployment,api}.md`, `docs/src/content/docs/{settings-directory,configuration}.mdx`, `SECURITY.md`): story 9 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)). Over a nested settings directory each tenant's folder now wires that tenant's verifier (`auth.jwks_url`, `auth.role_claim`), so a JWKS-issued token verifies only under the tenants whose `jwks_url` names its identity provider's key set — under any other tenant's header it is refused as invalid — where before one verifier, tenant `0`'s, accepted a token under any header. `auth.Config` is now the boot-config half alone (the HMAC secret and the operator key, shared by every tenant) and the new `auth.Wiring` a tenant's half; `NewAuthenticator` builds no verifier, `Reconfigure(id, wiring)` gives a tenant one — swapped atomically when its wiring changed, kept when it did not — `Prune` drops the verifiers of the tenants a reload stopped serving, rejected or removed alike (no work runs for a tenant that is not served; a folder adopted again gets a fresh verifier), and `Close` stops every JWKS refresh as a component of `App.Close`. This holds on every reload because the registry's `AfterAdopt` hooks run after every reload it applied, empty list included, so a per-tenant reload that rejects a folder drops its verifier then rather than at the next adoption. The middleware reads the request tenant's verifier through an injected `auth.TenantSource` — the store `api.TenantMW` resolved names its tenant with `settings.Store.Tenant()`, one read of the context — a tenant-exempt route (the ops tree) verifies as tenant `0`, and a tenant with no verifier fails closed. The operator key stamps the request tenant's `admin_role`, read from that tenant's policy, rather than tenant `0`'s. A JWKS key set is fetched on its own goroutine: the verifier is in place at once and *pending* until a set has been stored — the first fetch retried with backoff from one second to a minute, a stored set then kept fresh by the library hourly and, rate-limited, on an unknown key id — so neither boot nor a reload (which holds the registry's lock) waits on the endpoint, and **an unreachable JWKS no longer refuses boot** — it logs (`jwks refresh failed; no token validates until it succeeds`) and that tenant alone is affected. A token checked against a pending verifier is a new outcome, `auth.ErrVerifierPending`: every `/v1` route answers it `503 {"error": "token verifier not ready: …"}` with `Retry-After: 30` (`api.refuseUnverifiable`) rather than evaluate the request under the `default_role`, which could accept its data under a lesser role while another pod holding the keys would have served it as its own; a request without a token, and the operator key, are unaffected. Every fetch goes through one client that caps the response at 1 MiB, refusing a larger one as unreachable with `jwks response exceeds 1048576 bytes` as the logged cause. Nothing changes for a flat directory beyond that boot rule: its one tenant gets exactly the verifier it had. The boot warning for the secretless posture (no `auth.jwt_secret` and no `jwks_url`, whose token refusal landed in [#607](https://github.com/Wave-RF/WaveHouse/pull/607)) is now per tenant — each served tenant with no `jwks_url` while the boot secret is unset — where one line that any JWKS tenant silenced used to stand for the whole directory. diff --git a/docs/src/content/docs/configuration.mdx b/docs/src/content/docs/configuration.mdx index ee264bf45..193a6c21b 100644 --- a/docs/src/content/docs/configuration.mdx +++ b/docs/src/content/docs/configuration.mdx @@ -48,7 +48,7 @@ Each layer's implementation is chosen once, at boot. Today every layer has one b | `dedupe.backend` | `WH_DEDUPE_BACKEND` | `pebble` | Where ingest dedupe keeps the event ids it has seen. `pebble`: in this process, under `/pebble`, open while any tenant has dedupe on. | | `coord.backend` | `WH_COORD_BACKEND` | `local` | Reserved for the leases that will elect work only one process may do at a time, such as the sweeper. Nothing is elected yet: every process runs its own sweeper, and `local`, the only value, changes nothing. | -Settings for one backend go in a sub-block named after it, `.`, read only when that backend is selected. A sub-block for a backend this build does not have is an unknown key and refuses boot. `mq` and `dedupe` also appear in the settings directory's `config.json`, with different keys (`mq.max_bytes_gb`, `dedupe.enabled`, …): those are per-tenant tunables and stay there, and one written in `config.yaml` refuses boot as an unknown key. +Settings for one backend will go in a sub-block named after it, `.`, read only when that backend is selected. No backend has settings yet, so today any such sub-block, `mq.embedded` included, is an unknown key and refuses boot. `mq` and `dedupe` also appear in the settings directory's `config.json`, with different keys (`mq.max_bytes_gb`, `dedupe.enabled`, …): those are per-tenant tunables and stay there, and one written in `config.yaml` refuses boot as an unknown key. ### Server From 67df3173993f30d18cf92c00810fa1e886346ad6 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Thu, 24 Sep 2026 23:56:52 -0400 Subject: [PATCH 04/45] perf(cache): flat version index, pruned per tenant The local version index nested each table under its tenant's version and each scope under its table's, and was never pruned: every InvalidateTenant left the tenant's whole index behind. It now holds one version per tenant, (tenant, table) and (tenant, table, scope), bumped in place. A tenant bump drops the tenant's index and its next key gets a process-unique generation; a table bump drops the table's scopes. After each reload the wiring prunes the index to the tenants served. Fixes #262 for the local backend. Part of #613. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01EJr5tY4WQUy2sc4MbW67vL --- AGENTS.md | 4 +- CHANGELOG.md | 1 + docs/src/content/docs/architecture.md | 2 +- internal/app/app_test.go | 42 +++++ internal/app/wire.go | 16 +- internal/cache/local.go | 19 +- internal/cache/local_test.go | 33 ++++ internal/cache/version_manager.go | 232 +++++++++++++++++-------- internal/cache/version_manager_test.go | 192 ++++++++++++++------ 9 files changed, 406 insertions(+), 135 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index d3e34545e..6207f60a7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -29,9 +29,9 @@ One binary: Eighteen internal packages under `internal/` (plus `internal/testutil/` for shared test helpers): - **`api/`** — Chi HTTP router, JWT/JWKS middleware (from `auth/`), ingest/query/structured-query/SSE/schema/DLQ/pipes handlers -- **`app/`** — the process wiring: `New` builds every component from the boot config and the settings directory (each one wired in one place — what it opens, what it loops, what it releases — with the settings registry handed to its wiring function whole, the injection point of the per-tenant registry of #583: store-keyed getters for the handlers, `perTenant` for the async paths (with the tenant each message's `mq.Topic` names for the stream hub and the ingest worker), the `chconn.Pools` and the per-tenant `discoveries` reconciled from `AfterAdopt`, `shortestKeepalive` for the one setting folded over every tenant served, `gapWindows` and the `mq.max_bytes_gb` reconcile handing the MQ each served tenant's own gap window and byte budget, and `defaultSetting`/`onDefaultAdopt` for the one setting that still follows tenant `0`, a flat directory's ops-gate admin role; the auth verifiers are per tenant, reconfigured (rebuilt only on changed wiring) and pruned from `AfterAdopt`, and the same hook's `Hub.Prune` ends the open streams of a tenant no longer served), `Run` drives the long-lived ones under one `errgroup` until the context is cancelled or one fails, `Close` releases them in reverse order. `cmd/wavehouse` and `tests/integration` both boot through it +- **`app/`** — the process wiring: `New` builds every component from the boot config and the settings directory (each one wired in one place — what it opens, what it loops, what it releases — with the settings registry handed to its wiring function whole, the injection point of the per-tenant registry of #583: store-keyed getters for the handlers, `perTenant` for the async paths (with the tenant each message's `mq.Topic` names for the stream hub and the ingest worker), the `chconn.Pools` and the per-tenant `discoveries` reconciled from `AfterAdopt`, `shortestKeepalive` for the one setting folded over every tenant served, `gapWindows` and the `mq.max_bytes_gb` reconcile handing the MQ each served tenant's own gap window and byte budget, and `defaultSetting`/`onDefaultAdopt` for the one setting that still follows tenant `0`, a flat directory's ops-gate admin role; the auth verifiers are per tenant, reconfigured (rebuilt only on changed wiring) and pruned from `AfterAdopt`, the same hook's `Hub.Prune` ends the open streams of a tenant no longer served, and `LocalCache.Prune` drops its cache version index), `Run` drives the long-lived ones under one `errgroup` until the context is cancelled or one fails, `Close` releases them in reverse order. `cmd/wavehouse` and `tests/integration` both boot through it - **`auth/`** — JWT auth middleware: HMAC **or** JWKS verification with `alg` pinned to the active verifier, role extraction from a configurable claim path; always runs, never rejects (bad token → empty role + stashed reason). One verifier per tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 9): `Authenticator` keys them by `tenant.ID` — the request store's `settings.Store.Tenant()`, through an injected `TenantSource`; `tenant.Default` on the tenant-exempt routes — built from each tenant's `auth` block by `Reconfigure`, dropped by `Prune` once the tenant stops being served (rejected or removed), released by `Close`; the secrets (`Config`) are boot-level and shared. A JWKS key set is fetched off the boot and reload paths: until one has been stored the verifier is pending and a token-bearing request gets `503` + `Retry-After` from `api.refuseUnverifiable` (`auth.ErrVerifierPending`), never a `default_role` evaluation; refresh is library-managed (Eric, 2026-09-22), response capped at 1 MiB; the operator key's admin role is the request tenant's -- **`cache/`** — `Cache` interface → `LocalCache` (Ristretto: one pool for every tenant) + `VersionManager` (the invalidation index). Every key leads with the tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8) — `:query:` for a result and its singleflight, `..
.
.` for a namespace — so no cached read or coalesced flight crosses tenants, a bump through `Invalidate` names one tenant's namespaces and no other's, and `InvalidateTenant` advances the tenant version folded into every namespace and query key of one tenant, orphaning its every cached result in one step, pipe results included (no insert reaches a pipe result until [#343](https://github.com/Wave-RF/WaveHouse/pull/343)); `Lookup` returns a `Snapshot` of the versions it read and `Set` files the fill under it, so a write landing mid-query orphans the fill ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)), and every backend runs the conformance suite `internal/testutil/cachetest`; the one crossing is the wiring's, above the package: `internal/app` hands the ingest worker the cache through `sharedTables`, which repeats each of the worker's bumps under every tenant on the same ClickHouse address and database (`chconn.Pools.SharingTables`, whatever their user or tls block — they read the same tables), and orphans the whole cache of a tenant back on a pool after an absence, since it was out of that fan-out while away, or moved to another address or database, since it now reads other tables (story 6) +- **`cache/`** — `Cache` interface → `LocalCache` (Ristretto: one pool for every tenant) + `VersionManager` (the invalidation index). Every key leads with the tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8) — `:query:` for a result and its singleflight, a version per tenant, per (tenant, table) and per (tenant, table, scope), keyed by name and bumped in place (one entry per live namespace however often it is bumped, [#262](https://github.com/Wave-RF/WaveHouse/issues/262)) — so no cached read or coalesced flight crosses tenants, a bump through `Invalidate` names one tenant's namespaces and no other's, and `InvalidateTenant` drops the tenant's index so its next key gets a process-unique generation, orphaning its every cached result in one step, pipe results included (no insert reaches a pipe result until [#343](https://github.com/Wave-RF/WaveHouse/pull/343)); `Lookup` returns a `Snapshot` of the versions it read and `Set` files the fill under it, so a write landing mid-query orphans the fill ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)), and every backend runs the conformance suite `internal/testutil/cachetest`; the one crossing is the wiring's, above the package: `internal/app` hands the ingest worker the cache through `sharedTables`, which repeats each of the worker's bumps under every tenant on the same ClickHouse address and database (`chconn.Pools.SharingTables`, whatever their user or tls block — they read the same tables), and orphans the whole cache of a tenant back on a pool after an absence, since it was out of that fan-out while away, or moved to another address or database, since it now reads other tables (story 6) - **`chconn/`** — `Pools`, one `Manager` (a `driver.Conn`) per distinct `Identity{Addr, Database, Username, Password, TLS}` tuple among the served tenants, reconciled from the settings registry's `AfterAdopt` after every reload ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 6): tenants naming one tuple share its pool, sized to their largest `max_open_conns`/`max_idle_conns`; a tenant whose tuple changed is repointed; a tuple no tenant names is released after the longest `query_timeout` among the tenants it had (never dials; a resize swaps the connection with the same grace). The boot config's `clickhouse.max_total_conns` bounds the open pools' `max_open_conns` together: boot refuses naming sum and ceiling; at a reload a resize above it keeps the pool's size, and a tuple that cannot be opened (the ceiling, an unreadable certificate, or options the driver refuses) leaves its tenants on the pool they had or on none — logged, retried by the next reload. Every consumer resolves its tenant's pool per call: `For` (nil for a tenant on no pool, a `503`), `Target` (the tenant's own HTTP wiring over its pool's TLS config), `SharingTables`, `Ping` (every pool at once, ready at the first answer). `HTTPClients` keeps one `http.Client` per TLS config - **`chsql/`** — dependency-free ClickHouse SQL helpers shared by `query`/`policy` (avoids an import cycle): `QuoteIdent` (backtick-quote every identifier) + `BindUnsafe` (reject names with a literal `?`) - **`config/`** — YAML + env var config loading (cleanenv); strict on both sides (undeclared YAML key, unbound `WH_*` variable) and probes `data_dir` writability — boot is the validator, there is no dry run diff --git a/CHANGELOG.md b/CHANGELOG.md index 83f2fd832..50ea8f2ef 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -79,6 +79,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Fixed - **A write that lands while a cached read is running no longer re-homes the pre-write rows under the post-write key** (`internal/cache/{cache,local,version_manager}.go` (+ tests), `internal/testutil/cachetest` (new), `internal/api/{structured_query,pipes}.go` (+ tests), `internal/app/wire.go`, `docs/src/content/docs/{architecture,deployment}.md`, `AGENTS.md`): fixes [#382](https://github.com/Wave-RF/WaveHouse/issues/382), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `POST /v1/query` and pipe execution rebuilt the version-folded cache key after the query ran, so an insert invalidating the table mid-query filed the rows read before it under the new versions, and they were served as fresh until their TTL. The `Cache` interface now snapshots at lookup: `Lookup(ctx, tenant, sha, deps)` returns the `Entry` and a `Snapshot` of the versions it read, and `Set(ctx, snapshot, value, ttl)` stores under that snapshot, so such a fill is orphaned and the next request reads the post-write rows. The singleflight leader's snapshot is the one used; coalescing is unchanged. A `Lookup` whose dependencies name another tenant is refused (`ErrForeignDependency`). `Set` now errors only when the backend failed: a value the cache declines (larger than the pool, a non-positive TTL) is not an error. One behavior change: the tenant's version is folded into every key, a pipe result's included, so `InvalidateTenant` (a tenant back on a pool after an absence, or moved to another ClickHouse address or database) now drops that tenant's cached pipe results as well as its query results; before, a pipe result stayed until its TTL. Inserts still do not invalidate pipe results ([#343](https://github.com/Wave-RF/WaveHouse/pull/343)). A backend-agnostic conformance suite, `cachetest.Run`, pins what a hit, a miss and each kind of bump mean, and `LocalCache` runs it; the Redis-compatible backend will run the same suite. +- **The cache's version index no longer grows with every bump, and forgets a tenant no longer served** (`internal/cache/{local,version_manager}.go` (+ tests), `internal/app/wire.go` (+ tests), `docs/src/content/docs/architecture.md`, `AGENTS.md`): fixes [#262](https://github.com/Wave-RF/WaveHouse/issues/262) for the in-process cache, part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The index nested each table under its tenant's version and each scope under its table's, and never pruned, so every tenant invalidation left the tenant's whole index behind, and it grew with every tenant ever served. It now holds one version per tenant, per (tenant, table) and per (tenant, table, scope), bumped in place. A tenant invalidation drops the tenant's index and hands its next key a generation unique within the process, so nothing cached before it can match again, and a table bump drops the table's scope versions. After each settings reload the index of every tenant no longer served, removed or rejected, is dropped the same way; its cached results are orphaned with it, as they already were when such a tenant came back on a pool. No change to what is cached or served. The Redis-compatible backend (#613) bounds its versions with a TTL instead. - **An insert invalidates a table's cached results under every tenant the directory holds** (`internal/app/wire.go` (+ tests), `internal/settings/registry.go` (+ tests), `AGENTS.md`, `docs/src/content/docs/{deployment,architecture,ingest-pipeline}.md`): until [#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 6 gives each tenant its own ClickHouse, every tenant reads the same tables, but the ingest worker — which writes every event as tenant `0`'s until story 5 — bumped only tenant `0`'s cache namespaces after an insert, so another tenant's cached query could answer stale rows for up to its TTL (an hour at most). The cache the worker invalidates through now fans each bumped namespace out to every tenant the registry knows (the new `Registry.Known`), the named one and a rejected one included — a rejected tenant comes back into service with the entries it has, so leaving it out would let a folder repaired inside a TTL serve pre-insert rows; reads are untouched, so a tenant is still never served another's cached rows. The residual, a folder removed and restored inside a TTL, is closed since #610: a tenant back on a pool after an absence has its structured-query results orphaned at once (`Cache.InvalidateTenant`). Raised by CodeRabbit on #602. - **The `?token=` strip no longer repairs a query string that does not parse** (`internal/auth/auth.go`): `bearerToken` removed a query-string token by parsing the query, deleting `token`, and re-encoding what was left — and `url.ParseQuery` skips a pair it cannot read, so the re-encoding erased that pair. A handler that parses the query strictly in order to refuse a malformed one would then see a clean query: `GET /v1/ops/pipes?tenant=acme;x=1&token=…` would have answered `200` with the default tenant's pipes. The token is read exactly as before and a query that parses is rewritten exactly as before; a query that does not parse now loses its token pairs and nothing else, byte for byte. Pinned through `api.NewRouter` with the real authenticator, since a handler-level test never runs the middleware that rewrote the URL. diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 3ba9a9eb0..de4f6dfc1 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -111,7 +111,7 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ - **cache.go** — `Cache` interface: `Lookup`, `Set`, `Invalidate`, `InvalidateTenant`, `Close`, plus `QueryTimeToTTL`, which sets a result's TTL from how long its query took (10 s floor, 1 h ceiling). Every entry is one tenant's: `Lookup` takes the tenant, the caller's query key — `:query:`, built by the two cached handlers in `api/` (`queryCacheKey`, with the tenant read off the request's store — `settings.Store.Tenant`), which use it as their [singleflight](https://pkg.go.dev/golang.org/x/sync/singleflight) key too — and the `Namespace`s the result depends on, each of that tenant (another tenant's is `ErrForeignDependency`) and naming a table and scope: one for a structured query, none yet for a pipe (a pipe's table dependencies are [#343](https://github.com/Wave-RF/WaveHouse/pull/343)). `Lookup` returns the `Entry` (a nil value is a miss) and a `Snapshot` of the versions it read; on a miss the handler runs the query and passes that snapshot to `Set`, so a result is filed under the versions read *before* its query ran, and a write that lands while it runs orphans the fill rather than re-homing pre-write rows under the post-write versions ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)). The singleflight leader's snapshot is the one used. `Set` returns an error only when the backend failed; a value the cache declines — larger than it keeps, a non-positive TTL, a zero snapshot — is not one. What a hit, a miss and a bump mean is pinned by the conformance suite every backend runs, `internal/testutil/cachetest`. - **local.go** — `LocalCache`, the in-process L1 on [Ristretto](https://github.com/dgraph-io/ristretto): one pool shared by every tenant (a heavier tenant holds more of it), sized by the boot config's `cache.l1_max_cost`. -- **version_manager.go** — `VersionManager`, the invalidation index behind `Invalidate` and `InvalidateTenant`: a namespace key is `..
.
.`, and a query key is folded with the tenant's version and each dependency's namespace key and namespace version, so bumping a table (a scopeless write) or one scope — scope is reserved and empty today, so every write is the whole-table bump — orphans every dependent entry without touching the pool. The tenant leads every key ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8): the same table under two tenants is two namespaces, so a bump through `Invalidate` under one tenant never touches — and a read under one tenant is never served — the other's results, and the flat directory's single tenant simply carries the `0` prefix. `BumpTenant` (behind `InvalidateTenant`) advances the tenant version that leads every namespace key of one tenant, orphaning its every namespace and every cached query in one step (the tenant version is folded into every query key, so a pipe result with no dependencies is orphaned too) — a table no bump ever keyed included, which an enumeration of the index would miss — for a tenant back on a pool after an absence from the fan-out, or moved to another address or database (story 6). The index is per tenant; the cross-tenant invalidation an insert into a shared table needs is not the index's but the wiring's: `internal/app` hands the ingest worker a cache (`sharedTables`) that repeats each bump under every tenant on the same ClickHouse address and database. +- **version_manager.go** — `VersionManager`, the invalidation index behind `Invalidate` and `InvalidateTenant`: one version per tenant, per (tenant, table) and per (tenant, table, scope), each keyed by its name alone and bumped in place, so the index holds one entry per live tenant, table and scope however often each is bumped ([#262](https://github.com/Wave-RF/WaveHouse/issues/262)). A query key folds the tenant's version and, for each dependency, its tenant's, table's and scope's, so bumping a table (a scopeless write) orphans every scope of it, and bumping one scope orphans that scope and the whole-table view — scope is reserved and empty today, so every write is the whole-table bump — all without touching the pool. The tenant leads every key ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8): the same table under two tenants is two namespaces, so a bump through `Invalidate` under one tenant never touches — and a read under one tenant is never served — the other's results. A tenant's version is a *generation*, unique within the process and handed out by the first key built for the tenant; `BumpTenant` (behind `InvalidateTenant`) drops the tenant's whole index, so the next key gets a fresh generation no cached entry folds, orphaning every cached result of the tenant in one step — a pipe result with no dependencies, and a table no bump ever keyed, included — for a tenant back on a pool after an absence from the fan-out, or moved to another address or database (story 6). `LocalCache.Prune` does the same for every tenant no longer served, which `internal/app` runs after each settings reload, so a tenant removed or rejected stops holding its index. A table bump drops the table's scope versions with it, since every key they were folded into also folds the old table version; and a bump of a tenant with no index is a no-op, since no key folds its next generation yet. The index is per tenant; the cross-tenant invalidation an insert into a shared table needs is not the index's but the wiring's: `internal/app` hands the ingest worker a cache (`sharedTables`) that repeats each bump under every tenant on the same ClickHouse address and database. ### `config/` — Configuration diff --git a/internal/app/app_test.go b/internal/app/app_test.go index 2ecd77d98..1f665b6a1 100644 --- a/internal/app/app_test.go +++ b/internal/app/app_test.go @@ -16,6 +16,7 @@ import ( "os" "path/filepath" "strings" + "sync" "sync/atomic" "syscall" "testing" @@ -635,6 +636,47 @@ func TestReload_ReadmittedTenantCacheIsOrphaned(t *testing.T) { assert.Equal(t, []tenant.ID{"globex", "acme"}, mock.GetTenants(), "restored: the same") } +// pruneRecorder is a cache that records, at each Prune, which of the tenants +// it is asked about are still served. +type pruneRecorder struct { + testutil.MockCache + mu sync.Mutex + served []map[tenant.ID]bool +} + +func (p *pruneRecorder) Prune(served func(tenant.ID) bool) { + p.mu.Lock() + defer p.mu.Unlock() + p.served = append(p.served, map[tenant.ID]bool{"acme": served("acme"), "globex": served("globex")}) +} + +func (p *pruneRecorder) last() map[tenant.ID]bool { + p.mu.Lock() + defer p.mu.Unlock() + if len(p.served) == 0 { + return nil + } + return p.served[len(p.served)-1] +} + +// Every reload prunes the cache's version index down to the tenants served, +// so a tenant rejected or removed stops holding it (#262). +func TestReload_PrunesCacheIndexToServedTenants(t *testing.T) { + root := writeNestedSettings(t, map[string]map[string]any{"acme": nil, "globex": nil}) + a := newApp(t, testConfig(t, root), Options{}) + rec := &pruneRecorder{} + a.cache = rec + + rewriteSettings(t, filepath.Join(root, "globex"), invalidQuery) + a.tenants.Reload("test") + assert.Equal(t, map[tenant.ID]bool{"acme": true, "globex": false}, rec.last(), "rejected") + + rewriteSettings(t, filepath.Join(root, "globex"), nil) + require.NoError(t, os.RemoveAll(filepath.Join(root, "acme"))) + a.tenants.Reload("test") + assert.Equal(t, map[tenant.ID]bool{"acme": false, "globex": true}, rec.last(), "removed; the repaired one served again") +} + // keepalive is a config.json patch setting the stream block's keepalive pair. func keepalive(interval, buckets int) map[string]any { return map[string]any{"stream": map[string]any{"keepalive_interval": interval, "keepalive_buckets": buckets, "gap_window_minutes": 15}} diff --git a/internal/app/wire.go b/internal/app/wire.go index 4c9f1da21..8811ba63e 100644 --- a/internal/app/wire.go +++ b/internal/app/wire.go @@ -591,7 +591,16 @@ func (a *App) wireMQ() error { return nil } -// wireCache opens the L1 cache — the only tier in standalone mode. +// pruner is a cache whose version index lives in the process and would +// otherwise keep a tenant that stopped being served (cache.LocalCache). +type pruner interface { + Prune(served func(tenant.ID) bool) +} + +// wireCache opens the L1 cache — the only tier in standalone mode. After +// every reload a tenant no longer served, removed or rejected alike, has its +// version index dropped (#262); its cache is orphaned with it, as it would +// be anyway when it came back (wireClickHouse). func (a *App) wireCache() error { l1, err := cache.NewLocal(a.cfg.Cache.L1MaxCost) if err != nil { @@ -600,6 +609,11 @@ func (a *App) wireCache() error { // TODO: eventually this is where we can switch between ristretto, redis, tiered (both), etc a.cache = l1 a.add(component{name: "cache", close: withoutContext(l1.Close)}) + a.tenants.AfterAdopt(func([]tenant.ID) { + if p, ok := a.cache.(pruner); ok { + p.Prune(a.served) + } + }) return nil } diff --git a/internal/cache/local.go b/internal/cache/local.go index 1a755e058..3e0e14b5e 100644 --- a/internal/cache/local.go +++ b/internal/cache/local.go @@ -69,10 +69,11 @@ func (l *LocalCache) Set(_ context.Context, snap Snapshot, value []byte, ttl tim // view. Returns the number of namespaces processed. // // This bumps exactly what it's given. A whole-table bump already subsumes every -// per-scope bump for the same table (the table version is embedded in every -// namespace key), so a caller that knows a whole-table bump is coming should drop -// the now-redundant scope entries itself — the ingest worker does this as it -// builds the batch, where it already loops once and knows it's a single table. +// per-scope bump for the same table (every key that folds a scope version +// folds the table version too), so a caller that knows a whole-table bump is +// coming should drop the now-redundant scope entries itself — the ingest +// worker does this as it builds the batch, where it already loops once and +// knows it's a single table. func (l *LocalCache) Invalidate(_ context.Context, namespaces []Namespace) (uint64, error) { for _, ns := range namespaces { if ns.Scope == "" { @@ -85,13 +86,21 @@ func (l *LocalCache) Invalidate(_ context.Context, namespaces []Namespace) (uint } // InvalidateTenant orphans every cached result of tenant id, pipe results -// included: one version bump, nothing enumerated (see +// included: its version index is dropped, nothing enumerated (see // VersionManager.BumpTenant). func (l *LocalCache) InvalidateTenant(_ context.Context, id tenant.ID) error { l.versionManager.BumpTenant(id) return nil } +// Prune drops the version index of every tenant served rejects, orphaning +// its entries as InvalidateTenant would, so a tenant removed or rejected at +// a reload stops holding memory (#262). The entries themselves go with +// their TTL or Ristretto's eviction. +func (l *LocalCache) Prune(served func(tenant.ID) bool) { + l.versionManager.Prune(served) +} + // Wait blocks until all buffered writes have been applied. // Exposed for testing; production callers rarely need this. func (l *LocalCache) Wait() { diff --git a/internal/cache/local_test.go b/internal/cache/local_test.go index 2fdb3237e..ad77a40db 100644 --- a/internal/cache/local_test.go +++ b/internal/cache/local_test.go @@ -2,10 +2,13 @@ package cache_test import ( "testing" + "time" + "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" "github.com/Wave-RF/WaveHouse/internal/cache" + "github.com/Wave-RF/WaveHouse/internal/tenant" "github.com/Wave-RF/WaveHouse/internal/testutil/cachetest" ) @@ -23,3 +26,33 @@ func TestLocalCache_Conformance(t *testing.T) { t.Parallel() cachetest.Run(t, newLocal, cachetest.Options{MaxValueBytes: localMaxCost}) } + +// A tenant that stops being served has its index dropped: what it cached is +// orphaned — it misses when served again — and a tenant still served keeps +// its entries. +func TestLocalCache_Prune(t *testing.T) { + t.Parallel() + c, err := cache.NewLocal(localMaxCost) + require.NoError(t, err) + t.Cleanup(func() { _ = c.Close() }) + ctx := t.Context() + fill := func(id tenant.ID) { + _, snap, err := c.Lookup(ctx, id, "q", []cache.Namespace{{Tenant: id, Table: "events"}}) + require.NoError(t, err) + require.NoError(t, c.Set(ctx, snap, []byte("rows"), time.Minute)) + } + get := func(id tenant.ID) []byte { + e, _, err := c.Lookup(ctx, id, "q", []cache.Namespace{{Tenant: id, Table: "events"}}) + require.NoError(t, err) + return e.Value + } + fill("acme") + fill("globex") + c.Wait() + require.NotNil(t, get("acme")) + require.NotNil(t, get("globex")) + + c.Prune(func(id tenant.ID) bool { return id == "acme" }) + assert.NotNil(t, get("acme"), "still served") + assert.Nil(t, get("globex"), "pruned: orphaned, never revived") +} diff --git a/internal/cache/version_manager.go b/internal/cache/version_manager.go index d97bc70aa..dfe18b8c6 100644 --- a/internal/cache/version_manager.go +++ b/internal/cache/version_manager.go @@ -9,27 +9,44 @@ import ( "github.com/Wave-RF/WaveHouse/internal/tenant" ) -// VersionManager handles the safe tracking of table + scope versioning. -// It uses a standard map because versions must NEVER be evicted under memory pressure. -// TODO: this potentially could be bad/dangerous with a low amount of RAM available/high memory pressure AND a TON of tables/scopes per table... will need to work out eventually +// VersionManager is the invalidation index: one version per tenant, per +// (tenant, table) and per (tenant, table, scope), in maps keyed by name +// alone, never by another version (#262). A bump overwrites a version in +// place, so the index holds one entry per live tenant, table and scope +// however often each is bumped, and forgetting a tenant releases all of it. +// +// A query key folds all three versions of each dependency, which gives the +// lattice: a table bump orphans every scope, a scope bump that scope and the +// whole-table view, and a tenant bump everything of the tenant's. type VersionManager struct { mu sync.RWMutex - // tenantVersions leads every key of a tenant, so BumpTenant orphans the - // tenant's every namespace and query in one step — the ones no bump ever - // keyed included, which is what an enumeration of the maps would miss. - tenantVersions map[tenant.ID]uint64 // -> tenant_version - tableVersions map[string]uint64 // ..
-> table_version - namespaceVersions map[string]uint64 // ..
.. -> namespace_version + // tenants holds each tenant's index from the first query key built for + // it until the tenant is bumped or pruned. + tenants map[tenant.ID]*tenantVersions + + // lastGen is the last generation handed to a tenant; see tenantVersions.gen. + lastGen uint64 } -// NewVersionManager initializes the thread-safe version store. -func NewVersionManager() *VersionManager { - return &VersionManager{ - tenantVersions: make(map[tenant.ID]uint64), - tableVersions: make(map[string]uint64), - namespaceVersions: make(map[string]uint64), - } +// tenantVersions is one tenant's slice of the index. +type tenantVersions struct { + // gen is the tenant's version: unique within the process, so a tenant + // forgotten and recreated can never fold a generation an entry was + // cached under. That is what makes dropping the tenant's whole index a + // safe bump. + gen uint64 + tables map[string]*tableVersions +} + +// tableVersions is one table's version and its scopes'. A missing table or +// scope reads as 0: an entry is only ever removed together with a bump of +// the version above it (a table bump clears the scopes, a tenant bump +// drops the tables), so a 0 read after a removal never matches an entry +// cached before it. +type tableVersions struct { + version uint64 + scopes map[string]uint64 } // Namespace is one (tenant, table, scope) a cached result depends on. The @@ -41,76 +58,101 @@ type Namespace struct { Scope string } -// tableKeyLocked renders the table-versions key, -// "..
"; caller must hold vm.mu. A tenant id -// cannot contain a dot and callers encode the table dot-free, so the tokens -// can never run together. -func (vm *VersionManager) tableKeyLocked(id tenant.ID, table string) string { - return fmt.Sprintf("%s.%d.%s", id, vm.tenantVersions[id], table) -} - -// namespaceKeyLocked builds the namespace-table key; caller must hold vm.mu. -func (vm *VersionManager) namespaceKeyLocked(ns Namespace) string { - tk := vm.tableKeyLocked(ns.Tenant, ns.Table) - return fmt.Sprintf("%s.%d.%s", tk, vm.tableVersions[tk], ns.Scope) -} - -// NamespaceKey renders the namespace-table key for ns at its tenant's and -// table's current versions: -// "..
.." (scopeless -// scope is "", so e.g. ".0.
.."). -func (vm *VersionManager) NamespaceKey(ns Namespace) string { - vm.mu.RLock() - defer vm.mu.RUnlock() - return vm.namespaceKeyLocked(ns) +// NewVersionManager initializes the thread-safe version store. +func NewVersionManager() *VersionManager { + return &VersionManager{tenants: make(map[tenant.ID]*tenantVersions)} } // QueryKey builds the queries-table key for tenant id's result that depends // on deps: the query's sha (hash of SQL+params) folded with the tenant's -// version and every dependency's namespace key AND its namespace version, so -// a bump of the tenant or of any dependency misses the key — a result with no -// deps (a pipe) is orphaned by BumpTenant too. A structured query passes one -// Namespace; a pipe passes several. Deps are sorted so their order never -// changes the key. +// version and, for every dependency, its tenant's, table's and scope's +// versions, so a bump of the tenant or of any dependency misses the key — a +// result with no deps (a pipe) is orphaned by BumpTenant too. A structured +// query passes one Namespace, a pipe none yet (#343). Deps are sorted so +// their order never changes the key. Every version is read under one lock, +// so the key is one consistent snapshot. +// +// The first key built for a tenant creates its index at a fresh generation. func (vm *VersionManager) QueryKey(id tenant.ID, sha string, deps []Namespace) string { + vm.mu.RLock() + key, ok := vm.queryKeyLocked(id, sha, deps, false) + vm.mu.RUnlock() + if ok { + return key + } + vm.mu.Lock() + defer vm.mu.Unlock() + key, _ = vm.queryKeyLocked(id, sha, deps, true) + return key +} + +// queryKeyLocked renders QueryKey with vm.mu held — for writing when create +// is set, which creates the index of each tenant the key names that has +// none; otherwise such a tenant reports !ok. +func (vm *VersionManager) queryKeyLocked(id tenant.ID, sha string, deps []Namespace, create bool) (string, bool) { + index := func(id tenant.ID) (*tenantVersions, bool) { + tv := vm.tenants[id] + if tv == nil && create { + tv = vm.newTenantLocked(id) + } + return tv, tv != nil + } + own, ok := index(id) + if !ok { + return "", false + } segs := make([]string, len(deps)) - // Lock per dependency rather than across the whole loop: each dep's table + - // namespace versions are read together (consistent for that dep), but we don't - // hold the lock across all deps. A concurrent bump can land between deps; the - // caller files its fill under this key (a Snapshot), so a bump that lands - // anywhere after the read of a version orphans it. The sort/join run with - // no lock held. for i, d := range deps { - vm.mu.RLock() - nsKey := vm.namespaceKeyLocked(d) - segs[i] = fmt.Sprintf("%s.%d", nsKey, vm.namespaceVersions[nsKey]) - vm.mu.RUnlock() + tv, ok := index(d.Tenant) + if !ok { + return "", false + } + var table, scope uint64 + if t := tv.tables[d.Table]; t != nil { + table, scope = t.version, t.scopes[d.Scope] + } + segs[i] = fmt.Sprintf("%s.%d.%s.%d.%s.%d", d.Tenant, tv.gen, d.Table, table, d.Scope, scope) } - vm.mu.RLock() - tv := vm.tenantVersions[id] - vm.mu.RUnlock() sort.Strings(segs) - return fmt.Sprintf("%s|%s.%d|%s", sha, id, tv, strings.Join(segs, "|")) + return fmt.Sprintf("%s|%s.%d|%s", sha, id, own.gen, strings.Join(segs, "|")), true +} + +func (vm *VersionManager) newTenantLocked(id tenant.ID) *tenantVersions { + vm.lastGen++ + tv := &tenantVersions{gen: vm.lastGen, tables: make(map[string]*tableVersions)} + vm.tenants[id] = tv + return tv +} + +// tableLocked is the entry for a tenant's table, created at version 0, or +// nil when the tenant has no index: no key folds its current generation +// yet, so there is nothing a bump could orphan. Caller holds vm.mu for +// writing. +func (vm *VersionManager) tableLocked(id tenant.ID, table string) *tableVersions { + tv := vm.tenants[id] + if tv == nil { + return nil + } + t := tv.tables[table] + if t == nil { + t = &tableVersions{} + tv.tables[table] = t + } + return t } // BumpTable advances a tenant's table version, orphaning every namespace — and // every cached query — that depends on the table, in one step (the whole-table -// nuke). The same table under another tenant is untouched. +// nuke). The table's scope versions are dropped with it: every key they were +// folded into also folds the old table version. The same table under another +// tenant is untouched. func (vm *VersionManager) BumpTable(id tenant.ID, table string) { vm.mu.Lock() defer vm.mu.Unlock() - vm.tableVersions[vm.tableKeyLocked(id, table)]++ -} - -// BumpTenant advances a tenant's version, orphaning its every namespace — -// and every cached query, whatever its deps — in one step (the whole-tenant -// nuke): every namespace and query key of the tenant carries the version, so -// nothing has to be enumerated, and a table no bump ever keyed is orphaned -// like the rest. Other tenants are untouched. -func (vm *VersionManager) BumpTenant(id tenant.ID) { - vm.mu.Lock() - defer vm.mu.Unlock() - vm.tenantVersions[id]++ + if t := vm.tableLocked(id, table); t != nil { + t.version++ + t.scopes = nil + } } // BumpNamespace advances one (tenant, table, scope) namespace plus the table's @@ -119,8 +161,54 @@ func (vm *VersionManager) BumpTenant(id tenant.ID) { func (vm *VersionManager) BumpNamespace(ns Namespace) { vm.mu.Lock() defer vm.mu.Unlock() - vm.namespaceVersions[vm.namespaceKeyLocked(ns)]++ + t := vm.tableLocked(ns.Tenant, ns.Table) + if t == nil { + return + } + if t.scopes == nil { + t.scopes = make(map[string]uint64) + } + t.scopes[ns.Scope]++ if ns.Scope != "" { - vm.namespaceVersions[vm.namespaceKeyLocked(Namespace{Tenant: ns.Tenant, Table: ns.Table})]++ + t.scopes[""]++ + } +} + +// BumpTenant orphans every cached query of a tenant, whatever its deps, in +// one step (the whole-tenant nuke), by dropping the tenant's index: the next +// key built for it gets a fresh generation, which no cached entry folds. +// Nothing has to be enumerated, a table no bump ever keyed is orphaned like +// the rest, and the index the tenant held is released. Other tenants are +// untouched. +func (vm *VersionManager) BumpTenant(id tenant.ID) { + vm.mu.Lock() + defer vm.mu.Unlock() + delete(vm.tenants, id) +} + +// Prune drops the index of every tenant keep rejects, as BumpTenant would, +// so a tenant that stops being served stops holding memory; one served again +// starts over at a fresh generation. +func (vm *VersionManager) Prune(keep func(tenant.ID) bool) { + vm.mu.Lock() + defer vm.mu.Unlock() + for id := range vm.tenants { + if !keep(id) { + delete(vm.tenants, id) + } + } +} + +// size is the number of versions the index holds, for tests. +func (vm *VersionManager) size() int { + vm.mu.RLock() + defer vm.mu.RUnlock() + n := len(vm.tenants) + for _, tv := range vm.tenants { + n += len(tv.tables) + for _, t := range tv.tables { + n += len(t.scopes) + } } + return n } diff --git a/internal/cache/version_manager_test.go b/internal/cache/version_manager_test.go index e1a1c330b..c9278577e 100644 --- a/internal/cache/version_manager_test.go +++ b/internal/cache/version_manager_test.go @@ -8,45 +8,17 @@ import ( "github.com/Wave-RF/WaveHouse/internal/tenant" ) -func TestVersionManager_NamespaceKey(t *testing.T) { - t.Parallel() - vm := NewVersionManager() - - // The tenant leads at its default version (0), then the table at its - // default version (0); a scopeless namespace renders a trailing dot. The - // flat directory's tenant is "0". - assert.Equal(t, "acme.0.users.0.", vm.NamespaceKey(Namespace{Tenant: "acme", Table: "users"})) - assert.Equal(t, "acme.0.users.0.org_1", vm.NamespaceKey(Namespace{Tenant: "acme", Table: "users", Scope: "org_1"})) - assert.Equal(t, "0.0.users.0.", vm.NamespaceKey(Namespace{Tenant: tenant.Default, Table: "users"})) - - // The table version is embedded in every namespace key for that tenant's - // table, so a BumpTable is reflected across all its scopes at once — and - // nowhere else: the same table under another tenant keeps its version. - vm.BumpTable("acme", "users") - assert.Equal(t, "acme.0.users.1.", vm.NamespaceKey(Namespace{Tenant: "acme", Table: "users"})) - assert.Equal(t, "acme.0.users.1.org_1", vm.NamespaceKey(Namespace{Tenant: "acme", Table: "users", Scope: "org_1"})) - assert.Equal(t, "globex.0.users.0.", vm.NamespaceKey(Namespace{Tenant: "globex", Table: "users"})) - - // The tenant version leads every key of the tenant, so a BumpTenant moves - // every table of acme's — the never-bumped orders table included — to a - // fresh key space, at table version 0 again, and no other tenant's. - vm.BumpTenant("acme") - assert.Equal(t, "acme.1.users.0.", vm.NamespaceKey(Namespace{Tenant: "acme", Table: "users"})) - assert.Equal(t, "acme.1.orders.0.", vm.NamespaceKey(Namespace{Tenant: "acme", Table: "orders"})) - assert.Equal(t, "globex.0.users.0.", vm.NamespaceKey(Namespace{Tenant: "globex", Table: "users"})) -} - func TestVersionManager_QueryKey(t *testing.T) { t.Parallel() vm := NewVersionManager() - // One dependency at default versions: - // sha | . | ..
.... + // sha | . | ..
...; + // acme's index is created by its first key, at generation 1. key := vm.QueryKey("acme", "hash123", []Namespace{{Tenant: "acme", Table: "users", Scope: "org_1"}}) - assert.Equal(t, "hash123|acme.0|acme.0.users.0.org_1.0", key) + assert.Equal(t, "hash123|acme.1|acme.1.users.0.org_1.0", key) // No deps (a pipe) still folds the tenant version. - assert.Equal(t, "hash123|acme.0|", vm.QueryKey("acme", "hash123", nil)) + assert.Equal(t, "hash123|acme.1|", vm.QueryKey("acme", "hash123", nil)) // Dependency order must not change the key (segments are sorted). deps1 := []Namespace{{Tenant: "acme", Table: "a"}, {Tenant: "acme", Table: "b"}} @@ -59,6 +31,9 @@ func TestVersionManager_QueryKey(t *testing.T) { vm.QueryKey("acme", "h", []Namespace{{Tenant: "acme", Table: "users"}}), vm.QueryKey("globex", "h", []Namespace{{Tenant: "globex", Table: "users"}})) assert.NotEqual(t, vm.QueryKey("acme", "h", nil), vm.QueryKey("globex", "h", nil)) + + // Reading keys is stable: nothing but a bump moves a version. + assert.Equal(t, key, vm.QueryKey("acme", "hash123", []Namespace{{Tenant: "acme", Table: "users", Scope: "org_1"}})) } func TestVersionManager_BumpTable(t *testing.T) { @@ -69,16 +44,16 @@ func TestVersionManager_BumpTable(t *testing.T) { orders := []Namespace{{Tenant: "acme", Table: "orders", Scope: "org_1"}} globexUsers := []Namespace{{Tenant: "globex", Table: "users", Scope: "org_1"}} - usersBefore := vm.QueryKey(users[0].Tenant, "h", users) - ordersBefore := vm.QueryKey(orders[0].Tenant, "h", orders) - globexBefore := vm.QueryKey(globexUsers[0].Tenant, "h", globexUsers) + usersBefore := vm.QueryKey("acme", "h", users) + ordersBefore := vm.QueryKey("acme", "h", orders) + globexBefore := vm.QueryKey("globex", "h", globexUsers) // Bumping a table changes the key for that tenant's table but leaves other // tables — and the same table under another tenant — alone. vm.BumpTable("acme", "users") - assert.NotEqual(t, usersBefore, vm.QueryKey(users[0].Tenant, "h", users)) - assert.Equal(t, ordersBefore, vm.QueryKey(orders[0].Tenant, "h", orders)) - assert.Equal(t, globexBefore, vm.QueryKey(globexUsers[0].Tenant, "h", globexUsers)) + assert.NotEqual(t, usersBefore, vm.QueryKey("acme", "h", users)) + assert.Equal(t, ordersBefore, vm.QueryKey("acme", "h", orders)) + assert.Equal(t, globexBefore, vm.QueryKey("globex", "h", globexUsers)) } func TestVersionManager_BumpNamespace(t *testing.T) { @@ -90,24 +65,44 @@ func TestVersionManager_BumpNamespace(t *testing.T) { otherScope := []Namespace{{Tenant: "acme", Table: "users", Scope: "org_2"}} otherTenant := []Namespace{{Tenant: "globex", Table: "users", Scope: "org_1"}} - scopedBefore := vm.QueryKey(scoped[0].Tenant, "h", scoped) - wholeBefore := vm.QueryKey(wholeTable[0].Tenant, "h", wholeTable) - otherBefore := vm.QueryKey(otherScope[0].Tenant, "h", otherScope) - otherTenantBefore := vm.QueryKey(otherTenant[0].Tenant, "h", otherTenant) + scopedBefore := vm.QueryKey("acme", "h", scoped) + wholeBefore := vm.QueryKey("acme", "h", wholeTable) + otherBefore := vm.QueryKey("acme", "h", otherScope) + otherTenantBefore := vm.QueryKey("globex", "h", otherTenant) // Bumping (acme, users, org_1) changes that scope AND the whole-table view, // but leaves every other scope — and the same scope under another tenant — // valid. vm.BumpNamespace(Namespace{Tenant: "acme", Table: "users", Scope: "org_1"}) - assert.NotEqual(t, scopedBefore, vm.QueryKey(scoped[0].Tenant, "h", scoped)) - assert.NotEqual(t, wholeBefore, vm.QueryKey(wholeTable[0].Tenant, "h", wholeTable)) - assert.Equal(t, otherBefore, vm.QueryKey(otherScope[0].Tenant, "h", otherScope)) - assert.Equal(t, otherTenantBefore, vm.QueryKey(otherTenant[0].Tenant, "h", otherTenant)) + assert.NotEqual(t, scopedBefore, vm.QueryKey("acme", "h", scoped)) + assert.NotEqual(t, wholeBefore, vm.QueryKey("acme", "h", wholeTable)) + assert.Equal(t, otherBefore, vm.QueryKey("acme", "h", otherScope)) + assert.Equal(t, otherTenantBefore, vm.QueryKey("globex", "h", otherTenant)) +} + +// A table bump drops the table's scope versions, which then read as 0 again +// — safe only because every key a scope version was folded into also folds +// the table version the bump moved. Pinned so a table bump that forgot to +// advance the table version would revive the scoped entry. +func TestVersionManager_BumpTableDropsScopes(t *testing.T) { + t.Parallel() + vm := NewVersionManager() + scoped := []Namespace{{Tenant: "acme", Table: "users", Scope: "org_1"}} + + fresh := vm.QueryKey("acme", "h", scoped) + vm.BumpNamespace(scoped[0]) + bumped := vm.QueryKey("acme", "h", scoped) + vm.BumpTable("acme", "users") + after := vm.QueryKey("acme", "h", scoped) + + assert.NotEqual(t, fresh, after) + assert.NotEqual(t, bumped, after) + assert.Equal(t, 2, vm.size(), "the tenant and its table; the scopes went with the table bump") } // TestVersionManager_BumpTenant: a tenant's every namespace is orphaned in -// one step — a table that was never bumped (so has no key of its own to bump) -// included — and no other tenant's is touched. +// one step — a table that was never bumped included — and no other tenant's +// is touched. func TestVersionManager_BumpTenant(t *testing.T) { t.Parallel() vm := NewVersionManager() @@ -115,18 +110,107 @@ func TestVersionManager_BumpTenant(t *testing.T) { users := []Namespace{{Tenant: "acme", Table: "users", Scope: "org_1"}} orders := []Namespace{{Tenant: "acme", Table: "orders"}} globexUsers := []Namespace{{Tenant: "globex", Table: "users", Scope: "org_1"}} + vm.QueryKey("acme", "h", users) vm.BumpTable("acme", "users") - usersBefore := vm.QueryKey(users[0].Tenant, "h", users) - ordersBefore := vm.QueryKey(orders[0].Tenant, "h", orders) - globexBefore := vm.QueryKey(globexUsers[0].Tenant, "h", globexUsers) + usersBefore := vm.QueryKey("acme", "h", users) + ordersBefore := vm.QueryKey("acme", "h", orders) + globexBefore := vm.QueryKey("globex", "h", globexUsers) vm.BumpTenant("acme") - assert.NotEqual(t, usersBefore, vm.QueryKey(users[0].Tenant, "h", users)) - assert.NotEqual(t, ordersBefore, vm.QueryKey(orders[0].Tenant, "h", orders), "a table no bump ever keyed is orphaned too") - assert.Equal(t, globexBefore, vm.QueryKey(globexUsers[0].Tenant, "h", globexUsers)) + assert.NotEqual(t, usersBefore, vm.QueryKey("acme", "h", users)) + assert.NotEqual(t, ordersBefore, vm.QueryKey("acme", "h", orders), "a table no bump ever keyed is orphaned too") + assert.Equal(t, globexBefore, vm.QueryKey("globex", "h", globexUsers)) pipeBefore := vm.QueryKey("acme", "h", nil) vm.BumpTenant("acme") assert.NotEqual(t, pipeBefore, vm.QueryKey("acme", "h", nil), "a result with no deps is orphaned too") } + +// Dropping a tenant's index is a bump only because the index it gets back +// never repeats a generation: every key built before any of these drops must +// differ from every key built after it. A counter per tenant restarting at 0 +// fails this, reviving the first entry. +func TestVersionManager_GenerationsNeverRepeat(t *testing.T) { + t.Parallel() + vm := NewVersionManager() + deps := []Namespace{{Tenant: "acme", Table: "users"}} + seen := map[string]bool{} + for i := range 100 { + key := vm.QueryKey("acme", "h", deps) + assert.False(t, seen[key], "round %d revived %s", i, key) + seen[key] = true + if i%2 == 0 { + vm.BumpTenant("acme") + } else { + vm.Prune(func(tenant.ID) bool { return false }) + } + } +} + +// A bump of a tenant with no index is a no-op: no key folds its next +// generation yet, so nothing needs orphaning — and an insert still in flight +// for a tenant just pruned does not bring its index back. +func TestVersionManager_BumpWithoutIndex(t *testing.T) { + t.Parallel() + vm := NewVersionManager() + vm.BumpTable("acme", "users") + vm.BumpNamespace(Namespace{Tenant: "acme", Table: "users", Scope: "org_1"}) + vm.BumpTenant("acme") + assert.Zero(t, vm.size()) +} + +func TestVersionManager_Prune(t *testing.T) { + t.Parallel() + vm := NewVersionManager() + acme := []Namespace{{Tenant: "acme", Table: "users"}} + globex := []Namespace{{Tenant: "globex", Table: "users"}} + acmeBefore := vm.QueryKey("acme", "h", acme) + globexBefore := vm.QueryKey("globex", "h", globex) + vm.BumpTable("acme", "users") + vm.BumpTable("globex", "users") + acmeBumped := vm.QueryKey("acme", "h", acme) + globexBumped := vm.QueryKey("globex", "h", globex) + + vm.Prune(func(id tenant.ID) bool { return id == "globex" }) + assert.Equal(t, 2, vm.size(), "globex and its table; acme released") + assert.Equal(t, globexBumped, vm.QueryKey("globex", "h", globex), "a kept tenant is untouched") + + back := vm.QueryKey("acme", "h", acme) + assert.NotEqual(t, acmeBefore, back, "a pruned tenant never revives what it cached") + assert.NotEqual(t, acmeBumped, back) + assert.NotEqual(t, globexBefore, globexBumped) +} + +// The index holds one version per live tenant, table and scope, however often +// each is bumped (#262): the nested index this replaced kept every table and +// scope under every tenant version it had seen. +func TestVersionManager_SizeDoesNotGrowWithBumps(t *testing.T) { + t.Parallel() + vm := NewVersionManager() + touch := func() { + for _, id := range []tenant.ID{"acme", "globex"} { + for _, table := range []string{"users", "orders"} { + for _, scope := range []string{"", "org_1", "org_2"} { + vm.QueryKey(id, "h", []Namespace{{Tenant: id, Table: table, Scope: scope}}) + vm.BumpNamespace(Namespace{Tenant: id, Table: table, Scope: scope}) + } + } + } + } + touch() + settled := vm.size() + assert.Equal(t, 2+2*2+2*2*3, settled, "two tenants, two tables each, three scopes each") + + for i := range 10_000 { + switch i % 3 { + case 0: + vm.BumpTable("acme", []string{"users", "orders"}[i%2]) + case 1: + vm.BumpTenant("globex") + } + touch() + assert.LessOrEqual(t, vm.size(), settled) + } + assert.Equal(t, settled, vm.size()) +} From ed8022bf34127e5cdd751d916f7d7dde162971fc Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Fri, 25 Sep 2026 00:06:14 -0400 Subject: [PATCH 05/45] docs(cache): name the cache prune hook; pin LocalCache as a pruner Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01EJr5tY4WQUy2sc4MbW67vL --- AGENTS.md | 2 +- docs/src/content/docs/architecture.md | 2 +- internal/app/wire.go | 9 +++++++-- 3 files changed, 9 insertions(+), 4 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 6207f60a7..adbec1d81 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -29,7 +29,7 @@ One binary: Eighteen internal packages under `internal/` (plus `internal/testutil/` for shared test helpers): - **`api/`** — Chi HTTP router, JWT/JWKS middleware (from `auth/`), ingest/query/structured-query/SSE/schema/DLQ/pipes handlers -- **`app/`** — the process wiring: `New` builds every component from the boot config and the settings directory (each one wired in one place — what it opens, what it loops, what it releases — with the settings registry handed to its wiring function whole, the injection point of the per-tenant registry of #583: store-keyed getters for the handlers, `perTenant` for the async paths (with the tenant each message's `mq.Topic` names for the stream hub and the ingest worker), the `chconn.Pools` and the per-tenant `discoveries` reconciled from `AfterAdopt`, `shortestKeepalive` for the one setting folded over every tenant served, `gapWindows` and the `mq.max_bytes_gb` reconcile handing the MQ each served tenant's own gap window and byte budget, and `defaultSetting`/`onDefaultAdopt` for the one setting that still follows tenant `0`, a flat directory's ops-gate admin role; the auth verifiers are per tenant, reconfigured (rebuilt only on changed wiring) and pruned from `AfterAdopt`, the same hook's `Hub.Prune` ends the open streams of a tenant no longer served, and `LocalCache.Prune` drops its cache version index), `Run` drives the long-lived ones under one `errgroup` until the context is cancelled or one fails, `Close` releases them in reverse order. `cmd/wavehouse` and `tests/integration` both boot through it +- **`app/`** — the process wiring: `New` builds every component from the boot config and the settings directory (each one wired in one place — what it opens, what it loops, what it releases — with the settings registry handed to its wiring function whole, the injection point of the per-tenant registry of #583: store-keyed getters for the handlers, `perTenant` for the async paths (with the tenant each message's `mq.Topic` names for the stream hub and the ingest worker), the `chconn.Pools` and the per-tenant `discoveries` reconciled from `AfterAdopt`, `shortestKeepalive` for the one setting folded over every tenant served, `gapWindows` and the `mq.max_bytes_gb` reconcile handing the MQ each served tenant's own gap window and byte budget, and `defaultSetting`/`onDefaultAdopt` for the one setting that still follows tenant `0`, a flat directory's ops-gate admin role; the auth verifiers are per tenant, reconfigured (rebuilt only on changed wiring) and pruned from `AfterAdopt`, the same hook's `Hub.Prune` ends the open streams of a tenant no longer served, and `wireCache`'s hook drops, through `LocalCache.Prune`, the cache version index of a tenant no longer served ([#262](https://github.com/Wave-RF/WaveHouse/issues/262))), `Run` drives the long-lived ones under one `errgroup` until the context is cancelled or one fails, `Close` releases them in reverse order. `cmd/wavehouse` and `tests/integration` both boot through it - **`auth/`** — JWT auth middleware: HMAC **or** JWKS verification with `alg` pinned to the active verifier, role extraction from a configurable claim path; always runs, never rejects (bad token → empty role + stashed reason). One verifier per tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 9): `Authenticator` keys them by `tenant.ID` — the request store's `settings.Store.Tenant()`, through an injected `TenantSource`; `tenant.Default` on the tenant-exempt routes — built from each tenant's `auth` block by `Reconfigure`, dropped by `Prune` once the tenant stops being served (rejected or removed), released by `Close`; the secrets (`Config`) are boot-level and shared. A JWKS key set is fetched off the boot and reload paths: until one has been stored the verifier is pending and a token-bearing request gets `503` + `Retry-After` from `api.refuseUnverifiable` (`auth.ErrVerifierPending`), never a `default_role` evaluation; refresh is library-managed (Eric, 2026-09-22), response capped at 1 MiB; the operator key's admin role is the request tenant's - **`cache/`** — `Cache` interface → `LocalCache` (Ristretto: one pool for every tenant) + `VersionManager` (the invalidation index). Every key leads with the tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8) — `:query:` for a result and its singleflight, a version per tenant, per (tenant, table) and per (tenant, table, scope), keyed by name and bumped in place (one entry per live namespace however often it is bumped, [#262](https://github.com/Wave-RF/WaveHouse/issues/262)) — so no cached read or coalesced flight crosses tenants, a bump through `Invalidate` names one tenant's namespaces and no other's, and `InvalidateTenant` drops the tenant's index so its next key gets a process-unique generation, orphaning its every cached result in one step, pipe results included (no insert reaches a pipe result until [#343](https://github.com/Wave-RF/WaveHouse/pull/343)); `Lookup` returns a `Snapshot` of the versions it read and `Set` files the fill under it, so a write landing mid-query orphans the fill ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)), and every backend runs the conformance suite `internal/testutil/cachetest`; the one crossing is the wiring's, above the package: `internal/app` hands the ingest worker the cache through `sharedTables`, which repeats each of the worker's bumps under every tenant on the same ClickHouse address and database (`chconn.Pools.SharingTables`, whatever their user or tls block — they read the same tables), and orphans the whole cache of a tenant back on a pool after an absence, since it was out of that fan-out while away, or moved to another address or database, since it now reads other tables (story 6) - **`chconn/`** — `Pools`, one `Manager` (a `driver.Conn`) per distinct `Identity{Addr, Database, Username, Password, TLS}` tuple among the served tenants, reconciled from the settings registry's `AfterAdopt` after every reload ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 6): tenants naming one tuple share its pool, sized to their largest `max_open_conns`/`max_idle_conns`; a tenant whose tuple changed is repointed; a tuple no tenant names is released after the longest `query_timeout` among the tenants it had (never dials; a resize swaps the connection with the same grace). The boot config's `clickhouse.max_total_conns` bounds the open pools' `max_open_conns` together: boot refuses naming sum and ceiling; at a reload a resize above it keeps the pool's size, and a tuple that cannot be opened (the ceiling, an unreadable certificate, or options the driver refuses) leaves its tenants on the pool they had or on none — logged, retried by the next reload. Every consumer resolves its tenant's pool per call: `For` (nil for a tenant on no pool, a `503`), `Target` (the tenant's own HTTP wiring over its pool's TLS config), `SharingTables`, `Ping` (every pool at once, ready at the first answer). `HTTPClients` keeps one `http.Client` per TLS config diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index de4f6dfc1..368c93680 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -90,7 +90,7 @@ The API layer uses [Chi](https://github.com/go-chi/chi) for routing with Request ### `app/` — Process wiring - **app.go** — `New(ctx, Options)` builds every component from the boot config (`Options.Config`) and the settings directory it names, in dependency order: settings registry, observability, ClickHouse pools, schema discovery, the dedupe stores, embedded NATS (ingest + DLQ streams), cache, sweeper, streaming (hub, MQ→hub bridge, keepalive wheel), ingest worker, auth, reload triggers, HTTP. Each is one `component` value — what it opens, what it loops, what it releases — so a failure part-way releases what was already opened and returns the error. `Run(ctx)` drives every loop under one `errgroup` until `ctx` is canceled (a clean stop: every loop drains, the API server and the ingest worker within `server.shutdown_timeout`; open SSE streams are ended as the drain begins rather than waited on) or a component fails, which stops the rest and returns that error. `Close(ctx)` releases what `New` opened, newest first, under the caller's release budget (`ReleaseTimeout`, 5s), a real bound: a remote implementation's close gives up at the deadline itself, and a close that ignores the context (the local stores) is abandoned at it, with the components below it left unreleased rather than overlapping it, both named in the error — and then flushes telemetry under its own 3s budget, so the flush that reports on the stop is never handed a deadline a slow close already spent. The SIGHUP registration is released last of all. `Handler`, `Registry`, and `MQ` expose the pieces a harness needs; `Options.Listener` lets one serve the API on its own listener instead of `server.port`. -- **wire.go** — one `wire*` function per component, each handed the settings registry whole and deriving the per-call getters the internal packages take (`DLQFor`, `DedupeFor`, `GapWindow`, …) and registering its `AfterAdopt` hook there where it has one. Those wiring functions are where the per-tenant registry of [#583](https://github.com/Wave-RF/WaveHouse/issues/583) is injected, not `main`: `wireSettings` opens the `settings.Registry`, the HTTP handlers get store-keyed getters (method expressions such as `(*settings.Store).Policy`), and `perTenant` adapts a store accessor into the `func(tenant.ID) T` getter the async packages take, with the tenant each message's `mq.Topic` names for the stream hub and the ingest worker — a tenant the registry is not serving is logged and read as the zero value, except in `dlqFor`, the ingest worker's DLQ switch, where it reads as on so a message the worker cannot read is parked rather than dropped, and a removed or rejected tenant's queued rows are parked rather than left unacked, where they would hold the ack floor and stop the sweeper. The ClickHouse pools (`chconn.Pools`) and the per-tenant schema registries (`discoveries`, in `discoveries.go`) are reconciled from `AfterAdopt` after every reload ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 6): `wireClickHouse` builds each served tenant's `chconn.Member` from its store and logs what the reconcile refused; `wireDiscovery` builds a registry over `pools.For` for each newly served tenant — a flat directory's tenant `0` refreshed synchronously first, as before — runs its loop under the App's stop context, stops the loop of a tenant no longer served, and drives the `BootState` from the first tenant's first discovery, sticky from there; before that, a diagnostic naming a tenant a reload stopped serving goes back to the no-tenant one. The handlers resolve both per request through store-keyed getters (`chConnFor`, `registryFor`, `chTargetFor`, `queryTimeout`), the hub and the ingest worker through tenant-keyed ones (`discoveries.For`, `pools.Target`) called with the tenant the message's topic names; a tenant on no pool is an untyped nil connection, the handlers' `503`. The ingest worker is handed the cache through `sharedTables`, which bumps each namespace the worker invalidates under every tenant on the same ClickHouse address and database (`pools.SharingTables`), and the pools hook orphans the whole cache — structured-query and pipe results — of a tenant back on a pool after an absence (`Cache.InvalidateTenant`), since it was out of that fan-out while away, and of a tenant moved to another address or database, since it now reads other tables (both returned by `Pools.Reconcile`). The one setting that still follows the default tenant is read per request, the admin role of a flat directory's ops gate: `defaultSetting` reads the store tenant `0` last adopted (`App.defaultStore`, tracked by an `onDefaultAdopt` hook that runs only after a reload that adopted it), so a `0` folder that a reload rejects or removes leaves it as it was. The auth verifiers are per tenant: `wireAuth` builds one for each tenant being served, its `AfterAdopt` hook reconfigures the adopted tenants' (rebuilt only when their wiring changed) and prunes the ones no longer served, and the operator key's admin role is read from the request tenant's policy. `wireStreaming`'s hook prunes the stream hub the same way (`Hub.Prune`, with the one `served` predicate the auth and dedupe hooks use too), ending the open streams of a tenant no longer served. One setting is shared by folding over the tenants being served rather than by following tenant `0`: the keepalive wheel runs at the shortest `stream.keepalive_interval` among them (`shortestKeepalive`), re-derived after every reload the registry applies — an adoption, a rejection, or a removal — so a dropped tenant's interval leaves the wheel at once ([#597](https://github.com/Wave-RF/WaveHouse/issues/597)). The sweeper is handed each served tenant's own `stream.gap_window_minutes` (`gapWindows`, read every sweep), since each tenant's events have a queue of their own. The dedupe stores are per tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 7): `wireDedupe` builds a `dedupe.Stores` over the `Tenant` factory of the embedded Pebble implementation (`dedupe.NewEmbedded`), handing it `data_dir` once; the implementation decides where every tenant's store lives — one instance, each key led by its tenant (story 3) — and one reconcile closure, the boot apply and the `AfterAdopt` hook alike, sets every store to what the registry says: open exactly when its tenant is served with `dedupe.enabled` on, closed with its seen ids kept when the tenant is switched off, rejected, or removed. An instance that cannot open follows the registry's rule for the shape: fatal at boot over a flat directory, fail-closed for every tenant with dedupe on over a nested one. The system gauges report that one instance's figures (`Embedded.Stats`), not a sum over tenants. The ingest handler picks the tenant's store off the request's `settings.Store` (`Store.Tenant()`). The reload triggers only start in `Run`, after `New` has registered every hook, so the watcher's first reload already drives all of them: SIGHUP in both shapes, the directory watcher for a flat directory only. `wireMQ` hands each served tenant's `mq.max_bytes_gb` to `mq.Broker.SetMaxBytes` at boot and again after every reload, under the App's stop context, which opens that tenant's queue the first time; a queue that cannot be opened or resized follows the registry's rule for the shape — fatal at boot over a flat directory, logged over a nested one — and is retried by the next reload. How the budget is split across the tenant's streams, the time bounds, the rollback, and the dead-letter shrink guard are `internal/mq`'s. +- **wire.go** — one `wire*` function per component, each handed the settings registry whole and deriving the per-call getters the internal packages take (`DLQFor`, `DedupeFor`, `GapWindow`, …) and registering its `AfterAdopt` hook there where it has one. Those wiring functions are where the per-tenant registry of [#583](https://github.com/Wave-RF/WaveHouse/issues/583) is injected, not `main`: `wireSettings` opens the `settings.Registry`, the HTTP handlers get store-keyed getters (method expressions such as `(*settings.Store).Policy`), and `perTenant` adapts a store accessor into the `func(tenant.ID) T` getter the async packages take, with the tenant each message's `mq.Topic` names for the stream hub and the ingest worker — a tenant the registry is not serving is logged and read as the zero value, except in `dlqFor`, the ingest worker's DLQ switch, where it reads as on so a message the worker cannot read is parked rather than dropped, and a removed or rejected tenant's queued rows are parked rather than left unacked, where they would hold the ack floor and stop the sweeper. The ClickHouse pools (`chconn.Pools`) and the per-tenant schema registries (`discoveries`, in `discoveries.go`) are reconciled from `AfterAdopt` after every reload ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 6): `wireClickHouse` builds each served tenant's `chconn.Member` from its store and logs what the reconcile refused; `wireDiscovery` builds a registry over `pools.For` for each newly served tenant — a flat directory's tenant `0` refreshed synchronously first, as before — runs its loop under the App's stop context, stops the loop of a tenant no longer served, and drives the `BootState` from the first tenant's first discovery, sticky from there; before that, a diagnostic naming a tenant a reload stopped serving goes back to the no-tenant one. The handlers resolve both per request through store-keyed getters (`chConnFor`, `registryFor`, `chTargetFor`, `queryTimeout`), the hub and the ingest worker through tenant-keyed ones (`discoveries.For`, `pools.Target`) called with the tenant the message's topic names; a tenant on no pool is an untyped nil connection, the handlers' `503`. The ingest worker is handed the cache through `sharedTables`, which bumps each namespace the worker invalidates under every tenant on the same ClickHouse address and database (`pools.SharingTables`), and the pools hook orphans the whole cache — structured-query and pipe results — of a tenant back on a pool after an absence (`Cache.InvalidateTenant`), since it was out of that fan-out while away, and of a tenant moved to another address or database, since it now reads other tables (both returned by `Pools.Reconcile`). The one setting that still follows the default tenant is read per request, the admin role of a flat directory's ops gate: `defaultSetting` reads the store tenant `0` last adopted (`App.defaultStore`, tracked by an `onDefaultAdopt` hook that runs only after a reload that adopted it), so a `0` folder that a reload rejects or removes leaves it as it was. The auth verifiers are per tenant: `wireAuth` builds one for each tenant being served, its `AfterAdopt` hook reconfigures the adopted tenants' (rebuilt only when their wiring changed) and prunes the ones no longer served, and the operator key's admin role is read from the request tenant's policy. `wireStreaming`'s hook prunes the stream hub the same way (`Hub.Prune`, with the one `served` predicate the auth, dedupe and cache hooks use too), ending the open streams of a tenant no longer served, and `wireCache`'s hook prunes the cache's version index the same way (`LocalCache.Prune`, [#262](https://github.com/Wave-RF/WaveHouse/issues/262)), so a tenant no longer served stops holding it. One setting is shared by folding over the tenants being served rather than by following tenant `0`: the keepalive wheel runs at the shortest `stream.keepalive_interval` among them (`shortestKeepalive`), re-derived after every reload the registry applies — an adoption, a rejection, or a removal — so a dropped tenant's interval leaves the wheel at once ([#597](https://github.com/Wave-RF/WaveHouse/issues/597)). The sweeper is handed each served tenant's own `stream.gap_window_minutes` (`gapWindows`, read every sweep), since each tenant's events have a queue of their own. The dedupe stores are per tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 7): `wireDedupe` builds a `dedupe.Stores` over the `Tenant` factory of the embedded Pebble implementation (`dedupe.NewEmbedded`), handing it `data_dir` once; the implementation decides where every tenant's store lives — one instance, each key led by its tenant (story 3) — and one reconcile closure, the boot apply and the `AfterAdopt` hook alike, sets every store to what the registry says: open exactly when its tenant is served with `dedupe.enabled` on, closed with its seen ids kept when the tenant is switched off, rejected, or removed. An instance that cannot open follows the registry's rule for the shape: fatal at boot over a flat directory, fail-closed for every tenant with dedupe on over a nested one. The system gauges report that one instance's figures (`Embedded.Stats`), not a sum over tenants. The ingest handler picks the tenant's store off the request's `settings.Store` (`Store.Tenant()`). The reload triggers only start in `Run`, after `New` has registered every hook, so the watcher's first reload already drives all of them: SIGHUP in both shapes, the directory watcher for a flat directory only. `wireMQ` hands each served tenant's `mq.max_bytes_gb` to `mq.Broker.SetMaxBytes` at boot and again after every reload, under the App's stop context, which opens that tenant's queue the first time; a queue that cannot be opened or resized follows the registry's rule for the shape — fatal at boot over a flat directory, logged over a nested one — and is retried by the next reload. How the budget is split across the tenant's streams, the time bounds, the rollback, and the dead-letter shrink guard are `internal/mq`'s. ### `stream/` — SSE keepalive & fan-out diff --git a/internal/app/wire.go b/internal/app/wire.go index 8811ba63e..900c19992 100644 --- a/internal/app/wire.go +++ b/internal/app/wire.go @@ -143,8 +143,9 @@ func gapWindows(tenants *settings.Registry) map[tenant.ID]time.Duration { } // served reports whether the registry is serving tenant id: what the -// per-tenant resources — verifiers, dedupe stores, open streams — are pruned -// by once a reload removes or rejects their tenant. +// per-tenant resources — verifiers, dedupe stores, open streams, the cache +// version index — are pruned by once a reload removes or rejects their +// tenant. func (a *App) served(id tenant.ID) bool { _, ok := a.tenants.For(id) return ok @@ -597,6 +598,10 @@ type pruner interface { Prune(served func(tenant.ID) bool) } +// The hook below asserts pruner at run time; this keeps LocalCache from +// silently dropping out of it. +var _ pruner = (*cache.LocalCache)(nil) + // wireCache opens the L1 cache — the only tier in standalone mode. After // every reload a tenant no longer served, removed or rejected alike, has its // version index dropped (#262); its cache is orphaned with it, as it would From 50b7e0c3727eb4fb76eda458c2d5dd05c3aaa12c Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Fri, 25 Sep 2026 00:10:10 -0400 Subject: [PATCH 06/45] feat(cache): Redis-compatible shared cache backend RedisCache keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process. Versions are random tokens under the tenant's hash tag; a value carries the tokens it was filed under, so a lookup is one pipelined MGET+GET and a lost token can only cause misses. Failures bypass the cache behind a circuit breaker, and undelivered invalidations are retried until they land. Built and tested against Redis, Valkey, Dragonfly and a Redis Cluster node; not yet selectable by config (E4). Part of #613. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01EJr5tY4WQUy2sc4MbW67vL --- AGENTS.md | 6 +- CHANGELOG.md | 1 + Makefile | 2 +- docs/src/content/docs/architecture.md | 8 +- docs/src/content/docs/development.md | 4 +- go.mod | 7 +- go.sum | 4 + internal/cache/breaker.go | 65 +++ internal/cache/breaker_test.go | 59 +++ internal/cache/cache.go | 3 +- internal/cache/export_test.go | 12 + internal/cache/metrics.go | 106 ++++ internal/cache/pending.go | 78 +++ internal/cache/pending_test.go | 45 ++ internal/cache/redis.go | 644 +++++++++++++++++++++++ internal/cache/redis_codec.go | 194 +++++++ internal/cache/redis_codec_test.go | 198 +++++++ internal/cache/redis_integration_test.go | 418 +++++++++++++++ internal/cache/redis_test.go | 231 ++++++++ 19 files changed, 2074 insertions(+), 11 deletions(-) create mode 100644 internal/cache/breaker.go create mode 100644 internal/cache/breaker_test.go create mode 100644 internal/cache/export_test.go create mode 100644 internal/cache/metrics.go create mode 100644 internal/cache/pending.go create mode 100644 internal/cache/pending_test.go create mode 100644 internal/cache/redis.go create mode 100644 internal/cache/redis_codec.go create mode 100644 internal/cache/redis_codec_test.go create mode 100644 internal/cache/redis_integration_test.go create mode 100644 internal/cache/redis_test.go diff --git a/AGENTS.md b/AGENTS.md index d3e34545e..9455b6fe6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -31,7 +31,7 @@ Eighteen internal packages under `internal/` (plus `internal/testutil/` for shar - **`api/`** — Chi HTTP router, JWT/JWKS middleware (from `auth/`), ingest/query/structured-query/SSE/schema/DLQ/pipes handlers - **`app/`** — the process wiring: `New` builds every component from the boot config and the settings directory (each one wired in one place — what it opens, what it loops, what it releases — with the settings registry handed to its wiring function whole, the injection point of the per-tenant registry of #583: store-keyed getters for the handlers, `perTenant` for the async paths (with the tenant each message's `mq.Topic` names for the stream hub and the ingest worker), the `chconn.Pools` and the per-tenant `discoveries` reconciled from `AfterAdopt`, `shortestKeepalive` for the one setting folded over every tenant served, `gapWindows` and the `mq.max_bytes_gb` reconcile handing the MQ each served tenant's own gap window and byte budget, and `defaultSetting`/`onDefaultAdopt` for the one setting that still follows tenant `0`, a flat directory's ops-gate admin role; the auth verifiers are per tenant, reconfigured (rebuilt only on changed wiring) and pruned from `AfterAdopt`, and the same hook's `Hub.Prune` ends the open streams of a tenant no longer served), `Run` drives the long-lived ones under one `errgroup` until the context is cancelled or one fails, `Close` releases them in reverse order. `cmd/wavehouse` and `tests/integration` both boot through it - **`auth/`** — JWT auth middleware: HMAC **or** JWKS verification with `alg` pinned to the active verifier, role extraction from a configurable claim path; always runs, never rejects (bad token → empty role + stashed reason). One verifier per tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 9): `Authenticator` keys them by `tenant.ID` — the request store's `settings.Store.Tenant()`, through an injected `TenantSource`; `tenant.Default` on the tenant-exempt routes — built from each tenant's `auth` block by `Reconfigure`, dropped by `Prune` once the tenant stops being served (rejected or removed), released by `Close`; the secrets (`Config`) are boot-level and shared. A JWKS key set is fetched off the boot and reload paths: until one has been stored the verifier is pending and a token-bearing request gets `503` + `Retry-After` from `api.refuseUnverifiable` (`auth.ErrVerifierPending`), never a `default_role` evaluation; refresh is library-managed (Eric, 2026-09-22), response capped at 1 MiB; the operator key's admin role is the request tenant's -- **`cache/`** — `Cache` interface → `LocalCache` (Ristretto: one pool for every tenant) + `VersionManager` (the invalidation index). Every key leads with the tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8) — `:query:` for a result and its singleflight, `..
.
.` for a namespace — so no cached read or coalesced flight crosses tenants, a bump through `Invalidate` names one tenant's namespaces and no other's, and `InvalidateTenant` advances the tenant version folded into every namespace and query key of one tenant, orphaning its every cached result in one step, pipe results included (no insert reaches a pipe result until [#343](https://github.com/Wave-RF/WaveHouse/pull/343)); `Lookup` returns a `Snapshot` of the versions it read and `Set` files the fill under it, so a write landing mid-query orphans the fill ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)), and every backend runs the conformance suite `internal/testutil/cachetest`; the one crossing is the wiring's, above the package: `internal/app` hands the ingest worker the cache through `sharedTables`, which repeats each of the worker's bumps under every tenant on the same ClickHouse address and database (`chconn.Pools.SharingTables`, whatever their user or tls block — they read the same tables), and orphans the whole cache of a tenant back on a pool after an absence, since it was out of that fan-out while away, or moved to another address or database, since it now reads other tables (story 6) +- **`cache/`** — `Cache` interface → `LocalCache` (Ristretto: one pool for every tenant) + `VersionManager` (the invalidation index), and `RedisCache`, the Redis-compatible shared backend (random version tokens under the tenant's hash tag, one-round-trip lookups, bypass on failure behind a circuit breaker, deferred invalidations retried; built and tested, not yet selectable by config — [#613](https://github.com/Wave-RF/WaveHouse/issues/613) E4). Every key leads with the tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8) — `:query:` for a result and its singleflight, `..
.
.` for a namespace — so no cached read or coalesced flight crosses tenants, a bump through `Invalidate` names one tenant's namespaces and no other's, and `InvalidateTenant` advances the tenant version folded into every namespace and query key of one tenant, orphaning its every cached result in one step, pipe results included (no insert reaches a pipe result until [#343](https://github.com/Wave-RF/WaveHouse/pull/343)); `Lookup` returns a `Snapshot` of the versions it read and `Set` files the fill under it, so a write landing mid-query orphans the fill ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)), and every backend runs the conformance suite `internal/testutil/cachetest`; the one crossing is the wiring's, above the package: `internal/app` hands the ingest worker the cache through `sharedTables`, which repeats each of the worker's bumps under every tenant on the same ClickHouse address and database (`chconn.Pools.SharingTables`, whatever their user or tls block — they read the same tables), and orphans the whole cache of a tenant back on a pool after an absence, since it was out of that fan-out while away, or moved to another address or database, since it now reads other tables (story 6) - **`chconn/`** — `Pools`, one `Manager` (a `driver.Conn`) per distinct `Identity{Addr, Database, Username, Password, TLS}` tuple among the served tenants, reconciled from the settings registry's `AfterAdopt` after every reload ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 6): tenants naming one tuple share its pool, sized to their largest `max_open_conns`/`max_idle_conns`; a tenant whose tuple changed is repointed; a tuple no tenant names is released after the longest `query_timeout` among the tenants it had (never dials; a resize swaps the connection with the same grace). The boot config's `clickhouse.max_total_conns` bounds the open pools' `max_open_conns` together: boot refuses naming sum and ceiling; at a reload a resize above it keeps the pool's size, and a tuple that cannot be opened (the ceiling, an unreadable certificate, or options the driver refuses) leaves its tenants on the pool they had or on none — logged, retried by the next reload. Every consumer resolves its tenant's pool per call: `For` (nil for a tenant on no pool, a `503`), `Target` (the tenant's own HTTP wiring over its pool's TLS config), `SharingTables`, `Ping` (every pool at once, ready at the first answer). `HTTPClients` keeps one `http.Client` per TLS config - **`chsql/`** — dependency-free ClickHouse SQL helpers shared by `query`/`policy` (avoids an import cycle): `QuoteIdent` (backtick-quote every identifier) + `BindUnsafe` (reject names with a literal `?`) - **`config/`** — YAML + env var config loading (cleanenv); strict on both sides (undeclared YAML key, unbound `WH_*` variable) and probes `data_dir` writability — boot is the validator, there is no dry run @@ -127,7 +127,7 @@ Tooling notes (the non-obvious bits `make help` won't tell you): - **Shared mocks in `internal/testutil/`**: Use `MockPublisher` (records `Publish` and `DeadLetter`), `MockCache`, `MockDeduplicator`, `MockSubscriber`, `MockMessage`, `MockPurger`, `MockDeadLetterStats` instead of creating ad-hoc mocks. See `testutil/mocks.go`. - **JWT helpers**: Use `testutil.MakeJWT(t, claims)` and `testutil.MakeExpiredJWT(t, claims)` for auth tests. See `testutil/jwt.go`. - **Schema helpers**: Use `testutil.NewTestSchemaRegistry(t, tables)` for schema-aware tests — it builds the registry through the real discovery path (`Refresh` against a mock ClickHouse connection), so timestamp specs are precomputed like production. -- **Cache backends**: every `cache.Cache` implementation runs `cachetest.Run` (`internal/testutil/cachetest`), the backend-agnostic conformance suite; a behavior the contract promises goes there, not in one backend's tests. +- **Cache backends**: every `cache.Cache` implementation runs `cachetest.Run` (`internal/testutil/cachetest`), the backend-agnostic conformance suite; a behavior the contract promises goes there, not in one backend's tests. `RedisCache` runs it from `internal/cache/redis_integration_test.go` (`//go:build integration`, pinned Redis, Valkey, Dragonfly and Redis Cluster containers), which `make test-integration` includes. - **Policy helpers**: Use `policy.Static(p)` for a fixed `policy.Source` in tests. - **Pipes helpers**: Use `pipes.Static(queries...)` for a fixed `pipes.Source` in tests. - **Response assertions**: Use `testutil.AssertJSONResponse(t, rec, status, expected)` and `testutil.AssertJSONContains(t, rec, status, substring)`. @@ -428,7 +428,7 @@ cmd/ → Binary entry points (thin — argv, logger, config, internal/api/ → HTTP layer (handlers, router, middleware, schema/DLQ/pipes endpoints) internal/app/ → Process wiring (build every component, run them under one errgroup, release in reverse) internal/auth/ → JWT/JWKS authentication middleware (HMAC or JWKS, role extraction from claims) -internal/cache/ → Query cache (interface, Ristretto L1, tenant-led version index) +internal/cache/ → Query cache (interface, Ristretto L1, tenant-led version index, Redis-compatible shared backend) internal/chconn/ → ClickHouse pools, one per connection tuple among the served tenants (reconciled on settings reload) internal/chsql/ → Shared ClickHouse SQL helpers (identifier quoting + bind-safety) internal/config/ → Configuration structs + loader diff --git a/CHANGELOG.md b/CHANGELOG.md index 83f2fd832..6a39bc8df 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added +- **A Redis-compatible shared cache backend, built but not yet selectable** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values over 1 KiB are zstd-compressed and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row open a circuit breaker that skips the server until a probe succeeds, and deferred invalidations are retried until they land, collapsing to one tenant-wide bump per tenant past 100,000 keys. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending`, `wavehouse_cache_value_bytes`, `wavehouse_cache_oversize_total`, `wavehouse_cache_set_failures_total{backend,reason}`. Nothing selects it yet: the `cache.backend` switch and the `cache.redis.*` settings arrive with the wiring (E4), so every deployment still runs `LocalCache`. Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression and server-stops-answering cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. - **A tenant removed or rejected at runtime has its open streams ended** (`internal/stream/{hub,subscriber,bucket}.go` (+ tests), `internal/api/stream.go` (+ tests), `internal/api/router.go`, `internal/settings/{registry,tree}.go` (+ tests), `internal/ingest/worker.go` (+ tests), `internal/app/wire.go` (+ tests), `clients/ts/src/stream/sse.ts`, `docs/src/content/docs/{api,deployment,architecture,ingest-pipeline}.md`, `docs/src/content/docs/settings-directory.mdx`, `AGENTS.md`): story 3 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)). A `GET /v1/stream` used to outlive its tenant: the hub read no policy for it and withheld every row while the keepalive wheel held the connection open, so the client could not tell it from a quiet table. After every reload the hub now evicts the subscribers of each tenant no longer served, removed or rejected alike (`Hub.Prune`, through the close-once `Subscriber.Evict`), and the handler ends the stream: a gap-fill in progress included, and a stream `TenantMW` admitted just before the reload but registered just after it, which the handler checks for as it registers. The client's reconnect gets `404` (removed) or `503` (rejected); the SDK stops on the first and retries the second, resuming from `Last-Event-ID` once the folder is back — except in a browser going cross-origin where tenant `0` is not served or its CORS list does not admit the page, which cannot read either refusal (both are decorated from tenant `0`'s list) and re-dials as after a dropped connection. A flat directory never stops serving tenant `0`, so nothing changes there. Removing a tenant is deleting its folder, then reloading the whole directory, the last folder included: a server started with tenant folders reads the emptied directory as no folder left, where it read as a change of shape and the reload was rejected whole, leaving that tenant served; at boot an empty directory still reads as the four files, missing. Reloading the deleted folder by name leaves its tenant rejected. `GET /v1/health` keeps resolving a tenant, deliberately: its `404` tells a caller with no token no more than every tenant route's does, since they all answer before authenticating, and resolving answers a served tenant's ping from that tenant's own CORS list. A batch queued for a tenant with no ClickHouse connection — one no longer served, or one no pool could be opened for (such as by the connection ceiling) — skips the row-by-row retry, which no row of it could pass, and meets its DLQ switch once, whole, logged once per batch rather than twice per row; a tenant no longer served reads the switch as on, so its queued rows are parked under its own subject rather than dropped, inserted into another tenant's ClickHouse, or left unacked, redelivered for as long as the tenant is away and holding its queue's ack floor, which the purge waits on. Over a nested directory `/livez` no longer keeps naming a tenant that stopped being served before any tenant completed a first discovery: the diagnostic goes back to `no tenant has completed a first discovery yet`. - **One ClickHouse pool per tuple and one schema registry per tenant** (`internal/chconn/chconn.go` (+ tests), `internal/discovery/discovery.go` (+ tests), `internal/app/discoveries.go` (new), `internal/app/{app,wire}.go` (+ tests), `internal/api/{schema,ingest,structured_query,pipes,query,health,errors,clickhouse_exec}.go` (+ tests), `internal/stream/hub.go`, `internal/ingest/worker.go`, `internal/cache/{cache,local,version_manager}.go` (+ tests), `internal/testutil/{testutil,mocks}.go`, `tests/integration/{setup,tenants,boot_resilience,query_limits}_test.go`, `clients/ts/src/{schema,table,sql,client,types}.ts` (+ tests), `tests/e2e/sdk/admin.test.ts`, `docs/src/content/docs/{api,deployment,architecture,ingest-pipeline}.md`, `docs/src/content/docs/{settings-directory,configuration,access-control,reverse-proxy}.mdx`, `docs/src/content/docs/sdk/{admin,reference,queries}.md`, `AGENTS.md`): the second slice of story 6 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)), with no behavior change for a settings directory that holds the four files beyond the three noted at the end. The process opens one native pool per distinct `clickhouse.addr` / `database` / `username` / password / `tls` tuple among the served tenants (`chconn.Identity`, `chconn.Pools`), shared by the tenants naming it and sized to their largest `max_open_conns` and `max_idle_conns`; `http_port`, `http_scheme`, `headers` and `query_timeout` stay each tenant's own. Every reload reconciles the pools: a new tuple opens (never dials), a tenant whose tuple changed is repointed, a tuple no tenant names closes after the longest `query_timeout` among the tenants it had, and a pool whose largest ask changed is resized with the same grace. The boot config's `clickhouse.max_total_conns` now bounds the open pools together: boot is refused naming the sum and the ceiling; at a reload a resize above it is refused with the pool kept at its size, and a tuple that cannot be opened — the ceiling, a certificate file that cannot be read, or options the driver refuses — leaves its tenants on the pool they had (the keep-previous-wiring rule of the connection ceiling) or on none when they had none; both are logged and the next reload retries. A tenant on no pool fails closed: `503` with `Retry-After: 30` on `POST /v1/query`, `GET/POST /v1/pipes/{name}`, `POST /v1/ops/query` and `POST /v1/ops/schema/refresh`, ahead of the cache. Each served tenant gets a `discovery.SchemaRegistry` of its own over its pool, kept fresh by its own loop — the boot retry until the first success, then `schema.refresh_interval` with the first refresh at a random point within the interval so tenants adopted together do not refresh together — created and stopped from the settings reload and stopped under `App.Close`; `wavehouse_schema_refresh_failures_total{tenant}` counts a loop's failed attempts. `GET /v1/ops/schema`, `POST /v1/ops/schema/refresh` and `POST /v1/ops/query` take the strict `?tenant=` the pipe reads take (absent is tenant `0`, `400` malformed, `404` unknown, `503` rejected); the SDK sends it as the `tenant` option of `wh.schema.list()`, `wh.schema.refresh()`, `wh.from(t).schema()` and `wh.sql()`. Over a nested directory `/livez` (and `/v1/health`) is `503` with the latest discovery failure, naming its tenant, while no tenant has completed a first discovery, then `200` for the rest of the process lifetime; `/readyz` pings every open pool at once, is ready at the first answer, and names every pool that did not answer when none does — one tenant's ClickHouse outage is its log line and counter, never a probe failure. The ingest worker inserts each batch into its own tenant's ClickHouse — the tenant the message's topic names, through that tenant's HTTP wiring (`chconn.Pools.Target`); a tenant on no pool takes the failure path an unreachable ClickHouse takes — and its cache invalidation fans out to the tenants on the same address and database as the batch's tenant, whatever their user or `tls` block (they read the same tables), rather than to every known tenant, and a tenant adopted after an absence — rejected or removed, so out of that fan-out — or moved to another address or database has its cached structured-query results orphaned in one step (`Cache.InvalidateTenant`, a tenant generation in every version key), so a repaired folder never serves query rows cached before the inserts it missed (a pipe result names no table, so neither this nor any insert invalidates it: it stays until its TTL expires). Three changes reach the single-tenant directory too: a table lookup before the first discovery is a `503` with `Retry-After: 5` (`schema not loaded yet`) rather than a `404`, on `POST /v1/ingest`, `POST /v1/query` and `GET /v1/ops/schema` (the list included, where `[]` would read as no tables); the first periodic refresh fires at a random point within the interval rather than a full interval after boot; and a reload that moves `clickhouse.addr` or `clickhouse.database` now orphans the structured-query results cached before it, where they were served until their TTL. Until [#529](https://github.com/Wave-RF/WaveHouse/issues/529) every tenant's user authenticates with `WH_CH_PASSWORD`, so the tuple is in effect the address, database, username and `tls` block. - **One token verifier per tenant, built off the boot and reload paths** (`internal/auth/auth.go` (+ tests), `internal/api/router.go` (+ tests), `internal/app/{app,wire}.go` (+ tests), `go.mod`, `docs/src/content/docs/sdk/{reference,streaming}.md`, `docs/src/content/docs/{architecture,deployment,api}.md`, `docs/src/content/docs/{settings-directory,configuration}.mdx`, `SECURITY.md`): story 9 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)). Over a nested settings directory each tenant's folder now wires that tenant's verifier (`auth.jwks_url`, `auth.role_claim`), so a JWKS-issued token verifies only under the tenants whose `jwks_url` names its identity provider's key set — under any other tenant's header it is refused as invalid — where before one verifier, tenant `0`'s, accepted a token under any header. `auth.Config` is now the boot-config half alone (the HMAC secret and the operator key, shared by every tenant) and the new `auth.Wiring` a tenant's half; `NewAuthenticator` builds no verifier, `Reconfigure(id, wiring)` gives a tenant one — swapped atomically when its wiring changed, kept when it did not — `Prune` drops the verifiers of the tenants a reload stopped serving, rejected or removed alike (no work runs for a tenant that is not served; a folder adopted again gets a fresh verifier), and `Close` stops every JWKS refresh as a component of `App.Close`. This holds on every reload because the registry's `AfterAdopt` hooks run after every reload it applied, empty list included, so a per-tenant reload that rejects a folder drops its verifier then rather than at the next adoption. The middleware reads the request tenant's verifier through an injected `auth.TenantSource` — the store `api.TenantMW` resolved names its tenant with `settings.Store.Tenant()`, one read of the context — a tenant-exempt route (the ops tree) verifies as tenant `0`, and a tenant with no verifier fails closed. The operator key stamps the request tenant's `admin_role`, read from that tenant's policy, rather than tenant `0`'s. A JWKS key set is fetched on its own goroutine: the verifier is in place at once and *pending* until a set has been stored — the first fetch retried with backoff from one second to a minute, a stored set then kept fresh by the library hourly and, rate-limited, on an unknown key id — so neither boot nor a reload (which holds the registry's lock) waits on the endpoint, and **an unreachable JWKS no longer refuses boot** — it logs (`jwks refresh failed; no token validates until it succeeds`) and that tenant alone is affected. A token checked against a pending verifier is a new outcome, `auth.ErrVerifierPending`: every `/v1` route answers it `503 {"error": "token verifier not ready: …"}` with `Retry-After: 30` (`api.refuseUnverifiable`) rather than evaluate the request under the `default_role`, which could accept its data under a lesser role while another pod holding the keys would have served it as its own; a request without a token, and the operator key, are unaffected. Every fetch goes through one client that caps the response at 1 MiB, refusing a larger one as unreachable with `jwks response exceeds 1048576 bytes` as the logged cause. Nothing changes for a flat directory beyond that boot rule: its one tenant gets exactly the verifier it had. The boot warning for the secretless posture (no `auth.jwt_secret` and no `jwks_url`, whose token refusal landed in [#607](https://github.com/Wave-RF/WaveHouse/pull/607)) is now per tenant — each served tenant with no `jwks_url` while the boot secret is unset — where one line that any JWKS tenant silenced used to stand for the whole directory. diff --git a/Makefile b/Makefile index 8dcbe35d0..5fff09f2b 100644 --- a/Makefile +++ b/Makefile @@ -764,7 +764,7 @@ test-integration: go-mod-download ## Run Go integration tests + render coverage @rm -rf $(COV_INT)/data && mkdir -p $(COV_INT)/data @GOCOVERDIR="$(CURDIR)/$(COV_INT)/data" go tool gotestsum --format $(GOTESTSUM_FMT) -- \ -tags="integration $(TAGS)" -timeout 240s -coverpkg=./... -race -count=1 \ - ./tests/integration/... $(ARGS) \ + ./tests/integration/... ./internal/cache/... $(ARGS) \ -args -test.gocoverdir="$(CURDIR)/$(COV_INT)/data" @if [ -z "$(COV_DEFER)" ]; then go run ./scripts/cov render integration; fi diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 3ba9a9eb0..53850f49f 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -54,7 +54,7 @@ internal/ ├── api/ HTTP layer (Chi router, handlers, middleware, the cached read paths' singleflight) ├── app/ Process wiring: build every component, run them under one errgroup, release in reverse ├── auth/ JWT/JWKS authentication middleware (HMAC or JWKS, role extraction) -├── cache/ Query cache: Ristretto L1 + the tenant-led version index +├── cache/ Query cache: Ristretto L1 + the tenant-led version index; the Redis-compatible shared backend ├── chconn/ One ClickHouse pool per connection tuple among the served tenants, reconciled on reload under the ceiling ├── chsql/ Shared ClickHouse SQL helpers (identifier quoting, bind-safety) ├── config/ YAML + env var configuration loading @@ -112,6 +112,11 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ - **cache.go** — `Cache` interface: `Lookup`, `Set`, `Invalidate`, `InvalidateTenant`, `Close`, plus `QueryTimeToTTL`, which sets a result's TTL from how long its query took (10 s floor, 1 h ceiling). Every entry is one tenant's: `Lookup` takes the tenant, the caller's query key — `:query:`, built by the two cached handlers in `api/` (`queryCacheKey`, with the tenant read off the request's store — `settings.Store.Tenant`), which use it as their [singleflight](https://pkg.go.dev/golang.org/x/sync/singleflight) key too — and the `Namespace`s the result depends on, each of that tenant (another tenant's is `ErrForeignDependency`) and naming a table and scope: one for a structured query, none yet for a pipe (a pipe's table dependencies are [#343](https://github.com/Wave-RF/WaveHouse/pull/343)). `Lookup` returns the `Entry` (a nil value is a miss) and a `Snapshot` of the versions it read; on a miss the handler runs the query and passes that snapshot to `Set`, so a result is filed under the versions read *before* its query ran, and a write that lands while it runs orphans the fill rather than re-homing pre-write rows under the post-write versions ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)). The singleflight leader's snapshot is the one used. `Set` returns an error only when the backend failed; a value the cache declines — larger than it keeps, a non-positive TTL, a zero snapshot — is not one. What a hit, a miss and a bump mean is pinned by the conformance suite every backend runs, `internal/testutil/cachetest`. - **local.go** — `LocalCache`, the in-process L1 on [Ristretto](https://github.com/dgraph-io/ristretto): one pool shared by every tenant (a heavier tenant holds more of it), sized by the boot config's `cache.l1_max_cost`. - **version_manager.go** — `VersionManager`, the invalidation index behind `Invalidate` and `InvalidateTenant`: a namespace key is `..
.
.`, and a query key is folded with the tenant's version and each dependency's namespace key and namespace version, so bumping a table (a scopeless write) or one scope — scope is reserved and empty today, so every write is the whole-table bump — orphans every dependent entry without touching the pool. The tenant leads every key ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8): the same table under two tenants is two namespaces, so a bump through `Invalidate` under one tenant never touches — and a read under one tenant is never served — the other's results, and the flat directory's single tenant simply carries the `0` prefix. `BumpTenant` (behind `InvalidateTenant`) advances the tenant version that leads every namespace key of one tenant, orphaning its every namespace and every cached query in one step (the tenant version is folded into every query key, so a pipe result with no dependencies is orphaned too) — a table no bump ever keyed included, which an enumeration of the index would miss — for a tenant back on a pool after an absence from the fan-out, or moved to another address or database (story 6). The index is per tenant; the cross-tenant invalidation an insert into a shared table needs is not the index's but the wiring's: `internal/app` hands the ingest worker a cache (`sharedTables`) that repeats each bump under every tenant on the same ClickHouse address and database. +- **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — only `GET`, `SET` and `MGET`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, so a token lost to eviction, expiry or a restart is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing. Nothing selects this backend yet — the boot config and wiring arrive with [#613](https://github.com/Wave-RF/WaveHouse/issues/613)'s E4. +- **redis_codec.go** — the key schema (`tokenKeys`, `bumpKeys`, `valueKey`) and the value frame: format, flags, expiry, the token list, the payload, zstd-compressed from `CompressMinBytes` when that is smaller. A stored value is capped at `MaxValueBytes` and a decoded one at eight times that, which refuses a zip bomb planted in a shared server; an unknown format, as a newer process writes during a rolling upgrade, is a miss. +- **breaker.go** — the circuit breaker: `BreakerThreshold` (5) consecutive failures open it, and while open every operation skips the server at once; after `BreakerOpenFor` (5 s) one background `PING` decides whether it closes. A reply from the server, error replies included, counts as a success; a caller that gave up first counts as nothing. +- **pending.go** — invalidations the server did not take, kept per token key (repeats coalesce) and retried by a background loop with backoff from 100 ms to 10 s until they land. Landing late is still correct: a fresh token orphans the pre-write entries and any fill made meanwhile. Past `PendingMax` (100,000) keys the set collapses to one tenant bump per affected tenant. Bumps still owed when a process stops are lost after one last attempt, and the entries they would orphan are served until their TTL — the same failure as a worker stopping between an insert and its invalidation. +- **metrics.go** — the shared backend's instruments (meter `wavehouse-cache`, `backend="redis"`, no tenant attribute): `wavehouse_cache_lookups_total` by `result` (`hit`, `miss`, `stale` — filed under since-bumped tokens, `bypass` — server skipped, `error`), `wavehouse_cache_op_duration_seconds` by `op` (`lookup`, `set`, `invalidate`), `wavehouse_cache_breaker_open` (1 while bypassed, including before the first connection), `wavehouse_cache_invalidations_total` by `result` (`ok`, `deferred`), `wavehouse_cache_invalidations_pending` (bumps owed: entries they would orphan may be served stale meanwhile), `wavehouse_cache_value_bytes` (stored size), `wavehouse_cache_oversize_total` and `wavehouse_cache_set_failures_total` by `reason` (`oom`, `timeout`, `other`). ### `config/` — Configuration @@ -360,6 +365,7 @@ Client GET /v1/stream | Analytics DB | ClickHouse | Primary data store + schema source of truth | | Message Queue | NATS + JetStream | Durable event streaming | | L1 Cache | Ristretto v2 | In-process memory cache | +| Shared cache | [rueidis](https://github.com/redis/rueidis) | Redis-compatible client for the shared backend (not yet selectable) | | Embedded KV | Pebble | Optional deduplication | | Config | cleanenv | YAML + env var config loading | | Release | GoReleaser | Cross-platform binary builds | diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index 01f82e735..8e6a85036 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -341,7 +341,7 @@ Each test target writes `covdata` to `tmp/coverage//data/`, renders a tex | -------- | -------- | ------- | ------- | | Unit tests | `internal/*/_test.go` | No | `make test` | | SDK unit tests | `clients/ts/src/**/*.test.ts` | No | `make test-ts` (always includes coverage + gate) | -| Integration tests (Go) | `tests/integration/*_test.go` | Yes | `make test-integration` | +| Integration tests (Go) | `tests/integration/*_test.go`, `internal/cache/*_integration_test.go` | Yes | `make test-integration` | | E2E tests (SDK) | `tests/e2e/sdk/*.test.ts` | Yes | `make test-e2e` | - **Unit tests** live beside the code they test (e.g., `internal/discovery/discovery_test.go`). They use mocks or embedded NATS (in-process, no Docker needed). @@ -352,7 +352,7 @@ Shared test utilities live in `internal/testutil/`. The packages log through `sl ### Adding New Tests - **Unit test for `internal/foo/`** → create `internal/foo/foo_test.go` (same package). -- **Integration test needing Docker** → add a subtest under `tests/integration/` (e.g. a new file with `//go:build integration`). +- **Integration test needing Docker** → add a subtest under `tests/integration/` (e.g. a new file with `//go:build integration`). A test of one package against its own external server — the shared cache backend against Redis, Valkey and Dragonfly containers — lives beside the package instead (`internal/cache/redis_integration_test.go`, same build tag), and the package is listed in the `test-integration` target. - **E2E test via SDK** → add a `tests/e2e/sdk/*.test.ts` file. These tests exercise the full pipeline (ingest → ClickHouse → query) through the TypeScript SDK. Run with `make test-e2e`. - **Test helpers** → add to `internal/testutil/` (Go) or `tests/e2e/sdk/helpers.ts` (E2E). diff --git a/go.mod b/go.mod index ca418e893..d75355341 100644 --- a/go.mod +++ b/go.mod @@ -26,9 +26,13 @@ require ( github.com/golang-jwt/jwt/v5 v5.3.1 github.com/google/uuid v1.6.0 github.com/ilyakaznacheev/cleanenv v1.5.0 + github.com/klauspost/compress v1.19.2 + github.com/moby/moby/api v1.55.0 + github.com/moby/moby/client v0.5.1 github.com/nats-io/nats-server/v2 v2.14.6 github.com/nats-io/nats.go v1.53.1 github.com/prometheus/client_golang v1.24.1 + github.com/redis/rueidis v1.0.78 github.com/samber/slog-multi v1.8.0 github.com/samber/slog-sampling v1.7.0 github.com/stretchr/testify v1.12.1 @@ -132,7 +136,6 @@ require ( github.com/jedib0t/go-pretty/v6 v6.7.10 // indirect github.com/jmespath/go-jmespath v0.4.0 // indirect github.com/joho/godotenv v1.5.1 // indirect - github.com/klauspost/compress v1.19.2 // indirect github.com/knadh/profiler v0.2.0 // indirect github.com/kr/pretty v0.3.1 // indirect github.com/kr/text v0.2.0 // indirect @@ -147,8 +150,6 @@ require ( github.com/minio/highwayhash v1.0.4 // indirect github.com/moby/docker-image-spec v1.3.1 // indirect github.com/moby/go-archive v0.3.0 // indirect - github.com/moby/moby/api v1.55.0 // indirect - github.com/moby/moby/client v0.5.1 // indirect github.com/moby/patternmatcher v0.6.1 // indirect github.com/moby/sys/sequential v0.7.0 // indirect github.com/moby/sys/user v0.4.1 // indirect diff --git a/go.sum b/go.sum index 71dc27272..3b203e221 100644 --- a/go.sum +++ b/go.sum @@ -286,6 +286,8 @@ github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw= github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c= github.com/nikolaydubina/treemap v1.2.5 h1:oSC5z/qnsGLbkU2IihSrh2pS7uDjUq7ipGj8aw8bfII= github.com/nikolaydubina/treemap v1.2.5/go.mod h1:8+wLGh917AyeJqBN1D5KM26tv6W/XfvsY+nfJd04/u8= +github.com/onsi/gomega v1.42.1 h1:iN1rCUX+44NZ1Dc97MPoeFYbFR0vh8zxoxMFwKdyZ6I= +github.com/onsi/gomega v1.42.1/go.mod h1:REff/hsDsodHoKlWsP2mAPhu1+5/6hVYNf9rIEBpeSg= github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U= github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM= github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJwooC2xJA040= @@ -320,6 +322,8 @@ github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+ github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY= github.com/puzpuzpuz/xsync/v4 v4.5.0 h1:vOSWu6b57/emh+L/Cw0BeQfvxa/cogFywXHeGUxQxAg= github.com/puzpuzpuz/xsync/v4 v4.5.0/go.mod h1:VJDmTCJMBt8igNxnkQd86r+8KUeN1quSfNKu5bLYFQo= +github.com/redis/rueidis v1.0.78 h1:hJXpEgC9IYfdwY4hCdaGYsfK+oUaAqvhI/GMy5akVJI= +github.com/redis/rueidis v1.0.78/go.mod h1:L8mnCQJJaSNL6I4pIR6Rz732HTGS9vmuXm0yT9dRvjo= github.com/rivo/uniseg v0.1.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc= github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc= github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ= diff --git a/internal/cache/breaker.go b/internal/cache/breaker.go new file mode 100644 index 000000000..c90ab3a91 --- /dev/null +++ b/internal/cache/breaker.go @@ -0,0 +1,65 @@ +package cache + +import ( + "sync" + "time" +) + +// breaker stops a failing cache server from costing every request its full +// timeout: after threshold consecutive failures it opens, and while open +// callers skip the server entirely. Once openFor has passed, one caller is +// told to probe; the probe's outcome closes the breaker or reopens it. +type breaker struct { + threshold int + openFor time.Duration + now func() time.Time + + mu sync.Mutex + failures int + open bool + openedAt time.Time + probing bool +} + +func newBreaker(threshold int, openFor time.Duration, now func() time.Time) *breaker { + return &breaker{threshold: threshold, openFor: openFor, now: now} +} + +// allow reports whether a call may go to the server, and whether the caller +// should start the one probe that decides whether an open breaker closes. +func (b *breaker) allow() (ok, probe bool) { + b.mu.Lock() + defer b.mu.Unlock() + if !b.open { + return true, false + } + if b.probing || b.now().Sub(b.openedAt) < b.openFor { + return false, false + } + b.probing = true + return false, true +} + +// success records a call the server answered, and closes the breaker. +func (b *breaker) success() { + b.mu.Lock() + defer b.mu.Unlock() + b.failures, b.open, b.probing = 0, false, false +} + +// failure records a call the server did not answer in time, and opens the +// breaker at the threshold — or at once, for a failed probe. +func (b *breaker) failure() { + b.mu.Lock() + defer b.mu.Unlock() + b.failures++ + if b.probing || b.failures >= b.threshold { + b.open, b.openedAt, b.probing = true, b.now(), false + } +} + +func (b *breaker) isOpen() bool { + b.mu.Lock() + defer b.mu.Unlock() + return b.open +} diff --git a/internal/cache/breaker_test.go b/internal/cache/breaker_test.go new file mode 100644 index 000000000..e6dc2296c --- /dev/null +++ b/internal/cache/breaker_test.go @@ -0,0 +1,59 @@ +package cache + +import ( + "testing" + "time" + + "github.com/stretchr/testify/assert" +) + +type fakeClock struct{ t time.Time } + +func (c *fakeClock) now() time.Time { return c.t } + +func TestBreaker(t *testing.T) { + t.Parallel() + clock := &fakeClock{t: time.Unix(0, 0)} + b := newBreaker(3, 5*time.Second, clock.now) + allow := func() (bool, bool) { return b.allow() } + + ok, probe := allow() + assert.True(t, ok) + assert.False(t, probe) + + b.failure() + b.failure() + b.success() // a success resets the run + b.failure() + b.failure() + assert.False(t, b.isOpen(), "two in a row is below the threshold") + b.failure() + assert.True(t, b.isOpen()) + + ok, probe = allow() + assert.False(t, ok, "open: skip the server") + assert.False(t, probe, "not due for a probe yet") + + clock.t = clock.t.Add(5 * time.Second) + ok, probe = allow() + assert.False(t, ok) + assert.True(t, probe, "due: this caller probes") + ok, probe = allow() + assert.False(t, ok) + assert.False(t, probe, "one probe at a time") + + b.failure() // the probe failed: open for another period + assert.True(t, b.isOpen()) + clock.t = clock.t.Add(4 * time.Second) + _, probe = allow() + assert.False(t, probe) + clock.t = clock.t.Add(time.Second) + _, probe = allow() + assert.True(t, probe) + + b.success() + assert.False(t, b.isOpen()) + ok, probe = allow() + assert.True(t, ok) + assert.False(t, probe) +} diff --git a/internal/cache/cache.go b/internal/cache/cache.go index c9e66a96b..11fbc8512 100644 --- a/internal/cache/cache.go +++ b/internal/cache/cache.go @@ -19,7 +19,8 @@ type Entry struct { // the query runs orphans the fill rather than re-homing pre-write rows under // the post-bump versions (#382). The zero Snapshot makes Set a no-op. type Snapshot struct { - key string // the backend's key for the entry at the observed versions + key string // the backend's key for the entry at the observed versions + tokens []byte // RedisCache: the version tokens read, in tokenKeys order } // ErrForeignDependency is a Lookup whose dependencies name a tenant other diff --git a/internal/cache/export_test.go b/internal/cache/export_test.go new file mode 100644 index 000000000..7c392c13e --- /dev/null +++ b/internal/cache/export_test.go @@ -0,0 +1,12 @@ +package cache + +// Hooks for the integration tests in package cache_test. + +// Pending reports how many token bumps r still owes the server. +func Pending(r *RedisCache) int { return r.pending.len() } + +// Bypassed reports whether r is skipping the server. +func Bypassed(r *RedisCache) bool { return r.bypassed() } + +// DecodedFactor is how many times MaxValueBytes a value may decompress to. +const DecodedFactor = decodedFactor diff --git a/internal/cache/metrics.go b/internal/cache/metrics.go new file mode 100644 index 000000000..cf76eb7b3 --- /dev/null +++ b/internal/cache/metrics.go @@ -0,0 +1,106 @@ +package cache + +import ( + "context" + "errors" + "time" + + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" +) + +// Lookup outcomes for the "result" attribute of wavehouse_cache_lookups_total. +const ( + resultHit = "hit" + resultMiss = "miss" // nothing stored + resultStale = "stale" // stored under versions since bumped + resultBypass = "bypass" // server skipped: breaker open or not yet connected + resultError = "error" // server failed or timed out +) + +// metrics are a shared-cache backend's instruments. No tenant attribute: +// lookups are the hot path, and tenants are unbounded. +type metrics struct { + backend attribute.KeyValue + lookups metric.Int64Counter + duration metric.Float64Histogram + invalidation metric.Int64Counter + valueBytes metric.Int64Histogram + oversize metric.Int64Counter + setFailures metric.Int64Counter + registration metric.Registration +} + +// newMetrics builds the instruments on the global meter provider, with +// gauges read from breakerOpen and pending on every collection. Call it +// after observability.InitProvider, like every other instrument. +func newMetrics(backend string, breakerOpen func() bool, pending func() int) (*metrics, error) { + meter := otel.Meter("wavehouse-cache") + m := &metrics{backend: attribute.String("backend", backend)} + var errs [9]error + m.lookups, errs[0] = meter.Int64Counter("wavehouse_cache_lookups_total", + metric.WithDescription("Shared-cache lookups by result: hit, miss, stale (stored under since-bumped versions), bypass (server skipped), error")) + m.duration, errs[1] = meter.Float64Histogram("wavehouse_cache_op_duration_seconds", + metric.WithDescription("Shared-cache round-trip time by op: lookup, set, invalidate"), metric.WithUnit("s"), + metric.WithExplicitBucketBoundaries(.0001, .00025, .0005, .001, .0025, .005, .01, .025, .05, .1, .25)) + m.invalidation, errs[2] = meter.Int64Counter("wavehouse_cache_invalidations_total", + metric.WithDescription("Version-token bumps by result: ok, or deferred to the pending retry set")) + m.valueBytes, errs[3] = meter.Int64Histogram("wavehouse_cache_value_bytes", + metric.WithDescription("Size of each value written to the shared cache, after compression"), metric.WithUnit("By"), + metric.WithExplicitBucketBoundaries(256, 1<<10, 4<<10, 16<<10, 64<<10, 256<<10, 1<<20, 4<<20)) + m.oversize, errs[4] = meter.Int64Counter("wavehouse_cache_oversize_total", + metric.WithDescription("Results not cached because they exceed the value size limit")) + m.setFailures, errs[5] = meter.Int64Counter("wavehouse_cache_set_failures_total", + metric.WithDescription("Shared-cache writes that failed, by reason: oom, timeout, other")) + breakerGauge, err := meter.Int64ObservableGauge("wavehouse_cache_breaker_open", + metric.WithDescription("1 while the shared cache is being bypassed (circuit breaker open, or never connected), else 0")) + errs[6] = err + pendingGauge, err := meter.Int64ObservableGauge("wavehouse_cache_invalidations_pending", + metric.WithDescription("Version-token bumps not yet delivered to the shared cache; entries they would orphan may be served stale meanwhile")) + errs[7] = err + m.registration, errs[8] = meter.RegisterCallback(func(_ context.Context, o metric.Observer) error { + var open int64 + if breakerOpen() { + open = 1 + } + o.ObserveInt64(breakerGauge, open, metric.WithAttributes(m.backend)) + o.ObserveInt64(pendingGauge, int64(pending()), metric.WithAttributes(m.backend)) + return nil + }, breakerGauge, pendingGauge) + if err := errors.Join(errs[:]...); err != nil { + return nil, err + } + return m, nil +} + +func (m *metrics) lookup(result string) { + m.lookups.Add(context.Background(), 1, metric.WithAttributes(m.backend, attribute.String("result", result))) +} + +func (m *metrics) op(op string, start time.Time) { + m.duration.Record(context.Background(), time.Since(start).Seconds(), + metric.WithAttributes(m.backend, attribute.String("op", op))) +} + +func (m *metrics) invalidated(result string, n int) { + if n > 0 { + m.invalidation.Add(context.Background(), int64(n), metric.WithAttributes(m.backend, attribute.String("result", result))) + } +} + +func (m *metrics) stored(n int) { + m.valueBytes.Record(context.Background(), int64(n), metric.WithAttributes(m.backend)) +} + +func (m *metrics) tooLarge() { + m.oversize.Add(context.Background(), 1, metric.WithAttributes(m.backend)) +} + +func (m *metrics) setFailed(reason string) { + m.setFailures.Add(context.Background(), 1, metric.WithAttributes(m.backend, attribute.String("reason", reason))) +} + +func (m *metrics) close() { + _ = m.registration.Unregister() +} diff --git a/internal/cache/pending.go b/internal/cache/pending.go new file mode 100644 index 000000000..c19b21786 --- /dev/null +++ b/internal/cache/pending.go @@ -0,0 +1,78 @@ +package cache + +import ( + "sync" + + "github.com/Wave-RF/WaveHouse/internal/tenant" +) + +// pendingBumps holds the token bumps an invalidation could not deliver, until +// a retry lands them. Repeats of one key coalesce; past max keys the set +// collapses to one tenant bump per affected tenant — coarser, never less. +// Landing a bump late is still correct: a fresh token orphans the pre-write +// entries and any fill in between. +type pendingBumps struct { + prefix string + max int + + mu sync.Mutex + keys map[string]pendingKey + gen uint64 +} + +type pendingKey struct { + tenant tenant.ID + gen uint64 // when last added; take drops a key only if it is unchanged +} + +func newPendingBumps(prefix string, maxKeys int) *pendingBumps { + return &pendingBumps{prefix: prefix, max: maxKeys, keys: map[string]pendingKey{}} +} + +// add records keys of tenant id as owed a bump. +func (p *pendingBumps) add(id tenant.ID, keys ...string) { + p.mu.Lock() + defer p.mu.Unlock() + p.gen++ + for _, k := range keys { + p.keys[k] = pendingKey{tenant: id, gen: p.gen} + } + if len(p.keys) <= p.max { + return + } + collapsed := make(map[string]pendingKey, len(p.keys)) + for _, pk := range p.keys { + collapsed[tenantTokenKey(p.prefix, pk.tenant)] = pendingKey{tenant: pk.tenant, gen: p.gen} + } + p.keys = collapsed +} + +// snapshot returns the keys owed a bump with the generation each was added +// at, for a later done. +func (p *pendingBumps) snapshot() map[string]uint64 { + p.mu.Lock() + defer p.mu.Unlock() + out := make(map[string]uint64, len(p.keys)) + for k, pk := range p.keys { + out[k] = pk.gen + } + return out +} + +// done drops keys whose bump landed — unless one was added again since the +// snapshot, whose bump the landed one may have preceded. +func (p *pendingBumps) done(landed map[string]uint64) { + p.mu.Lock() + defer p.mu.Unlock() + for k, gen := range landed { + if pk, ok := p.keys[k]; ok && pk.gen == gen { + delete(p.keys, k) + } + } +} + +func (p *pendingBumps) len() int { + p.mu.Lock() + defer p.mu.Unlock() + return len(p.keys) +} diff --git a/internal/cache/pending_test.go b/internal/cache/pending_test.go new file mode 100644 index 000000000..cc8f331da --- /dev/null +++ b/internal/cache/pending_test.go @@ -0,0 +1,45 @@ +package cache + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestPendingBumps_Coalesce(t *testing.T) { + t.Parallel() + p := newPendingBumps("wh", 10) + p.add("acme", "k1", "k2") + p.add("acme", "k1") + assert.Equal(t, 2, p.len()) + + snap := p.snapshot() + p.done(snap) + assert.Zero(t, p.len()) +} + +// A key added again after the snapshot a drain worked from stays owed: the +// drain's bump may have landed before the write the new add is for. +func TestPendingBumps_ReAddedKeyStays(t *testing.T) { + t.Parallel() + p := newPendingBumps("wh", 10) + p.add("acme", "k1", "k2") + snap := p.snapshot() + p.add("acme", "k1") + p.done(snap) + assert.Equal(t, map[string]uint64{"k1": 2}, p.snapshot()) +} + +func TestPendingBumps_OverflowCollapsesToTenants(t *testing.T) { + t.Parallel() + p := newPendingBumps("wh", 3) + p.add("acme", "wh:{acme}:B:a", "wh:{acme}:B:b") + p.add("globex", "wh:{globex}:B:a") + assert.Equal(t, 3, p.len()) + + p.add("acme", "wh:{acme}:B:c") + got := p.snapshot() + assert.Len(t, got, 2) + assert.Contains(t, got, "wh:{acme}:T") + assert.Contains(t, got, "wh:{globex}:T") +} diff --git a/internal/cache/redis.go b/internal/cache/redis.go new file mode 100644 index 000000000..ede95a534 --- /dev/null +++ b/internal/cache/redis.go @@ -0,0 +1,644 @@ +package cache + +import ( + "bytes" + "context" + "crypto/tls" + "errors" + "fmt" + "log/slog" + "net" + "strings" + "sync" + "sync/atomic" + "time" + + "github.com/redis/rueidis" + + "github.com/Wave-RF/WaveHouse/internal/tenant" +) + +// Redis deployment modes for RedisConfig.Mode. +const ( + RedisStandalone = "standalone" + RedisCluster = "cluster" + RedisSentinel = "sentinel" +) + +// Defaults for RedisConfig's zero values. +const ( + DefaultRedisKeyPrefix = "wh" + DefaultRedisTimeout = 100 * time.Millisecond + DefaultRedisDialTimeout = time.Second + DefaultRedisMaxValueBytes = 1 << 20 + DefaultRedisCompressMinBytes = 1 << 10 + DefaultRedisVersionTTL = 7 * 24 * time.Hour + DefaultRedisPendingMax = 100_000 + defaultBreakerThreshold = 5 + defaultBreakerOpenFor = 5 * time.Second +) + +const ( + drainMinBackoff = 100 * time.Millisecond + drainMaxBackoff = 10 * time.Second + drainIdle = time.Second + drainBatch = 1000 + dialMaxBackoff = 30 * time.Second + closeDrainBudget = time.Second +) + +var errBypassed = errors.New("cache: redis unavailable") + +// RedisConfig configures a RedisCache. A zero field takes its Default* +// value, except CompressMinBytes, where 0 means never compress. +type RedisConfig struct { + Addrs []string // host:port; several are seeds (cluster) or sentinels + Mode string // RedisStandalone (""), RedisCluster or RedisSentinel + SentinelMaster string // the master set name, Mode RedisSentinel + Username string + Password string + DB int // standalone and sentinel only + TLS *tls.Config // nil: plaintext + + KeyPrefix string // leads every key; separates deployments sharing a server + Timeout time.Duration // per operation + DialTimeout time.Duration + MaxValueBytes int // largest value stored, after compression + CompressMinBytes int // zstd-compress values at least this large + VersionTTL time.Duration // idle lifetime of a version token, jittered ±10% + PendingMax int // undelivered bumps kept before collapsing to tenant bumps + + BreakerThreshold int // consecutive failures that open the breaker + BreakerOpenFor time.Duration // how long it stays open before a probe +} + +func (c RedisConfig) withDefaults() (RedisConfig, error) { + if len(c.Addrs) == 0 { + return c, errors.New("cache: redis needs at least one address") + } + for _, a := range c.Addrs { + if _, _, err := net.SplitHostPort(a); err != nil { + return c, fmt.Errorf("cache: redis address %q: %w", a, err) + } + } + switch c.Mode { + case "": + c.Mode = RedisStandalone + case RedisStandalone, RedisCluster, RedisSentinel: + default: + return c, fmt.Errorf("cache: redis mode %q: want %s, %s or %s", c.Mode, RedisStandalone, RedisCluster, RedisSentinel) + } + if c.Mode == RedisCluster && c.DB != 0 { + return c, errors.New("cache: redis cluster has only database 0") + } + if c.Mode == RedisSentinel && c.SentinelMaster == "" { + return c, errors.New("cache: redis sentinel mode needs the master set name") + } + if c.DB < 0 { + return c, fmt.Errorf("cache: redis db %d is negative", c.DB) + } + if strings.ContainsAny(c.KeyPrefix, "{}") { + return c, fmt.Errorf("cache: redis key prefix %q may not contain a hash tag brace", c.KeyPrefix) + } + for _, v := range []struct { + name string + n int64 + }{ + {"timeout", int64(c.Timeout)}, + {"dial timeout", int64(c.DialTimeout)}, + {"max value bytes", int64(c.MaxValueBytes)}, + {"compress min bytes", int64(c.CompressMinBytes)}, + {"version ttl", int64(c.VersionTTL)}, + {"pending max", int64(c.PendingMax)}, + {"breaker threshold", int64(c.BreakerThreshold)}, + {"breaker open for", int64(c.BreakerOpenFor)}, + } { + if v.n < 0 { + return c, fmt.Errorf("cache: redis %s is negative", v.name) + } + } + c.KeyPrefix = cmpOr(c.KeyPrefix, DefaultRedisKeyPrefix) + c.Timeout = cmpOr(c.Timeout, DefaultRedisTimeout) + c.DialTimeout = cmpOr(c.DialTimeout, DefaultRedisDialTimeout) + c.MaxValueBytes = cmpOr(c.MaxValueBytes, DefaultRedisMaxValueBytes) + c.VersionTTL = cmpOr(c.VersionTTL, DefaultRedisVersionTTL) + c.PendingMax = cmpOr(c.PendingMax, DefaultRedisPendingMax) + c.BreakerThreshold = cmpOr(c.BreakerThreshold, defaultBreakerThreshold) + c.BreakerOpenFor = cmpOr(c.BreakerOpenFor, defaultBreakerOpenFor) + return c, nil +} + +func cmpOr[T comparable](v, def T) T { + var zero T + if v == zero { + return def + } + return v +} + +func (c RedisConfig) clientOption() rueidis.ClientOption { + opt := rueidis.ClientOption{ + InitAddress: c.Addrs, + Username: c.Username, + Password: c.Password, + SelectDB: c.DB, + TLSConfig: c.TLS, + Dialer: net.Dialer{Timeout: c.DialTimeout}, + ClientName: "wavehouse", + DisableCache: true, // no client-side caching until the near-cache (E5) + ForceSingleClient: c.Mode == RedisStandalone, + } + if c.Mode == RedisSentinel { + opt.Sentinel = rueidis.SentinelOption{MasterSet: c.SentinelMaster, TLSConfig: c.TLS, Dialer: opt.Dialer} + } + return opt +} + +// RedisCache is a Cache shared by every process pointed at one Redis — +// or Valkey, Dragonfly, ElastiCache, MemoryDB: it uses only GET, SET and +// MGET, no scripts and no client tracking. +// +// Versions are random tokens, one per tenant, per table and per scope, +// under the tenant's hash tag; a bump sets a fresh one. A value carries the +// tokens it was computed under and is a hit only while they are all still +// current, so a lost token (eviction, expiry, a restart) can only cause +// misses. A lookup is one round trip. The server failing or timing out is +// a miss, a skipped fill and a deferred invalidation — never a failed query. +type RedisCache struct { + cfg RedisConfig + opt rueidis.ClientOption + maxDecoded int + + client atomic.Pointer[rueidis.Client] // nil until the first connection + codec *codec + breaker *breaker + pending *pendingBumps + metrics *metrics + + ctx context.Context // cancelled by Close + cancel context.CancelFunc + wg sync.WaitGroup + wake chan struct{} + closeOnce sync.Once +} + +var _ Cache = (*RedisCache)(nil) + +// NewRedis builds a RedisCache. A malformed cfg is an error; a server that +// cannot be reached within the dial timeout is not — the cache starts +// bypassed and keeps dialing in the background. +func NewRedis(cfg RedisConfig) (*RedisCache, error) { + cfg, err := cfg.withDefaults() + if err != nil { + return nil, err + } + maxDecoded := cfg.MaxValueBytes * decodedFactor + cd, err := newCodec(cfg.CompressMinBytes, maxDecoded) + if err != nil { + return nil, fmt.Errorf("cache: zstd: %w", err) + } + ctx, cancel := context.WithCancel(context.Background()) + r := &RedisCache{ + cfg: cfg, + opt: cfg.clientOption(), + maxDecoded: maxDecoded, + codec: cd, + breaker: newBreaker(cfg.BreakerThreshold, cfg.BreakerOpenFor, time.Now), + pending: newPendingBumps(cfg.KeyPrefix, cfg.PendingMax), + ctx: ctx, + cancel: cancel, + wake: make(chan struct{}, 1), + } + if r.metrics, err = newMetrics("redis", r.bypassed, r.pending.len); err != nil { + cancel() + cd.close() + return nil, fmt.Errorf("cache: metrics: %w", err) + } + r.wg.Add(1) + go r.drainLoop() + if err := r.dial(); err != nil { + r.wg.Add(1) + go r.dialLoop() + } + return r, nil +} + +func (r *RedisCache) dial() error { + c, err := rueidis.NewClient(r.opt) + if err != nil { + level := slog.LevelWarn + if isAuthError(err) { + level = slog.LevelError + } + slog.Log(r.ctx, level, "cache: redis unreachable; bypassing the cache and retrying", + "addrs", r.cfg.Addrs, "error", err) + return err + } + if r.ctx.Err() != nil { + c.Close() + return r.ctx.Err() + } + r.client.Store(&c) + r.nudge() + return nil +} + +func (r *RedisCache) dialLoop() { + defer r.wg.Done() + backoff := time.Second + for { + select { + case <-r.ctx.Done(): + return + case <-time.After(backoff): + } + if r.dial() == nil { + slog.InfoContext(r.ctx, "cache: redis connected", "addrs", r.cfg.Addrs) + return + } + backoff = min(backoff*2, dialMaxBackoff) + } +} + +func isAuthError(err error) bool { + msg := err.Error() + return strings.Contains(msg, "WRONGPASS") || strings.Contains(msg, "NOAUTH") || strings.Contains(msg, "NOPERM") +} + +// conn returns the client to send to, or nil when the server is to be +// skipped: not connected yet, or the breaker open. The caller that finds an +// open breaker due for a probe starts it. +func (r *RedisCache) conn() rueidis.Client { + cp := r.client.Load() + if cp == nil { + return nil + } + ok, probe := r.breaker.allow() + if probe { + go r.probe(*cp) + } + if !ok { + return nil + } + return *cp +} + +func (r *RedisCache) probe(c rueidis.Client) { + ctx, cancel := context.WithTimeout(r.ctx, r.cfg.Timeout) + defer cancel() + if err := c.Do(ctx, c.B().Ping().Build()).Error(); err != nil { + r.breaker.failure() + return + } + r.breaker.success() + slog.InfoContext(ctx, "cache: redis reachable again; cache back in use") + r.nudge() +} + +// bypassed reports whether operations are skipping the server. +func (r *RedisCache) bypassed() bool { + return r.client.Load() == nil || r.breaker.isOpen() +} + +// record feeds an operation's outcome to the breaker. A reply from the +// server, even an error reply, shows it is up; a caller that gave up first +// shows nothing about it. +func (r *RedisCache) record(parent context.Context, err error) { + if err == nil || rueidis.IsRedisNil(err) { + r.breaker.success() + return + } + if _, ok := rueidis.IsRedisErr(err); ok { + r.breaker.success() + return + } + if parent.Err() != nil { + return + } + r.breaker.failure() +} + +// Lookup reads the tokens deps fold and the entry for sha in one pipelined +// round trip. A token that does not exist yet is created, never read as a +// value, so its first use is a miss. +func (r *RedisCache) Lookup(ctx context.Context, id tenant.ID, sha string, deps []Namespace) (Entry, Snapshot, error) { + for _, d := range deps { + if d.Tenant != id { + return Entry{}, Snapshot{}, fmt.Errorf("%w: %q under %q", ErrForeignDependency, d.Tenant, id) + } + } + keys := tokenKeys(r.cfg.KeyPrefix, id, deps) + if len(keys) > maxTokenKeys { + return Entry{}, Snapshot{}, fmt.Errorf("cache: %d dependencies is more than a value can record", len(deps)) + } + c := r.conn() + if c == nil { + r.metrics.lookup(resultBypass) + return Entry{}, Snapshot{}, nil + } + defer r.metrics.op("lookup", time.Now()) + opCtx, cancel := context.WithTimeout(ctx, r.cfg.Timeout) + defer cancel() + + vkey := valueKey(r.cfg.KeyPrefix, id, sha, deps) + res := c.DoMulti(opCtx, c.B().Mget().Key(keys...).Build(), c.B().Get().Key(vkey).Build()) + tokens, missing, err := readTokens(res[0]) + if err != nil { + return r.lookupFailed(ctx, err) + } + val, err := res[1].AsBytes() + if err != nil && !rueidis.IsRedisNil(err) { + return r.lookupFailed(ctx, err) + } + r.record(ctx, nil) + + if len(missing) > 0 { + if tokens, err = r.createTokens(opCtx, c, keys, missing); err != nil { + return r.lookupFailed(ctx, err) + } + r.metrics.lookup(resultMiss) + return Entry{}, Snapshot{key: vkey, tokens: tokens}, nil + } + snap := Snapshot{key: vkey, tokens: tokens} + if val == nil { + r.metrics.lookup(resultMiss) + return Entry{}, snap, nil + } + stored, expiresAt, payload, err := r.codec.decode(val) + if err != nil { + slog.DebugContext(ctx, "cache: unreadable value; treating as a miss", "key", vkey, "error", err) + r.metrics.lookup(resultMiss) + return Entry{}, snap, nil + } + remaining := time.Until(expiresAt) + if !bytes.Equal(stored, tokens) || remaining <= 0 { + r.metrics.lookup(resultStale) + return Entry{}, snap, nil + } + r.metrics.lookup(resultHit) + return Entry{Value: payload, TTL: remaining}, snap, nil +} + +func (r *RedisCache) lookupFailed(ctx context.Context, err error) (Entry, Snapshot, error) { + r.record(ctx, err) + r.metrics.lookup(resultError) + return Entry{}, Snapshot{}, fmt.Errorf("cache: redis lookup: %w", err) +} + +// readTokens concatenates an MGET reply's tokens, listing the indexes of the +// keys that do not exist. +func readTokens(res rueidis.RedisResult) (tokens []byte, missing []int, err error) { + msgs, err := res.ToArray() + if err != nil { + return nil, nil, err + } + tokens = make([]byte, 0, len(msgs)*tokenLen) + for i := range msgs { + if msgs[i].IsNil() { + missing = append(missing, i) + tokens = append(tokens, make([]byte, tokenLen)...) + continue + } + s, err := msgs[i].ToString() + if err != nil { + return nil, nil, err + } + if len(s) != tokenLen { + return nil, nil, fmt.Errorf("version token is %d bytes, not %d: is another program writing under this key prefix?", len(s), tokenLen) + } + tokens = append(tokens, s...) + } + return tokens, missing, nil +} + +// createTokens sets each missing token — only if still missing, as another +// process may create it first — and reads them all back, in one round trip: +// the tokens share a slot, so the pipeline runs in order on one node. +func (r *RedisCache) createTokens(ctx context.Context, c rueidis.Client, keys []string, missing []int) ([]byte, error) { + cmds := make(rueidis.Commands, 0, len(missing)+1) + for _, i := range missing { + tok := newToken() + cmds = append(cmds, c.B().Set().Key(keys[i]).Value(rueidis.BinaryString(tok)).Nx().Ex(jitter(r.cfg.VersionTTL, tok)).Build()) + } + cmds = append(cmds, c.B().Mget().Key(keys...).Build()) + res := c.DoMulti(ctx, cmds...) + for _, rr := range res[:len(missing)] { + if err := rr.Error(); err != nil && !rueidis.IsRedisNil(err) { + return nil, err + } + } + tokens, still, err := readTokens(res[len(missing)]) + if err != nil { + return nil, err + } + if len(still) > 0 { + return nil, errors.New("version token vanished as it was created: is the server evicting everything?") + } + return tokens, nil +} + +// Set stores value with the tokens snap read, for ttl. A value over the size +// limit is not stored; neither is anything while the server is bypassed. +func (r *RedisCache) Set(ctx context.Context, snap Snapshot, value []byte, ttl time.Duration) error { + if snap.key == "" || ttl <= 0 { + return nil + } + if len(value) > r.maxDecoded { + r.metrics.tooLarge() + return nil + } + b := r.codec.encode(snap.tokens, time.Now().Add(ttl), value) + if len(b) > r.cfg.MaxValueBytes { + r.metrics.tooLarge() + return nil + } + c := r.conn() + if c == nil { + return nil + } + defer r.metrics.op("set", time.Now()) + opCtx, cancel := context.WithTimeout(ctx, r.cfg.Timeout) + defer cancel() + err := c.Do(opCtx, c.B().Set().Key(snap.key).Value(rueidis.BinaryString(b)).Px(max(ttl, time.Millisecond)).Build()).Error() + r.record(ctx, err) + if err != nil { + r.metrics.setFailed(setFailureReason(err)) + return fmt.Errorf("cache: redis set: %w", err) + } + r.metrics.stored(len(b)) + return nil +} + +func setFailureReason(err error) string { + if re, ok := rueidis.IsRedisErr(err); ok && strings.HasPrefix(re.Error(), "OOM") { + return "oom" + } + if errors.Is(err, context.DeadlineExceeded) { + return "timeout" + } + return "other" +} + +// Invalidate sets a fresh token for every token the namespaces' writes +// reach, in one pipelined round trip. Bumps the server does not take are +// kept and retried until it does, and reported as an error meanwhile. +func (r *RedisCache) Invalidate(ctx context.Context, namespaces []Namespace) (uint64, error) { + owner := map[string]tenant.ID{} + for _, ns := range namespaces { + for _, k := range bumpKeys(r.cfg.KeyPrefix, ns) { + owner[k] = ns.Tenant + } + } + return uint64(len(namespaces)), r.bump(ctx, owner) +} + +// InvalidateTenant sets a fresh tenant token, orphaning every entry of id. +func (r *RedisCache) InvalidateTenant(ctx context.Context, id tenant.ID) error { + return r.bump(ctx, map[string]tenant.ID{tenantTokenKey(r.cfg.KeyPrefix, id): id}) +} + +func (r *RedisCache) bump(ctx context.Context, owner map[string]tenant.ID) error { + if len(owner) == 0 { + return nil + } + c := r.conn() + if c == nil { + r.deferBumps(owner) + return fmt.Errorf("%w: %d invalidations deferred", errBypassed, len(owner)) + } + defer r.metrics.op("invalidate", time.Now()) + opCtx, cancel := context.WithTimeout(ctx, r.cfg.Timeout) + defer cancel() + keys := make([]string, 0, len(owner)) + cmds := make(rueidis.Commands, 0, len(owner)) + for k := range owner { + keys = append(keys, k) + cmds = append(cmds, r.bumpCmd(c, k)) + } + failed := map[string]tenant.ID{} + var firstErr error + for i, rr := range c.DoMulti(opCtx, cmds...) { + if err := rr.Error(); err != nil { + failed[keys[i]] = owner[keys[i]] + firstErr = cmpOr(firstErr, err) + } + } + r.record(ctx, firstErr) + r.metrics.invalidated("ok", len(owner)-len(failed)) + if len(failed) > 0 { + r.deferBumps(failed) + return fmt.Errorf("cache: redis invalidate (%d deferred): %w", len(failed), firstErr) + } + return nil +} + +func (r *RedisCache) bumpCmd(c rueidis.Client, key string) rueidis.Completed { + tok := newToken() + return c.B().Set().Key(key).Value(rueidis.BinaryString(tok)).Ex(jitter(r.cfg.VersionTTL, tok)).Build() +} + +func (r *RedisCache) deferBumps(owner map[string]tenant.ID) { + for k, id := range owner { + r.pending.add(id, k) + } + r.metrics.invalidated("deferred", len(owner)) +} + +// nudge wakes the drain loop now, rather than at its next tick. +func (r *RedisCache) nudge() { + select { + case r.wake <- struct{}{}: + default: + } +} + +func (r *RedisCache) drainLoop() { + defer r.wg.Done() + backoff := drainMinBackoff + timer := time.NewTimer(drainIdle) + defer timer.Stop() + for { + select { + case <-r.ctx.Done(): + return + case <-r.wake: + backoff = drainMinBackoff + case <-timer.C: + } + wait := drainIdle + if r.breaker.isOpen() { + r.conn() // starts the probe when due, so an idle process recovers too + } + if r.pending.len() > 0 { + if r.drain(r.ctx) { + backoff = drainMinBackoff + } else { + wait, backoff = backoff, min(backoff*2, drainMaxBackoff) + } + } + timer.Reset(wait) + } +} + +// drain delivers the pending bumps, reporting whether none remain. +func (r *RedisCache) drain(ctx context.Context) bool { + owed := r.pending.snapshot() + keys := make([]string, 0, len(owed)) + for k := range owed { + keys = append(keys, k) + } + for len(keys) > 0 { + batch := keys[:min(drainBatch, len(keys))] + keys = keys[len(batch):] + c := r.conn() + if c == nil { + return false + } + cmds := make(rueidis.Commands, 0, len(batch)) + for _, k := range batch { + cmds = append(cmds, r.bumpCmd(c, k)) + } + opCtx, cancel := context.WithTimeout(ctx, r.cfg.Timeout) + landed := map[string]uint64{} + var firstErr error + for i, rr := range c.DoMulti(opCtx, cmds...) { + if err := rr.Error(); err != nil { + firstErr = cmpOr(firstErr, err) + continue + } + landed[batch[i]] = owed[batch[i]] + } + cancel() + r.record(ctx, firstErr) + r.pending.done(landed) + r.metrics.invalidated("ok", len(landed)) + if firstErr != nil { + return false + } + } + return r.pending.len() == 0 +} + +// Close stops the background loops, makes one last attempt at the pending +// bumps, and closes the connection. Bumps still undelivered are lost: the +// entries they would orphan are served until their TTL. +func (r *RedisCache) Close() error { + r.closeOnce.Do(func() { + r.cancel() + r.wg.Wait() + if r.pending.len() > 0 { + ctx, cancel := context.WithTimeout(context.Background(), closeDrainBudget) + r.drain(ctx) + cancel() + if n := r.pending.len(); n > 0 { + slog.Warn("cache: closing with undelivered invalidations; entries they orphan stay cached until their TTL", "pending", n) + } + } + if cp := r.client.Load(); cp != nil { + (*cp).Close() + } + r.codec.close() + r.metrics.close() + }) + return nil +} diff --git a/internal/cache/redis_codec.go b/internal/cache/redis_codec.go new file mode 100644 index 000000000..cf6f4b5ec --- /dev/null +++ b/internal/cache/redis_codec.go @@ -0,0 +1,194 @@ +package cache + +import ( + "cmp" + "crypto/rand" + "crypto/sha256" + "encoding/binary" + "encoding/hex" + "errors" + "fmt" + "slices" + "time" + + "github.com/klauspost/compress/zstd" + + "github.com/Wave-RF/WaveHouse/internal/tenant" +) + +// tokenLen is the size of a version token: random, so a token key that is +// lost (evicted, expired, flushed) and recreated can never match a value +// stored under its predecessor, as a counter restarting at 0 would. +const tokenLen = 8 + +// decodedFactor bounds a value's decompressed size at this multiple of the +// stored-size limit, refusing a zip bomb planted in a shared server. +const decodedFactor = 8 + +// Value layout: format, flags, expires-at (unix ms), token count, tokens, +// payload. Big-endian. +const ( + valueFormat = 1 + flagZstd = 1 << 0 + headerLen = 1 + 1 + 8 + 2 + maxTokenKeys = 1<<16 - 1 +) + +var errCorruptValue = errors.New("cache: corrupt value") + +func newToken() []byte { + b := make([]byte, tokenLen) + _, _ = rand.Read(b) // never fails (crypto/rand, Go ≥ 1.24) + return b +} + +// tenantTokenKey is the key of tenant id's token. Every token key carries +// the tenant as a hash tag, so all of a tenant's tokens share one cluster +// slot and a lookup reads them with one MGET. +func tenantTokenKey(prefix string, id tenant.ID) string { + return prefix + ":{" + string(id) + "}:T" +} + +func tableTokenKey(prefix string, id tenant.ID, table string) string { + return prefix + ":{" + string(id) + "}:B:" + table +} + +func scopeTokenKey(prefix string, id tenant.ID, table, scope string) string { + return prefix + ":{" + string(id) + "}:S:" + table + ":" + scope +} + +// sortedDeps returns deps in canonical order without duplicates. +func sortedDeps(deps []Namespace) []Namespace { + out := slices.Clone(deps) + slices.SortFunc(out, func(a, b Namespace) int { + return cmp.Or(cmp.Compare(a.Table, b.Table), cmp.Compare(a.Scope, b.Scope)) + }) + return slices.Compact(out) +} + +// tokenKeys lists the tokens a result for deps of tenant id is filed under, +// in canonical order: the tenant's, then each dep's table and scope tokens. +// A dep with scope s folds B:table (bumped by a whole-table write) and +// S:table:s (bumped by a write to s, and — for s == "" — by any scoped write +// to the table), the same lattice LocalCache's version index encodes. +func tokenKeys(prefix string, id tenant.ID, deps []Namespace) []string { + keys := []string{tenantTokenKey(prefix, id)} + for _, d := range sortedDeps(deps) { + keys = append(keys, tableTokenKey(prefix, id, d.Table), scopeTokenKey(prefix, id, d.Table, d.Scope)) + } + slices.Sort(keys[1:]) + return append(keys[:1], slices.Compact(keys[1:])...) +} + +// bumpKeys lists the tokens an invalidation of ns replaces: a whole-table +// write the table's, a scoped write its scope's and the whole-table view's. +func bumpKeys(prefix string, ns Namespace) []string { + if ns.Scope == "" { + return []string{tableTokenKey(prefix, ns.Tenant, ns.Table)} + } + return []string{scopeTokenKey(prefix, ns.Tenant, ns.Table, ns.Scope), scopeTokenKey(prefix, ns.Tenant, ns.Table, "")} +} + +// valueKey names the entry for sha over deps. It carries no versions, so a +// refill overwrites in place, and no hash tag, so one tenant's values spread +// across a cluster's shards. +func valueKey(prefix string, id tenant.ID, sha string, deps []Namespace) string { + h := sha256.New() + h.Write([]byte(sha)) + h.Write([]byte{0}) + for _, d := range sortedDeps(deps) { + h.Write([]byte(d.Table)) + h.Write([]byte{0}) + h.Write([]byte(d.Scope)) + h.Write([]byte{0}) + } + return prefix + ":q:" + string(id) + ":" + hex.EncodeToString(h.Sum(nil)) +} + +// codec compresses and frames values. Its zstd encoder and decoder are safe +// for concurrent EncodeAll/DecodeAll. +type codec struct { + enc *zstd.Encoder + dec *zstd.Decoder + compressMin int + maxDecoded int +} + +func newCodec(compressMin, maxDecoded int) (*codec, error) { + enc, err := zstd.NewWriter(nil, zstd.WithEncoderLevel(zstd.SpeedFastest)) + if err != nil { + return nil, err + } + dec, err := zstd.NewReader(nil, zstd.WithDecoderMaxMemory(uint64(maxDecoded)), zstd.WithDecoderConcurrency(0)) //nolint:gosec // maxDecoded is a positive config-derived int + if err != nil { + return nil, err + } + return &codec{enc: enc, dec: dec, compressMin: compressMin, maxDecoded: maxDecoded}, nil +} + +func (c *codec) close() { + _ = c.enc.Close() + c.dec.Close() +} + +// encode frames payload with the tokens it was computed under, compressing +// it when that is enabled, the payload is large enough, and it helps. +func (c *codec) encode(tokens []byte, expiresAt time.Time, payload []byte) []byte { + var flags byte + body := payload + if c.compressMin > 0 && len(payload) >= c.compressMin { + if z := c.enc.EncodeAll(payload, nil); len(z) < len(payload) { + body, flags = z, flagZstd + } + } + out := make([]byte, headerLen, headerLen+len(tokens)+len(body)) + out[0] = valueFormat + out[1] = flags + binary.BigEndian.PutUint64(out[2:10], uint64(expiresAt.UnixMilli())) + binary.BigEndian.PutUint16(out[10:12], uint16(len(tokens)/tokenLen)) //nolint:gosec // bounded by maxTokenKeys at Lookup + out = append(out, tokens...) + return append(out, body...) +} + +// decode is encode's inverse. An unknown format — a newer process's value +// during a rolling upgrade — is an error, which the caller reads as a miss. +func (c *codec) decode(b []byte) (tokens []byte, expiresAt time.Time, payload []byte, err error) { + if len(b) < headerLen { + return nil, time.Time{}, nil, errCorruptValue + } + if b[0] != valueFormat { + return nil, time.Time{}, nil, fmt.Errorf("%w: format %d", errCorruptValue, b[0]) + } + flags := b[1] + expiresAt = time.UnixMilli(int64(binary.BigEndian.Uint64(b[2:10]))) //nolint:gosec // written by encode + n := int(binary.BigEndian.Uint16(b[10:12])) * tokenLen + if len(b) < headerLen+n { + return nil, time.Time{}, nil, errCorruptValue + } + tokens = b[headerLen : headerLen+n] + payload = b[headerLen+n:] + switch flags { + case 0: + case flagZstd: + if payload, err = c.dec.DecodeAll(payload, nil); err != nil { + return nil, time.Time{}, nil, fmt.Errorf("%w: %w", errCorruptValue, err) + } + if len(payload) > c.maxDecoded { + return nil, time.Time{}, nil, fmt.Errorf("%w: decodes to %d bytes", errCorruptValue, len(payload)) + } + default: + return nil, time.Time{}, nil, fmt.Errorf("%w: flags %#x", errCorruptValue, flags) + } + return tokens, expiresAt, payload, nil +} + +// jitter spreads d by ±10%, drawing on the token being written so a batch of +// bumps doesn't expire together. +func jitter(d time.Duration, token []byte) time.Duration { + span := int64(d) / 5 + if span <= 0 { + return d + } + r := int64(binary.BigEndian.Uint64(token) % uint64(span)) //nolint:gosec // r < span, an int64 + return d - time.Duration(span/2) + time.Duration(r) +} diff --git a/internal/cache/redis_codec_test.go b/internal/cache/redis_codec_test.go new file mode 100644 index 000000000..68c423d0f --- /dev/null +++ b/internal/cache/redis_codec_test.go @@ -0,0 +1,198 @@ +package cache + +import ( + "bytes" + "strings" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestTokenKeys(t *testing.T) { + t.Parallel() + tests := []struct { + name string + deps []Namespace + want []string + }{ + {"no deps is the tenant token alone", nil, []string{"wh:{acme}:T"}}, + { + "a dep folds its table and scope tokens", + []Namespace{{Tenant: "acme", Table: "events", Scope: "org_1"}}, + []string{"wh:{acme}:T", "wh:{acme}:B:events", "wh:{acme}:S:events:org_1"}, + }, + { + "a scopeless dep reads the whole-table view", + []Namespace{{Tenant: "acme", Table: "events"}}, + []string{"wh:{acme}:T", "wh:{acme}:B:events", "wh:{acme}:S:events:"}, + }, + { + "shared table tokens and duplicate deps appear once, sorted", + []Namespace{ + {Tenant: "acme", Table: "orders"}, + {Tenant: "acme", Table: "events", Scope: "b"}, + {Tenant: "acme", Table: "events", Scope: "a"}, + {Tenant: "acme", Table: "orders"}, + }, + []string{ + "wh:{acme}:T", "wh:{acme}:B:events", "wh:{acme}:B:orders", + "wh:{acme}:S:events:a", "wh:{acme}:S:events:b", "wh:{acme}:S:orders:", + }, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + assert.Equal(t, tt.want, tokenKeys("wh", "acme", tt.deps)) + }) + } +} + +func TestBumpKeys(t *testing.T) { + t.Parallel() + assert.Equal(t, []string{"p:{acme}:B:events"}, bumpKeys("p", Namespace{Tenant: "acme", Table: "events"})) + assert.Equal(t, []string{"p:{acme}:S:events:org_1", "p:{acme}:S:events:"}, + bumpKeys("p", Namespace{Tenant: "acme", Table: "events", Scope: "org_1"})) +} + +// Every key a bump writes is one some lookup reads: otherwise the bump +// orphans nothing. +func TestBumpKeysAreReadByLookups(t *testing.T) { + t.Parallel() + for _, ns := range []Namespace{{Tenant: "acme", Table: "events"}, {Tenant: "acme", Table: "events", Scope: "org_1"}} { + read := map[string]bool{} + for _, scope := range []string{"", ns.Scope} { + for _, k := range tokenKeys("wh", "acme", []Namespace{{Tenant: "acme", Table: ns.Table, Scope: scope}}) { + read[k] = true + } + } + for _, k := range bumpKeys("wh", ns) { + assert.True(t, read[k], k) + } + } +} + +func TestValueKey(t *testing.T) { + t.Parallel() + a, b := Namespace{Tenant: "acme", Table: "events"}, Namespace{Tenant: "acme", Table: "orders", Scope: "x"} + k := valueKey("wh", "acme", "acme:query:abc", []Namespace{a, b}) + assert.True(t, strings.HasPrefix(k, "wh:q:acme:"), k) + assert.NotContains(t, k, "{", "values carry no hash tag, so they spread across shards") + assert.Equal(t, k, valueKey("wh", "acme", "acme:query:abc", []Namespace{b, a, b}), "order and duplicates do not matter") + assert.NotEqual(t, k, valueKey("wh", "acme", "acme:query:abc", []Namespace{a})) + assert.NotEqual(t, k, valueKey("wh", "acme", "acme:query:abd", []Namespace{a, b})) + assert.NotEqual(t, + valueKey("wh", "acme", "q", []Namespace{{Tenant: "acme", Table: "ab", Scope: "c"}}), + valueKey("wh", "acme", "q", []Namespace{{Tenant: "acme", Table: "a", Scope: "bc"}})) +} + +func newTestCodec(t *testing.T, compressMin, maxDecoded int) *codec { + t.Helper() + c, err := newCodec(compressMin, maxDecoded) + require.NoError(t, err) + t.Cleanup(c.close) + return c +} + +func TestCodec_RoundTrip(t *testing.T) { + t.Parallel() + c := newTestCodec(t, 64, 1<<20) + tokens := append(newToken(), newToken()...) + exp := time.UnixMilli(time.Now().Add(time.Minute).UnixMilli()) + tests := []struct { + name string + payload []byte + compressed bool + }{ + {"empty", []byte{}, false}, + {"below the threshold", []byte(`[{"a":1}]`), false}, + {"compressible", bytes.Repeat([]byte(`{"user":"u1","n":42},`), 200), true}, + {"incompressible stays raw", randomBytes(4096), false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + b := c.encode(tokens, exp, tt.payload) + assert.Equal(t, tt.compressed, b[1]&flagZstd != 0) + if tt.compressed { + assert.Less(t, len(b), len(tt.payload)) + } + gotTokens, gotExp, gotPayload, err := c.decode(b) + require.NoError(t, err) + assert.Equal(t, tokens, gotTokens) + assert.True(t, exp.Equal(gotExp)) + assert.Equal(t, tt.payload, gotPayload) + }) + } +} + +func TestCodec_CompressionOff(t *testing.T) { + t.Parallel() + c := newTestCodec(t, 0, 1<<20) + b := c.encode(newToken(), time.Now(), bytes.Repeat([]byte("a"), 4096)) + assert.Zero(t, b[1]) +} + +func TestCodec_RefusesBadValues(t *testing.T) { + t.Parallel() + c := newTestCodec(t, 1, 1<<10) + good := c.encode(newToken(), time.Now(), []byte("rows")) + bomb := newTestCodec(t, 1, 1<<30).encode(newToken(), time.Now(), make([]byte, 1<<20)) + require.Less(t, len(bomb), 1<<10, "the bomb is small when stored") + + withByte := func(i int, v byte) []byte { + b := bytes.Clone(good) + b[i] = v + return b + } + tests := []struct { + name string + b []byte + }{ + {"shorter than the header", good[:headerLen-1]}, + {"unknown format", withByte(0, 2)}, + {"unknown flags", withByte(1, 0x80)}, + {"fewer tokens than it declares", withByte(11, 200)}, + {"not zstd", append(withByte(1, flagZstd)[:headerLen+tokenLen], "not zstd"...)}, + {"decodes past the limit", bomb}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + _, _, _, err := c.decode(tt.b) + require.ErrorIs(t, err, errCorruptValue) + }) + } +} + +func TestJitter(t *testing.T) { + t.Parallel() + d := 100 * time.Second + for range 1000 { + j := jitter(d, newToken()) + assert.GreaterOrEqual(t, j, 90*time.Second) + assert.Less(t, j, 110*time.Second) + } + assert.Equal(t, time.Nanosecond, jitter(time.Nanosecond, newToken())) +} + +func TestNewToken(t *testing.T) { + t.Parallel() + seen := map[string]bool{} + for range 1000 { + tok := newToken() + require.Len(t, tok, tokenLen) + require.False(t, seen[string(tok)]) + seen[string(tok)] = true + } +} + +func randomBytes(n int) []byte { + b := make([]byte, 0, n) + for len(b) < n { + b = append(b, newToken()...) + } + return b[:n] +} diff --git a/internal/cache/redis_integration_test.go b/internal/cache/redis_integration_test.go new file mode 100644 index 000000000..fb902e735 --- /dev/null +++ b/internal/cache/redis_integration_test.go @@ -0,0 +1,418 @@ +//go:build integration + +package cache_test + +import ( + "bytes" + "context" + "fmt" + "net" + "strconv" + "strings" + "sync/atomic" + "testing" + "time" + + "github.com/moby/moby/api/types/container" + "github.com/moby/moby/api/types/network" + "github.com/moby/moby/client" + "github.com/redis/rueidis" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "github.com/testcontainers/testcontainers-go" + "github.com/testcontainers/testcontainers-go/wait" + + "github.com/Wave-RF/WaveHouse/internal/cache" + "github.com/Wave-RF/WaveHouse/internal/tenant" + "github.com/Wave-RF/WaveHouse/internal/testutil/cachetest" +) + +// Pinned: the servers the shared cache is documented to run on. +const ( + redisImage = "redis:8.10.2-alpine" + valkeyImage = "valkey/valkey:8.1.10-alpine" + dragonflyImage = "docker.dragonflydb.io/dragonflydb/dragonfly:v2.0.0" +) + +// maxValue is the stored-size limit the tests run with; small, so the +// oversize case stays cheap. +const maxValue = 64 << 10 + +type server struct { + ctr testcontainers.Container + addr string + mode string +} + +// noPersistence keeps the image's VOLUME /data off an anonymous volume. +func noPersistence(hc *container.HostConfig) { + hc.Tmpfs = map[string]string{"/data": ""} +} + +func startContainer(t *testing.T, req testcontainers.ContainerRequest, port string) (testcontainers.Container, string) { + t.Helper() + ctx := context.Background() + if req.HostConfigModifier == nil { + req.HostConfigModifier = noPersistence + } + req.WaitingFor = wait.ForListeningPort(port).WithStartupTimeout(90 * time.Second) + ctr, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{ContainerRequest: req, Started: true}) + testcontainers.CleanupContainer(t, ctr) + require.NoError(t, err) + host, err := ctr.Host(ctx) + require.NoError(t, err) + mapped, err := ctr.MappedPort(ctx, port) + require.NoError(t, err) + return ctr, net.JoinHostPort(host, mapped.Port()) +} + +func startStandalone(t *testing.T, image string, cmd ...string) *server { + t.Helper() + ctr, addr := startContainer(t, testcontainers.ContainerRequest{ + Image: image, Cmd: cmd, ExposedPorts: []string{"6379/tcp"}, + }, "6379/tcp") + s := &server{ctr: ctr, addr: addr, mode: cache.RedisStandalone} + waitReady(t, s, nil) + return s +} + +func startRedis(t *testing.T) *server { + return startStandalone(t, redisImage, "redis-server", "--save", "", "--appendonly", "no") +} + +func startValkey(t *testing.T) *server { + return startStandalone(t, valkeyImage, "valkey-server", "--save", "", "--appendonly", "no") +} + +func startDragonfly(t *testing.T) *server { + return startStandalone(t, dragonflyImage, "--proactor_threads=2", "--maxmemory=512mb") +} + +// startCluster runs a one-node Redis Cluster owning every slot: enough for +// the server to enforce cluster semantics — CROSSSLOT on a multi-key +// command, MOVED routing through the client — which is what the key schema +// must survive. The node announces 127.0.0.1 on a host port bound to the +// same number, so the address the client learns from CLUSTER SLOTS is +// dialable from the test. +func startCluster(t *testing.T) *server { + t.Helper() + port := freePort(t) + p := network.MustParsePort(port + "/tcp") + ctr, _ := startContainer(t, testcontainers.ContainerRequest{ + Image: redisImage, + Cmd: []string{ + "redis-server", "--port", port, "--cluster-enabled", "yes", "--cluster-port", "16379", + "--cluster-announce-ip", "127.0.0.1", "--save", "", "--appendonly", "no", + }, + ExposedPorts: []string{port + "/tcp"}, + HostConfigModifier: func(hc *container.HostConfig) { + noPersistence(hc) + hc.PortBindings = network.PortMap{p: {{HostPort: port}}} + }, + }, port+"/tcp") + code, out, err := ctr.Exec(context.Background(), []string{"redis-cli", "-p", port, "cluster", "addslotsrange", "0", "16383"}) + require.NoError(t, err) + require.Zero(t, code, "%v", out) + s := &server{ctr: ctr, addr: net.JoinHostPort("127.0.0.1", port), mode: cache.RedisCluster} + waitReady(t, s, func(c rueidis.Client) error { + info, err := c.Do(context.Background(), c.B().ClusterInfo().Build()).ToString() + if err == nil && !strings.Contains(info, "cluster_state:ok") { + err = fmt.Errorf("cluster not ready: %q", info) + } + return err + }) + return s +} + +func freePort(t *testing.T) string { + t.Helper() + var lc net.ListenConfig + for { + ln, err := lc.Listen(context.Background(), "tcp", "127.0.0.1:0") + require.NoError(t, err) + port := strconv.Itoa(ln.Addr().(*net.TCPAddr).Port) + require.NoError(t, ln.Close()) + if port != "16379" { + return port + } + } +} + +// raw opens a plain client on s, for the test to reach under the cache. +func raw(t *testing.T, s *server) rueidis.Client { + t.Helper() + c, err := rueidis.NewClient(rueidis.ClientOption{ + InitAddress: []string{s.addr}, DisableCache: true, ForceSingleClient: s.mode == cache.RedisStandalone, + }) + require.NoError(t, err) + t.Cleanup(c.Close) + return c +} + +// waitReady blocks until s answers — a listening port is not yet a server +// that takes commands — and, given ready, until ready passes too. +func waitReady(t *testing.T, s *server, ready func(rueidis.Client) error) { + t.Helper() + var last error + require.Eventually(t, func() bool { + c, err := rueidis.NewClient(rueidis.ClientOption{ + InitAddress: []string{s.addr}, DisableCache: true, ForceSingleClient: true, + }) + if last = err; err != nil { + return false + } + defer c.Close() + if last = c.Do(context.Background(), c.B().Ping().Build()).Error(); last != nil { + return false + } + if ready != nil { + last = ready(c) + } + return last == nil + }, 60*time.Second, 100*time.Millisecond, "server %s not ready: %v", s.addr, last) +} + +var prefixes atomic.Uint64 + +func uniquePrefix() string { return fmt.Sprintf("t%d", prefixes.Add(1)) } + +// open builds a RedisCache on s. Its timeout is generous: the suite runs +// in parallel under -race, and a timed-out lookup is a miss the conformance +// cases would read as a wrong answer. +func open(t *testing.T, s *server, prefix string, tune ...func(*cache.RedisConfig)) *cache.RedisCache { + t.Helper() + cfg := cache.RedisConfig{ + Addrs: []string{s.addr}, Mode: s.mode, KeyPrefix: prefix, + Timeout: 5 * time.Second, MaxValueBytes: maxValue, CompressMinBytes: cache.DefaultRedisCompressMinBytes, + } + for _, f := range tune { + f(&cfg) + } + c, err := cache.NewRedis(cfg) + require.NoError(t, err) + t.Cleanup(func() { _ = c.Close() }) + return c +} + +func TestRedis_Conformance(t *testing.T) { + t.Parallel() + servers := []struct { + name string + start func(*testing.T) *server + }{ + {"redis", startRedis}, + {"valkey", startValkey}, + {"dragonfly", startDragonfly}, + {"redis cluster", startCluster}, + } + for _, sv := range servers { + t.Run(sv.name, func(t *testing.T) { + t.Parallel() + s := sv.start(t) + cachetest.Run(t, + func(t *testing.T) cache.Cache { return open(t, s, uniquePrefix()) }, + cachetest.Options{ + // The raw-size bound: a value past it is refused before + // compression, and the suite's oversize value is zeros, + // which would compress under the stored-size one. + MaxValueBytes: maxValue * cache.DecodedFactor, + NewPair: func(t *testing.T) (cache.Cache, cache.Cache) { + p := uniquePrefix() + return open(t, s, p), open(t, s, p) + }, + }) + t.Run("cross-tenant invalidation spans slots", func(t *testing.T) { + t.Parallel() + testCrossTenantInvalidate(t, open(t, s, uniquePrefix())) + }) + t.Run("compressed values round-trip", func(t *testing.T) { + t.Parallel() + testCompression(t, s) + }) + }) + } +} + +// The ingest worker's shared-tables fan-out bumps one table under several +// tenants in one call: tokens in as many slots, one pipeline. +func testCrossTenantInvalidate(t *testing.T, c *cache.RedisCache) { + ctx := context.Background() + var deps [][]cache.Namespace + for i := range 20 { + id := tenantID(i) + d := []cache.Namespace{{Tenant: id, Table: "events"}} + deps = append(deps, d) + _, snap, err := c.Lookup(ctx, id, "q", d) + require.NoError(t, err) + require.NoError(t, c.Set(ctx, snap, []byte("rows"), time.Minute)) + } + var all []cache.Namespace + for _, d := range deps { + all = append(all, d...) + } + n, err := c.Invalidate(ctx, all) + require.NoError(t, err) + assert.Equal(t, uint64(len(all)), n) + for i, d := range deps { + e, _, err := c.Lookup(ctx, tenantID(i), "q", d) + require.NoError(t, err) + assert.Nil(t, e.Value, tenantID(i)) + } +} + +func tenantID(i int) tenant.ID { return tenant.ID(fmt.Sprintf("tenant-%d", i)) } + +func testCompression(t *testing.T, s *server) { + ctx := context.Background() + prefix := uniquePrefix() + c := open(t, s, prefix) + rows := bytes.Repeat([]byte(`{"user_id":"u-1","event":"click","value":42.5},`), 10_000) + require.Greater(t, len(rows), maxValue, "stored only because it compresses under the limit") + deps := []cache.Namespace{{Tenant: "acme", Table: "events"}} + _, snap, err := c.Lookup(ctx, "acme", "big", deps) + require.NoError(t, err) + require.NoError(t, c.Set(ctx, snap, rows, time.Minute)) + e, _, err := c.Lookup(ctx, "acme", "big", deps) + require.NoError(t, err) + assert.Equal(t, rows, e.Value) + + r := raw(t, s) + keys, err := r.Do(ctx, r.B().Keys().Pattern(prefix+":q:*").Build()).AsStrSlice() + require.NoError(t, err) + require.Len(t, keys, 1) + stored, err := r.Do(ctx, r.B().Strlen().Key(keys[0]).Build()).AsInt64() + require.NoError(t, err) + assert.Less(t, stored, int64(len(rows)/10)) +} + +// A token that is lost — evicted, expired, flushed, a restart without +// persistence — is recreated fresh, so a value stored under its predecessor +// can only miss. A counter recreated at its initial value would serve the +// value filed at that value again: this test fails for one. +func TestRedis_LostTokensAreMisses(t *testing.T) { + t.Parallel() + ctx := context.Background() + s := startRedis(t) + r := raw(t, s) + prefix := uniquePrefix() + c := open(t, s, prefix) + deps := []cache.Namespace{{Tenant: "acme", Table: "events", Scope: "org_1"}} + fill := func(t *testing.T, sha string, deps []cache.Namespace) { + t.Helper() + _, snap, err := c.Lookup(ctx, "acme", sha, deps) + require.NoError(t, err) + require.NoError(t, c.Set(ctx, snap, []byte("rows"), time.Minute)) + e, _, err := c.Lookup(ctx, "acme", sha, deps) + require.NoError(t, err) + require.Equal(t, "rows", string(e.Value)) + } + // Twice: the first lookup recreates the lost tokens, and it is the + // second, reading them back, that a recreated counter would fool. + requireMiss := func(t *testing.T, sha string, deps []cache.Namespace) { + t.Helper() + for range 2 { + e, _, err := c.Lookup(ctx, "acme", sha, deps) + require.NoError(t, err) + require.Nil(t, e.Value) + } + } + keys := func(t *testing.T, pattern string) []string { + t.Helper() + keys, err := r.Do(ctx, r.B().Keys().Pattern(pattern).Build()).AsStrSlice() + require.NoError(t, err) + return keys + } + + for _, lost := range []string{"T", "B:events", "S:events:org_1", "*"} { + fill(t, "q", deps) + fill(t, "pipe", nil) + tokens := keys(t, prefix+":{acme}:"+lost) + require.NotEmpty(t, tokens, lost) + require.NoError(t, r.Do(ctx, r.B().Del().Key(tokens...).Build()).Error()) + require.Len(t, keys(t, prefix+":q:*"), 2, "lost %s: the values survive, only their tokens are gone", lost) + requireMiss(t, "q", deps) + if lost == "T" || lost == "*" { + requireMiss(t, "pipe", nil) + } + } + + fill(t, "q", deps) + require.NoError(t, r.Do(ctx, r.B().Flushall().Build()).Error()) + requireMiss(t, "q", deps) + fill(t, "q", deps) +} + +func dockerClient(t *testing.T) *testcontainers.DockerClient { + t.Helper() + d, err := testcontainers.NewDockerClientWithOpts(context.Background()) + require.NoError(t, err) + t.Cleanup(func() { _ = d.Close() }) + return d +} + +// A server that stops answering costs a request at most about the op +// timeout, then nothing: the breaker opens and the cache is bypassed. +// Invalidations made meanwhile are kept and land once it answers again, and +// a process that boots while it is down starts bypassed and connects later. +func TestRedis_ServerStopsAnswering(t *testing.T) { + t.Parallel() + ctx := context.Background() + s := startRedis(t) + d := dockerClient(t) + prefix := uniquePrefix() + const timeout = 100 * time.Millisecond + a := open(t, s, prefix, func(c *cache.RedisConfig) { + c.Timeout, c.BreakerThreshold, c.BreakerOpenFor = timeout, 3, 300*time.Millisecond + }) + deps := []cache.Namespace{{Tenant: "acme", Table: "events"}} + _, snap, err := a.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + require.NoError(t, a.Set(ctx, snap, []byte("pre-write rows"), time.Minute)) + + _, err = d.ContainerPause(ctx, s.ctr.GetContainerID(), client.ContainerPauseOptions{}) + require.NoError(t, err) + paused := true + unpause := func() { + if paused { + paused = false + _, err := d.ContainerUnpause(ctx, s.ctr.GetContainerID(), client.ContainerUnpauseOptions{}) + require.NoError(t, err) + } + } + t.Cleanup(unpause) + + for i := range 3 { + start := time.Now() + e, snap, err := a.Lookup(ctx, "acme", "q", deps) + require.Error(t, err, "lookup %d", i) + assert.Nil(t, e.Value) + assert.Less(t, time.Since(start), 10*timeout, "a lookup costs at most about the timeout") + require.NoError(t, a.Set(ctx, snap, []byte("rows"), time.Minute), "the failed lookup's snapshot files nothing") + } + require.True(t, cache.Bypassed(a), "three timeouts open the breaker") + start := time.Now() + e, _, err := a.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err, "bypassed is a miss, not a failure") + assert.Nil(t, e.Value) + assert.Less(t, time.Since(start), timeout/2, "bypassed costs no round trip") + + _, err = a.Invalidate(ctx, deps) + require.Error(t, err) + assert.Equal(t, 1, cache.Pending(a)) + + bootStart := time.Now() + late := open(t, s, prefix, func(c *cache.RedisConfig) { c.DialTimeout = 200 * time.Millisecond }) + assert.Less(t, time.Since(bootStart), 5*time.Second, "an unanswering server does not hold boot") + assert.True(t, cache.Bypassed(late)) + + unpause() + require.Eventually(t, func() bool { return cache.Pending(a) == 0 && !cache.Bypassed(a) }, 15*time.Second, 50*time.Millisecond, + "the deferred bump lands once the server answers") + require.Eventually(t, func() bool { return !cache.Bypassed(late) }, 15*time.Second, 50*time.Millisecond, + "the late process connects") + + b := open(t, s, prefix) + e, _, err = b.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + assert.Nil(t, e.Value, "the fill from before the deferred bump is orphaned for every process") +} diff --git a/internal/cache/redis_test.go b/internal/cache/redis_test.go new file mode 100644 index 000000000..9feff6327 --- /dev/null +++ b/internal/cache/redis_test.go @@ -0,0 +1,231 @@ +package cache + +import ( + "context" + "errors" + "net" + "testing" + "time" + + "github.com/redis/rueidis" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/attribute" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" +) + +func TestRedisConfig_Validation(t *testing.T) { + t.Parallel() + ok := RedisConfig{Addrs: []string{"redis:6379"}} + tests := []struct { + name string + mutate func(c *RedisConfig) + wantErr string + }{ + {"no address", func(c *RedisConfig) { c.Addrs = nil }, "at least one address"}, + {"address without a port", func(c *RedisConfig) { c.Addrs = []string{"redis"} }, `address "redis"`}, + {"unknown mode", func(c *RedisConfig) { c.Mode = "ring" }, `mode "ring"`}, + {"cluster with a db", func(c *RedisConfig) { c.Mode, c.DB = RedisCluster, 1 }, "only database 0"}, + {"sentinel without a master", func(c *RedisConfig) { c.Mode = RedisSentinel }, "master set name"}, + {"negative db", func(c *RedisConfig) { c.DB = -1 }, "negative"}, + {"hash tag in the prefix", func(c *RedisConfig) { c.KeyPrefix = "{wh}" }, "brace"}, + {"negative timeout", func(c *RedisConfig) { c.Timeout = -time.Second }, "timeout is negative"}, + {"negative size", func(c *RedisConfig) { c.MaxValueBytes = -1 }, "max value bytes is negative"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + c := ok + tt.mutate(&c) + _, err := NewRedis(c) + require.ErrorContains(t, err, tt.wantErr) + }) + } +} + +func TestRedisConfig_Defaults(t *testing.T) { + t.Parallel() + c, err := RedisConfig{Addrs: []string{"redis:6379"}}.withDefaults() + require.NoError(t, err) + assert.Equal(t, RedisStandalone, c.Mode) + assert.Equal(t, DefaultRedisKeyPrefix, c.KeyPrefix) + assert.Equal(t, DefaultRedisTimeout, c.Timeout) + assert.Equal(t, DefaultRedisMaxValueBytes, c.MaxValueBytes) + assert.Equal(t, DefaultRedisVersionTTL, c.VersionTTL) + assert.Zero(t, c.CompressMinBytes, "0 means never compress, not the default") + assert.True(t, c.clientOption().ForceSingleClient) + + c.Mode, c.SentinelMaster = RedisSentinel, "mymaster" + assert.Equal(t, "mymaster", c.clientOption().Sentinel.MasterSet) + c.Mode = RedisCluster + assert.False(t, c.clientOption().ForceSingleClient) +} + +// closedAddr is an address nothing listens on: dials are refused at once. +func closedAddr(t *testing.T) string { + t.Helper() + var lc net.ListenConfig + ln, err := lc.Listen(context.Background(), "tcp", "127.0.0.1:0") + require.NoError(t, err) + addr := ln.Addr().String() + require.NoError(t, ln.Close()) + return addr +} + +// An unreachable server does not fail construction: the cache is bypassed +// — a miss that files nothing, a no-op fill, a deferred invalidation — and +// keeps dialing. +func TestRedis_UnreachableIsBypassed(t *testing.T) { + t.Parallel() + ctx := context.Background() + r, err := NewRedis(RedisConfig{Addrs: []string{closedAddr(t)}, DialTimeout: 100 * time.Millisecond, PendingMax: 2}) + require.NoError(t, err) + t.Cleanup(func() { _ = r.Close() }) + assert.True(t, r.bypassed()) + + deps := []Namespace{{Tenant: "acme", Table: "events"}} + e, snap, err := r.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + assert.Nil(t, e.Value) + assert.Empty(t, snap.key, "nothing to file under") + + _, _, err = r.Lookup(ctx, "acme", "q", []Namespace{{Tenant: "globex", Table: "events"}}) + require.ErrorIs(t, err, ErrForeignDependency) + + require.NoError(t, r.Set(ctx, Snapshot{key: "k"}, []byte("rows"), time.Minute)) + + n, err := r.Invalidate(ctx, []Namespace{{Tenant: "acme", Table: "events", Scope: "org_1"}}) + require.ErrorIs(t, err, errBypassed) + assert.Equal(t, uint64(1), n) + assert.Equal(t, 2, r.pending.len()) + require.ErrorIs(t, r.InvalidateTenant(ctx, "globex"), errBypassed) + owed := r.pending.snapshot() + assert.Len(t, owed, 2, "past PendingMax: one bump per tenant") + assert.Contains(t, owed, "wh:{acme}:T") + assert.Contains(t, owed, "wh:{globex}:T") + + require.NoError(t, r.Close()) + require.NoError(t, r.Close(), "idempotent") +} + +func TestRedis_SetDeclinesWithoutTouchingTheServer(t *testing.T) { + t.Parallel() + r, err := NewRedis(RedisConfig{Addrs: []string{closedAddr(t)}, DialTimeout: 100 * time.Millisecond, MaxValueBytes: 64}) + require.NoError(t, err) + t.Cleanup(func() { _ = r.Close() }) + ctx := context.Background() + snap := Snapshot{key: "k", tokens: newToken()} + for _, tt := range []struct { + name string + snap Snapshot + value []byte + ttl time.Duration + }{ + {"zero snapshot", Snapshot{}, []byte("rows"), time.Minute}, + {"zero ttl", snap, []byte("rows"), 0}, + {"over the stored limit", snap, randomBytes(100), time.Minute}, + {"over the decoded limit", snap, make([]byte, 64*decodedFactor+1), time.Minute}, + } { + require.NoError(t, r.Set(ctx, tt.snap, tt.value, tt.ttl), tt.name) + } +} + +func TestRedis_Record(t *testing.T) { + t.Parallel() + r := &RedisCache{breaker: newBreaker(1, time.Hour, time.Now)} + live, cancelled := context.Background(), cancelledCtx() + + r.record(live, rueidis.Nil) + r.record(live, &rueidis.RedisError{}) + r.record(cancelled, context.Canceled) + assert.False(t, r.breaker.isOpen(), "a reply, even an error reply, or the caller giving up says nothing against the server") + + r.record(live, context.DeadlineExceeded) + assert.True(t, r.breaker.isOpen()) +} + +func cancelledCtx() context.Context { + ctx, cancel := context.WithCancel(context.Background()) + cancel() + return ctx +} + +func TestSetFailureReason(t *testing.T) { + t.Parallel() + assert.Equal(t, "timeout", setFailureReason(context.DeadlineExceeded)) + assert.Equal(t, "other", setFailureReason(errors.New("broken pipe"))) +} + +func TestIsAuthError(t *testing.T) { + t.Parallel() + assert.True(t, isAuthError(errors.New("WRONGPASS invalid username-password pair"))) + assert.True(t, isAuthError(errors.New("NOAUTH authentication required"))) + assert.False(t, isAuthError(errors.New("dial tcp: connection refused"))) +} + +func TestReadTokens_RefusesForeignValues(t *testing.T) { + t.Parallel() + _, _, err := readTokens(rueidis.RedisResult{}) + require.Error(t, err) +} + +func TestRedisMetrics(t *testing.T) { + // No t.Parallel(): swaps the global meter provider. + saved := otel.GetMeterProvider() + reader := sdkmetric.NewManualReader() + mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader)) + otel.SetMeterProvider(mp) + t.Cleanup(func() { + _ = mp.Shutdown(context.Background()) + otel.SetMeterProvider(saved) + }) + + r, err := NewRedis(RedisConfig{Addrs: []string{closedAddr(t)}, DialTimeout: 100 * time.Millisecond}) + require.NoError(t, err) + t.Cleanup(func() { _ = r.Close() }) + ctx := context.Background() + _, _, _ = r.Lookup(ctx, "acme", "q", nil) + _, _ = r.Invalidate(ctx, []Namespace{{Tenant: "acme", Table: "events"}}) + r.metrics.op("set", time.Now()) + r.metrics.stored(10) + r.metrics.tooLarge() + r.metrics.setFailed("oom") + + var rm metricdata.ResourceMetrics + require.NoError(t, reader.Collect(ctx, &rm)) + got := map[string]metricdata.Aggregation{} + for _, sm := range rm.ScopeMetrics { + for _, m := range sm.Metrics { + got[m.Name] = m.Data + } + } + sumOf := func(name, key, value string) int64 { + t.Helper() + var n int64 + switch d := got[name].(type) { + case metricdata.Sum[int64]: + for _, dp := range d.DataPoints { + if v, ok := dp.Attributes.Value(attribute.Key(key)); key == "" || ok && v.AsString() == value { + n += dp.Value + } + } + case metricdata.Gauge[int64]: + for _, dp := range d.DataPoints { + n += dp.Value + } + default: + t.Fatalf("%s: %T", name, got[name]) + } + return n + } + assert.Equal(t, int64(1), sumOf("wavehouse_cache_lookups_total", "result", resultBypass)) + assert.Equal(t, int64(1), sumOf("wavehouse_cache_invalidations_total", "result", "deferred")) + assert.Equal(t, int64(1), sumOf("wavehouse_cache_invalidations_pending", "", "")) + assert.Equal(t, int64(1), sumOf("wavehouse_cache_breaker_open", "", "")) + assert.Equal(t, int64(1), sumOf("wavehouse_cache_oversize_total", "", "")) + assert.Equal(t, int64(1), sumOf("wavehouse_cache_set_failures_total", "reason", "oom")) + assert.Contains(t, got, "wavehouse_cache_op_duration_seconds") + assert.Contains(t, got, "wavehouse_cache_value_bytes") +} From e346426faccb0b05eeb9738b4cafb5abfeb11703 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Fri, 25 Sep 2026 00:20:07 -0400 Subject: [PATCH 07/45] fix(cache): count only transport failures against the breaker Review round: a reply the backend cannot use (a foreign value under a token key, an unreadable MGET) no longer opens the breaker, and such a token is replaced; the probe follows the same rule. Close drains the pending bumps past an open breaker, and starts no probe once closing. VersionTTL under 2s is refused (it would round to EX 0). Docs: PING in the command list, the breaker rule, compression wording. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01EJr5tY4WQUy2sc4MbW67vL --- CHANGELOG.md | 2 +- docs/src/content/docs/architecture.md | 6 +- internal/cache/redis.go | 100 +++++++++++++++-------- internal/cache/redis_integration_test.go | 63 ++++++++++++++ internal/cache/redis_test.go | 9 +- 5 files changed, 137 insertions(+), 43 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6a39bc8df..08960f289 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **A Redis-compatible shared cache backend, built but not yet selectable** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values over 1 KiB are zstd-compressed and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row open a circuit breaker that skips the server until a probe succeeds, and deferred invalidations are retried until they land, collapsing to one tenant-wide bump per tenant past 100,000 keys. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending`, `wavehouse_cache_value_bytes`, `wavehouse_cache_oversize_total`, `wavehouse_cache_set_failures_total{backend,reason}`. Nothing selects it yet: the `cache.backend` switch and the `cache.redis.*` settings arrive with the wiring (E4), so every deployment still runs `LocalCache`. Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression and server-stops-answering cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. +- **A Redis-compatible shared cache backend, built but not yet selectable** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row open a circuit breaker that skips the server until a probe succeeds, and deferred invalidations are retried until they land, collapsing to one tenant-wide bump per tenant past 100,000 keys. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending`, `wavehouse_cache_value_bytes`, `wavehouse_cache_oversize_total`, `wavehouse_cache_set_failures_total{backend,reason}`. Nothing selects it yet: the `cache.backend` switch and the `cache.redis.*` settings arrive with the wiring (E4), so every deployment still runs `LocalCache`. Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression and server-stops-answering cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. - **A tenant removed or rejected at runtime has its open streams ended** (`internal/stream/{hub,subscriber,bucket}.go` (+ tests), `internal/api/stream.go` (+ tests), `internal/api/router.go`, `internal/settings/{registry,tree}.go` (+ tests), `internal/ingest/worker.go` (+ tests), `internal/app/wire.go` (+ tests), `clients/ts/src/stream/sse.ts`, `docs/src/content/docs/{api,deployment,architecture,ingest-pipeline}.md`, `docs/src/content/docs/settings-directory.mdx`, `AGENTS.md`): story 3 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)). A `GET /v1/stream` used to outlive its tenant: the hub read no policy for it and withheld every row while the keepalive wheel held the connection open, so the client could not tell it from a quiet table. After every reload the hub now evicts the subscribers of each tenant no longer served, removed or rejected alike (`Hub.Prune`, through the close-once `Subscriber.Evict`), and the handler ends the stream: a gap-fill in progress included, and a stream `TenantMW` admitted just before the reload but registered just after it, which the handler checks for as it registers. The client's reconnect gets `404` (removed) or `503` (rejected); the SDK stops on the first and retries the second, resuming from `Last-Event-ID` once the folder is back — except in a browser going cross-origin where tenant `0` is not served or its CORS list does not admit the page, which cannot read either refusal (both are decorated from tenant `0`'s list) and re-dials as after a dropped connection. A flat directory never stops serving tenant `0`, so nothing changes there. Removing a tenant is deleting its folder, then reloading the whole directory, the last folder included: a server started with tenant folders reads the emptied directory as no folder left, where it read as a change of shape and the reload was rejected whole, leaving that tenant served; at boot an empty directory still reads as the four files, missing. Reloading the deleted folder by name leaves its tenant rejected. `GET /v1/health` keeps resolving a tenant, deliberately: its `404` tells a caller with no token no more than every tenant route's does, since they all answer before authenticating, and resolving answers a served tenant's ping from that tenant's own CORS list. A batch queued for a tenant with no ClickHouse connection — one no longer served, or one no pool could be opened for (such as by the connection ceiling) — skips the row-by-row retry, which no row of it could pass, and meets its DLQ switch once, whole, logged once per batch rather than twice per row; a tenant no longer served reads the switch as on, so its queued rows are parked under its own subject rather than dropped, inserted into another tenant's ClickHouse, or left unacked, redelivered for as long as the tenant is away and holding its queue's ack floor, which the purge waits on. Over a nested directory `/livez` no longer keeps naming a tenant that stopped being served before any tenant completed a first discovery: the diagnostic goes back to `no tenant has completed a first discovery yet`. - **One ClickHouse pool per tuple and one schema registry per tenant** (`internal/chconn/chconn.go` (+ tests), `internal/discovery/discovery.go` (+ tests), `internal/app/discoveries.go` (new), `internal/app/{app,wire}.go` (+ tests), `internal/api/{schema,ingest,structured_query,pipes,query,health,errors,clickhouse_exec}.go` (+ tests), `internal/stream/hub.go`, `internal/ingest/worker.go`, `internal/cache/{cache,local,version_manager}.go` (+ tests), `internal/testutil/{testutil,mocks}.go`, `tests/integration/{setup,tenants,boot_resilience,query_limits}_test.go`, `clients/ts/src/{schema,table,sql,client,types}.ts` (+ tests), `tests/e2e/sdk/admin.test.ts`, `docs/src/content/docs/{api,deployment,architecture,ingest-pipeline}.md`, `docs/src/content/docs/{settings-directory,configuration,access-control,reverse-proxy}.mdx`, `docs/src/content/docs/sdk/{admin,reference,queries}.md`, `AGENTS.md`): the second slice of story 6 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)), with no behavior change for a settings directory that holds the four files beyond the three noted at the end. The process opens one native pool per distinct `clickhouse.addr` / `database` / `username` / password / `tls` tuple among the served tenants (`chconn.Identity`, `chconn.Pools`), shared by the tenants naming it and sized to their largest `max_open_conns` and `max_idle_conns`; `http_port`, `http_scheme`, `headers` and `query_timeout` stay each tenant's own. Every reload reconciles the pools: a new tuple opens (never dials), a tenant whose tuple changed is repointed, a tuple no tenant names closes after the longest `query_timeout` among the tenants it had, and a pool whose largest ask changed is resized with the same grace. The boot config's `clickhouse.max_total_conns` now bounds the open pools together: boot is refused naming the sum and the ceiling; at a reload a resize above it is refused with the pool kept at its size, and a tuple that cannot be opened — the ceiling, a certificate file that cannot be read, or options the driver refuses — leaves its tenants on the pool they had (the keep-previous-wiring rule of the connection ceiling) or on none when they had none; both are logged and the next reload retries. A tenant on no pool fails closed: `503` with `Retry-After: 30` on `POST /v1/query`, `GET/POST /v1/pipes/{name}`, `POST /v1/ops/query` and `POST /v1/ops/schema/refresh`, ahead of the cache. Each served tenant gets a `discovery.SchemaRegistry` of its own over its pool, kept fresh by its own loop — the boot retry until the first success, then `schema.refresh_interval` with the first refresh at a random point within the interval so tenants adopted together do not refresh together — created and stopped from the settings reload and stopped under `App.Close`; `wavehouse_schema_refresh_failures_total{tenant}` counts a loop's failed attempts. `GET /v1/ops/schema`, `POST /v1/ops/schema/refresh` and `POST /v1/ops/query` take the strict `?tenant=` the pipe reads take (absent is tenant `0`, `400` malformed, `404` unknown, `503` rejected); the SDK sends it as the `tenant` option of `wh.schema.list()`, `wh.schema.refresh()`, `wh.from(t).schema()` and `wh.sql()`. Over a nested directory `/livez` (and `/v1/health`) is `503` with the latest discovery failure, naming its tenant, while no tenant has completed a first discovery, then `200` for the rest of the process lifetime; `/readyz` pings every open pool at once, is ready at the first answer, and names every pool that did not answer when none does — one tenant's ClickHouse outage is its log line and counter, never a probe failure. The ingest worker inserts each batch into its own tenant's ClickHouse — the tenant the message's topic names, through that tenant's HTTP wiring (`chconn.Pools.Target`); a tenant on no pool takes the failure path an unreachable ClickHouse takes — and its cache invalidation fans out to the tenants on the same address and database as the batch's tenant, whatever their user or `tls` block (they read the same tables), rather than to every known tenant, and a tenant adopted after an absence — rejected or removed, so out of that fan-out — or moved to another address or database has its cached structured-query results orphaned in one step (`Cache.InvalidateTenant`, a tenant generation in every version key), so a repaired folder never serves query rows cached before the inserts it missed (a pipe result names no table, so neither this nor any insert invalidates it: it stays until its TTL expires). Three changes reach the single-tenant directory too: a table lookup before the first discovery is a `503` with `Retry-After: 5` (`schema not loaded yet`) rather than a `404`, on `POST /v1/ingest`, `POST /v1/query` and `GET /v1/ops/schema` (the list included, where `[]` would read as no tables); the first periodic refresh fires at a random point within the interval rather than a full interval after boot; and a reload that moves `clickhouse.addr` or `clickhouse.database` now orphans the structured-query results cached before it, where they were served until their TTL. Until [#529](https://github.com/Wave-RF/WaveHouse/issues/529) every tenant's user authenticates with `WH_CH_PASSWORD`, so the tuple is in effect the address, database, username and `tls` block. - **One token verifier per tenant, built off the boot and reload paths** (`internal/auth/auth.go` (+ tests), `internal/api/router.go` (+ tests), `internal/app/{app,wire}.go` (+ tests), `go.mod`, `docs/src/content/docs/sdk/{reference,streaming}.md`, `docs/src/content/docs/{architecture,deployment,api}.md`, `docs/src/content/docs/{settings-directory,configuration}.mdx`, `SECURITY.md`): story 9 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)). Over a nested settings directory each tenant's folder now wires that tenant's verifier (`auth.jwks_url`, `auth.role_claim`), so a JWKS-issued token verifies only under the tenants whose `jwks_url` names its identity provider's key set — under any other tenant's header it is refused as invalid — where before one verifier, tenant `0`'s, accepted a token under any header. `auth.Config` is now the boot-config half alone (the HMAC secret and the operator key, shared by every tenant) and the new `auth.Wiring` a tenant's half; `NewAuthenticator` builds no verifier, `Reconfigure(id, wiring)` gives a tenant one — swapped atomically when its wiring changed, kept when it did not — `Prune` drops the verifiers of the tenants a reload stopped serving, rejected or removed alike (no work runs for a tenant that is not served; a folder adopted again gets a fresh verifier), and `Close` stops every JWKS refresh as a component of `App.Close`. This holds on every reload because the registry's `AfterAdopt` hooks run after every reload it applied, empty list included, so a per-tenant reload that rejects a folder drops its verifier then rather than at the next adoption. The middleware reads the request tenant's verifier through an injected `auth.TenantSource` — the store `api.TenantMW` resolved names its tenant with `settings.Store.Tenant()`, one read of the context — a tenant-exempt route (the ops tree) verifies as tenant `0`, and a tenant with no verifier fails closed. The operator key stamps the request tenant's `admin_role`, read from that tenant's policy, rather than tenant `0`'s. A JWKS key set is fetched on its own goroutine: the verifier is in place at once and *pending* until a set has been stored — the first fetch retried with backoff from one second to a minute, a stored set then kept fresh by the library hourly and, rate-limited, on an unknown key id — so neither boot nor a reload (which holds the registry's lock) waits on the endpoint, and **an unreachable JWKS no longer refuses boot** — it logs (`jwks refresh failed; no token validates until it succeeds`) and that tenant alone is affected. A token checked against a pending verifier is a new outcome, `auth.ErrVerifierPending`: every `/v1` route answers it `503 {"error": "token verifier not ready: …"}` with `Retry-After: 30` (`api.refuseUnverifiable`) rather than evaluate the request under the `default_role`, which could accept its data under a lesser role while another pod holding the keys would have served it as its own; a request without a token, and the operator key, are unaffected. Every fetch goes through one client that caps the response at 1 MiB, refusing a larger one as unreachable with `jwks response exceeds 1048576 bytes` as the logged cause. Nothing changes for a flat directory beyond that boot rule: its one tenant gets exactly the verifier it had. The boot warning for the secretless posture (no `auth.jwt_secret` and no `jwks_url`, whose token refusal landed in [#607](https://github.com/Wave-RF/WaveHouse/pull/607)) is now per tenant — each served tenant with no `jwks_url` while the boot secret is unset — where one line that any JWKS tenant silenced used to stand for the whole directory. diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 53850f49f..32f743875 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -112,10 +112,10 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ - **cache.go** — `Cache` interface: `Lookup`, `Set`, `Invalidate`, `InvalidateTenant`, `Close`, plus `QueryTimeToTTL`, which sets a result's TTL from how long its query took (10 s floor, 1 h ceiling). Every entry is one tenant's: `Lookup` takes the tenant, the caller's query key — `:query:`, built by the two cached handlers in `api/` (`queryCacheKey`, with the tenant read off the request's store — `settings.Store.Tenant`), which use it as their [singleflight](https://pkg.go.dev/golang.org/x/sync/singleflight) key too — and the `Namespace`s the result depends on, each of that tenant (another tenant's is `ErrForeignDependency`) and naming a table and scope: one for a structured query, none yet for a pipe (a pipe's table dependencies are [#343](https://github.com/Wave-RF/WaveHouse/pull/343)). `Lookup` returns the `Entry` (a nil value is a miss) and a `Snapshot` of the versions it read; on a miss the handler runs the query and passes that snapshot to `Set`, so a result is filed under the versions read *before* its query ran, and a write that lands while it runs orphans the fill rather than re-homing pre-write rows under the post-write versions ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)). The singleflight leader's snapshot is the one used. `Set` returns an error only when the backend failed; a value the cache declines — larger than it keeps, a non-positive TTL, a zero snapshot — is not one. What a hit, a miss and a bump mean is pinned by the conformance suite every backend runs, `internal/testutil/cachetest`. - **local.go** — `LocalCache`, the in-process L1 on [Ristretto](https://github.com/dgraph-io/ristretto): one pool shared by every tenant (a heavier tenant holds more of it), sized by the boot config's `cache.l1_max_cost`. - **version_manager.go** — `VersionManager`, the invalidation index behind `Invalidate` and `InvalidateTenant`: a namespace key is `..
.
.`, and a query key is folded with the tenant's version and each dependency's namespace key and namespace version, so bumping a table (a scopeless write) or one scope — scope is reserved and empty today, so every write is the whole-table bump — orphans every dependent entry without touching the pool. The tenant leads every key ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8): the same table under two tenants is two namespaces, so a bump through `Invalidate` under one tenant never touches — and a read under one tenant is never served — the other's results, and the flat directory's single tenant simply carries the `0` prefix. `BumpTenant` (behind `InvalidateTenant`) advances the tenant version that leads every namespace key of one tenant, orphaning its every namespace and every cached query in one step (the tenant version is folded into every query key, so a pipe result with no dependencies is orphaned too) — a table no bump ever keyed included, which an enumeration of the index would miss — for a tenant back on a pool after an absence from the fan-out, or moved to another address or database (story 6). The index is per tenant; the cross-tenant invalidation an insert into a shared table needs is not the index's but the wiring's: `internal/app` hands the ingest worker a cache (`sharedTables`) that repeats each bump under every tenant on the same ClickHouse address and database. -- **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — only `GET`, `SET` and `MGET`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, so a token lost to eviction, expiry or a restart is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing. Nothing selects this backend yet — the boot config and wiring arrive with [#613](https://github.com/Wave-RF/WaveHouse/issues/613)'s E4. +- **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — only `GET`, `SET`, `MGET` and `PING`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, and a token key holding anything but a token is replaced, so a token lost to eviction, expiry or a restart is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing. Nothing selects this backend yet — the boot config and wiring arrive with [#613](https://github.com/Wave-RF/WaveHouse/issues/613)'s E4. - **redis_codec.go** — the key schema (`tokenKeys`, `bumpKeys`, `valueKey`) and the value frame: format, flags, expiry, the token list, the payload, zstd-compressed from `CompressMinBytes` when that is smaller. A stored value is capped at `MaxValueBytes` and a decoded one at eight times that, which refuses a zip bomb planted in a shared server; an unknown format, as a newer process writes during a rolling upgrade, is a miss. -- **breaker.go** — the circuit breaker: `BreakerThreshold` (5) consecutive failures open it, and while open every operation skips the server at once; after `BreakerOpenFor` (5 s) one background `PING` decides whether it closes. A reply from the server, error replies included, counts as a success; a caller that gave up first counts as nothing. -- **pending.go** — invalidations the server did not take, kept per token key (repeats coalesce) and retried by a background loop with backoff from 100 ms to 10 s until they land. Landing late is still correct: a fresh token orphans the pre-write entries and any fill made meanwhile. Past `PendingMax` (100,000) keys the set collapses to one tenant bump per affected tenant. Bumps still owed when a process stops are lost after one last attempt, and the entries they would orphan are served until their TTL — the same failure as a worker stopping between an insert and its invalidation. +- **breaker.go** — the circuit breaker: `BreakerThreshold` (5) consecutive failures open it, and while open every operation skips the server at once; after `BreakerOpenFor` (5 s) one background `PING` decides whether it closes. Only a transport failure or a timeout counts against the server: any reply, an error reply or one the backend cannot use included, counts as a success, for operations and the probe alike, and a caller that gave up first counts as nothing. +- **pending.go** — invalidations the server did not take, kept per token key (repeats coalesce) and retried by a background loop with backoff from 100 ms to 10 s until they land. Landing late is still correct: a fresh token orphans the pre-write entries and any fill made meanwhile. Past `PendingMax` (100,000) keys the set collapses to one tenant bump per affected tenant. `Close` makes one last attempt at the bumps still owed, past the breaker (an open one is why they are owed); what that attempt cannot deliver is lost, and the entries they would orphan are served until their TTL — the same failure as a worker stopping between an insert and its invalidation. - **metrics.go** — the shared backend's instruments (meter `wavehouse-cache`, `backend="redis"`, no tenant attribute): `wavehouse_cache_lookups_total` by `result` (`hit`, `miss`, `stale` — filed under since-bumped tokens, `bypass` — server skipped, `error`), `wavehouse_cache_op_duration_seconds` by `op` (`lookup`, `set`, `invalidate`), `wavehouse_cache_breaker_open` (1 while bypassed, including before the first connection), `wavehouse_cache_invalidations_total` by `result` (`ok`, `deferred`), `wavehouse_cache_invalidations_pending` (bumps owed: entries they would orphan may be served stale meanwhile), `wavehouse_cache_value_bytes` (stored size), `wavehouse_cache_oversize_total` and `wavehouse_cache_set_failures_total` by `reason` (`oom`, `timeout`, `other`). ### `config/` — Configuration diff --git a/internal/cache/redis.go b/internal/cache/redis.go index ede95a534..30ed35ba6 100644 --- a/internal/cache/redis.go +++ b/internal/cache/redis.go @@ -47,7 +47,12 @@ const ( closeDrainBudget = time.Second ) -var errBypassed = errors.New("cache: redis unavailable") +var ( + errBypassed = errors.New("cache: redis unavailable") + // errMalformedReply is a reply the server sent that this code cannot + // use: the server is up, so it never counts against the breaker. + errMalformedReply = errors.New("cache: malformed reply") +) // RedisConfig configures a RedisCache. A zero field takes its Default* // value, except CompressMinBytes, where 0 means never compress. @@ -117,6 +122,9 @@ func (c RedisConfig) withDefaults() (RedisConfig, error) { return c, fmt.Errorf("cache: redis %s is negative", v.name) } } + if c.VersionTTL != 0 && c.VersionTTL < 2*time.Second { + return c, fmt.Errorf("cache: redis version ttl %s is under 2s: jittered, it would round to EX 0", c.VersionTTL) + } c.KeyPrefix = cmpOr(c.KeyPrefix, DefaultRedisKeyPrefix) c.Timeout = cmpOr(c.Timeout, DefaultRedisTimeout) c.DialTimeout = cmpOr(c.DialTimeout, DefaultRedisDialTimeout) @@ -155,8 +163,8 @@ func (c RedisConfig) clientOption() rueidis.ClientOption { } // RedisCache is a Cache shared by every process pointed at one Redis — -// or Valkey, Dragonfly, ElastiCache, MemoryDB: it uses only GET, SET and -// MGET, no scripts and no client tracking. +// or Valkey, Dragonfly, ElastiCache, MemoryDB: it uses only GET, SET, MGET +// and PING, no scripts and no client tracking. // // Versions are random tokens, one per tenant, per table and per scope, // under the tenant's hash tag; a bump sets a fresh one. A value carries the @@ -274,7 +282,7 @@ func (r *RedisCache) conn() rueidis.Client { return nil } ok, probe := r.breaker.allow() - if probe { + if probe && r.ctx.Err() == nil { go r.probe(*cp) } if !ok { @@ -286,11 +294,10 @@ func (r *RedisCache) conn() rueidis.Client { func (r *RedisCache) probe(c rueidis.Client) { ctx, cancel := context.WithTimeout(r.ctx, r.cfg.Timeout) defer cancel() - if err := c.Do(ctx, c.B().Ping().Build()).Error(); err != nil { - r.breaker.failure() + r.record(r.ctx, c.Do(ctx, c.B().Ping().Build()).Error()) + if r.breaker.isOpen() { return } - r.breaker.success() slog.InfoContext(ctx, "cache: redis reachable again; cache back in use") r.nudge() } @@ -301,14 +308,14 @@ func (r *RedisCache) bypassed() bool { } // record feeds an operation's outcome to the breaker. A reply from the -// server, even an error reply, shows it is up; a caller that gave up first +// server, even an error reply or one this code cannot use, shows it is up; a caller that gave up first // shows nothing about it. func (r *RedisCache) record(parent context.Context, err error) { if err == nil || rueidis.IsRedisNil(err) { r.breaker.success() return } - if _, ok := rueidis.IsRedisErr(err); ok { + if _, ok := rueidis.IsRedisErr(err); ok || errors.Is(err, errMalformedReply) { r.breaker.success() return } @@ -342,18 +349,23 @@ func (r *RedisCache) Lookup(ctx context.Context, id tenant.ID, sha string, deps vkey := valueKey(r.cfg.KeyPrefix, id, sha, deps) res := c.DoMulti(opCtx, c.B().Mget().Key(keys...).Build(), c.B().Get().Key(vkey).Build()) - tokens, missing, err := readTokens(res[0]) + for _, rr := range res { + if err := rr.Error(); err != nil && !rueidis.IsRedisNil(err) { + return r.lookupFailed(ctx, err) + } + } + r.record(ctx, nil) + tokens, missing, foreign, err := readTokens(res[0]) if err != nil { return r.lookupFailed(ctx, err) } val, err := res[1].AsBytes() if err != nil && !rueidis.IsRedisNil(err) { - return r.lookupFailed(ctx, err) + return r.lookupFailed(ctx, fmt.Errorf("%w: %w", errMalformedReply, err)) } - r.record(ctx, nil) - if len(missing) > 0 { - if tokens, err = r.createTokens(opCtx, c, keys, missing); err != nil { + if len(missing) > 0 || len(foreign) > 0 { + if tokens, err = r.createTokens(opCtx, c, keys, missing, foreign); err != nil { return r.lookupFailed(ctx, err) } r.metrics.lookup(resultMiss) @@ -386,11 +398,11 @@ func (r *RedisCache) lookupFailed(ctx context.Context, err error) (Entry, Snapsh } // readTokens concatenates an MGET reply's tokens, listing the indexes of the -// keys that do not exist. -func readTokens(res rueidis.RedisResult) (tokens []byte, missing []int, err error) { +// keys that do not exist and of those holding something that is not a token. +func readTokens(res rueidis.RedisResult) (tokens []byte, missing, foreign []int, err error) { msgs, err := res.ToArray() if err != nil { - return nil, nil, err + return nil, nil, nil, fmt.Errorf("%w: %w", errMalformedReply, err) } tokens = make([]byte, 0, len(msgs)*tokenLen) for i := range msgs { @@ -400,39 +412,47 @@ func readTokens(res rueidis.RedisResult) (tokens []byte, missing []int, err erro continue } s, err := msgs[i].ToString() - if err != nil { - return nil, nil, err - } - if len(s) != tokenLen { - return nil, nil, fmt.Errorf("version token is %d bytes, not %d: is another program writing under this key prefix?", len(s), tokenLen) + if err != nil || len(s) != tokenLen { + foreign = append(foreign, i) + tokens = append(tokens, make([]byte, tokenLen)...) + continue } tokens = append(tokens, s...) } - return tokens, missing, nil + return tokens, missing, foreign, nil } // createTokens sets each missing token — only if still missing, as another -// process may create it first — and reads them all back, in one round trip: -// the tokens share a slot, so the pipeline runs in order on one node. -func (r *RedisCache) createTokens(ctx context.Context, c rueidis.Client, keys []string, missing []int) ([]byte, error) { - cmds := make(rueidis.Commands, 0, len(missing)+1) +// process may create it first — replaces each foreign one, and reads them +// all back, in one round trip: the tokens share a slot, so the pipeline runs +// in order on one node. A fresh token can only cause misses, so replacing +// whatever held a token key is safe. +func (r *RedisCache) createTokens(ctx context.Context, c rueidis.Client, keys []string, missing, foreign []int) ([]byte, error) { + if len(foreign) > 0 { + slog.WarnContext(ctx, "cache: replacing values that are not version tokens; is another program writing under this key prefix?", + "keys", len(foreign), "prefix", r.cfg.KeyPrefix) + } + cmds := make(rueidis.Commands, 0, len(missing)+len(foreign)+1) for _, i := range missing { tok := newToken() cmds = append(cmds, c.B().Set().Key(keys[i]).Value(rueidis.BinaryString(tok)).Nx().Ex(jitter(r.cfg.VersionTTL, tok)).Build()) } + for _, i := range foreign { + cmds = append(cmds, r.bumpCmd(c, keys[i])) + } cmds = append(cmds, c.B().Mget().Key(keys...).Build()) res := c.DoMulti(ctx, cmds...) - for _, rr := range res[:len(missing)] { + for _, rr := range res { if err := rr.Error(); err != nil && !rueidis.IsRedisNil(err) { return nil, err } } - tokens, still, err := readTokens(res[len(missing)]) + tokens, still, bad, err := readTokens(res[len(res)-1]) if err != nil { return nil, err } - if len(still) > 0 { - return nil, errors.New("version token vanished as it was created: is the server evicting everything?") + if len(still) > 0 || len(bad) > 0 { + return nil, fmt.Errorf("%w: version token gone or replaced as it was written", errMalformedReply) } return tokens, nil } @@ -570,7 +590,7 @@ func (r *RedisCache) drainLoop() { r.conn() // starts the probe when due, so an idle process recovers too } if r.pending.len() > 0 { - if r.drain(r.ctx) { + if r.drain(r.ctx, r.conn) { backoff = drainMinBackoff } else { wait, backoff = backoff, min(backoff*2, drainMaxBackoff) @@ -580,8 +600,9 @@ func (r *RedisCache) drainLoop() { } } -// drain delivers the pending bumps, reporting whether none remain. -func (r *RedisCache) drain(ctx context.Context) bool { +// drain delivers the pending bumps through the client conn returns, +// reporting whether none remain. +func (r *RedisCache) drain(ctx context.Context, conn func() rueidis.Client) bool { owed := r.pending.snapshot() keys := make([]string, 0, len(owed)) for k := range owed { @@ -590,7 +611,7 @@ func (r *RedisCache) drain(ctx context.Context) bool { for len(keys) > 0 { batch := keys[:min(drainBatch, len(keys))] keys = keys[len(batch):] - c := r.conn() + c := conn() if c == nil { return false } @@ -628,7 +649,14 @@ func (r *RedisCache) Close() error { r.wg.Wait() if r.pending.len() > 0 { ctx, cancel := context.WithTimeout(context.Background(), closeDrainBudget) - r.drain(ctx) + // Past the breaker: an open one is why bumps are pending, and this + // is the last chance to deliver them. + r.drain(ctx, func() rueidis.Client { + if cp := r.client.Load(); cp != nil { + return *cp + } + return nil + }) cancel() if n := r.pending.len(); n > 0 { slog.Warn("cache: closing with undelivered invalidations; entries they orphan stay cached until their TTL", "pending", n) diff --git a/internal/cache/redis_integration_test.go b/internal/cache/redis_integration_test.go index fb902e735..155caca92 100644 --- a/internal/cache/redis_integration_test.go +++ b/internal/cache/redis_integration_test.go @@ -342,6 +342,33 @@ func TestRedis_LostTokensAreMisses(t *testing.T) { fill(t, "q", deps) } +// A token key holding something that is not a token — another program +// under the prefix, a different token size mid-upgrade — is a reply, not a +// failure: it is replaced, which can only cause misses, and the breaker +// stays closed. +func TestRedis_ForeignTokenIsReplaced(t *testing.T) { + t.Parallel() + ctx := context.Background() + s := startRedis(t) + r := raw(t, s) + prefix := uniquePrefix() + c := open(t, s, prefix, func(c *cache.RedisConfig) { c.BreakerThreshold = 1 }) + deps := []cache.Namespace{{Tenant: "acme", Table: "events"}} + _, snap, err := c.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + require.NoError(t, c.Set(ctx, snap, []byte("rows"), time.Minute)) + require.NoError(t, r.Do(ctx, r.B().Set().Key(prefix+":{acme}:B:events").Value("abc").Build()).Error()) + + e, snap, err := c.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + assert.Nil(t, e.Value) + assert.False(t, cache.Bypassed(c)) + require.NoError(t, c.Set(ctx, snap, []byte("new rows"), time.Minute)) + e, _, err = c.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + assert.Equal(t, "new rows", string(e.Value)) +} + func dockerClient(t *testing.T) *testcontainers.DockerClient { t.Helper() d, err := testcontainers.NewDockerClientWithOpts(context.Background()) @@ -416,3 +443,39 @@ func TestRedis_ServerStopsAnswering(t *testing.T) { require.NoError(t, err) assert.Nil(t, e.Value, "the fill from before the deferred bump is orphaned for every process") } + +// Close makes its last attempt at the pending bumps past the breaker: an +// open one is why they are pending, and the server may be back by now. +func TestRedis_CloseDeliversPastAnOpenBreaker(t *testing.T) { + t.Parallel() + ctx := context.Background() + s := startRedis(t) + d := dockerClient(t) + prefix := uniquePrefix() + a := open(t, s, prefix, func(c *cache.RedisConfig) { + c.Timeout, c.BreakerThreshold, c.BreakerOpenFor = 100*time.Millisecond, 1, time.Hour + }) + deps := []cache.Namespace{{Tenant: "acme", Table: "events"}} + _, snap, err := a.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + require.NoError(t, a.Set(ctx, snap, []byte("pre-write rows"), time.Minute)) + + _, err = d.ContainerPause(ctx, s.ctr.GetContainerID(), client.ContainerPauseOptions{}) + require.NoError(t, err) + _, _, err = a.Lookup(ctx, "acme", "q", deps) + require.Error(t, err) + require.True(t, cache.Bypassed(a)) + _, err = a.Invalidate(ctx, deps) + require.Error(t, err) + _, err = d.ContainerUnpause(ctx, s.ctr.GetContainerID(), client.ContainerUnpauseOptions{}) + require.NoError(t, err) + + require.True(t, cache.Bypassed(a), "the breaker stays open for its hour") + require.Equal(t, 1, cache.Pending(a)) + require.NoError(t, a.Close()) + assert.Zero(t, cache.Pending(a)) + + e, _, err := open(t, s, prefix).Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + assert.Nil(t, e.Value, "the bump Close delivered orphans the fill") +} diff --git a/internal/cache/redis_test.go b/internal/cache/redis_test.go index 9feff6327..9aafb4f12 100644 --- a/internal/cache/redis_test.go +++ b/internal/cache/redis_test.go @@ -3,6 +3,7 @@ package cache import ( "context" "errors" + "fmt" "net" "testing" "time" @@ -33,6 +34,7 @@ func TestRedisConfig_Validation(t *testing.T) { {"hash tag in the prefix", func(c *RedisConfig) { c.KeyPrefix = "{wh}" }, "brace"}, {"negative timeout", func(c *RedisConfig) { c.Timeout = -time.Second }, "timeout is negative"}, {"negative size", func(c *RedisConfig) { c.MaxValueBytes = -1 }, "max value bytes is negative"}, + {"version ttl under EX's resolution", func(c *RedisConfig) { c.VersionTTL = 1500 * time.Millisecond }, "under 2s"}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -140,6 +142,7 @@ func TestRedis_Record(t *testing.T) { r.record(live, rueidis.Nil) r.record(live, &rueidis.RedisError{}) r.record(cancelled, context.Canceled) + r.record(live, fmt.Errorf("%w: token is 3 bytes", errMalformedReply)) assert.False(t, r.breaker.isOpen(), "a reply, even an error reply, or the caller giving up says nothing against the server") r.record(live, context.DeadlineExceeded) @@ -165,10 +168,10 @@ func TestIsAuthError(t *testing.T) { assert.False(t, isAuthError(errors.New("dial tcp: connection refused"))) } -func TestReadTokens_RefusesForeignValues(t *testing.T) { +func TestReadTokens_NotAnArrayIsMalformed(t *testing.T) { t.Parallel() - _, _, err := readTokens(rueidis.RedisResult{}) - require.Error(t, err) + _, _, _, err := readTokens(rueidis.RedisResult{}) + require.ErrorIs(t, err, errMalformedReply) } func TestRedisMetrics(t *testing.T) { From d2a87ab7ef14abef09e16c5d4abe12c82aa34716 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Fri, 25 Sep 2026 00:25:55 -0400 Subject: [PATCH 08/45] fix(cache): replace token and value keys of the wrong type MGET reads a non-string token key as nil and SET NX will not overwrite it, so such a key disabled caching behind it; it is now replaced on a second round trip. A value key of the wrong type is a miss the fill's SET replaces, not a lookup failure. Docs: VersionTTL is a lifetime from the last bump, and each metric's labels are listed. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01EJr5tY4WQUy2sc4MbW67vL --- CHANGELOG.md | 2 +- docs/src/content/docs/architecture.md | 2 +- internal/cache/redis.go | 39 ++++++++++++++-------- internal/cache/redis_integration_test.go | 42 ++++++++++++++++-------- 4 files changed, 57 insertions(+), 28 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 08960f289..78a4e3ca0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **A Redis-compatible shared cache backend, built but not yet selectable** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row open a circuit breaker that skips the server until a probe succeeds, and deferred invalidations are retried until they land, collapsing to one tenant-wide bump per tenant past 100,000 keys. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending`, `wavehouse_cache_value_bytes`, `wavehouse_cache_oversize_total`, `wavehouse_cache_set_failures_total{backend,reason}`. Nothing selects it yet: the `cache.backend` switch and the `cache.redis.*` settings arrive with the wiring (E4), so every deployment still runs `LocalCache`. Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression and server-stops-answering cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. +- **A Redis-compatible shared cache backend, built but not yet selectable** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row open a circuit breaker that skips the server until a probe succeeds, and deferred invalidations are retried until they land, collapsing to one tenant-wide bump per tenant past 100,000 keys. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. Nothing selects it yet: the `cache.backend` switch and the `cache.redis.*` settings arrive with the wiring (E4), so every deployment still runs `LocalCache`. Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression and server-stops-answering cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. - **A tenant removed or rejected at runtime has its open streams ended** (`internal/stream/{hub,subscriber,bucket}.go` (+ tests), `internal/api/stream.go` (+ tests), `internal/api/router.go`, `internal/settings/{registry,tree}.go` (+ tests), `internal/ingest/worker.go` (+ tests), `internal/app/wire.go` (+ tests), `clients/ts/src/stream/sse.ts`, `docs/src/content/docs/{api,deployment,architecture,ingest-pipeline}.md`, `docs/src/content/docs/settings-directory.mdx`, `AGENTS.md`): story 3 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)). A `GET /v1/stream` used to outlive its tenant: the hub read no policy for it and withheld every row while the keepalive wheel held the connection open, so the client could not tell it from a quiet table. After every reload the hub now evicts the subscribers of each tenant no longer served, removed or rejected alike (`Hub.Prune`, through the close-once `Subscriber.Evict`), and the handler ends the stream: a gap-fill in progress included, and a stream `TenantMW` admitted just before the reload but registered just after it, which the handler checks for as it registers. The client's reconnect gets `404` (removed) or `503` (rejected); the SDK stops on the first and retries the second, resuming from `Last-Event-ID` once the folder is back — except in a browser going cross-origin where tenant `0` is not served or its CORS list does not admit the page, which cannot read either refusal (both are decorated from tenant `0`'s list) and re-dials as after a dropped connection. A flat directory never stops serving tenant `0`, so nothing changes there. Removing a tenant is deleting its folder, then reloading the whole directory, the last folder included: a server started with tenant folders reads the emptied directory as no folder left, where it read as a change of shape and the reload was rejected whole, leaving that tenant served; at boot an empty directory still reads as the four files, missing. Reloading the deleted folder by name leaves its tenant rejected. `GET /v1/health` keeps resolving a tenant, deliberately: its `404` tells a caller with no token no more than every tenant route's does, since they all answer before authenticating, and resolving answers a served tenant's ping from that tenant's own CORS list. A batch queued for a tenant with no ClickHouse connection — one no longer served, or one no pool could be opened for (such as by the connection ceiling) — skips the row-by-row retry, which no row of it could pass, and meets its DLQ switch once, whole, logged once per batch rather than twice per row; a tenant no longer served reads the switch as on, so its queued rows are parked under its own subject rather than dropped, inserted into another tenant's ClickHouse, or left unacked, redelivered for as long as the tenant is away and holding its queue's ack floor, which the purge waits on. Over a nested directory `/livez` no longer keeps naming a tenant that stopped being served before any tenant completed a first discovery: the diagnostic goes back to `no tenant has completed a first discovery yet`. - **One ClickHouse pool per tuple and one schema registry per tenant** (`internal/chconn/chconn.go` (+ tests), `internal/discovery/discovery.go` (+ tests), `internal/app/discoveries.go` (new), `internal/app/{app,wire}.go` (+ tests), `internal/api/{schema,ingest,structured_query,pipes,query,health,errors,clickhouse_exec}.go` (+ tests), `internal/stream/hub.go`, `internal/ingest/worker.go`, `internal/cache/{cache,local,version_manager}.go` (+ tests), `internal/testutil/{testutil,mocks}.go`, `tests/integration/{setup,tenants,boot_resilience,query_limits}_test.go`, `clients/ts/src/{schema,table,sql,client,types}.ts` (+ tests), `tests/e2e/sdk/admin.test.ts`, `docs/src/content/docs/{api,deployment,architecture,ingest-pipeline}.md`, `docs/src/content/docs/{settings-directory,configuration,access-control,reverse-proxy}.mdx`, `docs/src/content/docs/sdk/{admin,reference,queries}.md`, `AGENTS.md`): the second slice of story 6 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)), with no behavior change for a settings directory that holds the four files beyond the three noted at the end. The process opens one native pool per distinct `clickhouse.addr` / `database` / `username` / password / `tls` tuple among the served tenants (`chconn.Identity`, `chconn.Pools`), shared by the tenants naming it and sized to their largest `max_open_conns` and `max_idle_conns`; `http_port`, `http_scheme`, `headers` and `query_timeout` stay each tenant's own. Every reload reconciles the pools: a new tuple opens (never dials), a tenant whose tuple changed is repointed, a tuple no tenant names closes after the longest `query_timeout` among the tenants it had, and a pool whose largest ask changed is resized with the same grace. The boot config's `clickhouse.max_total_conns` now bounds the open pools together: boot is refused naming the sum and the ceiling; at a reload a resize above it is refused with the pool kept at its size, and a tuple that cannot be opened — the ceiling, a certificate file that cannot be read, or options the driver refuses — leaves its tenants on the pool they had (the keep-previous-wiring rule of the connection ceiling) or on none when they had none; both are logged and the next reload retries. A tenant on no pool fails closed: `503` with `Retry-After: 30` on `POST /v1/query`, `GET/POST /v1/pipes/{name}`, `POST /v1/ops/query` and `POST /v1/ops/schema/refresh`, ahead of the cache. Each served tenant gets a `discovery.SchemaRegistry` of its own over its pool, kept fresh by its own loop — the boot retry until the first success, then `schema.refresh_interval` with the first refresh at a random point within the interval so tenants adopted together do not refresh together — created and stopped from the settings reload and stopped under `App.Close`; `wavehouse_schema_refresh_failures_total{tenant}` counts a loop's failed attempts. `GET /v1/ops/schema`, `POST /v1/ops/schema/refresh` and `POST /v1/ops/query` take the strict `?tenant=` the pipe reads take (absent is tenant `0`, `400` malformed, `404` unknown, `503` rejected); the SDK sends it as the `tenant` option of `wh.schema.list()`, `wh.schema.refresh()`, `wh.from(t).schema()` and `wh.sql()`. Over a nested directory `/livez` (and `/v1/health`) is `503` with the latest discovery failure, naming its tenant, while no tenant has completed a first discovery, then `200` for the rest of the process lifetime; `/readyz` pings every open pool at once, is ready at the first answer, and names every pool that did not answer when none does — one tenant's ClickHouse outage is its log line and counter, never a probe failure. The ingest worker inserts each batch into its own tenant's ClickHouse — the tenant the message's topic names, through that tenant's HTTP wiring (`chconn.Pools.Target`); a tenant on no pool takes the failure path an unreachable ClickHouse takes — and its cache invalidation fans out to the tenants on the same address and database as the batch's tenant, whatever their user or `tls` block (they read the same tables), rather than to every known tenant, and a tenant adopted after an absence — rejected or removed, so out of that fan-out — or moved to another address or database has its cached structured-query results orphaned in one step (`Cache.InvalidateTenant`, a tenant generation in every version key), so a repaired folder never serves query rows cached before the inserts it missed (a pipe result names no table, so neither this nor any insert invalidates it: it stays until its TTL expires). Three changes reach the single-tenant directory too: a table lookup before the first discovery is a `503` with `Retry-After: 5` (`schema not loaded yet`) rather than a `404`, on `POST /v1/ingest`, `POST /v1/query` and `GET /v1/ops/schema` (the list included, where `[]` would read as no tables); the first periodic refresh fires at a random point within the interval rather than a full interval after boot; and a reload that moves `clickhouse.addr` or `clickhouse.database` now orphans the structured-query results cached before it, where they were served until their TTL. Until [#529](https://github.com/Wave-RF/WaveHouse/issues/529) every tenant's user authenticates with `WH_CH_PASSWORD`, so the tuple is in effect the address, database, username and `tls` block. - **One token verifier per tenant, built off the boot and reload paths** (`internal/auth/auth.go` (+ tests), `internal/api/router.go` (+ tests), `internal/app/{app,wire}.go` (+ tests), `go.mod`, `docs/src/content/docs/sdk/{reference,streaming}.md`, `docs/src/content/docs/{architecture,deployment,api}.md`, `docs/src/content/docs/{settings-directory,configuration}.mdx`, `SECURITY.md`): story 9 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)). Over a nested settings directory each tenant's folder now wires that tenant's verifier (`auth.jwks_url`, `auth.role_claim`), so a JWKS-issued token verifies only under the tenants whose `jwks_url` names its identity provider's key set — under any other tenant's header it is refused as invalid — where before one verifier, tenant `0`'s, accepted a token under any header. `auth.Config` is now the boot-config half alone (the HMAC secret and the operator key, shared by every tenant) and the new `auth.Wiring` a tenant's half; `NewAuthenticator` builds no verifier, `Reconfigure(id, wiring)` gives a tenant one — swapped atomically when its wiring changed, kept when it did not — `Prune` drops the verifiers of the tenants a reload stopped serving, rejected or removed alike (no work runs for a tenant that is not served; a folder adopted again gets a fresh verifier), and `Close` stops every JWKS refresh as a component of `App.Close`. This holds on every reload because the registry's `AfterAdopt` hooks run after every reload it applied, empty list included, so a per-tenant reload that rejects a folder drops its verifier then rather than at the next adoption. The middleware reads the request tenant's verifier through an injected `auth.TenantSource` — the store `api.TenantMW` resolved names its tenant with `settings.Store.Tenant()`, one read of the context — a tenant-exempt route (the ops tree) verifies as tenant `0`, and a tenant with no verifier fails closed. The operator key stamps the request tenant's `admin_role`, read from that tenant's policy, rather than tenant `0`'s. A JWKS key set is fetched on its own goroutine: the verifier is in place at once and *pending* until a set has been stored — the first fetch retried with backoff from one second to a minute, a stored set then kept fresh by the library hourly and, rate-limited, on an unknown key id — so neither boot nor a reload (which holds the registry's lock) waits on the endpoint, and **an unreachable JWKS no longer refuses boot** — it logs (`jwks refresh failed; no token validates until it succeeds`) and that tenant alone is affected. A token checked against a pending verifier is a new outcome, `auth.ErrVerifierPending`: every `/v1` route answers it `503 {"error": "token verifier not ready: …"}` with `Retry-After: 30` (`api.refuseUnverifiable`) rather than evaluate the request under the `default_role`, which could accept its data under a lesser role while another pod holding the keys would have served it as its own; a request without a token, and the operator key, are unaffected. Every fetch goes through one client that caps the response at 1 MiB, refusing a larger one as unreachable with `jwks response exceeds 1048576 bytes` as the logged cause. Nothing changes for a flat directory beyond that boot rule: its one tenant gets exactly the verifier it had. The boot warning for the secretless posture (no `auth.jwt_secret` and no `jwks_url`, whose token refusal landed in [#607](https://github.com/Wave-RF/WaveHouse/pull/607)) is now per tenant — each served tenant with no `jwks_url` while the boot secret is unset — where one line that any JWKS tenant silenced used to stand for the whole directory. diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 32f743875..0d617e972 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -112,7 +112,7 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ - **cache.go** — `Cache` interface: `Lookup`, `Set`, `Invalidate`, `InvalidateTenant`, `Close`, plus `QueryTimeToTTL`, which sets a result's TTL from how long its query took (10 s floor, 1 h ceiling). Every entry is one tenant's: `Lookup` takes the tenant, the caller's query key — `:query:`, built by the two cached handlers in `api/` (`queryCacheKey`, with the tenant read off the request's store — `settings.Store.Tenant`), which use it as their [singleflight](https://pkg.go.dev/golang.org/x/sync/singleflight) key too — and the `Namespace`s the result depends on, each of that tenant (another tenant's is `ErrForeignDependency`) and naming a table and scope: one for a structured query, none yet for a pipe (a pipe's table dependencies are [#343](https://github.com/Wave-RF/WaveHouse/pull/343)). `Lookup` returns the `Entry` (a nil value is a miss) and a `Snapshot` of the versions it read; on a miss the handler runs the query and passes that snapshot to `Set`, so a result is filed under the versions read *before* its query ran, and a write that lands while it runs orphans the fill rather than re-homing pre-write rows under the post-write versions ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)). The singleflight leader's snapshot is the one used. `Set` returns an error only when the backend failed; a value the cache declines — larger than it keeps, a non-positive TTL, a zero snapshot — is not one. What a hit, a miss and a bump mean is pinned by the conformance suite every backend runs, `internal/testutil/cachetest`. - **local.go** — `LocalCache`, the in-process L1 on [Ristretto](https://github.com/dgraph-io/ristretto): one pool shared by every tenant (a heavier tenant holds more of it), sized by the boot config's `cache.l1_max_cost`. - **version_manager.go** — `VersionManager`, the invalidation index behind `Invalidate` and `InvalidateTenant`: a namespace key is `..
.
.`, and a query key is folded with the tenant's version and each dependency's namespace key and namespace version, so bumping a table (a scopeless write) or one scope — scope is reserved and empty today, so every write is the whole-table bump — orphans every dependent entry without touching the pool. The tenant leads every key ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8): the same table under two tenants is two namespaces, so a bump through `Invalidate` under one tenant never touches — and a read under one tenant is never served — the other's results, and the flat directory's single tenant simply carries the `0` prefix. `BumpTenant` (behind `InvalidateTenant`) advances the tenant version that leads every namespace key of one tenant, orphaning its every namespace and every cached query in one step (the tenant version is folded into every query key, so a pipe result with no dependencies is orphaned too) — a table no bump ever keyed included, which an enumeration of the index would miss — for a tenant back on a pool after an absence from the fan-out, or moved to another address or database (story 6). The index is per tenant; the cross-tenant invalidation an insert into a shared table needs is not the index's but the wiring's: `internal/app` hands the ingest worker a cache (`sharedTables`) that repeats each bump under every tenant on the same ClickHouse address and database. -- **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — only `GET`, `SET`, `MGET` and `PING`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, and a token key holding anything but a token is replaced, so a token lost to eviction, expiry or a restart is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing. Nothing selects this backend yet — the boot config and wiring arrive with [#613](https://github.com/Wave-RF/WaveHouse/issues/613)'s E4. +- **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — only `GET`, `SET`, `MGET` and `PING`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, and a token key holding anything but a token (a string of another length, a list, a hash) is replaced, so a token lost to eviction, expiry or a restart is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing. Nothing selects this backend yet — the boot config and wiring arrive with [#613](https://github.com/Wave-RF/WaveHouse/issues/613)'s E4. - **redis_codec.go** — the key schema (`tokenKeys`, `bumpKeys`, `valueKey`) and the value frame: format, flags, expiry, the token list, the payload, zstd-compressed from `CompressMinBytes` when that is smaller. A stored value is capped at `MaxValueBytes` and a decoded one at eight times that, which refuses a zip bomb planted in a shared server; an unknown format, as a newer process writes during a rolling upgrade, is a miss. - **breaker.go** — the circuit breaker: `BreakerThreshold` (5) consecutive failures open it, and while open every operation skips the server at once; after `BreakerOpenFor` (5 s) one background `PING` decides whether it closes. Only a transport failure or a timeout counts against the server: any reply, an error reply or one the backend cannot use included, counts as a success, for operations and the probe alike, and a caller that gave up first counts as nothing. - **pending.go** — invalidations the server did not take, kept per token key (repeats coalesce) and retried by a background loop with backoff from 100 ms to 10 s until they land. Landing late is still correct: a fresh token orphans the pre-write entries and any fill made meanwhile. Past `PendingMax` (100,000) keys the set collapses to one tenant bump per affected tenant. `Close` makes one last attempt at the bumps still owed, past the breaker (an open one is why they are owed); what that attempt cannot deliver is lost, and the entries they would orphan are served until their TTL — the same failure as a worker stopping between an insert and its invalidation. diff --git a/internal/cache/redis.go b/internal/cache/redis.go index 30ed35ba6..f9ce1a0fe 100644 --- a/internal/cache/redis.go +++ b/internal/cache/redis.go @@ -70,7 +70,7 @@ type RedisConfig struct { DialTimeout time.Duration MaxValueBytes int // largest value stored, after compression CompressMinBytes int // zstd-compress values at least this large - VersionTTL time.Duration // idle lifetime of a version token, jittered ±10% + VersionTTL time.Duration // a token's lifetime from its last bump, jittered ±10%; reads do not extend it PendingMax int // undelivered bumps kept before collapsing to tenant bumps BreakerThreshold int // consecutive failures that open the breaker @@ -349,19 +349,26 @@ func (r *RedisCache) Lookup(ctx context.Context, id tenant.ID, sha string, deps vkey := valueKey(r.cfg.KeyPrefix, id, sha, deps) res := c.DoMulti(opCtx, c.B().Mget().Key(keys...).Build(), c.B().Get().Key(vkey).Build()) - for _, rr := range res { - if err := rr.Error(); err != nil && !rueidis.IsRedisNil(err) { - return r.lookupFailed(ctx, err) - } + if err := res[0].Error(); err != nil { + return r.lookupFailed(ctx, err) + } + // An error reply such as WRONGTYPE: the value key holds something else, + // which the fill's plain SET replaces, so it is a miss to fill. + valErr := res[1].Error() + _, valReplied := rueidis.IsRedisErr(valErr) + if valErr != nil && !rueidis.IsRedisNil(valErr) && !valReplied { + return r.lookupFailed(ctx, valErr) } r.record(ctx, nil) tokens, missing, foreign, err := readTokens(res[0]) if err != nil { return r.lookupFailed(ctx, err) } - val, err := res[1].AsBytes() - if err != nil && !rueidis.IsRedisNil(err) { - return r.lookupFailed(ctx, fmt.Errorf("%w: %w", errMalformedReply, err)) + var val []byte + if valErr == nil { + if val, err = res[1].AsBytes(); err != nil { + return r.lookupFailed(ctx, fmt.Errorf("%w: %w", errMalformedReply, err)) + } } if len(missing) > 0 || len(foreign) > 0 { @@ -423,8 +430,9 @@ func readTokens(res rueidis.RedisResult) (tokens []byte, missing, foreign []int, } // createTokens sets each missing token — only if still missing, as another -// process may create it first — replaces each foreign one, and reads them -// all back, in one round trip: the tokens share a slot, so the pipeline runs +// process may create it first — replaces each foreign one (a string that is +// not a token, or, on a second round trip, a key of another type), and reads +// them all back, in one round trip: the tokens share a slot, so the pipeline runs // in order on one node. A fresh token can only cause misses, so replacing // whatever held a token key is safe. func (r *RedisCache) createTokens(ctx context.Context, c rueidis.Client, keys []string, missing, foreign []int) ([]byte, error) { @@ -451,10 +459,15 @@ func (r *RedisCache) createTokens(ctx context.Context, c rueidis.Client, keys [] if err != nil { return nil, err } - if len(still) > 0 || len(bad) > 0 { - return nil, fmt.Errorf("%w: version token gone or replaced as it was written", errMalformedReply) + if len(still) == 0 && len(bad) == 0 { + return tokens, nil + } + // Still nil after SET NX: the key holds a list, hash or other non-string, + // which MGET reads as nil and NX will not overwrite. Replace it too. + if len(foreign) == 0 && len(still) > 0 { + return r.createTokens(ctx, c, keys, nil, still) } - return tokens, nil + return nil, fmt.Errorf("%w: version token gone or replaced as it was written", errMalformedReply) } // Set stores value with the tokens snap read, for ttl. A value over the size diff --git a/internal/cache/redis_integration_test.go b/internal/cache/redis_integration_test.go index 155caca92..422e372e7 100644 --- a/internal/cache/redis_integration_test.go +++ b/internal/cache/redis_integration_test.go @@ -342,10 +342,9 @@ func TestRedis_LostTokensAreMisses(t *testing.T) { fill(t, "q", deps) } -// A token key holding something that is not a token — another program -// under the prefix, a different token size mid-upgrade — is a reply, not a -// failure: it is replaced, which can only cause misses, and the breaker -// stays closed. +// A token or value key holding something else — another program under the +// prefix, a different token size mid-upgrade — is a reply, not a failure: +// it is replaced, which can only cause misses, and the breaker stays closed. func TestRedis_ForeignTokenIsReplaced(t *testing.T) { t.Parallel() ctx := context.Background() @@ -357,16 +356,33 @@ func TestRedis_ForeignTokenIsReplaced(t *testing.T) { _, snap, err := c.Lookup(ctx, "acme", "q", deps) require.NoError(t, err) require.NoError(t, c.Set(ctx, snap, []byte("rows"), time.Minute)) - require.NoError(t, r.Do(ctx, r.B().Set().Key(prefix+":{acme}:B:events").Value("abc").Build()).Error()) - - e, snap, err := c.Lookup(ctx, "acme", "q", deps) - require.NoError(t, err) - assert.Nil(t, e.Value) - assert.False(t, cache.Bypassed(c)) - require.NoError(t, c.Set(ctx, snap, []byte("new rows"), time.Minute)) - e, _, err = c.Lookup(ctx, "acme", "q", deps) + valueKeys, err := r.Do(ctx, r.B().Keys().Pattern(prefix+":q:*").Build()).AsStrSlice() require.NoError(t, err) - assert.Equal(t, "new rows", string(e.Value)) + require.Len(t, valueKeys, 1) + + for _, plant := range []struct { + name string + cmd rueidis.Completed + }{ + {"a short string under the table token", r.B().Set().Key(prefix + ":{acme}:B:events").Value("abc").Build()}, + {"", r.B().Del().Key(prefix + ":{acme}:T").Build()}, + {"a list under the tenant token", r.B().Rpush().Key(prefix + ":{acme}:T").Element("x").Build()}, + {"", r.B().Del().Key(valueKeys[0]).Build()}, + {"a hash under the value key", r.B().Hset().Key(valueKeys[0]).FieldValue().FieldValue("f", "v").Build()}, + } { + require.NoError(t, r.Do(ctx, plant.cmd).Error()) + if plant.name == "" { // the first half of a two-step plant + continue + } + e, snap, err := c.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err, plant.name) + assert.Nil(t, e.Value, plant.name) + assert.False(t, cache.Bypassed(c), plant.name) + require.NoError(t, c.Set(ctx, snap, []byte("new rows"), time.Minute), plant.name) + e, _, err = c.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err, plant.name) + assert.Equal(t, "new rows", string(e.Value), plant.name) + } } func dockerClient(t *testing.T) *testcontainers.DockerClient { From d67a46af2e63800bd8cb0721ff49e138e094787f Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Fri, 25 Sep 2026 00:54:26 -0400 Subject: [PATCH 09/45] test(cov): keep the unreachable Redis backend out of the e2e gate The e2e stack runs LocalCache and no config selects the Redis backend yet, so its files pulled the e2e suite to 56.4% (floor 60). The integration suite covers them against real servers, and per-suite excludes leave the merged total unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01EJr5tY4WQUy2sc4MbW67vL --- .testcoverage.yml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.testcoverage.yml b/.testcoverage.yml index aff1a694d..def851f08 100644 --- a/.testcoverage.yml +++ b/.testcoverage.yml @@ -73,3 +73,9 @@ exclude: - ^internal/settings/ - ^cmd/wavehouse/validate\.go$ - ^cmd/wavehouse/bootstrap\.go$ + # The Redis-compatible cache backend: the e2e stack runs LocalCache + # (no Redis server, and no config selects the backend yet — #613 E4), + # so the binary carries these files but e2e can never reach them; they + # pulled the e2e gate to 56.4%. The integration suite runs them against + # real servers, and the merged total still counts them. + - ^internal/cache/(redis|redis_codec|breaker|pending|metrics)\.go$ From c06808400e5a5eb24fdf65b26ac3531d31ed9884 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Fri, 25 Sep 2026 02:53:43 -0400 Subject: [PATCH 10/45] feat(config): cache.backend=redis selects the shared cache The boot config's cache.backend now takes redis, configured by a new cache.redis block (WH_CACHE_REDIS_*), and wireCache builds a RedisCache from it, reading the TLS files. A malformed block refuses boot; an unreachable server or a rejected password boots bypassed and keeps reconnecting. An integration test boots two instances over one Redis and one ClickHouse: an ingest through one is served fresh by the other inside the stale entry's TTL, and a paused Redis leaves queries succeeding. The e2e suite now runs against Redis, so its coverage exclude is gone. Part of #613 (E4). Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01EJr5tY4WQUy2sc4MbW67vL --- .testcoverage.yml | 6 - AGENTS.md | 4 +- CHANGELOG.md | 3 +- README.md | 2 +- config.yaml | 13 +- deployments/compose/dependencies.yaml | 13 + docs/src/content/docs/api.md | 2 +- docs/src/content/docs/architecture.md | 13 +- docs/src/content/docs/configuration.mdx | 78 ++++- docs/src/content/docs/deployment.md | 20 ++ docs/src/content/docs/development.md | 4 +- docs/src/content/docs/getting-started.md | 2 +- docs/src/content/docs/pipes.mdx | 2 +- docs/src/content/docs/settings-directory.mdx | 2 +- internal/app/app_test.go | 71 +++++ internal/app/wire.go | 41 ++- internal/config/backends.go | 40 ++- internal/config/backends_test.go | 2 +- internal/config/cache_redis.go | 153 ++++++++++ internal/config/cache_redis_test.go | 298 +++++++++++++++++++ scripts/orchestrator/main.go | 51 +++- tests/e2e/fixtures/config.yaml | 16 +- tests/integration/shared_cache_test.go | 271 +++++++++++++++++ 23 files changed, 1056 insertions(+), 51 deletions(-) create mode 100644 internal/config/cache_redis.go create mode 100644 internal/config/cache_redis_test.go create mode 100644 tests/integration/shared_cache_test.go diff --git a/.testcoverage.yml b/.testcoverage.yml index def851f08..aff1a694d 100644 --- a/.testcoverage.yml +++ b/.testcoverage.yml @@ -73,9 +73,3 @@ exclude: - ^internal/settings/ - ^cmd/wavehouse/validate\.go$ - ^cmd/wavehouse/bootstrap\.go$ - # The Redis-compatible cache backend: the e2e stack runs LocalCache - # (no Redis server, and no config selects the backend yet — #613 E4), - # so the binary carries these files but e2e can never reach them; they - # pulled the e2e gate to 56.4%. The integration suite runs them against - # real servers, and the merged total still counts them. - - ^internal/cache/(redis|redis_codec|breaker|pending|metrics)\.go$ diff --git a/AGENTS.md b/AGENTS.md index ebdd877c9..68c8f8df9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -31,10 +31,10 @@ Eighteen internal packages under `internal/` (plus `internal/testutil/` for shar - **`api/`** — Chi HTTP router, JWT/JWKS middleware (from `auth/`), ingest/query/structured-query/SSE/schema/DLQ/pipes handlers - **`app/`** — the process wiring: `New` builds every component from the boot config and the settings directory (each one wired in one place — what it opens, what it loops, what it releases — with the settings registry handed to its wiring function whole, the injection point of the per-tenant registry of #583: store-keyed getters for the handlers, `perTenant` for the async paths (with the tenant each message's `mq.Topic` names for the stream hub and the ingest worker), the `chconn.Pools` and the per-tenant `discoveries` reconciled from `AfterAdopt`, `shortestKeepalive` for the one setting folded over every tenant served, `gapWindows` and the `mq.max_bytes_gb` reconcile handing the MQ each served tenant's own gap window and byte budget, and `defaultSetting`/`onDefaultAdopt` for the one setting that still follows tenant `0`, a flat directory's ops-gate admin role; the auth verifiers are per tenant, reconfigured (rebuilt only on changed wiring) and pruned from `AfterAdopt`, the same hook's `Hub.Prune` ends the open streams of a tenant no longer served, and `wireCache`'s hook drops, through `LocalCache.Prune`, the cache version index of a tenant no longer served ([#262](https://github.com/Wave-RF/WaveHouse/issues/262))), `Run` drives the long-lived ones under one `errgroup` until the context is cancelled or one fails, `Close` releases them in reverse order. `cmd/wavehouse` and `tests/integration` both boot through it - **`auth/`** — JWT auth middleware: HMAC **or** JWKS verification with `alg` pinned to the active verifier, role extraction from a configurable claim path; always runs, never rejects (bad token → empty role + stashed reason). One verifier per tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 9): `Authenticator` keys them by `tenant.ID` — the request store's `settings.Store.Tenant()`, through an injected `TenantSource`; `tenant.Default` on the tenant-exempt routes — built from each tenant's `auth` block by `Reconfigure`, dropped by `Prune` once the tenant stops being served (rejected or removed), released by `Close`; the secrets (`Config`) are boot-level and shared. A JWKS key set is fetched off the boot and reload paths: until one has been stored the verifier is pending and a token-bearing request gets `503` + `Retry-After` from `api.refuseUnverifiable` (`auth.ErrVerifierPending`), never a `default_role` evaluation; refresh is library-managed (Eric, 2026-09-22), response capped at 1 MiB; the operator key's admin role is the request tenant's -- **`cache/`** — `Cache` interface → `LocalCache` (Ristretto: one pool for every tenant) + `VersionManager` (the invalidation index), and `RedisCache`, the Redis-compatible shared backend (random version tokens under the tenant's hash tag, one-round-trip lookups, bypass on failure behind a circuit breaker, deferred invalidations retried; built and tested, not yet selectable by config — [#613](https://github.com/Wave-RF/WaveHouse/issues/613) E4). Every key leads with the tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8) — `:query:` for a result and its singleflight, a version per tenant, per (tenant, table) and per (tenant, table, scope), keyed by name and bumped in place (one entry per live namespace however often it is bumped, [#262](https://github.com/Wave-RF/WaveHouse/issues/262)) — so no cached read or coalesced flight crosses tenants, a bump through `Invalidate` names one tenant's namespaces and no other's, and `InvalidateTenant` drops the tenant's index so its next key gets a process-unique generation, orphaning its every cached result in one step, pipe results included (no insert reaches a pipe result until [#343](https://github.com/Wave-RF/WaveHouse/pull/343)); `Lookup` returns a `Snapshot` of the versions it read and `Set` files the fill under it, so a write landing mid-query orphans the fill ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)), and every backend runs the conformance suite `internal/testutil/cachetest`; the one crossing is the wiring's, above the package: `internal/app` hands the ingest worker the cache through `sharedTables`, which repeats each of the worker's bumps under every tenant on the same ClickHouse address and database (`chconn.Pools.SharingTables`, whatever their user or tls block — they read the same tables), and orphans the whole cache of a tenant back on a pool after an absence, since it was out of that fan-out while away, or moved to another address or database, since it now reads other tables (story 6) +- **`cache/`** — `Cache` interface → `LocalCache` (Ristretto: one pool for every tenant) + `VersionManager` (the invalidation index), and `RedisCache`, the Redis-compatible shared backend (random version tokens under the tenant's hash tag, one-round-trip lookups, bypass on failure behind a circuit breaker, deferred invalidations retried; selected by `cache.backend: redis`, configured by the boot config's `cache.redis` block — [#613](https://github.com/Wave-RF/WaveHouse/issues/613)). Every key leads with the tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8) — `:query:` for a result and its singleflight, a version per tenant, per (tenant, table) and per (tenant, table, scope), keyed by name and bumped in place (one entry per live namespace however often it is bumped, [#262](https://github.com/Wave-RF/WaveHouse/issues/262)) — so no cached read or coalesced flight crosses tenants, a bump through `Invalidate` names one tenant's namespaces and no other's, and `InvalidateTenant` drops the tenant's index so its next key gets a process-unique generation, orphaning its every cached result in one step, pipe results included (no insert reaches a pipe result until [#343](https://github.com/Wave-RF/WaveHouse/pull/343)); `Lookup` returns a `Snapshot` of the versions it read and `Set` files the fill under it, so a write landing mid-query orphans the fill ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)), and every backend runs the conformance suite `internal/testutil/cachetest`; the one crossing is the wiring's, above the package: `internal/app` hands the ingest worker the cache through `sharedTables`, which repeats each of the worker's bumps under every tenant on the same ClickHouse address and database (`chconn.Pools.SharingTables`, whatever their user or tls block — they read the same tables), and orphans the whole cache of a tenant back on a pool after an absence, since it was out of that fan-out while away, or moved to another address or database, since it now reads other tables (story 6) - **`chconn/`** — `Pools`, one `Manager` (a `driver.Conn`) per distinct `Identity{Addr, Database, Username, Password, TLS}` tuple among the served tenants, reconciled from the settings registry's `AfterAdopt` after every reload ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 6): tenants naming one tuple share its pool, sized to their largest `max_open_conns`/`max_idle_conns`; a tenant whose tuple changed is repointed; a tuple no tenant names is released after the longest `query_timeout` among the tenants it had (never dials; a resize swaps the connection with the same grace). The boot config's `clickhouse.max_total_conns` bounds the open pools' `max_open_conns` together: boot refuses naming sum and ceiling; at a reload a resize above it keeps the pool's size, and a tuple that cannot be opened (the ceiling, an unreadable certificate, or options the driver refuses) leaves its tenants on the pool they had or on none — logged, retried by the next reload. Every consumer resolves its tenant's pool per call: `For` (nil for a tenant on no pool, a `503`), `Target` (the tenant's own HTTP wiring over its pool's TLS config), `SharingTables`, `Ping` (every pool at once, ready at the first answer). `HTTPClients` keeps one `http.Client` per TLS config - **`chsql/`** — dependency-free ClickHouse SQL helpers shared by `query`/`policy` (avoids an import cycle): `QuoteIdent` (backtick-quote every identifier) + `BindUnsafe` (reject names with a literal `?`) -- **`config/`** — YAML + env var config loading (cleanenv); strict on both sides (undeclared YAML key, unbound `WH_*` variable) and probes `data_dir` writability when a selected backend keeps state there (`NeedsDataDir`); `backends.go` holds each layer's `.backend` (only the in-process value today; `coord.backend` reserved) — boot is the validator, there is no dry run +- **`config/`** — YAML + env var config loading (cleanenv); strict on both sides (undeclared YAML key, unbound `WH_*` variable) and probes `data_dir` writability when a selected backend keeps state there (`NeedsDataDir`); `backends.go` holds each layer's `.backend` (the in-process value by default; `cache.backend` also takes `redis`, whose sub-block is `cache_redis.go`; `coord.backend` reserved) — boot is the validator, there is no dry run - **`dedupe/`** — `Deduplicator` interface → `Embedded` (Pebble: every tenant's seen ids in one instance at `data_dir/pebble`, each key led by its tenant, open while any tenant's store is — the layout is the implementation's call, and the wiring hands it `data_dir` once; its `Stats` feed the system gauges), wrapped by `Managed` whose open/closed state follows the hot-reloadable `dedupe.enabled` in the settings directory's `config.json`; `Stores` holds one `Managed` per tenant, built through a `Factory` (`func(tenant.ID) *Managed`, `Embedded.Tenant` in production; `Managed` opens its store through a function, so every backend gets the same switch), and reconciled from the registry's `AfterAdopt` hook — open exactly when the tenant is served with its switch on, closed with its seen ids kept otherwise ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) stories 7 and 3) - **`discovery/`** — `SchemaRegistry`, one per served tenant over a `Source` read once per refresh — the tenant's pool's connection and the database that pool was opened for, one snapshot, so a refused move keeps discovering the database the tenant's queries still use (`internal/app`'s `discoveries` builds, runs and stops them from `AfterAdopt` and `App.Close`: `RetryRefresh` until the first success, then `StartAutoRefresh` with a random first tick; `Lookup` answers `ErrNotLoaded` before the first success — the handlers' `503` with `Retry-After` — and `ErrUnknownTable` after; a failed loop attempt counts in `wavehouse_schema_refresh_failures_total{tenant}`), that introspects ClickHouse `system.columns` (name/type/nullability plus `default_expression` and 1-based `position`) and `system.tables` (each table's `create_table_query`, kept in-process and never serialized — an external-engine table renders its wiring there unconditionally — endpoint, bucket/host, database, username, S3 access key id; ClickHouse masks the password as `[HIDDEN]` from ~23.9, so the exposure is the topology, not the secret), records the server version, + `Validate()` for ingest payloads + `CanonicalizeTimestamps()` rewriting top-level `DateTime`/`DateTime64` column values to the canonical RFC 3339 UTC wire form pre-publish (Key Design Decision #19) - **`ingest/`** — Ingest worker pipeline (`worker.go`: JetStream input → per-table batch INSERT with DLQ output). The pipeline is **insert-only**. The wire format `EventMessage` (`types.go`) carries `{table_name, scope, received_timestamp, format, columns, row}` and nothing else — `row` is one positional `JSONCompactEachRow` line and `columns` names its slots, the table's **insertable** columns (a `MATERIALIZED`/`ALIAS` column cannot be named in an `INSERT`); the worker batches per (tenant, table, column list), the tenant read off each message's `mq.Topic`, and inserts each batch into its tenant's own ClickHouse (`chconn.Pools.Target`); the worker accepts whatever table name the envelope carries (table existence was already checked by the HTTP ingest handler, which `404`s an unknown table before publish; the worker doesn't re-validate), then bulk-INSERTs. In the embedded-NATS deployment (the default), the server runs with `DontListen: true` (`internal/mq/embedded.go`), so the only Publishers reachable on the `ingest.>` subjects are in-process Go code — today, only the HTTP `/v1/ingest?table={table}` handler. Non-insert mutations (`DELETE`/`UPDATE`/`TRUNCATE`/…) must go through `POST /v1/ops/query` under the admin role (the same `RequireAdmin` gate as the rest of `/v1/ops/*`), so non-admin callers never reach the proxy. A request with no token (or an invalid one) resolves to the `default_role`, which in a production config is not the admin role (setting them equal is a loudly-warned dev-only setting), so it can't reach this endpoint. Plus `Sweeper` (Active Sweeper for NATS message lifecycle) + `EventMessage`/`BufferConsumerName` types (`types.go`) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5bbe30535..ab628fdff 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **A Redis-compatible shared cache backend, built but not yet selectable** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row open a circuit breaker that skips the server until a probe succeeds, and deferred invalidations are retried until they land, collapsing to one tenant-wide bump per tenant past 100,000 keys. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. Nothing selects it yet: the `cache.backend` switch and the `cache.redis.*` settings arrive with the wiring (E4), so every deployment still runs `LocalCache`. Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression and server-stops-answering cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. +- **`cache.backend: redis` shares the query cache across instances** (`internal/config/{cache_redis,backends}.go` (+ tests), `internal/app/wire.go` (+ tests), `tests/integration/shared_cache_test.go`, `scripts/orchestrator/main.go`, `tests/e2e/fixtures/config.yaml`, `.testcoverage.yml`, `deployments/compose/dependencies.yaml`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md,settings-directory.mdx,getting-started.md,pipes.mdx,api.md,development.md}`, `README.md`, `AGENTS.md`): PR E4 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config's `cache.backend` now takes `redis`, configured by a new `cache.redis` block (`WH_CACHE_REDIS_*`): `addrs` (required), `mode` (`standalone`, `cluster`, `sentinel`), `sentinel_master`, `username`, `password` (a secret — set it through the environment), `db`, `tls.{enabled,ca_file,cert_file,key_file,server_name,insecure_skip_verify}`, `key_prefix` (`wh`), `timeout` (`100ms`), `dial_timeout` (`1s`), `max_value_bytes` (1 MiB), `compress_min_bytes` (`1024`; `-1` never compresses, since the loader reads a `0` in the file as unset) and `version_ttl` (`168h`). Every instance pointed at one server shares its cached results, and an insert on any instance invalidates every instance's. A malformed block — no address, an address without a port, an unknown mode, `db` other than `0` in cluster mode, an unreadable TLS file, a TLS key set while `tls.enabled` is off — refuses boot; an unreachable server, or one that rejects the password, does not: the process boots with the cache bypassed and keeps reconnecting, logging a rejected password at `ERROR` on every attempt, so a rotated secret cannot crash-loop every instance at once. `insecure_skip_verify`, and a `cache.redis.addrs` set while `cache.backend` is `local`, are logged at `WARN` at boot. The e2e suite now runs against a Redis testcontainer with `cache.backend: redis`, so the shared backend is exercised end to end, and the e2e coverage exclude for its files is gone; the unit and integration suites keep covering `local`. An integration test boots two instances over one Redis and one ClickHouse: a result one fills is a hit for the other, and a row ingested through one is served fresh by the other on its next query, well inside the stale entry's TTL; another pauses Redis and checks queries keep succeeding from ClickHouse, then turn back to hits. `deployments/compose/dependencies.yaml` gains an optional `redis` profile for local multi-instance work. The deployment guide gains a "Multiple instances and the shared cache" section: what each instance keeps to itself, what a reader on another instance can see and when, `maxmemory-policy`, and the metrics to alert on. +- **A Redis-compatible shared cache backend** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row open a circuit breaker that skips the server until a probe succeeds, and deferred invalidations are retried until they land, collapsing to one tenant-wide bump per tenant past 100,000 keys. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. `cache.backend: redis` selects it (the entry above). Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression and server-stops-answering cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. - **Each layer's implementation is chosen at boot** (`internal/config/{backends,config}.go` (+ tests), `internal/app/{app,wire}.go` (+ tests), `cmd/wavehouse/main.go`, `tests/integration/{setup,tenants}_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,settings-directory.mdx,architecture.md}`): the first step of running WaveHouse as more than one process ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config gains `mq.backend` (`WH_MQ_BACKEND`, default `embedded`), `cache.backend` (`WH_CACHE_BACKEND`, `local`), `dedupe.backend` (`WH_DEDUPE_BACKEND`, `pebble`) and `coord.backend` (`WH_COORD_BACKEND`, `local`). Each layer has only its in-process backend so far, and it is the default, so nothing changes for a config that sets none of them; a value with no backend refuses boot and names the valid ones. A backend's own settings will go in a `.` sub-block; no backend has settings yet, so every such sub-block is an unknown key for now and refuses boot. `internal/app` picks each implementation in one `switch` per layer (`wireMQ`, `wireCache`, `wireDedupe`), `data_dir` is probed only when a selected backend keeps state there (`Config.NeedsDataDir`), and boot logs at `WARN` each line of `Config.Warnings`, the combinations that are correct for one replica only once a shared queue exists. A `config.Config` built without `config.Load` must now name the `mq`, `cache` and `dedupe` backends: the zero value is not the default, and `app.New` refuses it. `coord.backend` is reserved: nothing reads it until the lease layer lands, and the sweeper still runs in every process. - **A tenant removed or rejected at runtime has its open streams ended** (`internal/stream/{hub,subscriber,bucket}.go` (+ tests), `internal/api/stream.go` (+ tests), `internal/api/router.go`, `internal/settings/{registry,tree}.go` (+ tests), `internal/ingest/worker.go` (+ tests), `internal/app/wire.go` (+ tests), `clients/ts/src/stream/sse.ts`, `docs/src/content/docs/{api,deployment,architecture,ingest-pipeline}.md`, `docs/src/content/docs/settings-directory.mdx`, `AGENTS.md`): story 3 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)). A `GET /v1/stream` used to outlive its tenant: the hub read no policy for it and withheld every row while the keepalive wheel held the connection open, so the client could not tell it from a quiet table. After every reload the hub now evicts the subscribers of each tenant no longer served, removed or rejected alike (`Hub.Prune`, through the close-once `Subscriber.Evict`), and the handler ends the stream: a gap-fill in progress included, and a stream `TenantMW` admitted just before the reload but registered just after it, which the handler checks for as it registers. The client's reconnect gets `404` (removed) or `503` (rejected); the SDK stops on the first and retries the second, resuming from `Last-Event-ID` once the folder is back — except in a browser going cross-origin where tenant `0` is not served or its CORS list does not admit the page, which cannot read either refusal (both are decorated from tenant `0`'s list) and re-dials as after a dropped connection. A flat directory never stops serving tenant `0`, so nothing changes there. Removing a tenant is deleting its folder, then reloading the whole directory, the last folder included: a server started with tenant folders reads the emptied directory as no folder left, where it read as a change of shape and the reload was rejected whole, leaving that tenant served; at boot an empty directory still reads as the four files, missing. Reloading the deleted folder by name leaves its tenant rejected. `GET /v1/health` keeps resolving a tenant, deliberately: its `404` tells a caller with no token no more than every tenant route's does, since they all answer before authenticating, and resolving answers a served tenant's ping from that tenant's own CORS list. A batch queued for a tenant with no ClickHouse connection — one no longer served, or one no pool could be opened for (such as by the connection ceiling) — skips the row-by-row retry, which no row of it could pass, and meets its DLQ switch once, whole, logged once per batch rather than twice per row; a tenant no longer served reads the switch as on, so its queued rows are parked under its own subject rather than dropped, inserted into another tenant's ClickHouse, or left unacked, redelivered for as long as the tenant is away and holding its queue's ack floor, which the purge waits on. Over a nested directory `/livez` no longer keeps naming a tenant that stopped being served before any tenant completed a first discovery: the diagnostic goes back to `no tenant has completed a first discovery yet`. - **One ClickHouse pool per tuple and one schema registry per tenant** (`internal/chconn/chconn.go` (+ tests), `internal/discovery/discovery.go` (+ tests), `internal/app/discoveries.go` (new), `internal/app/{app,wire}.go` (+ tests), `internal/api/{schema,ingest,structured_query,pipes,query,health,errors,clickhouse_exec}.go` (+ tests), `internal/stream/hub.go`, `internal/ingest/worker.go`, `internal/cache/{cache,local,version_manager}.go` (+ tests), `internal/testutil/{testutil,mocks}.go`, `tests/integration/{setup,tenants,boot_resilience,query_limits}_test.go`, `clients/ts/src/{schema,table,sql,client,types}.ts` (+ tests), `tests/e2e/sdk/admin.test.ts`, `docs/src/content/docs/{api,deployment,architecture,ingest-pipeline}.md`, `docs/src/content/docs/{settings-directory,configuration,access-control,reverse-proxy}.mdx`, `docs/src/content/docs/sdk/{admin,reference,queries}.md`, `AGENTS.md`): the second slice of story 6 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)), with no behavior change for a settings directory that holds the four files beyond the three noted at the end. The process opens one native pool per distinct `clickhouse.addr` / `database` / `username` / password / `tls` tuple among the served tenants (`chconn.Identity`, `chconn.Pools`), shared by the tenants naming it and sized to their largest `max_open_conns` and `max_idle_conns`; `http_port`, `http_scheme`, `headers` and `query_timeout` stay each tenant's own. Every reload reconciles the pools: a new tuple opens (never dials), a tenant whose tuple changed is repointed, a tuple no tenant names closes after the longest `query_timeout` among the tenants it had, and a pool whose largest ask changed is resized with the same grace. The boot config's `clickhouse.max_total_conns` now bounds the open pools together: boot is refused naming the sum and the ceiling; at a reload a resize above it is refused with the pool kept at its size, and a tuple that cannot be opened — the ceiling, a certificate file that cannot be read, or options the driver refuses — leaves its tenants on the pool they had (the keep-previous-wiring rule of the connection ceiling) or on none when they had none; both are logged and the next reload retries. A tenant on no pool fails closed: `503` with `Retry-After: 30` on `POST /v1/query`, `GET/POST /v1/pipes/{name}`, `POST /v1/ops/query` and `POST /v1/ops/schema/refresh`, ahead of the cache. Each served tenant gets a `discovery.SchemaRegistry` of its own over its pool, kept fresh by its own loop — the boot retry until the first success, then `schema.refresh_interval` with the first refresh at a random point within the interval so tenants adopted together do not refresh together — created and stopped from the settings reload and stopped under `App.Close`; `wavehouse_schema_refresh_failures_total{tenant}` counts a loop's failed attempts. `GET /v1/ops/schema`, `POST /v1/ops/schema/refresh` and `POST /v1/ops/query` take the strict `?tenant=` the pipe reads take (absent is tenant `0`, `400` malformed, `404` unknown, `503` rejected); the SDK sends it as the `tenant` option of `wh.schema.list()`, `wh.schema.refresh()`, `wh.from(t).schema()` and `wh.sql()`. Over a nested directory `/livez` (and `/v1/health`) is `503` with the latest discovery failure, naming its tenant, while no tenant has completed a first discovery, then `200` for the rest of the process lifetime; `/readyz` pings every open pool at once, is ready at the first answer, and names every pool that did not answer when none does — one tenant's ClickHouse outage is its log line and counter, never a probe failure. The ingest worker inserts each batch into its own tenant's ClickHouse — the tenant the message's topic names, through that tenant's HTTP wiring (`chconn.Pools.Target`); a tenant on no pool takes the failure path an unreachable ClickHouse takes — and its cache invalidation fans out to the tenants on the same address and database as the batch's tenant, whatever their user or `tls` block (they read the same tables), rather than to every known tenant, and a tenant adopted after an absence — rejected or removed, so out of that fan-out — or moved to another address or database has its cached structured-query results orphaned in one step (`Cache.InvalidateTenant`, a tenant generation in every version key), so a repaired folder never serves query rows cached before the inserts it missed (a pipe result names no table, so neither this nor any insert invalidates it: it stays until its TTL expires). Three changes reach the single-tenant directory too: a table lookup before the first discovery is a `503` with `Retry-After: 5` (`schema not loaded yet`) rather than a `404`, on `POST /v1/ingest`, `POST /v1/query` and `GET /v1/ops/schema` (the list included, where `[]` would read as no tables); the first periodic refresh fires at a random point within the interval rather than a full interval after boot; and a reload that moves `clickhouse.addr` or `clickhouse.database` now orphans the structured-query results cached before it, where they were served until their TTL. Until [#529](https://github.com/Wave-RF/WaveHouse/issues/529) every tenant's user authenticates with `WH_CH_PASSWORD`, so the tuple is in effect the address, database, username and `tls` block. diff --git a/README.md b/README.md index fce01a0a5..65f6da141 100644 --- a/README.md +++ b/README.md @@ -72,7 +72,7 @@ ClickHouse is a phenomenal OLAP database, but pointing a frontend right at it le If you're building user-facing analytics, WaveHouse is like **Supabase for ClickHouse**. Or an **open-source Tinybird** that pushes data to the frontend in real time over SSE, not just pull-based REST. - **Ingest** — async durable WAL (embedded NATS JetStream), `200 OK` instantly, background batch-flush; schema-validated against `system.columns`; optional ID-based dedup (idempotent ingest); dead-letter queue for failed inserts. -- **Query** — in-process Ristretto cache + `singleflight` coalescing; type-safe structured query AST; Tinybird-style named pipes (parameterized SQL endpoints). +- **Query** — result cache (in-process Ristretto, or a Redis shared by every instance) + `singleflight` coalescing; type-safe structured query AST; Tinybird-style named pipes (parameterized SQL endpoints). - **Real-time** — native SSE push, broadcast *before* the ClickHouse flush, with JetStream gap-fill for late/reconnecting clients. - **Security** — Hasura-style per-table, per-role column + row policies with JWT claim templating, defined in the hot-reloadable settings directory. - **Client** — `@wavehouse/sdk`: TypeScript client with query builder, live queries, streaming, and schema codegen; one runtime dependency (an SSE frame parser, ~1.4 KB gzipped). diff --git a/config.yaml b/config.yaml index 5519b78cd..aa9171e42 100644 --- a/config.yaml +++ b/config.yaml @@ -43,8 +43,8 @@ clickhouse: password: "" max_total_conns: 0 # ceiling on open native connections across pools; 0 = none -# Each layer's implementation, chosen at boot. Only the in-process backend -# exists for each today, and it is the default. +# Each layer's implementation, chosen at boot. The in-process backend is the +# default for each, and the only one for these three. mq: backend: embedded # NATS JetStream under /nats dedupe: @@ -52,11 +52,18 @@ dedupe: coord: backend: local # reserved: nothing is elected yet -# In-process L1 cache size. The query time-bucket +# The query-result cache: local (in-process, sized by l1_max_cost) or redis +# (one Redis-compatible server shared by every instance; see the redis block +# below and the Configuration page for every key). The query time-bucket # (query.timestamp_bucket_seconds) is a settings key. cache: backend: local l1_max_cost: 67108864 + # redis: # read only with backend: redis + # addrs: ["localhost:6379"] # `docker compose -f deployments/compose/dependencies.yaml --profile redis up -d` + # key_prefix: wh + # timeout: 100ms # per operation; slower is a miss, never a failed query + # The password is a secret: WH_CACHE_REDIS_PASSWORD, not this file. # Auth has no on/off switch — the JWT middleware always runs. A request with no # token, or an invalid/expired one, falls back to the policy default_role; diff --git a/deployments/compose/dependencies.yaml b/deployments/compose/dependencies.yaml index de147c726..4cc4f2fd6 100644 --- a/deployments/compose/dependencies.yaml +++ b/deployments/compose/dependencies.yaml @@ -33,5 +33,18 @@ services: timeout: 2s retries: 15 + # Optional shared cache for trying cache.backend=redis locally, e.g. two + # host-side instances on different ports: `--profile redis`. No + # persistence, and /data on tmpfs, so it leaves no volume behind. + redis: + profiles: [redis] + # Pinned to match internal/cache's integration suite. + image: redis:8.10.2-alpine + command: ["redis-server", "--save", "", "--appendonly", "no", "--maxmemory", "256mb", "--maxmemory-policy", "allkeys-lru"] + ports: + - "6379:6379" + tmpfs: + - /data + volumes: clickhouse-data: diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index 1634aaabc..c82739c07 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -560,7 +560,7 @@ The inbound request body is capped at 1 MiB; a body over the cap is rejected wit ### `GET/POST /v1/pipes/{name}` — Execute Named Pipe -Executes a pre-defined named query (pipe) with parameter binding. Parameters can be supplied via query string and/or JSON body. Results are cached in the shared L1 (Ristretto) with singleflight coalescing — same machinery as the structured query endpoint, keyed by [tenant](/deployment#multi-tenant-deployments) like it, and again, unlike `/v1/ops/query`. +Executes a pre-defined named query (pipe) with parameter binding. Parameters can be supplied via query string and/or JSON body. Results are cached in the query cache ([`cache.backend`](/configuration#backends): in-process, or a Redis shared by every instance) with singleflight coalescing — same machinery as the structured query endpoint, keyed by [tenant](/deployment#multi-tenant-deployments) like it, and again, unlike `/v1/ops/query`. **Query Parameters:** Any key matching a pipe parameter name. diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 2a06d9fff..6c59e5f16 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -28,7 +28,7 @@ flowchart TD MQ --> BC["Buffer Consumer
(batch flush)"] BC -.->|failed inserts| DLQ["DLQ"]:::fail - QH["Query Handler"] --> Cache["Cache
(Ristretto + singleflight)"] + QH["Query Handler"] --> Cache["Cache
(local or Redis + singleflight)"] SSH["SSE Handler"] --> Hub["Stream Hub
(project once per role)"] @@ -90,7 +90,7 @@ The API layer uses [Chi](https://github.com/go-chi/chi) for routing with Request ### `app/` — Process wiring - **app.go** — `New(ctx, Options)` builds every component from the boot config (`Options.Config`) and the settings directory it names, in dependency order: settings registry, observability (after which each `Config.Warnings` line is logged at `WARN`), ClickHouse pools, schema discovery, the dedupe stores, embedded NATS (ingest + DLQ streams), cache, sweeper, streaming (hub, MQ→hub bridge, keepalive wheel), ingest worker, auth, reload triggers, HTTP. Each is one `component` value — what it opens, what it loops, what it releases — so a failure part-way releases what was already opened and returns the error. `Run(ctx)` drives every loop under one `errgroup` until `ctx` is canceled (a clean stop: every loop drains, the API server and the ingest worker within `server.shutdown_timeout`; open SSE streams are ended as the drain begins rather than waited on) or a component fails, which stops the rest and returns that error. `Close(ctx)` releases what `New` opened, newest first, under the caller's release budget (`ReleaseTimeout`, 5s), a real bound: a remote implementation's close gives up at the deadline itself, and a close that ignores the context (the local stores) is abandoned at it, with the components below it left unreleased rather than overlapping it, both named in the error — and then flushes telemetry under its own 3s budget, so the flush that reports on the stop is never handed a deadline a slow close already spent. The SIGHUP registration is released last of all. `Handler`, `Registry`, and `MQ` expose the pieces a harness needs; `Options.Listener` lets one serve the API on its own listener instead of `server.port`. -- **wire.go** — one `wire*` function per component, each handed the settings registry whole and deriving the per-call getters the internal packages take (`DLQFor`, `DedupeFor`, `GapWindow`, …) and registering its `AfterAdopt` hook there where it has one. A layer with a choice of implementation — `wireMQ`, `wireCache`, `wireDedupe` — picks it there and nowhere else, in a `switch` on the boot config's `.backend` with one case per backend; the default case refuses boot, which only a `config.Config` built without `config.Load` reaches, since `Validate` refuses a value no case handles. Those wiring functions are where the per-tenant registry of [#583](https://github.com/Wave-RF/WaveHouse/issues/583) is injected, not `main`: `wireSettings` opens the `settings.Registry`, the HTTP handlers get store-keyed getters (method expressions such as `(*settings.Store).Policy`), and `perTenant` adapts a store accessor into the `func(tenant.ID) T` getter the async packages take, with the tenant each message's `mq.Topic` names for the stream hub and the ingest worker — a tenant the registry is not serving is logged and read as the zero value, except in `dlqFor`, the ingest worker's DLQ switch, where it reads as on so a message the worker cannot read is parked rather than dropped, and a removed or rejected tenant's queued rows are parked rather than left unacked, where they would hold the ack floor and stop the sweeper. The ClickHouse pools (`chconn.Pools`) and the per-tenant schema registries (`discoveries`, in `discoveries.go`) are reconciled from `AfterAdopt` after every reload ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 6): `wireClickHouse` builds each served tenant's `chconn.Member` from its store and logs what the reconcile refused; `wireDiscovery` builds a registry over `pools.For` for each newly served tenant — a flat directory's tenant `0` refreshed synchronously first, as before — runs its loop under the App's stop context, stops the loop of a tenant no longer served, and drives the `BootState` from the first tenant's first discovery, sticky from there; before that, a diagnostic naming a tenant a reload stopped serving goes back to the no-tenant one. The handlers resolve both per request through store-keyed getters (`chConnFor`, `registryFor`, `chTargetFor`, `queryTimeout`), the hub and the ingest worker through tenant-keyed ones (`discoveries.For`, `pools.Target`) called with the tenant the message's topic names; a tenant on no pool is an untyped nil connection, the handlers' `503`. The ingest worker is handed the cache through `sharedTables`, which bumps each namespace the worker invalidates under every tenant on the same ClickHouse address and database (`pools.SharingTables`), and the pools hook orphans the whole cache — structured-query and pipe results — of a tenant back on a pool after an absence (`Cache.InvalidateTenant`), since it was out of that fan-out while away, and of a tenant moved to another address or database, since it now reads other tables (both returned by `Pools.Reconcile`). The one setting that still follows the default tenant is read per request, the admin role of a flat directory's ops gate: `defaultSetting` reads the store tenant `0` last adopted (`App.defaultStore`, tracked by an `onDefaultAdopt` hook that runs only after a reload that adopted it), so a `0` folder that a reload rejects or removes leaves it as it was. The auth verifiers are per tenant: `wireAuth` builds one for each tenant being served, its `AfterAdopt` hook reconfigures the adopted tenants' (rebuilt only when their wiring changed) and prunes the ones no longer served, and the operator key's admin role is read from the request tenant's policy. `wireStreaming`'s hook prunes the stream hub the same way (`Hub.Prune`, with the one `served` predicate the auth, dedupe and cache hooks use too), ending the open streams of a tenant no longer served, and `wireCache`'s hook prunes the cache's version index the same way (`LocalCache.Prune`, [#262](https://github.com/Wave-RF/WaveHouse/issues/262)), so a tenant no longer served stops holding it. One setting is shared by folding over the tenants being served rather than by following tenant `0`: the keepalive wheel runs at the shortest `stream.keepalive_interval` among them (`shortestKeepalive`), re-derived after every reload the registry applies — an adoption, a rejection, or a removal — so a dropped tenant's interval leaves the wheel at once ([#597](https://github.com/Wave-RF/WaveHouse/issues/597)). The sweeper is handed each served tenant's own `stream.gap_window_minutes` (`gapWindows`, read every sweep), since each tenant's events have a queue of their own. The dedupe stores are per tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 7): `wireDedupe`'s `pebble` case builds a `dedupe.Stores` over the `Tenant` factory of the embedded Pebble implementation (`dedupe.NewEmbedded`), handing it `data_dir` once; the implementation decides where every tenant's store lives — one instance, each key led by its tenant (story 3) — and one reconcile closure, the boot apply and the `AfterAdopt` hook alike, sets every store to what the registry says: open exactly when its tenant is served with `dedupe.enabled` on, closed with its seen ids kept when the tenant is switched off, rejected, or removed. An instance that cannot open follows the registry's rule for the shape: fatal at boot over a flat directory, fail-closed for every tenant with dedupe on over a nested one. The system gauges report that one instance's figures (`Embedded.Stats`), not a sum over tenants. The ingest handler picks the tenant's store off the request's `settings.Store` (`Store.Tenant()`). The reload triggers only start in `Run`, after `New` has registered every hook, so the watcher's first reload already drives all of them: SIGHUP in both shapes, the directory watcher for a flat directory only. `wireMQ`'s `embedded` case hands each served tenant's `mq.max_bytes_gb` to `mq.Broker.SetMaxBytes` at boot and again after every reload, under the App's stop context, which opens that tenant's queue the first time; a queue that cannot be opened or resized follows the registry's rule for the shape — fatal at boot over a flat directory, logged over a nested one — and is retried by the next reload. How the budget is split across the tenant's streams, the time bounds, the rollback, and the dead-letter shrink guard are `internal/mq`'s. +- **wire.go** — one `wire*` function per component, each handed the settings registry whole and deriving the per-call getters the internal packages take (`DLQFor`, `DedupeFor`, `GapWindow`, …) and registering its `AfterAdopt` hook there where it has one. A layer with a choice of implementation — `wireMQ`, `wireCache`, `wireDedupe` — picks it there and nowhere else, in a `switch` on the boot config's `.backend` with one case per backend; the default case refuses boot, which only a `config.Config` built without `config.Load` reaches, since `Validate` refuses a value no case handles. `wireCache` has two: `local`, the in-process `LocalCache`, and `redis`, the shared `RedisCache` built from the `cache.redis` block, which boots bypassed rather than failing when its server is unreachable. Those wiring functions are where the per-tenant registry of [#583](https://github.com/Wave-RF/WaveHouse/issues/583) is injected, not `main`: `wireSettings` opens the `settings.Registry`, the HTTP handlers get store-keyed getters (method expressions such as `(*settings.Store).Policy`), and `perTenant` adapts a store accessor into the `func(tenant.ID) T` getter the async packages take, with the tenant each message's `mq.Topic` names for the stream hub and the ingest worker — a tenant the registry is not serving is logged and read as the zero value, except in `dlqFor`, the ingest worker's DLQ switch, where it reads as on so a message the worker cannot read is parked rather than dropped, and a removed or rejected tenant's queued rows are parked rather than left unacked, where they would hold the ack floor and stop the sweeper. The ClickHouse pools (`chconn.Pools`) and the per-tenant schema registries (`discoveries`, in `discoveries.go`) are reconciled from `AfterAdopt` after every reload ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 6): `wireClickHouse` builds each served tenant's `chconn.Member` from its store and logs what the reconcile refused; `wireDiscovery` builds a registry over `pools.For` for each newly served tenant — a flat directory's tenant `0` refreshed synchronously first, as before — runs its loop under the App's stop context, stops the loop of a tenant no longer served, and drives the `BootState` from the first tenant's first discovery, sticky from there; before that, a diagnostic naming a tenant a reload stopped serving goes back to the no-tenant one. The handlers resolve both per request through store-keyed getters (`chConnFor`, `registryFor`, `chTargetFor`, `queryTimeout`), the hub and the ingest worker through tenant-keyed ones (`discoveries.For`, `pools.Target`) called with the tenant the message's topic names; a tenant on no pool is an untyped nil connection, the handlers' `503`. The ingest worker is handed the cache through `sharedTables`, which bumps each namespace the worker invalidates under every tenant on the same ClickHouse address and database (`pools.SharingTables`), and the pools hook orphans the whole cache — structured-query and pipe results — of a tenant back on a pool after an absence (`Cache.InvalidateTenant`), since it was out of that fan-out while away, and of a tenant moved to another address or database, since it now reads other tables (both returned by `Pools.Reconcile`). The one setting that still follows the default tenant is read per request, the admin role of a flat directory's ops gate: `defaultSetting` reads the store tenant `0` last adopted (`App.defaultStore`, tracked by an `onDefaultAdopt` hook that runs only after a reload that adopted it), so a `0` folder that a reload rejects or removes leaves it as it was. The auth verifiers are per tenant: `wireAuth` builds one for each tenant being served, its `AfterAdopt` hook reconfigures the adopted tenants' (rebuilt only when their wiring changed) and prunes the ones no longer served, and the operator key's admin role is read from the request tenant's policy. `wireStreaming`'s hook prunes the stream hub the same way (`Hub.Prune`, with the one `served` predicate the auth, dedupe and cache hooks use too), ending the open streams of a tenant no longer served, and `wireCache`'s hook prunes the cache's version index the same way (`LocalCache.Prune`, [#262](https://github.com/Wave-RF/WaveHouse/issues/262)), so a tenant no longer served stops holding it. One setting is shared by folding over the tenants being served rather than by following tenant `0`: the keepalive wheel runs at the shortest `stream.keepalive_interval` among them (`shortestKeepalive`), re-derived after every reload the registry applies — an adoption, a rejection, or a removal — so a dropped tenant's interval leaves the wheel at once ([#597](https://github.com/Wave-RF/WaveHouse/issues/597)). The sweeper is handed each served tenant's own `stream.gap_window_minutes` (`gapWindows`, read every sweep), since each tenant's events have a queue of their own. The dedupe stores are per tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 7): `wireDedupe`'s `pebble` case builds a `dedupe.Stores` over the `Tenant` factory of the embedded Pebble implementation (`dedupe.NewEmbedded`), handing it `data_dir` once; the implementation decides where every tenant's store lives — one instance, each key led by its tenant (story 3) — and one reconcile closure, the boot apply and the `AfterAdopt` hook alike, sets every store to what the registry says: open exactly when its tenant is served with `dedupe.enabled` on, closed with its seen ids kept when the tenant is switched off, rejected, or removed. An instance that cannot open follows the registry's rule for the shape: fatal at boot over a flat directory, fail-closed for every tenant with dedupe on over a nested one. The system gauges report that one instance's figures (`Embedded.Stats`), not a sum over tenants. The ingest handler picks the tenant's store off the request's `settings.Store` (`Store.Tenant()`). The reload triggers only start in `Run`, after `New` has registered every hook, so the watcher's first reload already drives all of them: SIGHUP in both shapes, the directory watcher for a flat directory only. `wireMQ`'s `embedded` case hands each served tenant's `mq.max_bytes_gb` to `mq.Broker.SetMaxBytes` at boot and again after every reload, under the App's stop context, which opens that tenant's queue the first time; a queue that cannot be opened or resized follows the registry's rule for the shape — fatal at boot over a flat directory, logged over a nested one — and is retried by the next reload. How the budget is split across the tenant's streams, the time bounds, the rollback, and the dead-letter shrink guard are `internal/mq`'s. ### `stream/` — SSE keepalive & fan-out @@ -112,7 +112,7 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ - **cache.go** — `Cache` interface: `Lookup`, `Set`, `Invalidate`, `InvalidateTenant`, `Close`, plus `QueryTimeToTTL`, which sets a result's TTL from how long its query took (10 s floor, 1 h ceiling). Every entry is one tenant's: `Lookup` takes the tenant, the caller's query key — `:query:`, built by the two cached handlers in `api/` (`queryCacheKey`, with the tenant read off the request's store — `settings.Store.Tenant`), which use it as their [singleflight](https://pkg.go.dev/golang.org/x/sync/singleflight) key too — and the `Namespace`s the result depends on, each of that tenant (another tenant's is `ErrForeignDependency`) and naming a table and scope: one for a structured query, none yet for a pipe (a pipe's table dependencies are [#343](https://github.com/Wave-RF/WaveHouse/pull/343)). `Lookup` returns the `Entry` (a nil value is a miss) and a `Snapshot` of the versions it read; on a miss the handler runs the query and passes that snapshot to `Set`, so a result is filed under the versions read *before* its query ran, and a write that lands while it runs orphans the fill rather than re-homing pre-write rows under the post-write versions ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)). The singleflight leader's snapshot is the one used. `Set` returns an error only when the backend failed; a value the cache declines — larger than it keeps, a non-positive TTL, a zero snapshot — is not one. What a hit, a miss and a bump mean is pinned by the conformance suite every backend runs, `internal/testutil/cachetest`. - **local.go** — `LocalCache`, the in-process L1 on [Ristretto](https://github.com/dgraph-io/ristretto): one pool shared by every tenant (a heavier tenant holds more of it), sized by the boot config's `cache.l1_max_cost`. - **version_manager.go** — `VersionManager`, the invalidation index behind `Invalidate` and `InvalidateTenant`: one version per tenant, per (tenant, table) and per (tenant, table, scope), each keyed by its name alone and bumped in place, so the index holds one entry per live tenant, table and scope however often each is bumped ([#262](https://github.com/Wave-RF/WaveHouse/issues/262)). A query key folds the tenant's version and, for each dependency, its tenant's, table's and scope's, so bumping a table (a scopeless write) orphans every scope of it, and bumping one scope orphans that scope and the whole-table view — scope is reserved and empty today, so every write is the whole-table bump — all without touching the pool. The tenant leads every key ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8): the same table under two tenants is two namespaces, so a bump through `Invalidate` under one tenant never touches — and a read under one tenant is never served — the other's results. A tenant's version is a *generation*, unique within the process and handed out by the first key built for the tenant; `BumpTenant` (behind `InvalidateTenant`) drops the tenant's whole index, so the next key gets a fresh generation no cached entry folds, orphaning every cached result of the tenant in one step — a pipe result with no dependencies, and a table no bump ever keyed, included — for a tenant back on a pool after an absence from the fan-out, or moved to another address or database (story 6). `LocalCache.Prune` does the same for every tenant no longer served, which `internal/app` runs after each settings reload, so a tenant removed or rejected stops holding its index. A table bump drops the table's scope versions with it, since every key they were folded into also folds the old table version; and a bump of a tenant with no index is a no-op, since no key folds its next generation yet. The index is per tenant; the cross-tenant invalidation an insert into a shared table needs is not the index's but the wiring's: `internal/app` hands the ingest worker a cache (`sharedTables`) that repeats each bump under every tenant on the same ClickHouse address and database. -- **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — only `GET`, `SET`, `MGET` and `PING`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, and a token key holding anything but a token (a string of another length, a list, a hash) is replaced, so a token lost to eviction, expiry or a restart is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing. Nothing selects this backend yet — the boot config and wiring arrive with [#613](https://github.com/Wave-RF/WaveHouse/issues/613)'s E4. +- **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — only `GET`, `SET`, `MGET` and `PING`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, and a token key holding anything but a token (a string of another length, a list, a hash) is replaced, so a token lost to eviction, expiry or a restart is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing. `cache.backend: redis` selects it: `internal/app`'s `wireCache` maps the boot config's `cache.redis` block onto `RedisConfig`, reading the TLS files, and releases it with the other components. - **redis_codec.go** — the key schema (`tokenKeys`, `bumpKeys`, `valueKey`) and the value frame: format, flags, expiry, the token list, the payload, zstd-compressed from `CompressMinBytes` when that is smaller. A stored value is capped at `MaxValueBytes` and a decoded one at eight times that, which refuses a zip bomb planted in a shared server; an unknown format, as a newer process writes during a rolling upgrade, is a miss. - **breaker.go** — the circuit breaker: `BreakerThreshold` (5) consecutive failures open it, and while open every operation skips the server at once; after `BreakerOpenFor` (5 s) one background `PING` decides whether it closes. Only a transport failure or a timeout counts against the server: any reply, an error reply or one the backend cannot use included, counts as a success, for operations and the probe alike, and a caller that gave up first counts as nothing. - **pending.go** — invalidations the server did not take, kept per token key (repeats coalesce) and retried by a background loop with backoff from 100 ms to 10 s until they land. Landing late is still correct: a fresh token orphans the pre-write entries and any fill made meanwhile. Past `PendingMax` (100,000) keys the set collapses to one tenant bump per affected tenant. `Close` makes one last attempt at the bumps still owed, past the breaker (an open one is why they are owed); what that attempt cannot deliver is lost, and the entries they would orphan are served until their TTL — the same failure as a worker stopping between an insert and its invalidation. @@ -120,9 +120,10 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ ### `config/` — Configuration -- **config.go** — Loads *boot* configuration from a YAML file with environment variable overrides (using [cleanenv](https://github.com/ilyakaznacheev/cleanenv)); every key has a `WH_`-prefixed env var. Boot config is only what can't change under a running process — the implementation each layer runs on, resource sizing, listeners, observability exporters, the settings-directory path, and the secrets (`clickhouse.password`, `auth.jwt_secret`, `auth.operator_key`). Everything tenant-tunable lives in the settings directory (`settings/`). Both sources are strict: `Load` refuses to boot naming every YAML key the struct doesn't declare (`strict.go`) and every `WH_*` environment variable no field binds (`check.go`), so a tunable that moved to the settings directory can't be read, ignored, and believed. Boot is the validator for this half — there is no dry-run command. See [Configuration Reference](/configuration). +- **config.go** — Loads *boot* configuration from a YAML file with environment variable overrides (using [cleanenv](https://github.com/ilyakaznacheev/cleanenv)); every key has a `WH_`-prefixed env var. Boot config is only what can't change under a running process — the implementation each layer runs on, resource sizing, listeners, observability exporters, the settings-directory path, and the secrets (`clickhouse.password`, `cache.redis.password`, `auth.jwt_secret`, `auth.operator_key`). Everything tenant-tunable lives in the settings directory (`settings/`). Both sources are strict: `Load` refuses to boot naming every YAML key the struct doesn't declare (`strict.go`) and every `WH_*` environment variable no field binds (`check.go`), so a tunable that moved to the settings directory can't be read, ignored, and believed. Boot is the validator for this half — there is no dry-run command. See [Configuration Reference](/configuration). - **check.go** — `rejectUnboundEnv` is the environment half of the strict loader: `unboundEnv` walks the struct's `env` tags (plus the two process-level names, `WH_CONFIG` and `WH_LOG_LEVEL`) against the environment; `CheckDataDir` probes `data_dir` — run by `main` right after `Load` when `Config.NeedsDataDir` says a selected backend keeps state there, so an unusable `data_dir` refuses boot before anything dials out. It refuses an empty value (reachable through `WH_DATA_DIR=`) outright rather than probing the working directory; a path that exists and is not a directory; a dangling symlink at `data_dir` or any component above it (the walk to the nearest existing ancestor uses `Lstat`, so a failed mount is not skipped over as "does not exist"); and a directory the process cannot write to — or, when it does not exist, an unwritable nearest ancestor — probed by creating and removing one temp file. A permission denial, on the probe or on reaching the path through a parent without search permission, carries the UID-65532 hint, since a bind mount owned by root is the typical cause. -- **backends.go** — the `.backend` keys: one string type per layer (`MQBackend`, `CacheBackend`, `DedupeBackend`, `CoordBackend`), each with its list of the backends this build has, and a `validate` per layer block (`checkBackend`, run by `validateBackends` at the end of `Validate`), which refuses a value not on the list and names the ones that are. A backend's own settings go in a `.` sub-block that is its `validate`'s case to check. `Distributed` reports whether the MQ is shared with other processes, `NeedsDataDir` whether a selected backend keeps state under `data_dir`, and `Warnings` returns the valid combinations that are correct for one replica only (a shared MQ over a local cache or Pebble dedupe), which `app.New` logs at `WARN`. +- **backends.go** — the `.backend` keys: one string type per layer (`MQBackend`, `CacheBackend`, `DedupeBackend`, `CoordBackend`), each with its list of the backends this build has, and a `validate` per layer block (`checkBackend`, run by `validateBackends` at the end of `Validate`), which refuses a value not on the list and names the ones that are. A backend's own settings go in a `.` sub-block that is its `validate`'s case to check. `Distributed` reports whether the MQ is shared with other processes, `NeedsDataDir` whether a selected backend keeps state under `data_dir`, and `Warnings` returns what a valid configuration is still likely to get wrong — the combinations that are correct for one replica only (a shared MQ over a local cache or Pebble dedupe), a `cache.redis` block that is not read, certificate verification turned off — which `app.New` logs at `WARN`. +- **cache_redis.go** — `CacheRedisConfig`, the `cache.redis` sub-block, and its checks: an address, each `host:port`, a known mode, a sentinel's master name, `db` 0 in cluster mode, positive timeouts and sizes, a `version_ttl` of at least 2 s, and `compress_min_bytes` positive or `-1` (never: cleanenv reads a `0` in the file as unset and applies the default, so `0` cannot mean off). `CacheRedisTLS.Config` builds the `tls.Config`, reading the files; `Validate` calls it so an unreadable file refuses boot, and `internal/app` calls it again to build the connection. A TLS key set while `tls.enabled` is off is an error rather than a plaintext connection. - **strict.go** — `rejectUnknownKeys`, the YAML half: re-reads the file as a generic tree and walks it against the struct's `yaml` tags, listing every key the struct doesn't declare. cleanenv itself is lenient by design, which is exactly wrong for boot config once keys have moved to the settings directory. - **persistence.go** — `WarnIfFreshDataDir` logs the startup `WARN` when `data_dir` is missing or empty (on a redeploy, the sign that the volume didn't persist); `LogStorageInitError` attaches the UID-65532 `permissionHint` to a NATS or Pebble open failure that looks like a permission denial — the same hint string `CheckDataDir` uses. @@ -366,7 +367,7 @@ Client GET /v1/stream | Analytics DB | ClickHouse | Primary data store + schema source of truth | | Message Queue | NATS + JetStream | Durable event streaming | | L1 Cache | Ristretto v2 | In-process memory cache | -| Shared cache | [rueidis](https://github.com/redis/rueidis) | Redis-compatible client for the shared backend (not yet selectable) | +| Shared cache | [rueidis](https://github.com/redis/rueidis) | Redis-compatible client for the shared backend (`cache.backend: redis`) | | Embedded KV | Pebble | Optional deduplication | | Config | cleanenv | YAML + env var config loading | | Release | GoReleaser | Cross-platform binary builds | diff --git a/docs/src/content/docs/configuration.mdx b/docs/src/content/docs/configuration.mdx index 193a6c21b..82744a359 100644 --- a/docs/src/content/docs/configuration.mdx +++ b/docs/src/content/docs/configuration.mdx @@ -39,16 +39,16 @@ This page is boot config only — what the platform operator owns (wiring, lifec ### Backends -Each layer's implementation is chosen once, at boot. Today every layer has one backend, the in-process one, and it is the default, so a config that sets none of these keys runs as it always has. A value this build has no backend for refuses boot and names the valid ones. +Each layer's implementation is chosen once, at boot. Every layer defaults to its in-process backend, so a config that sets none of these keys runs as it always has; the cache also has a shared one, `redis`. A value this build has no backend for refuses boot and names the valid ones. | YAML Key | Env Var | Default | Description | | --- | --- | ------- | ----------- | | `mq.backend` | `WH_MQ_BACKEND` | `embedded` | The message queue. `embedded`: NATS JetStream inside this process, under `/nats`. It listens on no port, so no other process can reach its queue. | -| `cache.backend` | `WH_CACHE_BACKEND` | `local` | The query-result cache. `local`: in this process, sized by `cache.l1_max_cost`. | +| `cache.backend` | `WH_CACHE_BACKEND` | `local` | The query-result cache. `local`: in this process, sized by `cache.l1_max_cost`. `redis`: one Redis-compatible server shared by every process, configured by [`cache.redis`](#cache), so an insert one process makes invalidates what every process has cached. | | `dedupe.backend` | `WH_DEDUPE_BACKEND` | `pebble` | Where ingest dedupe keeps the event ids it has seen. `pebble`: in this process, under `/pebble`, open while any tenant has dedupe on. | | `coord.backend` | `WH_COORD_BACKEND` | `local` | Reserved for the leases that will elect work only one process may do at a time, such as the sweeper. Nothing is elected yet: every process runs its own sweeper, and `local`, the only value, changes nothing. | -Settings for one backend will go in a sub-block named after it, `.`, read only when that backend is selected. No backend has settings yet, so today any such sub-block, `mq.embedded` included, is an unknown key and refuses boot. `mq` and `dedupe` also appear in the settings directory's `config.json`, with different keys (`mq.max_bytes_gb`, `dedupe.enabled`, …): those are per-tenant tunables and stay there, and one written in `config.yaml` refuses boot as an unknown key. +Settings for one backend go in a sub-block named after it, `.`, read only when that backend is selected. `cache.redis` is the only one so far; any other, `mq.embedded` included, is an unknown key and refuses boot. A `cache.redis.addrs` set while `cache.backend` is `local` is logged at `WARN` at boot, since the block is not read. `mq` and `dedupe` also appear in the settings directory's `config.json`, with different keys (`mq.max_bytes_gb`, `dedupe.enabled`, …): those are per-tenant tunables and stay there, and one written in `config.yaml` refuses boot as an unknown key. ### Server @@ -120,7 +120,34 @@ Each tenant's queue has its own disk budget, `mq.max_bytes_gb`, a hot-reloadable | YAML Key | Env Var | Default | Description | | --- | --- | ------- | ----------- | -| `cache.l1_max_cost` | `WH_CACHE_L1_MAX_COST` | `67108864` | Maximum L1 cache size in bytes (~64 MB). The time-range bucket structured queries normalize to is `query.timestamp_bucket_seconds` in the [Settings Directory](/settings-directory#configjson-keys). | +| `cache.l1_max_cost` | `WH_CACHE_L1_MAX_COST` | `67108864` | Maximum size in bytes (~64 MB) of the `local` backend's in-process cache. The time-range bucket structured queries normalize to is `query.timestamp_bucket_seconds` in the [Settings Directory](/settings-directory#configjson-keys). | + +The `redis` backend's settings, read only when `cache.backend` is `redis`. It runs on Redis, Valkey, Dragonfly, ElastiCache (including Serverless) and MemoryDB: it sends only `GET`, `SET`, `MGET` and `PING`. [Deployment](/deployment#multiple-instances-and-the-shared-cache) covers sizing, `maxmemory-policy` and what a reader on another instance can see. + +| YAML Key | Env Var | Default | Description | +| --- | --- | ------- | ----------- | +| `cache.redis.addrs` | `WH_CACHE_REDIS_ADDRS` | — | **Required** with `backend: redis.` `host:port` of the server; several are a cluster's seed nodes or the sentinels. Comma-separated in the env var. | +| `cache.redis.mode` | `WH_CACHE_REDIS_MODE` | `standalone` | `standalone`, `cluster` or `sentinel`. | +| `cache.redis.sentinel_master` | `WH_CACHE_REDIS_SENTINEL_MASTER` | — | The master set name. Required with `mode: sentinel`. | +| `cache.redis.username` | `WH_CACHE_REDIS_USERNAME` | — | ACL user. Empty uses the server's `default` user. | +| `cache.redis.password` | `WH_CACHE_REDIS_PASSWORD` | — | A secret: set it through the environment (or your secret store's env injection), not in a tracked `config.yaml`. | +| `cache.redis.db` | `WH_CACHE_REDIS_DB` | `0` | Database number (`SELECT`). Standalone and sentinel only: a cluster has only database `0`, and any other value refuses boot. | +| `cache.redis.tls.enabled` | `WH_CACHE_REDIS_TLS_ENABLED` | `false` | Connect over TLS, verifying the server against the system roots or `ca_file`. Any other `tls` key set while this is off refuses boot, rather than connecting in plaintext. | +| `cache.redis.tls.ca_file` | `WH_CACHE_REDIS_TLS_CA_FILE` | — | PEM file of the authorities to trust instead of the system roots. | +| `cache.redis.tls.cert_file` | `WH_CACHE_REDIS_TLS_CERT_FILE` | — | Client certificate (PEM) for mutual TLS. Set together with `key_file`. | +| `cache.redis.tls.key_file` | `WH_CACHE_REDIS_TLS_KEY_FILE` | — | The client certificate's private key (PEM). | +| `cache.redis.tls.server_name` | `WH_CACHE_REDIS_TLS_SERVER_NAME` | — | Name to verify the server's certificate against, when it differs from the address. | +| `cache.redis.tls.insecure_skip_verify` | `WH_CACHE_REDIS_TLS_INSECURE_SKIP_VERIFY` | `false` | Accept any server certificate. Logged at `WARN` at boot: whoever can intercept the connection can read and replace cached results. | +| `cache.redis.key_prefix` | `WH_CACHE_REDIS_KEY_PREFIX` | `wh` | Leads every key, so several deployments can share one server. No `{` or `}`. | +| `cache.redis.timeout` | `WH_CACHE_REDIS_TIMEOUT` | `100ms` | Per operation. A lookup or fill that takes longer is a miss or a skipped fill, never a failed query. | +| `cache.redis.dial_timeout` | `WH_CACHE_REDIS_DIAL_TIMEOUT` | `1s` | Per connection attempt. | +| `cache.redis.max_value_bytes` | `WH_CACHE_REDIS_MAX_VALUE_BYTES` | `1048576` | Largest result stored, after compression (1 MiB). A larger one is returned to the caller but not cached. | +| `cache.redis.compress_min_bytes` | `WH_CACHE_REDIS_COMPRESS_MIN_BYTES` | `1024` | Results at least this large are zstd-compressed when that makes them smaller. `-1` never compresses. `0` is not "off": in the YAML file it reads as unset and takes the default, and in the env var it refuses boot. | +| `cache.redis.version_ttl` | `WH_CACHE_REDIS_VERSION_TTL` | `168h` | How long a table's or tenant's version token outlives its last write, so the tokens of dropped tables and removed tenants eventually expire. At least `2s`. An expired token only causes misses. | + +**When the server is unreachable or misbehaves, the cache is bypassed; queries are not.** A failure or a timeout makes the lookup a miss and the fill a no-op, and five in a row open a circuit breaker that skips the server entirely until a probe, every 5 s, gets an answer. Queries then go straight to ClickHouse, still coalesced per instance by `singleflight`. An invalidation the server did not take is kept and retried until it lands. `/readyz` does not depend on the cache. + +**Boot does not wait for the server.** A malformed block (an address without a port, `mode: cluster` with `db` other than `0`, an unreadable or unparsable TLS file) refuses boot. A server that cannot be reached, or that refuses the credentials, does not: the process boots with the cache bypassed and keeps reconnecting, with backoff up to 30 s. A rejected password (`WRONGPASS`, `NOAUTH`) is logged at `ERROR` on every attempt; any other failure at `WARN`. This is deliberate: a rotated Redis password must not crash-loop every instance at once. Watch `wavehouse_cache_breaker_open`, which reads `1` while the cache is bypassed, including before the first connection. ### Authentication @@ -210,8 +237,28 @@ mq: backend: embedded # in-process NATS JetStream under /nats cache: - backend: local - l1_max_cost: 67108864 + backend: local # local | redis + l1_max_cost: 67108864 # the local backend's size + redis: # read only with backend: redis + addrs: [] # required with backend: redis, e.g. ["redis:6379"] + mode: standalone # standalone | cluster | sentinel + sentinel_master: "" + username: "" + password: "" # a secret: set WH_CACHE_REDIS_PASSWORD instead + db: 0 + tls: + enabled: false + ca_file: "" + cert_file: "" + key_file: "" + server_name: "" + insecure_skip_verify: false + key_prefix: wh + timeout: 100ms + dial_timeout: 1s + max_value_bytes: 1048576 + compress_min_bytes: 1024 # -1 = never + version_ttl: 168h dedupe: backend: pebble # in-process Pebble under /pebble @@ -267,6 +314,25 @@ WH_CH_MAX_TOTAL_CONNS=0 WH_MQ_BACKEND=embedded WH_CACHE_BACKEND=local WH_CACHE_L1_MAX_COST=67108864 +# Read only with WH_CACHE_BACKEND=redis; WH_CACHE_REDIS_ADDRS is then required. +WH_CACHE_REDIS_ADDRS= +WH_CACHE_REDIS_MODE=standalone +WH_CACHE_REDIS_SENTINEL_MASTER= +WH_CACHE_REDIS_USERNAME= +WH_CACHE_REDIS_PASSWORD= +WH_CACHE_REDIS_DB=0 +WH_CACHE_REDIS_TLS_ENABLED=false +WH_CACHE_REDIS_TLS_CA_FILE= +WH_CACHE_REDIS_TLS_CERT_FILE= +WH_CACHE_REDIS_TLS_KEY_FILE= +WH_CACHE_REDIS_TLS_SERVER_NAME= +WH_CACHE_REDIS_TLS_INSECURE_SKIP_VERIFY=false +WH_CACHE_REDIS_KEY_PREFIX=wh +WH_CACHE_REDIS_TIMEOUT=100ms +WH_CACHE_REDIS_DIAL_TIMEOUT=1s +WH_CACHE_REDIS_MAX_VALUE_BYTES=1048576 +WH_CACHE_REDIS_COMPRESS_MIN_BYTES=1024 +WH_CACHE_REDIS_VERSION_TTL=168h WH_DEDUPE_BACKEND=pebble WH_COORD_BACKEND=local diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index 79dae7d6a..63f1b1a9c 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -397,6 +397,26 @@ The folder name is the tenant id, and each folder is a complete settings directo `X-Tenant-ID` is a generic name, and some gateways and service meshes stamp one on every request. WaveHouse used to ignore it; now, over a settings directory that holds the four files, any value other than `0` names an unknown tenant, so **every `/v1` route outside `/v1/ops/*` answers `404 unknown tenant: `** (a `400` when the value is not a tenant id at all, a dotted hostname, say) — the SDK's `/v1/health` reachability ping included, while the bare probes and the admin surface stay green. Strip the inbound header at the edge ([header forwarding](/reverse-proxy#header-and-auth-forwarding)) unless you are using it deliberately. +## Multiple instances and the shared cache + +Several WaveHouse instances can serve one ClickHouse behind a load balancer, but most of what each one holds is its own. The message queue is embedded, so an event is inserted by the instance that took its `POST /v1/ingest`, and reaches only that instance's SSE subscribers. The dedupe store is per instance too, so an id one instance has seen is new to another. + +The query-result cache is the layer that can be shared today. With the default `cache.backend: local`, each instance caches in its own memory, and an insert invalidates only the cache of the instance that made it. Every other instance keeps serving its cached results for the rows before the insert until each entry's TTL runs out, between 10 s and 1 h depending on how long the query took. With [`cache.backend: redis`](/configuration#cache), every instance reads and fills one Redis-compatible server, and an insert on any instance invalidates the cached results of every instance. + +**What another instance can see.** Ingest is already asynchronous: `/v1/ingest` answers before the batch is inserted. Once the inserting instance's worker has written the batch to ClickHouse, it replaces the table's version token in Redis, and from then on a lookup on any instance misses and reads the new rows. The cache adds no delay of its own beyond that single write. The exceptions: + +- **The server is unreachable from the inserting instance.** The invalidation is kept and retried until it lands (`wavehouse_cache_invalidations_pending` counts what is owed). Meanwhile other instances that can still reach the server keep serving the older results, for as long as the outage lasts and at most until each entry's TTL. An instance that stops while invalidations are still owed loses them, with the same bound. The same thing happens today when a process stops between an insert and its invalidation. +- **A failover to a replica that had not yet received the latest token writes** can bring back entries filed under the older tokens, bounded by the replication lag at the moment of failover and those entries' TTL. WaveHouse never reads from replicas. +- **Admin writes through `POST /v1/ops/query`** do not invalidate the cache ([#394](https://github.com/Wave-RF/WaveHouse/issues/394)). With a shared cache, the stale results they leave are served by every instance, not only one. + +**Sizing the server.** Every key WaveHouse writes has a TTL, and a version token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses, never bring back an entry it had invalidated. So set `maxmemory` and let the server evict: `maxmemory-policy allkeys-lru` (or `allkeys-lfu`, `volatile-lru`, `volatile-lfu`). Under `noeviction`, a full server refuses the writes. Fills then fail (counted by `wavehouse_cache_set_failures_total{reason="oom"}`), lookups keep working, and invalidations are kept and retried. A stored result is capped at `cache.redis.max_value_bytes` (1 MiB compressed). A tenant's version tokens share one hash tag, so each lookup reads them in one `MGET` in cluster mode as well. The results themselves carry no hash tag and spread across shards. Persistence is not needed: an empty server after a restart is a cold cache, not a wrong one. + +**Coalescing stays per instance.** `singleflight` collapses identical concurrent queries within each instance, so a cold hot query costs at most one ClickHouse query per instance, not one per request. + +**Metrics** (meter `wavehouse-cache`, every series labeled `backend="redis"`, no tenant label): `wavehouse_cache_lookups_total{result}` (`hit`, `miss`, `stale`, `bypass`, `error`), `wavehouse_cache_op_duration_seconds{op}` (`lookup`, `set`, `invalidate`), `wavehouse_cache_breaker_open` (1 while the cache is bypassed), `wavehouse_cache_invalidations_total{result}` (`ok`, `deferred`), `wavehouse_cache_invalidations_pending`, `wavehouse_cache_value_bytes`, `wavehouse_cache_oversize_total` and `wavehouse_cache_set_failures_total{reason}` (`oom`, `timeout`, `other`). Two signals are worth alerting on: `wavehouse_cache_breaker_open` at 1, or `wavehouse_cache_invalidations_pending` above 0, for more than a few minutes. + +For local development, `docker compose -f deployments/compose/dependencies.yaml --profile redis up -d` starts a Redis on `localhost:6379` with no persistence. + ## ClickHouse Schema WaveHouse uses a **Bring Your Own Schema** model. You create your tables in ClickHouse with whatever columns and engines you need. WaveHouse discovers the schemas automatically via `system.columns` and validates ingest data against them — see [Schema Validation](/api#post-v1ingesttabletable--ingest-data) for the rules a record must satisfy. diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index 8e6a85036..2a9ea7d5a 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -82,7 +82,7 @@ make dev WaveHouse is now running at `http://localhost:8080` in standalone mode with: - **Embedded NATS** (JetStream) — no external MQ needed -- **L1 cache only** (Ristretto) — no external cache needed +- **In-process cache** (Ristretto, `cache.backend: local`) — no external cache needed; to try the shared one, start Redis with `docker compose -f deployments/compose/dependencies.yaml --profile redis up -d` and set `WH_CACHE_BACKEND=redis WH_CACHE_REDIS_ADDRS=localhost:6379` - **Trial policy** — the dev settings directory `./settings` is seeded on first run with the compose stack's permissive `public` policy, so tokenless requests to the demo tables work (see [Test the API](#test-the-api)) - **Dedup disabled** by default — no Pebble needed - **Schema discovery** — automatically finds your ClickHouse tables @@ -454,7 +454,7 @@ WaveHouse/ │ ├── api/ # HTTP handlers, router, middleware │ ├── app/ # Process wiring (build every component, run under one errgroup, release in reverse) │ ├── auth/ # JWT/JWKS authentication middleware -│ ├── cache/ # L1 (Ristretto) + L2 caching +│ ├── cache/ # Query cache: in-process (Ristretto) or shared (Redis-compatible) │ ├── chconn/ # ClickHouse pools, one per connection tuple (reconciled on settings reload) │ ├── chsql/ # Shared ClickHouse SQL helpers (quoting + bind-safety) │ ├── config/ # YAML + env var configuration diff --git a/docs/src/content/docs/getting-started.md b/docs/src/content/docs/getting-started.md index f24c3ee6f..667137d1a 100644 --- a/docs/src/content/docs/getting-started.md +++ b/docs/src/content/docs/getting-started.md @@ -75,7 +75,7 @@ curl -s -X POST "http://localhost:8080/v1/query?table=clicks" \ -d '{"columns": ["page", "button", "score"], "limit": 10}' ``` -`POST /v1/query?table={table}` and `GET/POST /v1/pipes/{name}` are cached in-process (L1 Ristretto) with singleflight coalescing — duplicate concurrent queries hit ClickHouse once. For raw SQL there's `POST /v1/ops/query` (an admin escape hatch that never caches, emitting `Cache-Control: no-store`), but it's **admin-only** — the trial `public` role can't reach it. To use it, swap the public default for real auth: configure a JWT secret and present a token whose role is the policy [`admin_role`](/access-control#admin_role--the-privileged-role). +`POST /v1/query?table={table}` and `GET/POST /v1/pipes/{name}` are cached — in-process by default, or in a Redis shared by every instance with [`cache.backend: redis`](/configuration#cache) — with singleflight coalescing, so duplicate concurrent queries hit ClickHouse once. For raw SQL there's `POST /v1/ops/query` (an admin escape hatch that never caches, emitting `Cache-Control: no-store`), but it's **admin-only** — the trial `public` role can't reach it. To use it, swap the public default for real auth: configure a JWT secret and present a token whose role is the policy [`admin_role`](/access-control#admin_role--the-privileged-role). :::tip[Prefer a type-safe client?] The [TypeScript SDK](/sdk) wraps this endpoint in a chainable query builder with autocomplete on your table names and row types — plus live queries and streaming. The raw shapes are in the [structured query reference](/api#post-v1querytabletable--structured-query). diff --git a/docs/src/content/docs/pipes.mdx b/docs/src/content/docs/pipes.mdx index b8c7efa04..b64129225 100644 --- a/docs/src/content/docs/pipes.mdx +++ b/docs/src/content/docs/pipes.mdx @@ -169,7 +169,7 @@ curl -X POST http://localhost:8080/v1/pipes/top_pages \ -d '{"start_date": "2024-01-01", "limit": 20}' ``` -The response is a JSON array of rows. Results flow through the shared in-process L1 cache (Ristretto) with singleflight coalescing, so concurrent identical calls hit ClickHouse once; an `X-Cache: HIT` or `X-Cache: MISS` header tells you which path served the response. +The response is a JSON array of rows. Results flow through the query cache (in-process by default, or a Redis shared by every instance with [`cache.backend: redis`](/configuration#cache)) with singleflight coalescing, so concurrent identical calls hit ClickHouse once; an `X-Cache: HIT` or `X-Cache: MISS` header tells you which path served the response. | Status | Body | Cause | | ------ | ---- | ----- | diff --git a/docs/src/content/docs/settings-directory.mdx b/docs/src/content/docs/settings-directory.mdx index 561354627..922f3a3f2 100644 --- a/docs/src/content/docs/settings-directory.mdx +++ b/docs/src/content/docs/settings-directory.mdx @@ -179,7 +179,7 @@ The tenant tunables. Every key is required (a missing one is a validation error) } ``` -What stays in boot config is only what cannot change under a running process — the implementation each layer runs on (`mq.backend`, `cache.backend`, `dedupe.backend`, `coord.backend`), resource sizing (`data_dir`, `cache.l1_max_cost`, `clickhouse.max_total_conns`), the listeners, the observability exporters — and the **secrets**: `clickhouse.password`, `auth.jwt_secret`, `auth.operator_key`. Secrets never belong in a tracked JSON file, so they stay in the environment and are combined with the wiring here on every (re)connect; rotating one is a restart. See [Configuration](/configuration). Everything else lives here and reloads. +What stays in boot config is only what cannot change under a running process — the implementation each layer runs on (`mq.backend`, `cache.backend`, `dedupe.backend`, `coord.backend`) and a shared backend's connection (`cache.redis`), resource sizing (`data_dir`, `cache.l1_max_cost`, `clickhouse.max_total_conns`), the listeners, the observability exporters — and the **secrets**: `clickhouse.password`, `cache.redis.password`, `auth.jwt_secret`, `auth.operator_key`. Secrets never belong in a tracked JSON file, so they stay in the environment and are combined with the wiring here on every (re)connect; rotating one is a restart. See [Configuration](/configuration). Everything else lives here and reloads. ## Deduplication diff --git a/internal/app/app_test.go b/internal/app/app_test.go index cf727b6d4..2ce9ca5bb 100644 --- a/internal/app/app_test.go +++ b/internal/app/app_test.go @@ -702,6 +702,77 @@ func TestReload_PrunesCacheIndexToServedTenants(t *testing.T) { assert.Equal(t, map[tenant.ID]bool{"acme": false, "globex": true}, rec.last(), "removed; the repaired one served again") } +// redisTestConfig is testConfig with cache.backend=redis at addr, carrying +// the defaults Load would apply. +func redisTestConfig(t *testing.T, settingsDir, addr string) *config.Config { + t.Helper() + cfg := testConfig(t, settingsDir) + cfg.Cache = config.Cache{Backend: config.CacheRedis, Redis: config.CacheRedisConfig{ + Addrs: []string{addr}, Mode: config.RedisStandalone, KeyPrefix: "wh", + Timeout: 100 * time.Millisecond, DialTimeout: 200 * time.Millisecond, + MaxValueBytes: 1 << 20, CompressMinBytes: 1 << 10, VersionTTL: time.Hour, + }} + require.NoError(t, cfg.Validate()) + return cfg +} + +// cache.backend=redis wires the shared backend. A server that cannot be +// reached does not refuse boot: the cache starts bypassed, and the reload +// hook that prunes an in-process index leaves it alone. +func TestNew_RedisCacheBootsBypassedWhenUnreachable(t *testing.T) { + root := writeNestedSettings(t, map[string]map[string]any{"acme": nil, "globex": nil}) + a := newApp(t, redisTestConfig(t, root, closedAddr(t)), Options{}) + _, ok := a.cache.(*cache.RedisCache) + require.True(t, ok, "cache is %T", a.cache) + assert.Contains(t, componentNames(a), "cache") + + require.NoError(t, os.RemoveAll(filepath.Join(root, "acme"))) + a.tenants.Reload("test") // the prune hook must not trip on a non-pruner + + entry, snap, err := a.cache.Lookup(t.Context(), "globex", "sha", nil) + require.NoError(t, err) + assert.Nil(t, entry.Value, "bypassed: a miss") + assert.NoError(t, a.cache.Set(t.Context(), snap, []byte("v"), time.Minute), "and the fill a no-op") +} + +// A TLS file that went missing between validation and wiring refuses boot, +// naming the key. +func TestNew_RedisCacheRefusesAnUnreadableTLSFile(t *testing.T) { + guardGlobals(t) + cfg := redisTestConfig(t, writeSettings(t, nil), closedAddr(t)) + cfg.Cache.Redis.TLS = config.CacheRedisTLS{Enabled: true, CAFile: filepath.Join(t.TempDir(), "gone.pem")} + _, err := New(t.Context(), Options{Config: cfg}) + require.ErrorContains(t, err, "cache init: cache.redis.tls.ca_file") +} + +// The boot config's defaults are the backend's, and -1 is the backend's +// "never compress". Driven from Load, not a literal, so a default changed on +// one side only fails here. +func TestRedisConfig_FromLoadedDefaults(t *testing.T) { + t.Setenv("WH_SETTINGS_DIR", t.TempDir()) + t.Setenv("WH_CACHE_BACKEND", "redis") + t.Setenv("WH_CACHE_REDIS_ADDRS", "a:6379,b:6379") + t.Setenv("WH_CACHE_REDIS_PASSWORD", "pw") + loaded, err := config.Load(filepath.Join(t.TempDir(), "none.yaml")) + require.NoError(t, err) + got, err := redisConfig(loaded.Cache.Redis) + require.NoError(t, err) + assert.Equal(t, cache.RedisConfig{ + Addrs: []string{"a:6379", "b:6379"}, Mode: cache.RedisStandalone, Password: "pw", + KeyPrefix: cache.DefaultRedisKeyPrefix, Timeout: cache.DefaultRedisTimeout, + DialTimeout: cache.DefaultRedisDialTimeout, MaxValueBytes: cache.DefaultRedisMaxValueBytes, + CompressMinBytes: cache.DefaultRedisCompressMinBytes, VersionTTL: cache.DefaultRedisVersionTTL, + }, got) + + loaded.Cache.Redis.CompressMinBytes = -1 + loaded.Cache.Redis.Mode = config.RedisCluster + got, err = redisConfig(loaded.Cache.Redis) + require.NoError(t, err) + assert.Zero(t, got.CompressMinBytes) + assert.Equal(t, cache.RedisCluster, got.Mode) + assert.Equal(t, cache.RedisSentinel, config.RedisSentinel) +} + // keepalive is a config.json patch setting the stream block's keepalive pair. func keepalive(interval, buckets int) map[string]any { return map[string]any{"stream": map[string]any{"keepalive_interval": interval, "keepalive_buckets": buckets, "gap_window_minutes": 15}} diff --git a/internal/app/wire.go b/internal/app/wire.go index 3cbe71a82..e1fe828d8 100644 --- a/internal/app/wire.go +++ b/internal/app/wire.go @@ -627,7 +627,7 @@ var _ pruner = (*cache.LocalCache)(nil) // is chosen. After every reload a tenant no longer served, removed or // rejected alike, has its in-process version index dropped (#262); its cache // is orphaned with it, as it would be anyway when it came back -// (wireClickHouse). +// (wireClickHouse). A shared backend keeps no such index and is skipped. func (a *App) wireCache() error { var c cache.Cache switch b := a.cfg.Cache.Backend; b { @@ -637,6 +637,16 @@ func (a *App) wireCache() error { return fmt.Errorf("cache init: %w", err) } c = l1 + case config.CacheRedis: + rc, err := redisConfig(a.cfg.Cache.Redis) + if err != nil { + return fmt.Errorf("cache init: %w", err) + } + r, err := cache.NewRedis(rc) + if err != nil { + return fmt.Errorf("cache init: %w", err) + } + c = r default: return unreachableBackend("cache.backend", b) } @@ -650,6 +660,35 @@ func (a *App) wireCache() error { return nil } +// redisConfig maps the boot config's cache.redis block onto the backend's +// config. Load has applied every default and validated the block; the TLS +// files are read again here, so the connection uses what is on disk now. +func redisConfig(r config.CacheRedisConfig) (cache.RedisConfig, error) { + t, err := r.TLS.Config() + if err != nil { + return cache.RedisConfig{}, err + } + compressMin := r.CompressMinBytes + if compressMin < 0 { + compressMin = 0 // the backend's "never" + } + return cache.RedisConfig{ + Addrs: r.Addrs, + Mode: r.Mode, + SentinelMaster: r.SentinelMaster, + Username: r.Username, + Password: r.Password, + DB: r.DB, + TLS: t, + KeyPrefix: r.KeyPrefix, + Timeout: r.Timeout, + DialTimeout: r.DialTimeout, + MaxValueBytes: r.MaxValueBytes, + CompressMinBytes: compressMin, + VersionTTL: r.VersionTTL, + }, nil +} + // unreachableBackend is each layer switch's default case. config.Validate // refuses a backend with no case, so reaching it means a Config built by hand // without one (the zero value is not the default), or a case missing here. diff --git a/internal/config/backends.go b/internal/config/backends.go index f2ab93308..893d2304b 100644 --- a/internal/config/backends.go +++ b/internal/config/backends.go @@ -35,21 +35,34 @@ func (m MQ) validate() error { // CacheBackend names the query-result cache implementation. type CacheBackend string -// CacheLocal is the in-process Ristretto cache, sized by cache.l1_max_cost. -const CacheLocal CacheBackend = "local" +const ( + // CacheLocal is the in-process Ristretto cache, sized by + // cache.l1_max_cost. + CacheLocal CacheBackend = "local" + // CacheRedis is one Redis-compatible server shared by every process, + // configured by cache.redis. + CacheRedis CacheBackend = "redis" +) -var cacheBackends = []CacheBackend{CacheLocal} +var cacheBackends = []CacheBackend{CacheLocal, CacheRedis} // Cache selects and sizes the query-result cache. The time-range bucket // structured queries normalize to is a settings-directory key // (query.timestamp_bucket_seconds) — query shaping, not process memory. type Cache struct { - Backend CacheBackend `yaml:"backend" env:"WH_CACHE_BACKEND" env-default:"local"` - L1MaxCost int64 `yaml:"l1_max_cost" env:"WH_CACHE_L1_MAX_COST" env-default:"67108864"` + Backend CacheBackend `yaml:"backend" env:"WH_CACHE_BACKEND" env-default:"local"` + L1MaxCost int64 `yaml:"l1_max_cost" env:"WH_CACHE_L1_MAX_COST" env-default:"67108864"` + Redis CacheRedisConfig `yaml:"redis"` } func (c Cache) validate() error { - return checkBackend("cache.backend", "WH_CACHE_BACKEND", c.Backend, cacheBackends) + if err := checkBackend("cache.backend", "WH_CACHE_BACKEND", c.Backend, cacheBackends); err != nil { + return err + } + if c.Backend == CacheRedis { + return c.Redis.validate() + } + return nil } // DedupeBackend names where ingest dedupe keeps the ids it has seen. @@ -126,13 +139,20 @@ func (c *Config) NeedsDataDir() bool { } // Warnings returns what a valid configuration is still likely to get wrong, -// one line each, for boot to log at WARN. They are not errors because each is -// correct for a single replica, and one process cannot count its replicas. +// one line each, for boot to log at WARN. The shared-queue ones are not +// errors because each is correct for a single replica, and one process +// cannot count its replicas. func (c *Config) Warnings() []string { + var out []string + if c.Cache.Backend == CacheRedis && c.Cache.Redis.TLS.InsecureSkipVerify { + out = append(out, "cache.redis.tls.insecure_skip_verify is on: the cache accepts any certificate, so whoever can intercept the connection can read and replace cached query results") + } + if c.Cache.Backend != CacheRedis && c.Cache.Redis.hasAddrs() { + out = append(out, "cache.redis.addrs is set but cache.backend is "+string(c.Cache.Backend)+": the redis block is not read; set cache.backend=redis to share the cache") + } if !c.Distributed() { - return nil + return out } - var out []string if c.Cache.Backend == CacheLocal { out = append(out, "cache.backend=local with a shared mq.backend is correct for one replica only: an event ingested on another replica never invalidates this one's cache, so its reads stay stale until the cached entry expires") } diff --git a/internal/config/backends_test.go b/internal/config/backends_test.go index 0e70dc7a3..778336971 100644 --- a/internal/config/backends_test.go +++ b/internal/config/backends_test.go @@ -111,7 +111,7 @@ func TestValidate_UnknownBackend(t *testing.T) { want string }{ {"mq", func(c *Config) { c.MQ.Backend = "kafka" }, `mq.backend (WH_MQ_BACKEND) "kafka" is not a backend this build has; valid: embedded`}, - {"cache", func(c *Config) { c.Cache.Backend = "redis" }, `cache.backend (WH_CACHE_BACKEND) "redis" is not a backend this build has; valid: local`}, + {"cache", func(c *Config) { c.Cache.Backend = "memcached" }, `cache.backend (WH_CACHE_BACKEND) "memcached" is not a backend this build has; valid: local, redis`}, {"dedupe", func(c *Config) { c.Dedupe.Backend = "dynamodb" }, `dedupe.backend (WH_DEDUPE_BACKEND) "dynamodb" is not a backend this build has; valid: pebble`}, {"coord", func(c *Config) { c.Coord.Backend = "nats" }, `coord.backend (WH_COORD_BACKEND) "nats" is not a backend this build has; valid: local`}, // The zero value, which a Config built without Load carries. diff --git a/internal/config/cache_redis.go b/internal/config/cache_redis.go new file mode 100644 index 000000000..f0af67137 --- /dev/null +++ b/internal/config/cache_redis.go @@ -0,0 +1,153 @@ +package config + +import ( + "crypto/tls" + "crypto/x509" + "errors" + "fmt" + "net" + "os" + "strings" + "time" +) + +// Redis deployment modes for cache.redis.mode. +const ( + RedisStandalone = "standalone" + RedisCluster = "cluster" + RedisSentinel = "sentinel" +) + +// CacheRedisConfig configures cache.backend=redis: one Redis-compatible server +// (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB) shared by every process. +// Read only when that backend is selected. +type CacheRedisConfig struct { + // Addrs are host:port pairs: the server, or seeds for a cluster, or the + // sentinels. + Addrs []string `yaml:"addrs" env:"WH_CACHE_REDIS_ADDRS"` + Mode string `yaml:"mode" env:"WH_CACHE_REDIS_MODE" env-default:"standalone"` + SentinelMaster string `yaml:"sentinel_master" env:"WH_CACHE_REDIS_SENTINEL_MASTER"` + Username string `yaml:"username" env:"WH_CACHE_REDIS_USERNAME"` + Password string `yaml:"password" env:"WH_CACHE_REDIS_PASSWORD"` + DB int `yaml:"db" env:"WH_CACHE_REDIS_DB"` + TLS CacheRedisTLS `yaml:"tls"` + // KeyPrefix leads every key, so deployments can share one server. + KeyPrefix string `yaml:"key_prefix" env:"WH_CACHE_REDIS_KEY_PREFIX" env-default:"wh"` + Timeout time.Duration `yaml:"timeout" env:"WH_CACHE_REDIS_TIMEOUT" env-default:"100ms"` + DialTimeout time.Duration `yaml:"dial_timeout" env:"WH_CACHE_REDIS_DIAL_TIMEOUT" env-default:"1s"` + // MaxValueBytes is the largest value stored, after compression. + MaxValueBytes int `yaml:"max_value_bytes" env:"WH_CACHE_REDIS_MAX_VALUE_BYTES" env-default:"1048576"` + // CompressMinBytes is the smallest value zstd-compressed; -1 never + // compresses. Not 0: the loader reads a 0 in the file as unset and + // applies the default, so 0 cannot mean off. + CompressMinBytes int `yaml:"compress_min_bytes" env:"WH_CACHE_REDIS_COMPRESS_MIN_BYTES" env-default:"1024"` + // VersionTTL is how long a version token outlives its last bump. + VersionTTL time.Duration `yaml:"version_ttl" env:"WH_CACHE_REDIS_VERSION_TTL" env-default:"168h"` +} + +// CacheRedisTLS is cache.redis.tls. The files are paths, read at boot. +type CacheRedisTLS struct { + Enabled bool `yaml:"enabled" env:"WH_CACHE_REDIS_TLS_ENABLED"` + CAFile string `yaml:"ca_file" env:"WH_CACHE_REDIS_TLS_CA_FILE"` + CertFile string `yaml:"cert_file" env:"WH_CACHE_REDIS_TLS_CERT_FILE"` + KeyFile string `yaml:"key_file" env:"WH_CACHE_REDIS_TLS_KEY_FILE"` + ServerName string `yaml:"server_name" env:"WH_CACHE_REDIS_TLS_SERVER_NAME"` + InsecureSkipVerify bool `yaml:"insecure_skip_verify" env:"WH_CACHE_REDIS_TLS_INSECURE_SKIP_VERIFY"` +} + +// hasAddrs reports whether any address is set. An env file's blank +// `WH_CACHE_REDIS_ADDRS=` loads as one empty address, which is none. +func (r CacheRedisConfig) hasAddrs() bool { + return len(r.Addrs) > 1 || len(r.Addrs) == 1 && r.Addrs[0] != "" +} + +func (r CacheRedisConfig) validate() error { + if !r.hasAddrs() { + return errors.New("cache.backend=redis needs cache.redis.addrs (WH_CACHE_REDIS_ADDRS): the server's host:port, or a cluster's seeds, or the sentinels") + } + for _, a := range r.Addrs { + if _, _, err := net.SplitHostPort(a); err != nil { + return fmt.Errorf("cache.redis.addrs (WH_CACHE_REDIS_ADDRS) %q: want host:port: %w", a, err) + } + } + switch r.Mode { + case RedisStandalone, RedisCluster, RedisSentinel: + default: + return fmt.Errorf("cache.redis.mode (WH_CACHE_REDIS_MODE) %q: valid: %s, %s, %s", r.Mode, RedisStandalone, RedisCluster, RedisSentinel) + } + if r.Mode == RedisSentinel && r.SentinelMaster == "" { + return errors.New("cache.redis.mode=sentinel needs cache.redis.sentinel_master (WH_CACHE_REDIS_SENTINEL_MASTER), the master set name") + } + if r.DB < 0 { + return fmt.Errorf("cache.redis.db (WH_CACHE_REDIS_DB) %d is negative", r.DB) + } + if r.Mode == RedisCluster && r.DB != 0 { + return fmt.Errorf("cache.redis.db (WH_CACHE_REDIS_DB) %d: a Redis cluster has only database 0", r.DB) + } + if r.KeyPrefix == "" || strings.ContainsAny(r.KeyPrefix, "{}") { + return fmt.Errorf("cache.redis.key_prefix (WH_CACHE_REDIS_KEY_PREFIX) %q: want a non-empty prefix without a hash-tag brace", r.KeyPrefix) + } + for _, d := range []struct { + key string + v time.Duration + }{ + {"cache.redis.timeout (WH_CACHE_REDIS_TIMEOUT)", r.Timeout}, + {"cache.redis.dial_timeout (WH_CACHE_REDIS_DIAL_TIMEOUT)", r.DialTimeout}, + } { + if d.v <= 0 { + return fmt.Errorf("%s %s must be positive", d.key, d.v) + } + } + if r.VersionTTL < 2*time.Second { + return fmt.Errorf("cache.redis.version_ttl (WH_CACHE_REDIS_VERSION_TTL) %s is under 2s", r.VersionTTL) + } + if r.MaxValueBytes <= 0 { + return fmt.Errorf("cache.redis.max_value_bytes (WH_CACHE_REDIS_MAX_VALUE_BYTES) %d must be positive", r.MaxValueBytes) + } + if r.CompressMinBytes == 0 || r.CompressMinBytes < -1 { + return fmt.Errorf("cache.redis.compress_min_bytes (WH_CACHE_REDIS_COMPRESS_MIN_BYTES) %d: want a positive size, or -1 to never compress", r.CompressMinBytes) + } + if _, err := r.TLS.Config(); err != nil { + return err + } + return nil +} + +// Config builds the tls.Config the block describes, reading its files, or +// nil when TLS is off. A file set while TLS is off is an error rather than +// a silently plaintext connection. +func (t CacheRedisTLS) Config() (*tls.Config, error) { + if !t.Enabled { + if t != (CacheRedisTLS{}) { + return nil, errors.New("cache.redis.tls: files, server_name or insecure_skip_verify are set but cache.redis.tls.enabled (WH_CACHE_REDIS_TLS_ENABLED) is off") + } + return nil, nil + } + if (t.CertFile == "") != (t.KeyFile == "") { + return nil, errors.New("cache.redis.tls: cert_file and key_file must be set together") + } + cfg := &tls.Config{ + MinVersion: tls.VersionTLS12, + ServerName: t.ServerName, + InsecureSkipVerify: t.InsecureSkipVerify, //nolint:gosec // G402: the operator's cache.redis.tls.insecure_skip_verify, warned about at boot + } + if t.CAFile != "" { + pemBytes, err := os.ReadFile(t.CAFile) + if err != nil { + return nil, fmt.Errorf("cache.redis.tls.ca_file: %w", err) + } + pool := x509.NewCertPool() + if !pool.AppendCertsFromPEM(pemBytes) { + return nil, fmt.Errorf("cache.redis.tls.ca_file: no certificates in %s", t.CAFile) + } + cfg.RootCAs = pool + } + if t.CertFile != "" { + cert, err := tls.LoadX509KeyPair(t.CertFile, t.KeyFile) + if err != nil { + return nil, fmt.Errorf("cache.redis.tls.cert_file: %w", err) + } + cfg.Certificates = []tls.Certificate{cert} + } + return cfg, nil +} diff --git a/internal/config/cache_redis_test.go b/internal/config/cache_redis_test.go new file mode 100644 index 000000000..6d989c737 --- /dev/null +++ b/internal/config/cache_redis_test.go @@ -0,0 +1,298 @@ +package config + +import ( + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/x509" + "crypto/x509/pkix" + "encoding/pem" + "math/big" + "os" + "path/filepath" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// redisBackend is what Load produces for cache.backend=redis with only the +// address set. +func redisBackend() Config { + c := defaultBackends() + c.Cache.Backend = CacheRedis + c.Cache.Redis = CacheRedisConfig{ + Addrs: []string{"redis:6379"}, Mode: RedisStandalone, KeyPrefix: "wh", + Timeout: 100 * time.Millisecond, DialTimeout: time.Second, + MaxValueBytes: 1 << 20, CompressMinBytes: 1 << 10, VersionTTL: 168 * time.Hour, + } + return c +} + +func TestLoad_CacheRedisDefaults(t *testing.T) { + t.Setenv("WH_CACHE_BACKEND", "redis") + t.Setenv("WH_CACHE_REDIS_ADDRS", "redis:6379") + cfg, err := Load("nonexistent.yaml") + require.NoError(t, err) + want := redisBackend() + assert.Equal(t, want.Cache.Redis, cfg.Cache.Redis) + assert.Empty(t, cfg.Warnings()) +} + +func TestLoad_CacheRedisFromEnv(t *testing.T) { + dir := t.TempDir() + caFile, certFile, keyFile := writeTestPKI(t, dir) + for k, v := range map[string]string{ + "WH_CACHE_BACKEND": "redis", + "WH_CACHE_REDIS_ADDRS": "s1:26379,s2:26379", + "WH_CACHE_REDIS_MODE": "sentinel", + "WH_CACHE_REDIS_SENTINEL_MASTER": "mymaster", + "WH_CACHE_REDIS_USERNAME": "wavehouse", + "WH_CACHE_REDIS_PASSWORD": "s3cret", + "WH_CACHE_REDIS_DB": "2", + "WH_CACHE_REDIS_TLS_ENABLED": "true", + "WH_CACHE_REDIS_TLS_CA_FILE": caFile, + "WH_CACHE_REDIS_TLS_CERT_FILE": certFile, + "WH_CACHE_REDIS_TLS_KEY_FILE": keyFile, + "WH_CACHE_REDIS_TLS_SERVER_NAME": "redis.internal", + "WH_CACHE_REDIS_KEY_PREFIX": "staging", + "WH_CACHE_REDIS_TIMEOUT": "250ms", + "WH_CACHE_REDIS_DIAL_TIMEOUT": "3s", + "WH_CACHE_REDIS_MAX_VALUE_BYTES": "2048", + "WH_CACHE_REDIS_COMPRESS_MIN_BYTES": "-1", + "WH_CACHE_REDIS_VERSION_TTL": "24h", + "WH_CACHE_REDIS_TLS_INSECURE_SKIP_VERIFY": "false", + } { + t.Setenv(k, v) + } + cfg, err := Load("nonexistent.yaml") + require.NoError(t, err) + assert.Equal(t, CacheRedisConfig{ + Addrs: []string{"s1:26379", "s2:26379"}, Mode: RedisSentinel, SentinelMaster: "mymaster", + Username: "wavehouse", Password: "s3cret", DB: 2, + TLS: CacheRedisTLS{ + Enabled: true, CAFile: caFile, CertFile: certFile, KeyFile: keyFile, ServerName: "redis.internal", + }, + KeyPrefix: "staging", Timeout: 250 * time.Millisecond, DialTimeout: 3 * time.Second, + MaxValueBytes: 2048, CompressMinBytes: -1, VersionTTL: 24 * time.Hour, + }, cfg.Cache.Redis) + tc, err := cfg.Cache.Redis.TLS.Config() + require.NoError(t, err) + assert.Equal(t, "redis.internal", tc.ServerName) + assert.NotNil(t, tc.RootCAs) + assert.Len(t, tc.Certificates, 1) +} + +func TestLoad_CacheRedisFromYAML(t *testing.T) { + t.Parallel() + path := filepath.Join(t.TempDir(), "config.yaml") + require.NoError(t, os.WriteFile(path, []byte(` +settings: + dir: ./settings +cache: + backend: redis + redis: + addrs: ["n1:6379", "n2:6379"] + mode: cluster + key_prefix: prod + timeout: 50ms + version_ttl: 72h +`), 0o600)) + cfg, err := Load(path) + require.NoError(t, err) + r := cfg.Cache.Redis + assert.Equal(t, CacheRedis, cfg.Cache.Backend) + assert.Equal(t, []string{"n1:6379", "n2:6379"}, r.Addrs) + assert.Equal(t, RedisCluster, r.Mode) + assert.Equal(t, "prod", r.KeyPrefix) + assert.Equal(t, 50*time.Millisecond, r.Timeout) + assert.Equal(t, 72*time.Hour, r.VersionTTL) + assert.Equal(t, time.Second, r.DialTimeout, "an unset key takes its default") + assert.Equal(t, 1024, r.CompressMinBytes) +} + +// A 0 in the file is read as unset, so it takes the default instead of +// switching compression off: why "never" is -1. Pinned so a loader that +// starts honoring the 0 is noticed. +func TestLoad_CacheRedisCompressZeroInYAMLIsTheDefault(t *testing.T) { + t.Parallel() + path := filepath.Join(t.TempDir(), "config.yaml") + require.NoError(t, os.WriteFile(path, []byte(` +settings: + dir: ./settings +cache: + backend: redis + redis: + addrs: ["r:6379"] + compress_min_bytes: 0 +`), 0o600)) + cfg, err := Load(path) + require.NoError(t, err) + assert.Equal(t, 1024, cfg.Cache.Redis.CompressMinBytes) +} + +func TestLoad_CacheRedisRefusesUnknownKeys(t *testing.T) { + t.Parallel() + path := filepath.Join(t.TempDir(), "config.yaml") + require.NoError(t, os.WriteFile(path, []byte(` +settings: + dir: ./settings +cache: + backend: redis + redis: + addr: r:6379 + near_cache: + max_cost: 1 + tls: + ca: /x + memcached: + addrs: ["m:11211"] +`), 0o600)) + _, err := Load(path) + require.Error(t, err) + assert.Contains(t, err.Error(), "cache.memcached, cache.redis.addr, cache.redis.near_cache, cache.redis.tls.ca") +} + +// The documented env file lists WH_CACHE_REDIS_ADDRS blank: that is no +// address, not an unread block to warn about, and not a valid redis one. +func TestLoad_CacheRedisBlankAddrs(t *testing.T) { + t.Setenv("WH_CACHE_REDIS_ADDRS", "") + cfg, err := Load("nonexistent.yaml") + require.NoError(t, err) + assert.Empty(t, cfg.Warnings()) + + t.Setenv("WH_CACHE_BACKEND", "redis") + _, err = Load("nonexistent.yaml") + require.ErrorContains(t, err, "cache.backend=redis needs cache.redis.addrs") +} + +func TestUnboundEnv_KnowsTheCacheRedisVariables(t *testing.T) { + t.Parallel() + assert.Empty(t, unboundEnv([]string{ + "WH_CACHE_REDIS_ADDRS=r:6379", "WH_CACHE_REDIS_PASSWORD=x", "WH_CACHE_REDIS_TLS_CA_FILE=/ca.pem", + "WH_CACHE_REDIS_VERSION_TTL=1h", "WH_CACHE_REDIS_COMPRESS_MIN_BYTES=-1", + })) + assert.Equal(t, []string{"WH_CACHE_REDIS_ADDR"}, unboundEnv([]string{"WH_CACHE_REDIS_ADDR=r:6379"})) +} + +func TestValidate_CacheRedis(t *testing.T) { + t.Parallel() + dir := t.TempDir() + caFile, certFile, keyFile := writeTestPKI(t, dir) + notPEM := filepath.Join(dir, "not.pem") + require.NoError(t, os.WriteFile(notPEM, []byte("hello"), 0o600)) + cases := []struct { + name string + set func(*CacheRedisConfig) + want string // "" = valid + }{ + {"defaults", func(*CacheRedisConfig) {}, ""}, + {"no addrs", func(r *CacheRedisConfig) { r.Addrs = nil }, "cache.backend=redis needs cache.redis.addrs (WH_CACHE_REDIS_ADDRS)"}, + {"addr without port", func(r *CacheRedisConfig) { r.Addrs = []string{"redis"} }, `cache.redis.addrs (WH_CACHE_REDIS_ADDRS) "redis": want host:port`}, + {"mode", func(r *CacheRedisConfig) { r.Mode = "replica" }, `cache.redis.mode (WH_CACHE_REDIS_MODE) "replica": valid: standalone, cluster, sentinel`}, + {"sentinel without master", func(r *CacheRedisConfig) { r.Mode = RedisSentinel }, "needs cache.redis.sentinel_master"}, + {"sentinel", func(r *CacheRedisConfig) { r.Mode, r.SentinelMaster = RedisSentinel, "m" }, ""}, + {"cluster db", func(r *CacheRedisConfig) { r.Mode, r.DB = RedisCluster, 1 }, "a Redis cluster has only database 0"}, + {"standalone db", func(r *CacheRedisConfig) { r.DB = 3 }, ""}, + {"negative db", func(r *CacheRedisConfig) { r.DB = -1 }, "is negative"}, + {"empty prefix", func(r *CacheRedisConfig) { r.KeyPrefix = "" }, "cache.redis.key_prefix"}, + {"brace prefix", func(r *CacheRedisConfig) { r.KeyPrefix = "a{b}" }, "hash-tag brace"}, + {"zero timeout", func(r *CacheRedisConfig) { r.Timeout = 0 }, "cache.redis.timeout (WH_CACHE_REDIS_TIMEOUT) 0s must be positive"}, + {"negative dial timeout", func(r *CacheRedisConfig) { r.DialTimeout = -time.Second }, "cache.redis.dial_timeout"}, + {"short version ttl", func(r *CacheRedisConfig) { r.VersionTTL = time.Second }, "cache.redis.version_ttl (WH_CACHE_REDIS_VERSION_TTL) 1s is under 2s"}, + {"zero max value", func(r *CacheRedisConfig) { r.MaxValueBytes = 0 }, "cache.redis.max_value_bytes"}, + {"compress 0", func(r *CacheRedisConfig) { r.CompressMinBytes = 0 }, "or -1 to never compress"}, + {"compress -2", func(r *CacheRedisConfig) { r.CompressMinBytes = -2 }, "or -1 to never compress"}, + {"compress never", func(r *CacheRedisConfig) { r.CompressMinBytes = -1 }, ""}, + {"tls files while off", func(r *CacheRedisConfig) { r.TLS.CAFile = caFile }, "cache.redis.tls.enabled (WH_CACHE_REDIS_TLS_ENABLED) is off"}, + {"tls system roots", func(r *CacheRedisConfig) { r.TLS.Enabled = true }, ""}, + {"tls full", func(r *CacheRedisConfig) { + r.TLS = CacheRedisTLS{Enabled: true, CAFile: caFile, CertFile: certFile, KeyFile: keyFile} + }, ""}, + {"tls cert without key", func(r *CacheRedisConfig) { r.TLS = CacheRedisTLS{Enabled: true, CertFile: certFile} }, "cert_file and key_file must be set together"}, + {"tls missing ca", func(r *CacheRedisConfig) { + r.TLS = CacheRedisTLS{Enabled: true, CAFile: filepath.Join(dir, "missing.pem")} + }, "cache.redis.tls.ca_file"}, + {"tls ca not pem", func(r *CacheRedisConfig) { r.TLS = CacheRedisTLS{Enabled: true, CAFile: notPEM} }, "no certificates in"}, + {"tls bad pair", func(r *CacheRedisConfig) { + r.TLS = CacheRedisTLS{Enabled: true, CertFile: certFile, KeyFile: notPEM} + }, "cache.redis.tls.cert_file"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + cfg := redisBackend() + tc.set(&cfg.Cache.Redis) + err := cfg.Validate() + if tc.want == "" { + require.NoError(t, err) + return + } + require.Error(t, err) + assert.Contains(t, err.Error(), tc.want) + }) + } +} + +// The redis block is read only when selected: an invalid one under +// backend=local does not refuse boot, it warns that it is ignored. +func TestValidate_CacheRedisIgnoredUnlessSelected(t *testing.T) { + t.Parallel() + cfg := defaultBackends() + cfg.Cache.Redis.Addrs = []string{"no-port"} + require.NoError(t, cfg.Validate()) + got := cfg.Warnings() + require.Len(t, got, 1) + assert.Contains(t, got[0], "cache.redis.addrs is set but cache.backend is local") +} + +func TestWarnings_CacheRedis(t *testing.T) { + t.Parallel() + cfg := redisBackend() + assert.Empty(t, cfg.Warnings()) + cfg.Cache.Redis.TLS = CacheRedisTLS{Enabled: true, InsecureSkipVerify: true} + require.NoError(t, cfg.Validate()) + got := cfg.Warnings() + require.Len(t, got, 1) + assert.Contains(t, got[0], "cache.redis.tls.insecure_skip_verify is on") + + // A shared cache clears the shared-queue warning about a local one. + cfg = redisBackend() + cfg.MQ.Backend = "shared" + got = cfg.Warnings() + require.Len(t, got, 1) + assert.Contains(t, got[0], "dedupe.backend=pebble") +} + +// writeTestPKI writes a self-signed authority and a client certificate it +// signed, returning the three paths cache.redis.tls names. +func writeTestPKI(t *testing.T, dir string) (caFile, certFile, keyFile string) { + t.Helper() + caKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + require.NoError(t, err) + ca := &x509.Certificate{ + SerialNumber: big.NewInt(1), Subject: pkix.Name{CommonName: "test ca"}, + NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(time.Hour), + IsCA: true, BasicConstraintsValid: true, KeyUsage: x509.KeyUsageCertSign, + } + caDER, err := x509.CreateCertificate(rand.Reader, ca, ca, &caKey.PublicKey, caKey) + require.NoError(t, err) + leafKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + require.NoError(t, err) + leaf := &x509.Certificate{ + SerialNumber: big.NewInt(2), Subject: pkix.Name{CommonName: "wavehouse"}, + NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(time.Hour), + KeyUsage: x509.KeyUsageDigitalSignature, ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}, + } + leafDER, err := x509.CreateCertificate(rand.Reader, leaf, ca, &leafKey.PublicKey, caKey) + require.NoError(t, err) + keyDER, err := x509.MarshalECPrivateKey(leafKey) + require.NoError(t, err) + write := func(name, typ string, der []byte) string { + path := filepath.Join(dir, name) + require.NoError(t, os.WriteFile(path, pem.EncodeToMemory(&pem.Block{Type: typ, Bytes: der}), 0o600)) + return path + } + return write("ca.pem", "CERTIFICATE", caDER), write("client.pem", "CERTIFICATE", leafDER), write("client.key", "EC PRIVATE KEY", keyDER) +} diff --git a/scripts/orchestrator/main.go b/scripts/orchestrator/main.go index a128d402e..10eacba89 100644 --- a/scripts/orchestrator/main.go +++ b/scripts/orchestrator/main.go @@ -1,10 +1,12 @@ // E2E orchestrator — drives a clean, isolated E2E test session against -// one ClickHouse + one WaveHouse, then runs the vitest suite once. +// one ClickHouse + one Redis + one WaveHouse, then runs the vitest suite +// once. // // Lifecycle: // -// 1. Start ClickHouse via testcontainers-go (random host ports — no -// conflict with `make dev` or other compose stacks). +// 1. Start ClickHouse and Redis (the fixture's shared cache) via +// testcontainers-go (random host ports — no conflict with `make dev` or +// other compose stacks). // 2. Pick a random free TCP port on 127.0.0.1 and start bin/wavehouse-cov // bound to it (WH_SERVER_PORT) with auth enabled. Random port avoids // conflicts with `make dev`, dev servers, and previous runs that may @@ -44,6 +46,7 @@ import ( "syscall" "time" + "github.com/moby/moby/api/types/container" "github.com/testcontainers/testcontainers-go" "github.com/testcontainers/testcontainers-go/wait" ) @@ -181,6 +184,43 @@ func run() error { return fmt.Errorf("settings dir: %w", err) } + // The shared cache the fixture's cache.backend=redis names: the suite + // runs the backend a multi-instance deployment runs. No persistence, and + // /data on tmpfs so the image's VOLUME leaves no anonymous volume behind. + log.Println("→ starting Redis testcontainer...") + redis, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{ + ContainerRequest: testcontainers.ContainerRequest{ + // Pinned to match internal/cache's integration suite. + Image: "redis:8.10.2-alpine", + Cmd: []string{"redis-server", "--save", "", "--appendonly", "no"}, + ExposedPorts: []string{"6379/tcp"}, + HostConfigModifier: func(hc *container.HostConfig) { + hc.Tmpfs = map[string]string{"/data": ""} + }, + WaitingFor: wait.ForLog("Ready to accept connections").WithStartupTimeout(60 * time.Second), + }, + Started: true, + }) + if err != nil { + return fmt.Errorf("redis start: %w", err) + } + defer func() { + log.Println("→ terminating Redis testcontainer...") + if err := redis.Terminate(context.Background()); err != nil { + log.Printf(" redis terminate: %v", err) + } + }() + redisHost, err := redis.Host(ctx) + if err != nil { + return fmt.Errorf("redis host: %w", err) + } + redisPort, err := redis.MappedPort(ctx, "6379") + if err != nil { + return fmt.Errorf("redis port: %w", err) + } + redisAddr := net.JoinHostPort(redisHost, redisPort.Port()) + log.Printf("✓ Redis ready: %s", redisAddr) + whPort, err := pickFreePort(ctx) if err != nil { return fmt.Errorf("pick free port: %w", err) @@ -198,14 +238,15 @@ func run() error { // tests/e2e/fixtures/config.yaml and the tunables, policy, roles, and // pipes in tests/e2e/fixtures/settings — edit them there, not here. The // vars below are the per-run dynamic overrides (port, scratch paths, the - // patched settings copy) plus GOCOVERDIR and WH_CONFIG, which can't live - // in YAML. + // patched settings copy, the Redis address) plus GOCOVERDIR and + // WH_CONFIG, which can't live in YAML. whCmd.Env = append(os.Environ(), "GOCOVERDIR="+coverDir, "WH_CONFIG="+filepath.Join(repoRoot, "tests", "e2e", "fixtures", "config.yaml"), "WH_SERVER_PORT="+strconv.Itoa(whPort), "WH_SETTINGS_DIR="+settingsDir, "WH_DATA_DIR="+dataDir, + "WH_CACHE_REDIS_ADDRS="+redisAddr, ) if os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT") == "" { diff --git a/tests/e2e/fixtures/config.yaml b/tests/e2e/fixtures/config.yaml index 4ce30280b..69e8cb33e 100644 --- a/tests/e2e/fixtures/config.yaml +++ b/tests/e2e/fixtures/config.yaml @@ -1,8 +1,9 @@ # WaveHouse config for the e2e harness (scripts/orchestrator + tests/e2e). # -# Dynamic values — ClickHouse addr/HTTP port (testcontainer), WaveHouse -# server port (free port), WH_DATA_DIR (per-run scratch) — are injected by -# the orchestrator via env vars. Everything below is pinned here so the +# Dynamic values — ClickHouse addr/HTTP port (testcontainer), the Redis +# address (testcontainer, WH_CACHE_REDIS_ADDRS), WaveHouse server port (free +# port), WH_DATA_DIR (per-run scratch) — are injected by the orchestrator via +# env vars. Everything below is pinned here so the # rig config is visible and editable without recompiling Go. # JWT validation. The suite signs test tokens with this fixed dev secret; @@ -13,6 +14,15 @@ auth: # operator_key: "" # non-JWT full-access operator credential (Authorization: Operator, or X-Operator-Key); unset in the rig +# The shared cache, as a multi-instance deployment runs it; the in-process +# backend is covered by the unit and integration suites. The timeout is ten +# times the default so a loaded runner's slow round trip is not a bypassed +# lookup that a HIT assertion reads as a failure. +cache: + backend: redis + redis: + timeout: 1s + # Tenant tunables (schema refresh_interval 5s so schema-discovery tests don't # wait a minute; dedupe enabled + id_field; DLQ on; CORS "*"; a 1 GiB NATS # stream budget so the testcontainer stays tiny), the policy, and the pipes diff --git a/tests/integration/shared_cache_test.go b/tests/integration/shared_cache_test.go new file mode 100644 index 000000000..b665ccb67 --- /dev/null +++ b/tests/integration/shared_cache_test.go @@ -0,0 +1,271 @@ +//go:build integration + +package tests + +import ( + "context" + "fmt" + "io" + "net" + "net/http" + "net/url" + "strings" + "sync/atomic" + "testing" + "time" + + "github.com/moby/moby/api/types/container" + "github.com/moby/moby/client" + "github.com/redis/rueidis" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "github.com/testcontainers/testcontainers-go" + "github.com/testcontainers/testcontainers-go/wait" + + "github.com/Wave-RF/WaveHouse/internal/app" + "github.com/Wave-RF/WaveHouse/internal/config" +) + +// Pinned, as internal/cache's integration suite pins it. +const redisImage = "redis:8.10.2-alpine" + +// minCacheTTL is cache.QueryTimeToTTL's floor: a fill made less than this +// long ago cannot have expired, so a miss inside it is an invalidation. +const minCacheTTL = 10 * time.Second + +var cachePrefixes atomic.Uint64 + +// startRedis runs a throwaway Redis with no persistence, its /data on tmpfs +// so the image's VOLUME leaves no anonymous volume behind. +func startRedis(t *testing.T) (testcontainers.Container, string) { + t.Helper() + ctx := context.Background() + ctr, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{ + ContainerRequest: testcontainers.ContainerRequest{ + Image: redisImage, + Cmd: []string{"redis-server", "--save", "", "--appendonly", "no"}, + ExposedPorts: []string{"6379/tcp"}, + HostConfigModifier: func(hc *container.HostConfig) { + hc.Tmpfs = map[string]string{"/data": ""} + }, + WaitingFor: wait.ForLog("Ready to accept connections").WithStartupTimeout(90 * time.Second), + }, + Started: true, + }) + testcontainers.CleanupContainer(t, ctr) + require.NoError(t, err) + host, err := ctr.Host(ctx) + require.NoError(t, err) + port, err := ctr.MappedPort(ctx, "6379/tcp") + require.NoError(t, err) + return ctr, net.JoinHostPort(host, port.Port()) +} + +// bootRedisApp runs a second, independent WaveHouse — its own embedded NATS, +// ingest worker and data_dir — against the suite's ClickHouse, with +// cache.backend=redis. It returns the instance's base URL. +func bootRedisApp(t *testing.T, redisAddr, prefix string, timeout time.Duration) string { + t.Helper() + e := env(t) + ctx := context.Background() + settingsDir, err := writeTestSettings(e.ch) + require.NoError(t, err) + var lc net.ListenConfig + ln, err := lc.Listen(ctx, "tcp", "127.0.0.1:0") + require.NoError(t, err) + cfg := &config.Config{ + DataDir: t.TempDir(), + Server: config.Server{ShutdownTimeout: 10}, + ClickHouse: config.ClickHouse{Password: testCHPassword}, + MQ: config.MQ{Backend: config.MQEmbedded}, + Cache: config.Cache{Backend: config.CacheRedis, Redis: config.CacheRedisConfig{ + Addrs: []string{redisAddr}, Mode: config.RedisStandalone, KeyPrefix: prefix, + Timeout: timeout, DialTimeout: 2 * time.Second, + MaxValueBytes: 1 << 20, CompressMinBytes: 1 << 10, VersionTTL: time.Hour, + }}, + Dedupe: config.Dedupe{Backend: config.DedupePebble}, + Coord: config.Coord{Backend: config.CoordLocal}, + Settings: config.Settings{Dir: settingsDir}, + } + a, err := app.New(ctx, app.Options{Config: cfg, Listener: ln}) + require.NoError(t, err) + runCtx, stop := context.WithCancel(ctx) + runDone := make(chan error, 1) + go func() { runDone <- a.Run(runCtx) }() + t.Cleanup(func() { + stop() + assert.NoError(t, <-runDone) + closeCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + assert.NoError(t, a.Close(closeCtx)) + }) + baseURL := "http://" + ln.Addr().String() + require.NoError(t, waitForLive(ctx, baseURL, 30*time.Second)) + return baseURL +} + +// structuredQuery posts a select-all structured query and returns the +// status, the X-Cache header and the body. +func structuredQuery(t *testing.T, baseURL, table string) (int, string, string) { + t.Helper() + status, xc, body, err := tryStructuredQuery(baseURL, table) + require.NoError(t, err) + return status, xc, body +} + +// tryStructuredQuery is structuredQuery for an Eventually condition, which +// runs off the test goroutine and so must not call require. +func tryStructuredQuery(baseURL, table string) (int, string, string, error) { + req, err := http.NewRequestWithContext(context.Background(), http.MethodPost, + baseURL+"/v1/query?table="+url.QueryEscape(table), strings.NewReader(`{"select_all":true}`)) + if err != nil { + return 0, "", "", err + } + req.Header.Set("Content-Type", "application/json") + resp, err := http.DefaultClient.Do(req) + if err != nil { + return 0, "", "", err + } + defer func() { _ = resp.Body.Close() }() + body, err := io.ReadAll(resp.Body) + return resp.StatusCode, resp.Header.Get("X-Cache"), string(body), err +} + +func ingestRow(t *testing.T, baseURL, table, user string) { + t.Helper() + resp, err := http.Post(baseURL+"/v1/ingest?table="+url.QueryEscape(table), "application/json", + strings.NewReader(fmt.Sprintf(`{"user_id":%q,"value":1}`, user))) + require.NoError(t, err) + defer func() { _ = resp.Body.Close() }() + require.Equal(t, http.StatusOK, resp.StatusCode) +} + +// Two WaveHouse processes share one Redis and one ClickHouse. A result one +// fills is a hit for the other, and an insert one process's worker makes +// invalidates what the other cached: the other's next query is a miss that +// returns the new row, well inside the TTL the stale entry was filed with. +func TestSharedCache_IngestOnOneInstanceInvalidatesAnother(t *testing.T) { + table := createTable(t, "user_id String, value Float64", "ORDER BY user_id") + _, redisAddr := startRedis(t) + prefix := fmt.Sprintf("it%d", cachePrefixes.Add(1)) + a := bootRedisApp(t, redisAddr, prefix, 2*time.Second) + b := bootRedisApp(t, redisAddr, prefix, 2*time.Second) + + rc, err := rueidis.NewClient(rueidis.ClientOption{InitAddress: []string{redisAddr}, DisableCache: true, ForceSingleClient: true}) + require.NoError(t, err) + t.Cleanup(rc.Close) + tableToken := func() string { + v, err := rc.Do(context.Background(), rc.B().Get().Key(prefix+":{0}:B:"+table).Build()).ToString() + if rueidis.IsRedisNil(err) { + return "" + } + require.NoError(t, err) + return v + } + + status, xc, body := structuredQuery(t, b, table) + require.Equal(t, http.StatusOK, status, body) + require.Equal(t, "MISS", xc) + status, xc, _ = structuredQuery(t, a, table) + require.Equal(t, http.StatusOK, status) + require.Equal(t, "HIT", xc, "a fills, b hits: one cache") + + // An insert's worker and its invalidation run on whichever process took + // the ingest, so the discriminating window is the stale entry's TTL: if + // the batch window and load push the new row past it, the round proves + // nothing and runs again with a fresh fill. + for round := 1; ; round++ { + user := fmt.Sprintf("user-%d", round) + status, xc, body = structuredQuery(t, b, table) + require.Equal(t, http.StatusOK, status, body) + require.NotContains(t, body, user) + filled := time.Now() + if xc == "HIT" { + // The previous round's fill: refresh it so the TTL window starts now. + _, err := rc.Do(context.Background(), rc.B().Flushdb().Build()).ToString() + require.NoError(t, err) + status, xc, body = structuredQuery(t, b, table) + require.Equal(t, http.StatusOK, status, body) + require.Equal(t, "MISS", xc) + filled = time.Now() + } + before := tableToken() + + ingestRow(t, a, table, user) + var seenAt time.Time + require.Eventually(t, func() bool { + var err error + status, xc, body, err = tryStructuredQuery(b, table) + if err == nil && status == http.StatusOK && strings.Contains(body, user) { + seenAt = time.Now() + return true + } + return false + }, 30*time.Second, 100*time.Millisecond, "b never served the row ingested through a") + assert.NotEqual(t, before, tableToken(), "a's worker bumped the table token in the shared server") + if seenAt.Sub(filled) < minCacheTTL-time.Second { + assert.Equal(t, "MISS", xc, "the first answer carrying the new row is b's refill") + status, xc, _ = structuredQuery(t, b, table) + require.Equal(t, http.StatusOK, status) + assert.Equal(t, "HIT", xc, "b's refill is cached again") + return + } + require.Less(t, round, 3, "b served the new row only once its stale entry could have expired, in every round: the invalidation never reached it, or ingest is too slow here to tell") + t.Logf("round %d: row landed %s after the fill, past the TTL floor; retrying", round, seenAt.Sub(filled)) + } +} + +// A Redis that stops answering costs queries nothing but the cache: they +// keep succeeding, straight from ClickHouse, each a miss; an ingest made +// meanwhile is visible at once. Once it answers again, the cache serves hits. +func TestSharedCache_RedisDownQueriesBypass(t *testing.T) { + ctx := context.Background() + table := createTable(t, "user_id String, value Float64", "ORDER BY user_id") + ctr, redisAddr := startRedis(t) + const timeout = 200 * time.Millisecond + a := bootRedisApp(t, redisAddr, fmt.Sprintf("it%d", cachePrefixes.Add(1)), timeout) + + status, xc, _ := structuredQuery(t, a, table) + require.Equal(t, http.StatusOK, status) + require.Equal(t, "MISS", xc) + status, xc, _ = structuredQuery(t, a, table) + require.Equal(t, http.StatusOK, status) + require.Equal(t, "HIT", xc) + + d, err := testcontainers.NewDockerClientWithOpts(ctx) + require.NoError(t, err) + t.Cleanup(func() { _ = d.Close() }) + _, err = d.ContainerPause(ctx, ctr.GetContainerID(), client.ContainerPauseOptions{}) + require.NoError(t, err) + paused := true + unpause := func() { + if paused { + paused = false + _, err := d.ContainerUnpause(ctx, ctr.GetContainerID(), client.ContainerUnpauseOptions{}) + require.NoError(t, err) + } + } + t.Cleanup(unpause) + + for range 8 { + start := time.Now() + status, xc, body := structuredQuery(t, a, table) + require.Equal(t, http.StatusOK, status, body) + assert.Equal(t, "MISS", xc) + // A lookup and a fill each wait at most the timeout; the query itself + // is a few ms. Generous for -race under load. + assert.Less(t, time.Since(start), 5*timeout+2*time.Second) + } + + ingestRow(t, a, table, "while-down") + require.Eventually(t, func() bool { + status, _, body, err := tryStructuredQuery(a, table) + return err == nil && status == http.StatusOK && strings.Contains(body, "while-down") + }, 30*time.Second, 200*time.Millisecond, "a row ingested while the cache is down is served") + + unpause() + require.Eventually(t, func() bool { + _, xc, body, err := tryStructuredQuery(a, table) + return err == nil && xc == "HIT" && strings.Contains(body, "while-down") + }, 30*time.Second, 200*time.Millisecond, "the cache serves hits again once the server answers") +} From 13c6cb3014d211ed9f41f47c98ad49746599d0b4 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Fri, 25 Sep 2026 03:44:18 -0400 Subject: [PATCH 11/45] =?UTF-8?q?fix(cache):=20review=20round=20=E2=80=94?= =?UTF-8?q?=20e2e=20gate,=20trust=20boundary,=20#386,=20addr=20spaces?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Exclude LocalCache from the e2e gate now that e2e runs on Redis; document the shared server as a trust boundary, the noeviction and mutation-pipe (#386) staleness cases, and the connection commands an ACL user needs; refuse addresses with surrounding spaces. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01EJr5tY4WQUy2sc4MbW67vL --- .testcoverage.yml | 5 +++++ docs/src/content/docs/api.md | 6 +++--- docs/src/content/docs/architecture.md | 2 +- docs/src/content/docs/configuration.mdx | 6 +++--- docs/src/content/docs/deployment.md | 12 +++++++++++- internal/config/cache_redis.go | 6 ++++-- internal/config/cache_redis_test.go | 1 + 7 files changed, 28 insertions(+), 10 deletions(-) diff --git a/.testcoverage.yml b/.testcoverage.yml index aff1a694d..59ff6d0b7 100644 --- a/.testcoverage.yml +++ b/.testcoverage.yml @@ -73,3 +73,8 @@ exclude: - ^internal/settings/ - ^cmd/wavehouse/validate\.go$ - ^cmd/wavehouse/bootstrap\.go$ + # The in-process cache backend: the e2e stack runs cache.backend=redis + # (#613), so the binary carries LocalCache and its version index but e2e + # never reaches them. The unit suite and the integration suite's main + # app (cache.backend=local) cover them; the merged total still counts them. + - ^internal/cache/(local|version_manager)\.go$ diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index c82739c07..14f5d1308 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -422,7 +422,7 @@ ClickHouse's inline `FORMAT` clause (e.g. `SELECT 1 FORMAT CSV` or `… FORMAT P The proxy buffers the upstream response in memory before forwarding (no row-streaming yet), so a `SELECT *` from a large table can pin RAM on the API server. To avoid an admin OOMing themselves, responses larger than 64 MiB return 502 with a `clickhouse response exceeded N bytes` error. Narrow the query with `LIMIT`, or use a streaming client outside WaveHouse that talks to ClickHouse directly (the standard escape hatch — the same admin credentials work). ::: -This endpoint **does not cache, does not singleflight, and emits `Cache-Control: no-store`** — every request goes straight to ClickHouse, mutation or read, and downstream HTTP caches are explicitly told not to store the response. Raw SQL is an admin escape hatch with infrequent, ad-hoc traffic, so the L1/singleflight machinery would only add complexity without a real hit-rate win. Use [`POST /v1/query?table={table}`](#post-v1querytabletable--structured-query) or [`GET/POST /v1/pipes/{name}`](#getpost-v1pipesname--execute-named-pipe) for the cached read paths (dashboards, high-QPS clients, etc.) — both share an in-process L1 (Ristretto) with singleflight coalescing. +This endpoint **does not cache, does not singleflight, and emits `Cache-Control: no-store`** — every request goes straight to ClickHouse, mutation or read, and downstream HTTP caches are explicitly told not to store the response. Raw SQL is an admin escape hatch with infrequent, ad-hoc traffic, so the L1/singleflight machinery would only add complexity without a real hit-rate win. Use [`POST /v1/query?table={table}`](#post-v1querytabletable--structured-query) or [`GET/POST /v1/pipes/{name}`](#getpost-v1pipesname--execute-named-pipe) for the cached read paths (dashboards, high-QPS clients, etc.) — both go through the query cache ([`cache.backend`](/configuration#backends): in-process, or a Redis shared by every instance) with singleflight coalescing. :::note[Admin only] The route is mounted under `/v1/ops/*`, behind the `RequireAdmin` gate: only a caller whose JWT role equals the policy `admin_role` (`"admin"` by default) — or who presents the non-JWT [operator key](#authentication) — may use it. A tokenless request (or a valid token without a role claim) resolves to the `default_role` (not the admin role unless `default_role` is deliberately set to it — a loudly-warned dev-only setting) and is rejected with `403`; a present-but-invalid token — expired, malformed, bad signature — keeps its stashed verification error and fails loud with `401` instead. Raw SQL has no per-statement scope check (a full SQL parser would be needed to authorize predicates), so the role gate is the entire authorization story, shared with the rest of `/v1/ops/*` (see [Admin Endpoints](#admin-endpoints)). The normal surfaces for non-admin callers are `POST /v1/ingest?table={table}` for writes, `POST /v1/query?table={table}` for structured reads, and `GET/POST /v1/pipes/{name}` for pre-defined queries — none of which expose raw SQL. @@ -538,7 +538,7 @@ Table, column, and alias names may contain any characters ClickHouse accepts — **Response:** -JSON array of result rows. Top-level `DateTime`/`DateTime64` values are returned in canonical RFC 3339 UTC (`2026-06-21T04:00:00.123Z`) — `Nullable` timestamp columns included (a SQL `NULL` renders as JSON `null`), while timestamps nested inside `Array`/`Map`/`Tuple` columns are rendered in the column's declared zone, else the ClickHouse server's, as the driver returns them — byte-identical to the [SSE stream](#get-v1stream--server-sent-events-stream) for values [canonicalized at ingest](#timestamp-canonicalization) (a fail-open pass-through that ClickHouse accepted still comes back canonical here, though it streamed in the producer's spelling). The response carries an `X-Cache: HIT` or `X-Cache: MISS` header — this endpoint shares the in-process L1 (Ristretto) + singleflight machinery (unlike `/v1/ops/query`, which always hits ClickHouse), keyed by [tenant](/deployment#multi-tenant-deployments): a request is never served from, or coalesced with, another tenant's. +JSON array of result rows. Top-level `DateTime`/`DateTime64` values are returned in canonical RFC 3339 UTC (`2026-06-21T04:00:00.123Z`) — `Nullable` timestamp columns included (a SQL `NULL` renders as JSON `null`), while timestamps nested inside `Array`/`Map`/`Tuple` columns are rendered in the column's declared zone, else the ClickHouse server's, as the driver returns them — byte-identical to the [SSE stream](#get-v1stream--server-sent-events-stream) for values [canonicalized at ingest](#timestamp-canonicalization) (a fail-open pass-through that ClickHouse accepted still comes back canonical here, though it streamed in the producer's spelling). The response carries an `X-Cache: HIT` or `X-Cache: MISS` header — this endpoint shares the query cache + singleflight machinery (unlike `/v1/ops/query`, which always hits ClickHouse), keyed by [tenant](/deployment#multi-tenant-deployments): a request is never served from, or coalesced with, another tenant's. The inbound request body is capped at 1 MiB; a body over the cap is rejected with `413`. A query AST is bounded by nature (far under 1 MiB even with a large `in`-list), and the cap blocks a single-request memory-exhaustion vector on this public endpoint. Set a tighter or higher outer limit at your [reverse proxy](/reverse-proxy#request-body-size-limits) — but it can only narrow the effective limit, not raise it past this cap. @@ -575,7 +575,7 @@ Executes a pre-defined named query (pipe) with parameter binding. Parameters can **Response:** -JSON array of result rows, with `X-Cache: HIT` or `X-Cache: MISS` indicating whether the row came from the in-process L1. +JSON array of result rows, with `X-Cache: HIT` or `X-Cache: MISS` indicating whether the rows came from the query cache. The POST parameter body is capped at 1 MiB; a body over the cap is rejected with `413` (the same 1 MiB parameter/AST-body cap as [`POST /v1/query`](#post-v1querytabletable--structured-query) — see [reverse proxy → body limits](/reverse-proxy#request-body-size-limits)). A malformed-but-within-cap body is ignored rather than rejected, since parameters may legitimately come from the query string alone. diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 6c59e5f16..a16d01b9d 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -112,7 +112,7 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ - **cache.go** — `Cache` interface: `Lookup`, `Set`, `Invalidate`, `InvalidateTenant`, `Close`, plus `QueryTimeToTTL`, which sets a result's TTL from how long its query took (10 s floor, 1 h ceiling). Every entry is one tenant's: `Lookup` takes the tenant, the caller's query key — `:query:`, built by the two cached handlers in `api/` (`queryCacheKey`, with the tenant read off the request's store — `settings.Store.Tenant`), which use it as their [singleflight](https://pkg.go.dev/golang.org/x/sync/singleflight) key too — and the `Namespace`s the result depends on, each of that tenant (another tenant's is `ErrForeignDependency`) and naming a table and scope: one for a structured query, none yet for a pipe (a pipe's table dependencies are [#343](https://github.com/Wave-RF/WaveHouse/pull/343)). `Lookup` returns the `Entry` (a nil value is a miss) and a `Snapshot` of the versions it read; on a miss the handler runs the query and passes that snapshot to `Set`, so a result is filed under the versions read *before* its query ran, and a write that lands while it runs orphans the fill rather than re-homing pre-write rows under the post-write versions ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)). The singleflight leader's snapshot is the one used. `Set` returns an error only when the backend failed; a value the cache declines — larger than it keeps, a non-positive TTL, a zero snapshot — is not one. What a hit, a miss and a bump mean is pinned by the conformance suite every backend runs, `internal/testutil/cachetest`. - **local.go** — `LocalCache`, the in-process L1 on [Ristretto](https://github.com/dgraph-io/ristretto): one pool shared by every tenant (a heavier tenant holds more of it), sized by the boot config's `cache.l1_max_cost`. - **version_manager.go** — `VersionManager`, the invalidation index behind `Invalidate` and `InvalidateTenant`: one version per tenant, per (tenant, table) and per (tenant, table, scope), each keyed by its name alone and bumped in place, so the index holds one entry per live tenant, table and scope however often each is bumped ([#262](https://github.com/Wave-RF/WaveHouse/issues/262)). A query key folds the tenant's version and, for each dependency, its tenant's, table's and scope's, so bumping a table (a scopeless write) orphans every scope of it, and bumping one scope orphans that scope and the whole-table view — scope is reserved and empty today, so every write is the whole-table bump — all without touching the pool. The tenant leads every key ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8): the same table under two tenants is two namespaces, so a bump through `Invalidate` under one tenant never touches — and a read under one tenant is never served — the other's results. A tenant's version is a *generation*, unique within the process and handed out by the first key built for the tenant; `BumpTenant` (behind `InvalidateTenant`) drops the tenant's whole index, so the next key gets a fresh generation no cached entry folds, orphaning every cached result of the tenant in one step — a pipe result with no dependencies, and a table no bump ever keyed, included — for a tenant back on a pool after an absence from the fan-out, or moved to another address or database (story 6). `LocalCache.Prune` does the same for every tenant no longer served, which `internal/app` runs after each settings reload, so a tenant removed or rejected stops holding its index. A table bump drops the table's scope versions with it, since every key they were folded into also folds the old table version; and a bump of a tenant with no index is a no-op, since no key folds its next generation yet. The index is per tenant; the cross-tenant invalidation an insert into a shared table needs is not the index's but the wiring's: `internal/app` hands the ingest worker a cache (`sharedTables`) that repeats each bump under every tenant on the same ClickHouse address and database. -- **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — only `GET`, `SET`, `MGET` and `PING`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, and a token key holding anything but a token (a string of another length, a list, a hash) is replaced, so a token lost to eviction, expiry or a restart is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing. `cache.backend: redis` selects it: `internal/app`'s `wireCache` maps the boot config's `cache.redis` block onto `RedisConfig`, reading the TLS files, and releases it with the other components. +- **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — its data commands are only `GET`, `SET`, `MGET` and `PING`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, and a token key holding anything but a token (a string of another length, a list, a hash) is replaced, so a token lost to eviction, expiry or a restart is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing. `cache.backend: redis` selects it: `internal/app`'s `wireCache` maps the boot config's `cache.redis` block onto `RedisConfig`, reading the TLS files, and releases it with the other components. - **redis_codec.go** — the key schema (`tokenKeys`, `bumpKeys`, `valueKey`) and the value frame: format, flags, expiry, the token list, the payload, zstd-compressed from `CompressMinBytes` when that is smaller. A stored value is capped at `MaxValueBytes` and a decoded one at eight times that, which refuses a zip bomb planted in a shared server; an unknown format, as a newer process writes during a rolling upgrade, is a miss. - **breaker.go** — the circuit breaker: `BreakerThreshold` (5) consecutive failures open it, and while open every operation skips the server at once; after `BreakerOpenFor` (5 s) one background `PING` decides whether it closes. Only a transport failure or a timeout counts against the server: any reply, an error reply or one the backend cannot use included, counts as a success, for operations and the probe alike, and a caller that gave up first counts as nothing. - **pending.go** — invalidations the server did not take, kept per token key (repeats coalesce) and retried by a background loop with backoff from 100 ms to 10 s until they land. Landing late is still correct: a fresh token orphans the pre-write entries and any fill made meanwhile. Past `PendingMax` (100,000) keys the set collapses to one tenant bump per affected tenant. `Close` makes one last attempt at the bumps still owed, past the breaker (an open one is why they are owed); what that attempt cannot deliver is lost, and the entries they would orphan are served until their TTL — the same failure as a worker stopping between an insert and its invalidation. diff --git a/docs/src/content/docs/configuration.mdx b/docs/src/content/docs/configuration.mdx index 82744a359..f03a2854f 100644 --- a/docs/src/content/docs/configuration.mdx +++ b/docs/src/content/docs/configuration.mdx @@ -122,11 +122,11 @@ Each tenant's queue has its own disk budget, `mq.max_bytes_gb`, a hot-reloadable | --- | --- | ------- | ----------- | | `cache.l1_max_cost` | `WH_CACHE_L1_MAX_COST` | `67108864` | Maximum size in bytes (~64 MB) of the `local` backend's in-process cache. The time-range bucket structured queries normalize to is `query.timestamp_bucket_seconds` in the [Settings Directory](/settings-directory#configjson-keys). | -The `redis` backend's settings, read only when `cache.backend` is `redis`. It runs on Redis, Valkey, Dragonfly, ElastiCache (including Serverless) and MemoryDB: it sends only `GET`, `SET`, `MGET` and `PING`. [Deployment](/deployment#multiple-instances-and-the-shared-cache) covers sizing, `maxmemory-policy` and what a reader on another instance can see. +The `redis` backend's settings, read only when `cache.backend` is `redis`. It is tested on Redis, Valkey, Dragonfly and a Redis Cluster node, and its data commands are only `GET`, `SET`, `MGET` and `PING` (no scripts, no client tracking), which ElastiCache and MemoryDB also serve. An ACL user also needs the connection commands the client sends when it dials: `HELLO`, `CLIENT`, `SELECT`, and `CLUSTER` in cluster mode; without them it is refused (`NOPERM`) and the cache stays bypassed. Whoever can write to the server can replace cached query results, which are served after the access policy has already been applied, so treat the server as part of WaveHouse's trust boundary (see [Deployment](/deployment#multiple-instances-and-the-shared-cache)). [Deployment](/deployment#multiple-instances-and-the-shared-cache) covers sizing, `maxmemory-policy` and what a reader on another instance can see. | YAML Key | Env Var | Default | Description | | --- | --- | ------- | ----------- | -| `cache.redis.addrs` | `WH_CACHE_REDIS_ADDRS` | — | **Required** with `backend: redis.` `host:port` of the server; several are a cluster's seed nodes or the sentinels. Comma-separated in the env var. | +| `cache.redis.addrs` | `WH_CACHE_REDIS_ADDRS` | — | **Required** with `backend: redis`. `host:port` of the server; several are a cluster's seed nodes or the sentinels. Comma-separated in the env var. | | `cache.redis.mode` | `WH_CACHE_REDIS_MODE` | `standalone` | `standalone`, `cluster` or `sentinel`. | | `cache.redis.sentinel_master` | `WH_CACHE_REDIS_SENTINEL_MASTER` | — | The master set name. Required with `mode: sentinel`. | | `cache.redis.username` | `WH_CACHE_REDIS_USERNAME` | — | ACL user. Empty uses the server's `default` user. | @@ -138,7 +138,7 @@ The `redis` backend's settings, read only when `cache.backend` is `redis`. It ru | `cache.redis.tls.key_file` | `WH_CACHE_REDIS_TLS_KEY_FILE` | — | The client certificate's private key (PEM). | | `cache.redis.tls.server_name` | `WH_CACHE_REDIS_TLS_SERVER_NAME` | — | Name to verify the server's certificate against, when it differs from the address. | | `cache.redis.tls.insecure_skip_verify` | `WH_CACHE_REDIS_TLS_INSECURE_SKIP_VERIFY` | `false` | Accept any server certificate. Logged at `WARN` at boot: whoever can intercept the connection can read and replace cached results. | -| `cache.redis.key_prefix` | `WH_CACHE_REDIS_KEY_PREFIX` | `wh` | Leads every key, so several deployments can share one server. No `{` or `}`. | +| `cache.redis.key_prefix` | `WH_CACHE_REDIS_KEY_PREFIX` | `wh` | Leads every key, so several deployments can share one server, provided you trust each as much as the others: any of them can overwrite what the rest serve. No `{` or `}`. | | `cache.redis.timeout` | `WH_CACHE_REDIS_TIMEOUT` | `100ms` | Per operation. A lookup or fill that takes longer is a miss or a skipped fill, never a failed query. | | `cache.redis.dial_timeout` | `WH_CACHE_REDIS_DIAL_TIMEOUT` | `1s` | Per connection attempt. | | `cache.redis.max_value_bytes` | `WH_CACHE_REDIS_MAX_VALUE_BYTES` | `1048576` | Largest result stored, after compression (1 MiB). A larger one is returned to the caller but not cached. | diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index 63f1b1a9c..b94beb18c 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -151,6 +151,12 @@ WH_AUTH_JWT_SECRET= # as an admin secret — inject from your secret store, serve only over TLS. WH_AUTH_OPERATOR_KEY= +# Optional shared query cache for several instances (see Multiple instances +# and the shared cache below); the password is a secret like the ones above. +# WH_CACHE_BACKEND=redis +# WH_CACHE_REDIS_ADDRS=redis:6379 +# WH_CACHE_REDIS_PASSWORD= + # Settings directory (required): roles.json, policies.json, pipes.json, # config.json — the hot-reloadable configuration: the access-control policy # and its roles, the named pipes, and the tunables including the ClickHouse @@ -407,9 +413,13 @@ The query-result cache is the layer that can be shared today. With the default ` - **The server is unreachable from the inserting instance.** The invalidation is kept and retried until it lands (`wavehouse_cache_invalidations_pending` counts what is owed). Meanwhile other instances that can still reach the server keep serving the older results, for as long as the outage lasts and at most until each entry's TTL. An instance that stops while invalidations are still owed loses them, with the same bound. The same thing happens today when a process stops between an insert and its invalidation. - **A failover to a replica that had not yet received the latest token writes** can bring back entries filed under the older tokens, bounded by the replication lag at the moment of failover and those entries' TTL. WaveHouse never reads from replicas. +- **The server is full and `maxmemory-policy` is `noeviction`.** It refuses the token writes, so invalidations are kept and retried, and until one lands every instance serves the results from before the insert, up to their TTL. +- **A pipe that writes** (an `INSERT` in `pipes.json`) has its result cached like a read, so a repeated identical call is answered from the cache and the write does not run again ([#386](https://github.com/Wave-RF/WaveHouse/issues/386)). With a shared cache that holds on every instance, until the entry's TTL. - **Admin writes through `POST /v1/ops/query`** do not invalidate the cache ([#394](https://github.com/Wave-RF/WaveHouse/issues/394)). With a shared cache, the stale results they leave are served by every instance, not only one. -**Sizing the server.** Every key WaveHouse writes has a TTL, and a version token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses, never bring back an entry it had invalidated. So set `maxmemory` and let the server evict: `maxmemory-policy allkeys-lru` (or `allkeys-lfu`, `volatile-lru`, `volatile-lfu`). Under `noeviction`, a full server refuses the writes. Fills then fail (counted by `wavehouse_cache_set_failures_total{reason="oom"}`), lookups keep working, and invalidations are kept and retried. A stored result is capped at `cache.redis.max_value_bytes` (1 MiB compressed). A tenant's version tokens share one hash tag, so each lookup reads them in one `MGET` in cluster mode as well. The results themselves carry no hash tag and spread across shards. Persistence is not needed: an empty server after a restart is a cold cache, not a wrong one. +**Sizing the server.** Every key WaveHouse writes has a TTL, and a version token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses, never bring back an entry it had invalidated. So set `maxmemory` and let the server evict: `maxmemory-policy allkeys-lru` (or `allkeys-lfu`, `volatile-lru`, `volatile-lfu`). Under `noeviction`, a full server refuses the writes. Fills then fail (counted by `wavehouse_cache_set_failures_total{reason="oom"}`), only lookups whose version tokens already exist keep working, and invalidations are kept and retried, so the pre-insert results above stay served: avoid `noeviction`. A stored result is capped at `cache.redis.max_value_bytes` (1 MiB compressed). A tenant's version tokens share one hash tag, so each lookup reads them in one `MGET` in cluster mode as well. The results themselves carry no hash tag and spread across shards. Persistence is not needed: an empty server after a restart is a cold cache, not a wrong one. + +**The server is inside the trust boundary.** A cached result is served after the access policy has filtered it, so whoever can write to the server can change what any caller reads. Keep it on a private network, require a password or ACL user (`WH_CACHE_REDIS_PASSWORD`), use TLS across links you do not trust, and share it only with deployments you trust as much as this one. **Coalescing stays per instance.** `singleflight` collapses identical concurrent queries within each instance, so a cold hot query costs at most one ClickHouse query per instance, not one per request. diff --git a/internal/config/cache_redis.go b/internal/config/cache_redis.go index f0af67137..3ae3114ea 100644 --- a/internal/config/cache_redis.go +++ b/internal/config/cache_redis.go @@ -55,8 +55,7 @@ type CacheRedisTLS struct { InsecureSkipVerify bool `yaml:"insecure_skip_verify" env:"WH_CACHE_REDIS_TLS_INSECURE_SKIP_VERIFY"` } -// hasAddrs reports whether any address is set. An env file's blank -// `WH_CACHE_REDIS_ADDRS=` loads as one empty address, which is none. +// hasAddrs reports whether any address is set; a YAML `addrs: [""]` is none. func (r CacheRedisConfig) hasAddrs() bool { return len(r.Addrs) > 1 || len(r.Addrs) == 1 && r.Addrs[0] != "" } @@ -66,6 +65,9 @@ func (r CacheRedisConfig) validate() error { return errors.New("cache.backend=redis needs cache.redis.addrs (WH_CACHE_REDIS_ADDRS): the server's host:port, or a cluster's seeds, or the sentinels") } for _, a := range r.Addrs { + if strings.TrimSpace(a) != a { + return fmt.Errorf("cache.redis.addrs (WH_CACHE_REDIS_ADDRS) %q: no spaces around an address", a) + } if _, _, err := net.SplitHostPort(a); err != nil { return fmt.Errorf("cache.redis.addrs (WH_CACHE_REDIS_ADDRS) %q: want host:port: %w", a, err) } diff --git a/internal/config/cache_redis_test.go b/internal/config/cache_redis_test.go index 6d989c737..e1b5783a8 100644 --- a/internal/config/cache_redis_test.go +++ b/internal/config/cache_redis_test.go @@ -189,6 +189,7 @@ func TestValidate_CacheRedis(t *testing.T) { }{ {"defaults", func(*CacheRedisConfig) {}, ""}, {"no addrs", func(r *CacheRedisConfig) { r.Addrs = nil }, "cache.backend=redis needs cache.redis.addrs (WH_CACHE_REDIS_ADDRS)"}, + {"addr with space", func(r *CacheRedisConfig) { r.Addrs = []string{"a:6379", " b:6379"} }, "no spaces around an address"}, {"addr without port", func(r *CacheRedisConfig) { r.Addrs = []string{"redis"} }, `cache.redis.addrs (WH_CACHE_REDIS_ADDRS) "redis": want host:port`}, {"mode", func(r *CacheRedisConfig) { r.Mode = "replica" }, `cache.redis.mode (WH_CACHE_REDIS_MODE) "replica": valid: standalone, cluster, sentinel`}, {"sentinel without master", func(r *CacheRedisConfig) { r.Mode = RedisSentinel }, "needs cache.redis.sentinel_master"}, From 40d817d44dc4de8dba61ab804042262f3385eb26 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Fri, 25 Sep 2026 19:47:09 -0400 Subject: [PATCH 12/45] refactor(cache): the Redis codec escapes the raw names it keys A Namespace now carries raw table and scope names, so the Redis codec builds its token keys with keyenc.AppendJoin after the fixed ":{tenant}:B:" and ":S:" pieces; the {tenant} hash tag is placed as it was, since a tenant id is its own escaped form. A ':' in a table or scope no longer reads as the separator: table "a:b" with scope "c" and table "a" with scope "b:c" had one scope token. valueKey hashed each dep's raw table and scope ended by NULs, so a name holding a NUL could hash the same input as another set of names: table "a\x00b" and table "a" with scope "b\x00" shared a value key. It now hashes the escaped sha and each dep's escaped, joined form, each ended by a NUL, which escaping never writes. The key layout is otherwise unchanged. The conformance cases added with the raw-name change run against Redis, Valkey, Dragonfly and a Redis Cluster node, and the old codec fails "names never run together" on all four. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01B1tJWUp6oaDoH1usLwGtLF --- CHANGELOG.md | 2 +- docs/src/content/docs/architecture.md | 4 ++-- internal/cache/redis_codec.go | 24 +++++++++++-------- internal/cache/redis_codec_test.go | 33 +++++++++++++++++++++++---- 4 files changed, 46 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4227d58fa..3c1edc923 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **A Redis-compatible shared cache backend, built but not yet selectable** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row open a circuit breaker that skips the server until a probe succeeds, and deferred invalidations are retried until they land, collapsing to one tenant-wide bump per tenant past 100,000 keys. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. Nothing selects it yet: the `cache.backend` switch and the `cache.redis.*` settings arrive with the wiring (E4), so every deployment still runs `LocalCache`. Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression and server-stops-answering cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. +- **A Redis-compatible shared cache backend, built but not yet selectable** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag, with the table and scope escaped into the key (`internal/keyenc`) so no two names share a token; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row open a circuit breaker that skips the server until a probe succeeds, and deferred invalidations are retried until they land, collapsing to one tenant-wide bump per tenant past 100,000 keys. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. Nothing selects it yet: the `cache.backend` switch and the `cache.redis.*` settings arrive with the wiring (E4), so every deployment still runs `LocalCache`. Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression and server-stops-answering cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. - **A tenant removed or rejected at runtime has its open streams ended** (`internal/stream/{hub,subscriber,bucket}.go` (+ tests), `internal/api/stream.go` (+ tests), `internal/api/router.go`, `internal/settings/{registry,tree}.go` (+ tests), `internal/ingest/worker.go` (+ tests), `internal/app/wire.go` (+ tests), `clients/ts/src/stream/sse.ts`, `docs/src/content/docs/{api,deployment,architecture,ingest-pipeline}.md`, `docs/src/content/docs/settings-directory.mdx`, `AGENTS.md`): story 3 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)). A `GET /v1/stream` used to outlive its tenant: the hub read no policy for it and withheld every row while the keepalive wheel held the connection open, so the client could not tell it from a quiet table. After every reload the hub now evicts the subscribers of each tenant no longer served, removed or rejected alike (`Hub.Prune`, through the close-once `Subscriber.Evict`), and the handler ends the stream: a gap-fill in progress included, and a stream `TenantMW` admitted just before the reload but registered just after it, which the handler checks for as it registers. The client's reconnect gets `404` (removed) or `503` (rejected); the SDK stops on the first and retries the second, resuming from `Last-Event-ID` once the folder is back — except in a browser going cross-origin where tenant `0` is not served or its CORS list does not admit the page, which cannot read either refusal (both are decorated from tenant `0`'s list) and re-dials as after a dropped connection. A flat directory never stops serving tenant `0`, so nothing changes there. Removing a tenant is deleting its folder, then reloading the whole directory, the last folder included: a server started with tenant folders reads the emptied directory as no folder left, where it read as a change of shape and the reload was rejected whole, leaving that tenant served; at boot an empty directory still reads as the four files, missing. Reloading the deleted folder by name leaves its tenant rejected. `GET /v1/health` keeps resolving a tenant, deliberately: its `404` tells a caller with no token no more than every tenant route's does, since they all answer before authenticating, and resolving answers a served tenant's ping from that tenant's own CORS list. A batch queued for a tenant with no ClickHouse connection — one no longer served, or one no pool could be opened for (such as by the connection ceiling) — skips the row-by-row retry, which no row of it could pass, and meets its DLQ switch once, whole, logged once per batch rather than twice per row; a tenant no longer served reads the switch as on, so its queued rows are parked under its own subject rather than dropped, inserted into another tenant's ClickHouse, or left unacked, redelivered for as long as the tenant is away and holding its queue's ack floor, which the purge waits on. Over a nested directory `/livez` no longer keeps naming a tenant that stopped being served before any tenant completed a first discovery: the diagnostic goes back to `no tenant has completed a first discovery yet`. - **One ClickHouse pool per tuple and one schema registry per tenant** (`internal/chconn/chconn.go` (+ tests), `internal/discovery/discovery.go` (+ tests), `internal/app/discoveries.go` (new), `internal/app/{app,wire}.go` (+ tests), `internal/api/{schema,ingest,structured_query,pipes,query,health,errors,clickhouse_exec}.go` (+ tests), `internal/stream/hub.go`, `internal/ingest/worker.go`, `internal/cache/{cache,local,version_manager}.go` (+ tests), `internal/testutil/{testutil,mocks}.go`, `tests/integration/{setup,tenants,boot_resilience,query_limits}_test.go`, `clients/ts/src/{schema,table,sql,client,types}.ts` (+ tests), `tests/e2e/sdk/admin.test.ts`, `docs/src/content/docs/{api,deployment,architecture,ingest-pipeline}.md`, `docs/src/content/docs/{settings-directory,configuration,access-control,reverse-proxy}.mdx`, `docs/src/content/docs/sdk/{admin,reference,queries}.md`, `AGENTS.md`): the second slice of story 6 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)), with no behavior change for a settings directory that holds the four files beyond the three noted at the end. The process opens one native pool per distinct `clickhouse.addr` / `database` / `username` / password / `tls` tuple among the served tenants (`chconn.Identity`, `chconn.Pools`), shared by the tenants naming it and sized to their largest `max_open_conns` and `max_idle_conns`; `http_port`, `http_scheme`, `headers` and `query_timeout` stay each tenant's own. Every reload reconciles the pools: a new tuple opens (never dials), a tenant whose tuple changed is repointed, a tuple no tenant names closes after the longest `query_timeout` among the tenants it had, and a pool whose largest ask changed is resized with the same grace. The boot config's `clickhouse.max_total_conns` now bounds the open pools together: boot is refused naming the sum and the ceiling; at a reload a resize above it is refused with the pool kept at its size, and a tuple that cannot be opened — the ceiling, a certificate file that cannot be read, or options the driver refuses — leaves its tenants on the pool they had (the keep-previous-wiring rule of the connection ceiling) or on none when they had none; both are logged and the next reload retries. A tenant on no pool fails closed: `503` with `Retry-After: 30` on `POST /v1/query`, `GET/POST /v1/pipes/{name}`, `POST /v1/ops/query` and `POST /v1/ops/schema/refresh`, ahead of the cache. Each served tenant gets a `discovery.SchemaRegistry` of its own over its pool, kept fresh by its own loop — the boot retry until the first success, then `schema.refresh_interval` with the first refresh at a random point within the interval so tenants adopted together do not refresh together — created and stopped from the settings reload and stopped under `App.Close`; `wavehouse_schema_refresh_failures_total{tenant}` counts a loop's failed attempts. `GET /v1/ops/schema`, `POST /v1/ops/schema/refresh` and `POST /v1/ops/query` take the strict `?tenant=` the pipe reads take (absent is tenant `0`, `400` malformed, `404` unknown, `503` rejected); the SDK sends it as the `tenant` option of `wh.schema.list()`, `wh.schema.refresh()`, `wh.from(t).schema()` and `wh.sql()`. Over a nested directory `/livez` (and `/v1/health`) is `503` with the latest discovery failure, naming its tenant, while no tenant has completed a first discovery, then `200` for the rest of the process lifetime; `/readyz` pings every open pool at once, is ready at the first answer, and names every pool that did not answer when none does — one tenant's ClickHouse outage is its log line and counter, never a probe failure. The ingest worker inserts each batch into its own tenant's ClickHouse — the tenant the message's topic names, through that tenant's HTTP wiring (`chconn.Pools.Target`); a tenant on no pool takes the failure path an unreachable ClickHouse takes — and its cache invalidation fans out to the tenants on the same address and database as the batch's tenant, whatever their user or `tls` block (they read the same tables), rather than to every known tenant, and a tenant adopted after an absence — rejected or removed, so out of that fan-out — or moved to another address or database has its cached structured-query results orphaned in one step (`Cache.InvalidateTenant`, a tenant generation in every version key), so a repaired folder never serves query rows cached before the inserts it missed (a pipe result names no table, so neither this nor any insert invalidates it: it stays until its TTL expires). Three changes reach the single-tenant directory too: a table lookup before the first discovery is a `503` with `Retry-After: 5` (`schema not loaded yet`) rather than a `404`, on `POST /v1/ingest`, `POST /v1/query` and `GET /v1/ops/schema` (the list included, where `[]` would read as no tables); the first periodic refresh fires at a random point within the interval rather than a full interval after boot; and a reload that moves `clickhouse.addr` or `clickhouse.database` now orphans the structured-query results cached before it, where they were served until their TTL. Until [#529](https://github.com/Wave-RF/WaveHouse/issues/529) every tenant's user authenticates with `WH_CH_PASSWORD`, so the tuple is in effect the address, database, username and `tls` block. - **One token verifier per tenant, built off the boot and reload paths** (`internal/auth/auth.go` (+ tests), `internal/api/router.go` (+ tests), `internal/app/{app,wire}.go` (+ tests), `go.mod`, `docs/src/content/docs/sdk/{reference,streaming}.md`, `docs/src/content/docs/{architecture,deployment,api}.md`, `docs/src/content/docs/{settings-directory,configuration}.mdx`, `SECURITY.md`): story 9 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)). Over a nested settings directory each tenant's folder now wires that tenant's verifier (`auth.jwks_url`, `auth.role_claim`), so a JWKS-issued token verifies only under the tenants whose `jwks_url` names its identity provider's key set — under any other tenant's header it is refused as invalid — where before one verifier, tenant `0`'s, accepted a token under any header. `auth.Config` is now the boot-config half alone (the HMAC secret and the operator key, shared by every tenant) and the new `auth.Wiring` a tenant's half; `NewAuthenticator` builds no verifier, `Reconfigure(id, wiring)` gives a tenant one — swapped atomically when its wiring changed, kept when it did not — `Prune` drops the verifiers of the tenants a reload stopped serving, rejected or removed alike (no work runs for a tenant that is not served; a folder adopted again gets a fresh verifier), and `Close` stops every JWKS refresh as a component of `App.Close`. This holds on every reload because the registry's `AfterAdopt` hooks run after every reload it applied, empty list included, so a per-tenant reload that rejects a folder drops its verifier then rather than at the next adoption. The middleware reads the request tenant's verifier through an injected `auth.TenantSource` — the store `api.TenantMW` resolved names its tenant with `settings.Store.Tenant()`, one read of the context — a tenant-exempt route (the ops tree) verifies as tenant `0`, and a tenant with no verifier fails closed. The operator key stamps the request tenant's `admin_role`, read from that tenant's policy, rather than tenant `0`'s. A JWKS key set is fetched on its own goroutine: the verifier is in place at once and *pending* until a set has been stored — the first fetch retried with backoff from one second to a minute, a stored set then kept fresh by the library hourly and, rate-limited, on an unknown key id — so neither boot nor a reload (which holds the registry's lock) waits on the endpoint, and **an unreachable JWKS no longer refuses boot** — it logs (`jwks refresh failed; no token validates until it succeeds`) and that tenant alone is affected. A token checked against a pending verifier is a new outcome, `auth.ErrVerifierPending`: every `/v1` route answers it `503 {"error": "token verifier not ready: …"}` with `Retry-After: 30` (`api.refuseUnverifiable`) rather than evaluate the request under the `default_role`, which could accept its data under a lesser role while another pod holding the keys would have served it as its own; a request without a token, and the operator key, are unaffected. Every fetch goes through one client that caps the response at 1 MiB, refusing a larger one as unreachable with `jwks response exceeds 1048576 bytes` as the logged cause. Nothing changes for a flat directory beyond that boot rule: its one tenant gets exactly the verifier it had. The boot warning for the secretless posture (no `auth.jwt_secret` and no `jwks_url`, whose token refusal landed in [#607](https://github.com/Wave-RF/WaveHouse/pull/607)) is now per tenant — each served tenant with no `jwks_url` while the boot secret is unset — where one line that any JWKS tenant silenced used to stand for the whole directory. diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index cb96ae2b8..975db37ee 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -113,8 +113,8 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ - **cache.go** — `Cache` interface: `Lookup`, `Set`, `Invalidate`, `InvalidateTenant`, `Close`, plus `QueryTimeToTTL`, which sets a result's TTL from how long its query took (10 s floor, 1 h ceiling). Every entry is one tenant's: `Lookup` takes the tenant, the caller's query key — `:query:`, built by the two cached handlers in `api/` (`queryCacheKey`, with the tenant read off the request's store — `settings.Store.Tenant`), which use it as their [singleflight](https://pkg.go.dev/golang.org/x/sync/singleflight) key too — and the `Namespace`s the result depends on, each of that tenant (another tenant's is `ErrForeignDependency`) and naming a table and scope by their raw names, which the cache escapes where it builds a key: one for a structured query, none yet for a pipe (a pipe's table dependencies are [#343](https://github.com/Wave-RF/WaveHouse/pull/343)). `Lookup` returns the `Entry` (a nil value is a miss) and a `Snapshot` of the versions it read; on a miss the handler runs the query and passes that snapshot to `Set`, so a result is filed under the versions read *before* its query ran, and a write that lands while it runs orphans the fill rather than re-homing pre-write rows under the post-write versions ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)). The singleflight leader's snapshot is the one used. `Set` returns an error only when the backend failed; a value the cache declines — larger than it keeps, a non-positive TTL, a zero snapshot — is not one. What a hit, a miss and a bump mean is pinned by the conformance suite every backend runs, `internal/testutil/cachetest`. - **local.go** — `LocalCache`, the in-process L1 on [Ristretto](https://github.com/dgraph-io/ristretto): one pool shared by every tenant (a heavier tenant holds more of it), sized by the boot config's `cache.l1_max_cost`. - **version_manager.go** — `VersionManager`, the invalidation index behind `Invalidate` and `InvalidateTenant`: a namespace key is `..
.
.`, its fields joined with `keyenc.Join` so a dot in a table or scope name is escaped rather than read as a separator, and a query key, `|.||…` with the sha escaped, folds in the tenant's version and each dependency's namespace key and namespace version, so bumping a table (a scopeless write) or one scope — scope is reserved and empty today, so every write is the whole-table bump — orphans every dependent entry without touching the pool. The tenant leads every key ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8): the same table under two tenants is two namespaces, so a bump through `Invalidate` under one tenant never touches — and a read under one tenant is never served — the other's results, and the flat directory's single tenant simply carries the `0` prefix. `BumpTenant` (behind `InvalidateTenant`) advances the tenant version that leads every namespace key of one tenant, orphaning its every namespace and every cached query in one step (the tenant version is folded into every query key, so a pipe result with no dependencies is orphaned too) — a table no bump ever keyed included, which an enumeration of the index would miss — for a tenant back on a pool after an absence from the fan-out, or moved to another address or database (story 6). The index is per tenant; the cross-tenant invalidation an insert into a shared table needs is not the index's but the wiring's: `internal/app` hands the ingest worker a cache (`sharedTables`) that repeats each bump under every tenant on the same ClickHouse address and database. -- **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — only `GET`, `SET`, `MGET` and `PING`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, and a token key holding anything but a token (a string of another length, a list, a hash) is replaced, so a token lost to eviction, expiry or a restart is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing. Nothing selects this backend yet — the boot config and wiring arrive with [#613](https://github.com/Wave-RF/WaveHouse/issues/613)'s E4. -- **redis_codec.go** — the key schema (`tokenKeys`, `bumpKeys`, `valueKey`) and the value frame: format, flags, expiry, the token list, the payload, zstd-compressed from `CompressMinBytes` when that is smaller. A stored value is capped at `MaxValueBytes` and a decoded one at eight times that, which refuses a zip bomb planted in a shared server; an unknown format, as a newer process writes during a rolling upgrade, is a miss. +- **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — only `GET`, `SET`, `MGET` and `PING`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot; the table and scope are escaped with `keyenc` after the fixed prefix, so a `:` in a name is never read as the separator. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, the hash over their escaped forms, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, and a token key holding anything but a token (a string of another length, a list, a hash) is replaced, so a token lost to eviction, expiry or a restart is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing. Nothing selects this backend yet — the boot config and wiring arrive with [#613](https://github.com/Wave-RF/WaveHouse/issues/613)'s E4. +- **redis_codec.go** — the key schema (`tokenKeys`, `bumpKeys`, `valueKey`, which take a `Namespace`'s raw names and escape them) and the value frame: format, flags, expiry, the token list, the payload, zstd-compressed from `CompressMinBytes` when that is smaller. A stored value is capped at `MaxValueBytes` and a decoded one at eight times that, which refuses a zip bomb planted in a shared server; an unknown format, as a newer process writes during a rolling upgrade, is a miss. - **breaker.go** — the circuit breaker: `BreakerThreshold` (5) consecutive failures open it, and while open every operation skips the server at once; after `BreakerOpenFor` (5 s) one background `PING` decides whether it closes. Only a transport failure or a timeout counts against the server: any reply, an error reply or one the backend cannot use included, counts as a success, for operations and the probe alike, and a caller that gave up first counts as nothing. - **pending.go** — invalidations the server did not take, kept per token key (repeats coalesce) and retried by a background loop with backoff from 100 ms to 10 s until they land. Landing late is still correct: a fresh token orphans the pre-write entries and any fill made meanwhile. Past `PendingMax` (100,000) keys the set collapses to one tenant bump per affected tenant. `Close` makes one last attempt at the bumps still owed, past the breaker (an open one is why they are owed); what that attempt cannot deliver is lost, and the entries they would orphan are served until their TTL — the same failure as a worker stopping between an insert and its invalidation. - **metrics.go** — the shared backend's instruments (meter `wavehouse-cache`, `backend="redis"`, no tenant attribute): `wavehouse_cache_lookups_total` by `result` (`hit`, `miss`, `stale` — filed under since-bumped tokens, `bypass` — server skipped, `error`), `wavehouse_cache_op_duration_seconds` by `op` (`lookup`, `set`, `invalidate`), `wavehouse_cache_breaker_open` (1 while bypassed, including before the first connection), `wavehouse_cache_invalidations_total` by `result` (`ok`, `deferred`), `wavehouse_cache_invalidations_pending` (bumps owed: entries they would orphan may be served stale meanwhile), `wavehouse_cache_value_bytes` (stored size), `wavehouse_cache_oversize_total` and `wavehouse_cache_set_failures_total` by `reason` (`oom`, `timeout`, `other`). diff --git a/internal/cache/redis_codec.go b/internal/cache/redis_codec.go index cf6f4b5ec..1ce16f0cb 100644 --- a/internal/cache/redis_codec.go +++ b/internal/cache/redis_codec.go @@ -13,6 +13,7 @@ import ( "github.com/klauspost/compress/zstd" + "github.com/Wave-RF/WaveHouse/internal/keyenc" "github.com/Wave-RF/WaveHouse/internal/tenant" ) @@ -44,17 +45,20 @@ func newToken() []byte { // tenantTokenKey is the key of tenant id's token. Every token key carries // the tenant as a hash tag, so all of a tenant's tokens share one cluster -// slot and a lookup reads them with one MGET. +// slot and a lookup reads them with one MGET. The tenant goes in verbatim: +// its grammar is keyenc's kept bytes, so it is its own escaped form. func tenantTokenKey(prefix string, id tenant.ID) string { return prefix + ":{" + string(id) + "}:T" } +// tableTokenKey and scopeTokenKey take raw names and escape them after the +// fixed prefix (keyenc), so no ':' in a name reads as the separator. func tableTokenKey(prefix string, id tenant.ID, table string) string { - return prefix + ":{" + string(id) + "}:B:" + table + return string(keyenc.AppendJoin([]byte(prefix+":{"+string(id)+"}:B:"), ':', table)) } func scopeTokenKey(prefix string, id tenant.ID, table, scope string) string { - return prefix + ":{" + string(id) + "}:S:" + table + ":" + scope + return string(keyenc.AppendJoin([]byte(prefix+":{"+string(id)+"}:S:"), ':', table, scope)) } // sortedDeps returns deps in canonical order without duplicates. @@ -91,16 +95,16 @@ func bumpKeys(prefix string, ns Namespace) []string { // valueKey names the entry for sha over deps. It carries no versions, so a // refill overwrites in place, and no hash tag, so one tenant's values spread -// across a cluster's shards. +// across a cluster's shards. It hashes the escaped sha and each dep's +// escaped, joined table and scope, each ended by a NUL, which escaping never +// writes, so no two sets of names hash the same input. func valueKey(prefix string, id tenant.ID, sha string, deps []Namespace) string { h := sha256.New() - h.Write([]byte(sha)) - h.Write([]byte{0}) + b := keyenc.AppendEscape(nil, sha) + h.Write(append(b, 0)) for _, d := range sortedDeps(deps) { - h.Write([]byte(d.Table)) - h.Write([]byte{0}) - h.Write([]byte(d.Scope)) - h.Write([]byte{0}) + b = keyenc.AppendJoin(b[:0], ':', d.Table, d.Scope) + h.Write(append(b, 0)) } return prefix + ":q:" + string(id) + ":" + hex.EncodeToString(h.Sum(nil)) } diff --git a/internal/cache/redis_codec_test.go b/internal/cache/redis_codec_test.go index 68c423d0f..c956a6097 100644 --- a/internal/cache/redis_codec_test.go +++ b/internal/cache/redis_codec_test.go @@ -28,6 +28,11 @@ func TestTokenKeys(t *testing.T) { []Namespace{{Tenant: "acme", Table: "events"}}, []string{"wh:{acme}:T", "wh:{acme}:B:events", "wh:{acme}:S:events:"}, }, + { + "names arrive raw and are escaped after the hash tag", + []Namespace{{Tenant: "acme", Table: "default.clicks", Scope: "org:1"}}, + []string{"wh:{acme}:T", "wh:{acme}:B:default%2Eclicks", "wh:{acme}:S:default%2Eclicks:org%3A1"}, + }, { "shared table tokens and duplicate deps appear once, sorted", []Namespace{ @@ -55,13 +60,26 @@ func TestBumpKeys(t *testing.T) { assert.Equal(t, []string{"p:{acme}:B:events"}, bumpKeys("p", Namespace{Tenant: "acme", Table: "events"})) assert.Equal(t, []string{"p:{acme}:S:events:org_1", "p:{acme}:S:events:"}, bumpKeys("p", Namespace{Tenant: "acme", Table: "events", Scope: "org_1"})) + assert.Equal(t, []string{"p:{acme}:S:my%20table:a%3Ab", "p:{acme}:S:my%20table:"}, + bumpKeys("p", Namespace{Tenant: "acme", Table: "my table", Scope: "a:b"})) + + // A ':' in a name never reads as the separator: table "a:b" with scope + // "c" and table "a" with scope "b:c" bump two tokens. + assert.NotEqual(t, + bumpKeys("p", Namespace{Tenant: "acme", Table: "a:b", Scope: "c"}), + bumpKeys("p", Namespace{Tenant: "acme", Table: "a", Scope: "b:c"})) } // Every key a bump writes is one some lookup reads: otherwise the bump // orphans nothing. func TestBumpKeysAreReadByLookups(t *testing.T) { t.Parallel() - for _, ns := range []Namespace{{Tenant: "acme", Table: "events"}, {Tenant: "acme", Table: "events", Scope: "org_1"}} { + for _, ns := range []Namespace{ + {Tenant: "acme", Table: "events"}, + {Tenant: "acme", Table: "events", Scope: "org_1"}, + {Tenant: "acme", Table: "default.clicks"}, + {Tenant: "acme", Table: "my table", Scope: "org:1"}, + } { read := map[string]bool{} for _, scope := range []string{"", ns.Scope} { for _, k := range tokenKeys("wh", "acme", []Namespace{{Tenant: "acme", Table: ns.Table, Scope: scope}}) { @@ -83,9 +101,16 @@ func TestValueKey(t *testing.T) { assert.Equal(t, k, valueKey("wh", "acme", "acme:query:abc", []Namespace{b, a, b}), "order and duplicates do not matter") assert.NotEqual(t, k, valueKey("wh", "acme", "acme:query:abc", []Namespace{a})) assert.NotEqual(t, k, valueKey("wh", "acme", "acme:query:abd", []Namespace{a, b})) - assert.NotEqual(t, - valueKey("wh", "acme", "q", []Namespace{{Tenant: "acme", Table: "ab", Scope: "c"}}), - valueKey("wh", "acme", "q", []Namespace{{Tenant: "acme", Table: "a", Scope: "bc"}})) + // Names that would run together unescaped hash apart: at the table and + // scope boundary, at a NUL, and across dependencies. + for _, pair := range [][2][]Namespace{ + {{{Tenant: "acme", Table: "ab", Scope: "c"}}, {{Tenant: "acme", Table: "a", Scope: "bc"}}}, + {{{Tenant: "acme", Table: "a\x00b"}}, {{Tenant: "acme", Table: "a", Scope: "b\x00"}}}, + {{{Tenant: "acme", Table: "a:b"}}, {{Tenant: "acme", Table: "a", Scope: "b"}}}, + {{{Tenant: "acme", Table: "a"}, {Tenant: "acme", Table: "b"}}, {{Tenant: "acme", Table: "a", Scope: "\x00b\x00"}}}, + } { + assert.NotEqual(t, valueKey("wh", "acme", "q", pair[0]), valueKey("wh", "acme", "q", pair[1]), "%q", pair) + } } func newTestCodec(t *testing.T, compressMin, maxDecoded int) *codec { From 1e70d64506fd6f847776a55aee215b8b61974ad8 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Fri, 25 Sep 2026 20:10:07 -0400 Subject: [PATCH 13/45] docs(cache): the Redis token-key layout is a protocol between builds Every process sharing a server reads and bumps the table and scope token keys for itself, so two builds that lay them out differently (a layout change, or a change to what keyenc keeps) split them: one build's bumps miss the entries the other filed, which are served until their TTL for the whole rolling deploy. The token-key comment says such a change needs a compatibility step or a documented flush; a valueKey change only orphans values. architecture.md and AGENTS.md name the shared backend's Redis keys among keyenc's users and state that rolling-upgrade consequence. development.md lists the shared backend in the cache tree line and says internal/cache's integration tests start their own Redis, Valkey, Dragonfly and Redis Cluster containers; the tech-stack row reads "Shared Cache". Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01B1tJWUp6oaDoH1usLwGtLF --- AGENTS.md | 2 +- docs/src/content/docs/architecture.md | 4 ++-- docs/src/content/docs/development.md | 4 ++-- internal/cache/redis_codec.go | 9 +++++++++ 4 files changed, 14 insertions(+), 5 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 692e1ba57..0f1dd85ad 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -38,7 +38,7 @@ Nineteen internal packages under `internal/` (plus `internal/testutil/` for shar - **`dedupe/`** — `Deduplicator` interface → `Embedded` (Pebble: every tenant's seen ids in one instance at `data_dir/pebble`, each key led by its tenant, open while any tenant's store is — the layout is the implementation's call, and the wiring hands it `data_dir` once; its `Stats` feed the system gauges), wrapped by `Managed` whose open/closed state follows the hot-reloadable `dedupe.enabled` in the settings directory's `config.json`; `Stores` holds one `Managed` per tenant, built through a `Factory` (`func(tenant.ID) *Managed`, `Embedded.Tenant` in production; `Managed` opens its store through a function, so every backend gets the same switch), and reconciled from the registry's `AfterAdopt` hook — open exactly when the tenant is served with its switch on, closed with its seen ids kept otherwise ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) stories 7 and 3) - **`discovery/`** — `SchemaRegistry`, one per served tenant over a `Source` read once per refresh — the tenant's pool's connection and the database that pool was opened for, one snapshot, so a refused move keeps discovering the database the tenant's queries still use (`internal/app`'s `discoveries` builds, runs and stops them from `AfterAdopt` and `App.Close`: `RetryRefresh` until the first success, then `StartAutoRefresh` with a random first tick; `Lookup` answers `ErrNotLoaded` before the first success — the handlers' `503` with `Retry-After` — and `ErrUnknownTable` after; a failed loop attempt counts in `wavehouse_schema_refresh_failures_total{tenant}`), that introspects ClickHouse `system.columns` (name/type/nullability plus `default_expression` and 1-based `position`) and `system.tables` (each table's `create_table_query`, kept in-process and never serialized — an external-engine table renders its wiring there unconditionally — endpoint, bucket/host, database, username, S3 access key id; ClickHouse masks the password as `[HIDDEN]` from ~23.9, so the exposure is the topology, not the secret), records the server version, + `Validate()` for ingest payloads + `CanonicalizeTimestamps()` rewriting top-level `DateTime`/`DateTime64` column values to the canonical RFC 3339 UTC wire form pre-publish (Key Design Decision #19) - **`ingest/`** — Ingest worker pipeline (`worker.go`: JetStream input → per-table batch INSERT with DLQ output). The pipeline is **insert-only**. The wire format `EventMessage` (`types.go`) carries `{table_name, scope, received_timestamp, format, columns, row}` and nothing else — `row` is one positional `JSONCompactEachRow` line and `columns` names its slots, the table's **insertable** columns (a `MATERIALIZED`/`ALIAS` column cannot be named in an `INSERT`); the worker batches per (tenant, table, column list), the tenant read off each message's `mq.Topic`, and inserts each batch into its tenant's own ClickHouse (`chconn.Pools.Target`); the worker accepts whatever table name the envelope carries (table existence was already checked by the HTTP ingest handler, which `404`s an unknown table before publish; the worker doesn't re-validate), then bulk-INSERTs. In the embedded-NATS deployment (the default), the server runs with `DontListen: true` (`internal/mq/embedded.go`), so the only Publishers reachable on the `ingest.>` subjects are in-process Go code — today, only the HTTP `/v1/ingest?table={table}` handler. Non-insert mutations (`DELETE`/`UPDATE`/`TRUNCATE`/…) must go through `POST /v1/ops/query` under the admin role (the same `RequireAdmin` gate as the rest of `/v1/ops/*`), so non-admin callers never reach the proxy. A request with no token (or an invalid one) resolves to the `default_role`, which in a production config is not the admin role (setting them equal is a loudly-warned dev-only setting), so it can't reach this endpoint. Plus `Sweeper` (Active Sweeper for NATS message lifecycle) + `EventMessage`/`BufferConsumerName` types (`types.go`) -- **`keyenc/`** — the one escaping composite keys are built from: `Escape` keeps `[A-Za-z0-9_-]` (exactly the tenant-id grammar, so a tenant id is its own escaped form) and writes every other byte as `%XX`, `Unescape` is `url.PathUnescape` (lenient: either hex case, and a byte left unescaped reads as itself, so a `%2D` an earlier build wrote still reads), `Join`/`AppendJoin` escape each field and put a separator between them (they panic on no fields, and on a separator the escaping could write or one outside ASCII) and `Split` reverses them. The package that builds a key takes raw names and escapes them itself, so no caller has to and no field reaches a key unescaped: NATS subjects (`Join`/`Split` after the verbatim tenant) and the cache's keys (`cache.VersionManager`) use it; changing what it keeps orphans every stored key +- **`keyenc/`** — the one escaping composite keys are built from: `Escape` keeps `[A-Za-z0-9_-]` (exactly the tenant-id grammar, so a tenant id is its own escaped form) and writes every other byte as `%XX`, `Unescape` is `url.PathUnescape` (lenient: either hex case, and a byte left unescaped reads as itself, so a `%2D` an earlier build wrote still reads), `Join`/`AppendJoin` escape each field and put a separator between them (they panic on no fields, and on a separator the escaping could write or one outside ASCII) and `Split` reverses them. The package that builds a key takes raw names and escapes them itself, so no caller has to and no field reaches a key unescaped: NATS subjects (`Join`/`Split` after the verbatim tenant) and the cache's keys — the version index and the shared backend's Redis keys (`internal/cache`) — use it; changing what it keeps orphans every stored key, and on the shared backend, whose keys every process builds for itself, splits them between builds for the length of a rolling upgrade (a bump one build makes misses the entries the other filed, served until their TTL) - **`mq/`** — the message-queue boundary: the **only** package that imports NATS/JetStream (Key Design Decision #20). and the only one that knows how the broker works. Everything else addresses events by `Topic{Tenant, Table, Scope}` (a validated tenant id and raw names — the tenant leads every subject, `ingest..
`, so one wildcard selects a tenant's traffic, and a topic without one is refused) and states intent through the interfaces — `Publisher` (`ErrQueueFull` is the backpressure signal), `Subscriber`, `ConsumerManager`/`Consumer`/`ConsumerConfig` (the ingest worker's durable consumer), `DeadLetterer` and `DeadLetterStats` (park a message, count what is parked), `Purger` (drop what is both acked and older than a cutoff — the sweeper), `Replayer` (SSE gap-fill) — composed into `Broker`, which adds each tenant's byte budget (`SetMaxBytes`/`MaxBytes`: the `mq.max_bytes_gb` reload, which opens a tenant's queue the first time) and `Stats` (the system gauges' source). Every interface speaks per tenant, never per stream: the embedded implementation gives each tenant a queue of its own (a stream pair, `INGEST_`/`DLQ_`), and nothing outside the package may assume that layout — an external implementation may keep one shared stream. Subjects, prefixes, wildcards, stream names, sequences, and ack floors are private to the one implementation, `EmbeddedNATS` (`embedded.go`, `subject.go`, `purge.go`, `deadletter.go`), whose subject tokens are escaped by the shared `internal/keyenc`; `internal/app` constructs it and hands everything else a `mq.Broker` - **`observability/`** — OpenTelemetry pipeline: `InitProvider` wires trace/metric/log providers via OTLP gRPC (each signal independently gated). A top-level `Prometheus` config block drives an optional `/metrics` scrape endpoint that runs independently of OTLP push — standalone (Alloy/Mimir scrape, no collector), alongside OTLP, or off. `NewLogger` produces a slog handler that fans out to stdout AND OTLP (stdout always 100%, OTLP sample-rate-aware). `TraceHandler` injects trace_id/span_id from active spans. `tracer.go` provides W3C trace context propagation over message headers (`InjectHeaders`/`ExtractHeaders` on a plain header map; `internal/mq` injects on every publish and extracts on the `Subscribe` path, so this package never sees a NATS type). - **`pipes/`** — Named query pipes: `NamedQuery` type + `BindParams` + `Source` (read per request; `settings.Store` in production, `Static(q...)` in tests) diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index c2a77b1c7..e18ace9c1 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -209,7 +209,7 @@ The hot-reloadable half of configuration: a directory of four JSON files (`confi ### `keyenc/` — Key Escaping -- **keyenc.go** — The one escaping composite keys are built from, so a name can never be mistaken for a separator: `Escape` keeps ASCII letters, digits, `_` and `-` — exactly the tenant-id grammar, so a tenant id is its own escaped form — and writes every other byte as `%XX` (uppercase hex); `Unescape` is `url.PathUnescape`, which decodes `%XX` in either case and takes any other byte as itself, so a `%2D` for `-` that an earlier build wrote still reads. `Join`/`AppendJoin` escape each field and put a separator between them, panicking on no fields and on a separator the escaping could write or one outside ASCII, and `Split` reverses them. The package that builds a key takes raw names and escapes them itself, so no caller has to: NATS subject tokens (`internal/mq`) and the cache's keys (`internal/cache`) both use it. Keys built from it are stored, so changing what it keeps orphans them. +- **keyenc.go** — The one escaping composite keys are built from, so a name can never be mistaken for a separator: `Escape` keeps ASCII letters, digits, `_` and `-` — exactly the tenant-id grammar, so a tenant id is its own escaped form — and writes every other byte as `%XX` (uppercase hex); `Unescape` is `url.PathUnescape`, which decodes `%XX` in either case and takes any other byte as itself, so a `%2D` for `-` that an earlier build wrote still reads. `Join`/`AppendJoin` escape each field and put a separator between them, panicking on no fields and on a separator the escaping could write or one outside ASCII, and `Split` reverses them. The package that builds a key takes raw names and escapes them itself, so no caller has to: NATS subject tokens (`internal/mq`) and the cache's keys — the version index and the shared backend's Redis keys (`internal/cache`) — use it. Keys built from it are stored, so changing what it keeps orphans them — and on the shared backend, whose keys every process builds for itself, it splits them for the length of a rolling upgrade: a bump one build makes does not reach the entries the other build filed, which are served until their TTL. ## Data Flows @@ -373,7 +373,7 @@ Client GET /v1/stream | Analytics DB | ClickHouse | Primary data store + schema source of truth | | Message Queue | NATS + JetStream | Durable event streaming | | L1 Cache | Ristretto v2 | In-process memory cache | -| Shared cache | [rueidis](https://github.com/redis/rueidis) | Redis-compatible client for the shared backend (not yet selectable) | +| Shared Cache | [rueidis](https://github.com/redis/rueidis) | Redis-compatible client for the shared backend (not yet selectable) | | Embedded KV | Pebble | Optional deduplication | | Config | cleanenv | YAML + env var config loading | | Release | GoReleaser | Cross-platform binary builds | diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index a62476ee7..272ab8cd1 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -345,7 +345,7 @@ Each test target writes `covdata` to `tmp/coverage//data/`, renders a tex | E2E tests (SDK) | `tests/e2e/sdk/*.test.ts` | Yes | `make test-e2e` | - **Unit tests** live beside the code they test (e.g., `internal/discovery/discovery_test.go`). They use mocks or embedded NATS (in-process, no Docker needed). -- **Integration tests** use the `//go:build integration` build tag. `TestMain` starts one ClickHouse testcontainer and boots the production wiring against it through `app.New` (embedded NATS, ingest worker, sweeper, hub, the API server on a random loopback port); tests reach it via `env(t)` and create their own tables. DLQ tests use `assert.Eventually` with a 30-second timeout for the 5-second ingest worker batch window. +- **Integration tests** use the `//go:build integration` build tag. In `tests/integration`, `TestMain` starts one ClickHouse testcontainer and boots the production wiring against it through `app.New` (embedded NATS, ingest worker, sweeper, hub, the API server on a random loopback port); tests reach it via `env(t)` and create their own tables. DLQ tests use `assert.Eventually` with a 30-second timeout for the 5-second ingest worker batch window. `internal/cache`'s integration tests start their own containers instead — Redis, Valkey, Dragonfly and a one-node Redis Cluster — for the shared backend. Shared test utilities live in `internal/testutil/`. The packages log through `slog.Default()`, so tests reach log output through `internal/testutil/logtest`: `logtest.Silence()` in a package's `TestMain` discards it, and `logtest.Capture(t, level)` routes it to a buffer for a test that asserts on log lines — such a test must not call `t.Parallel()`, because the default logger is process-wide. @@ -454,7 +454,7 @@ WaveHouse/ │ ├── api/ # HTTP handlers, router, middleware │ ├── app/ # Process wiring (build every component, run under one errgroup, release in reverse) │ ├── auth/ # JWT/JWKS authentication middleware -│ ├── cache/ # Query cache: Ristretto L1 + the tenant-led version index +│ ├── cache/ # Query cache: Ristretto L1 + the tenant-led version index; the Redis-compatible shared backend │ ├── chconn/ # ClickHouse pools, one per connection tuple (reconciled on settings reload) │ ├── chsql/ # Shared ClickHouse SQL helpers (quoting + bind-safety) │ ├── config/ # YAML + env var configuration diff --git a/internal/cache/redis_codec.go b/internal/cache/redis_codec.go index 1ce16f0cb..e9ec64963 100644 --- a/internal/cache/redis_codec.go +++ b/internal/cache/redis_codec.go @@ -53,6 +53,15 @@ func tenantTokenKey(prefix string, id tenant.ID) string { // tableTokenKey and scopeTokenKey take raw names and escape them after the // fixed prefix (keyenc), so no ':' in a name reads as the separator. +// +// Their layout is a protocol between builds: every process on the server +// reads and bumps these keys for itself, so two builds that lay them out +// differently — a change here, or to what keyenc keeps — split them, and one +// build's bumps miss the entries the other filed, which are served until +// their TTL for the whole rolling deploy. Such a change needs a +// compatibility step (bump both layouts through the transition) or a +// documented flush. The tenant token, placed verbatim, stays shared; a +// valueKey change only orphans values, which is safe to roll. func tableTokenKey(prefix string, id tenant.ID, table string) string { return string(keyenc.AppendJoin([]byte(prefix+":{"+string(id)+"}:B:"), ':', table)) } From e6a824a4d747bdea658d77c57ac0ae8f8457c922 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Fri, 25 Sep 2026 20:23:09 -0400 Subject: [PATCH 14/45] docs(cache): a token-key layout change needs both layouts read and bumped The compatibility step the token-key comment named did not close the split: an old build cannot be changed, so it keeps bumping only old keys, and a new build that bumps both but files under the new layout alone still misses every write the old build handles. The comment now says the new build must read and bump both layouts, folding the old tokens into what it files, until no old build is left; or the upgrade must never run two builds against the server at once. The VersionTTL floor's error no longer claims every value under 2s rounds to EX 0: EX has one-second resolution, and only below about 1.1s can the jittered TTL truncate to EX 0. The 2s floor stays. CONTRIBUTING.md, development.md and AGENTS.md say integration tests may live beside a package tested against its own external server, and that the suite also starts Redis, Valkey, Dragonfly and a one-node Redis Cluster. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01B1tJWUp6oaDoH1usLwGtLF --- AGENTS.md | 4 ++-- CONTRIBUTING.md | 2 +- docs/src/content/docs/development.md | 2 +- internal/cache/redis.go | 2 +- internal/cache/redis_codec.go | 10 ++++++---- 5 files changed, 11 insertions(+), 9 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index a7f4bb306..9ba3cdd75 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -151,7 +151,7 @@ If `make ci` passes locally, your commit has crossed the same gates CI will run ### Running `make ci` (for agents) -`make ci` is **self-contained**: the integration suite (`tests/integration/`) and the E2E orchestrator (`scripts/orchestrator/`) each boot ClickHouse via **testcontainers on random host ports**. The only prerequisite is a running **Docker daemon** — do **not** `make deps-up` or start ClickHouse first (`deps-up` is for `make dev` only). +`make ci` is **self-contained**: the integration suite (`tests/integration/`) and the E2E orchestrator (`scripts/orchestrator/`) each boot ClickHouse via **testcontainers on random host ports**, and the shared cache backend's integration tests (`internal/cache/`) start their own Redis, Valkey, Dragonfly and one-node Redis Cluster containers the same way. The only prerequisite is a running **Docker daemon** — do **not** `make deps-up` or start ClickHouse first (`deps-up` is for `make dev` only). Run it via the **background Bash tool** (`run_in_background: true`) and wait for the completion notification; the harness re-invokes you on exit, so polling the log with `tail` only burns context: @@ -447,7 +447,7 @@ internal/stream/ → SSE fan-out (event Hub: project once per role, Subsc internal/tenant/ → Tenant id (type, grammar, reserved default, request header name) internal/testutil/ → Shared test helpers (mocks, JWT + schema helpers; logtest/ captures or silences the default logger; cachetest/ is the conformance suite every cache.Cache backend runs) tests/ → Integration & E2E tests -tests/integration/ → Go integration tests (//go:build integration; ClickHouse testcontainer) +tests/integration/ → Go integration tests (//go:build integration; ClickHouse testcontainer). A package tested against its own external server keeps them beside it: internal/cache/redis_integration_test.go (Redis, Valkey, Dragonfly, Redis Cluster testcontainers) tests/e2e/ → E2E test stack (scripts/orchestrator boots a ClickHouse testcontainer + the wavehouse-cov binary) tests/e2e/fixtures/ → Idempotent ClickHouse DDL scripts for test tables tests/e2e/sdk/ → E2E integration tests via TypeScript SDK (Vitest) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c02dfa1d9..70bbb00ba 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -39,7 +39,7 @@ Open a [feature request issue](https://github.com/Wave-RF/WaveHouse/issues/new?t The pre-push hook (installed by `make tools`) blocks a push until the tree has been validated locally: a code change needs `make ci`, a docs/prose-only change needs only `make verify` (the same split CI makes). `make lint` / `make test` / `make build` are fast inner-loop subsets. -2. Write tests for new functionality. Unit tests go alongside the code in `internal/`. Integration tests go in `tests/` with the `//go:build integration` tag. +2. Write tests for new functionality. Unit tests go alongside the code in `internal/`. Integration tests carry the `//go:build integration` tag and go in `tests/integration/`, or beside the package when they test one package against its own external server (e.g. `internal/cache/redis_integration_test.go`), with the package added to the `test-integration` target. 3. Update documentation if your change affects: - API endpoints → update `docs/src/content/docs/api.md` diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index 272ab8cd1..df84fb8ee 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -16,7 +16,7 @@ You need these on your `PATH` before any `make` recipe will work end-to-end: | **Go** | 1.26+ (matches `go.mod`) | Compiles `cmd/wavehouse`; also runs the pinned `tool` deps (`gotestsum`, `gofumpt`, `goimports`, `govulncheck`, `deadcode`, `gsa`, `goda`) via `go tool` | [go.dev/dl](https://go.dev/dl/) | | **GNU Make** | **4.0+** | The Makefile uses `--output-sync=target` (Make 4 only) and bash-pinned recipes. macOS ships with BSD Make 3.81, which **will not work** | macOS: `brew install make` then use `gmake` or put `$(brew --prefix make)/libexec/gnubin` on your PATH. Linux: usually already installed | | **bash** | 4+ recommended | Recipes are pinned to `bash`; the helper scripts under `scripts/` use `set -euo pipefail` and bash arrays | macOS default is bash 3.2 (works for current recipes, but `brew install bash` is safer); Linux distros ship 4+ | -| **Docker** *(or Podman)* | Engine 20.10+ with the Compose **v2** plugin (`docker compose`, no hyphen) | Compose stacks under `deployments/compose/`; the E2E and integration suites boot ClickHouse via testcontainers (no compose file) | [Docker Desktop](https://docs.docker.com/get-docker/), [colima](https://github.com/abiosoft/colima), or [Podman](https://podman.io) with `podman-compose` / the `podman compose` plugin. The testcontainers Go library also honors `DOCKER_HOST` for rootless Podman setups | +| **Docker** *(or Podman)* | Engine 20.10+ with the Compose **v2** plugin (`docker compose`, no hyphen) | Compose stacks under `deployments/compose/`; the E2E and integration suites boot ClickHouse via testcontainers (no compose file), and the integration suite also starts Redis, Valkey, Dragonfly (pulled from `docker.dragonflydb.io`) and a one-node Redis Cluster for the shared cache backend | [Docker Desktop](https://docs.docker.com/get-docker/), [colima](https://github.com/abiosoft/colima), or [Podman](https://podman.io) with `podman-compose` / the `podman compose` plugin. The testcontainers Go library also honors `DOCKER_HOST` for rootless Podman setups | | **Node.js** | 22 LTS — pinned via `.nvmrc` at the repo root | Runtime for pnpm and the Vitest suites. Pinned to match CI (`setup-node` uses 22) and to avoid Node-major surprises; older Vitest versions in this repo were known to crash on Node 26 with a V8 heap-allocation abort | [nodejs.org](https://nodejs.org/) or `nvm use` / `fnm use` / `volta` (all read `.nvmrc`) | | **pnpm** | 11.21+ (pinned via `packageManager` in the root `package.json`) | Package manager for the TypeScript SDK, E2E test harness, and docs site (managed as a single pnpm workspace from the repo root); `make build-ts`, `make test-ts`, `make test-e2e`, `make build-docs`, `make dev-docs`, `make preview-docs` all shell out to `pnpm` | `corepack enable && corepack prepare pnpm@11.21.0 --activate` (recommended), or `npm i -g pnpm` | | **git** + **curl** | any recent | `git` for source + version metadata in builds; `curl` is used by the Makefile to fetch the pinned `golangci-lint` binary into `.bin/` | usually preinstalled | diff --git a/internal/cache/redis.go b/internal/cache/redis.go index f9ce1a0fe..01c6a6c41 100644 --- a/internal/cache/redis.go +++ b/internal/cache/redis.go @@ -123,7 +123,7 @@ func (c RedisConfig) withDefaults() (RedisConfig, error) { } } if c.VersionTTL != 0 && c.VersionTTL < 2*time.Second { - return c, fmt.Errorf("cache: redis version ttl %s is under 2s: jittered, it would round to EX 0", c.VersionTTL) + return c, fmt.Errorf("cache: redis version ttl %s is under 2s: EX has one-second resolution, and jittered below ~1.1s it could be EX 0", c.VersionTTL) } c.KeyPrefix = cmpOr(c.KeyPrefix, DefaultRedisKeyPrefix) c.Timeout = cmpOr(c.Timeout, DefaultRedisTimeout) diff --git a/internal/cache/redis_codec.go b/internal/cache/redis_codec.go index e9ec64963..96cddb452 100644 --- a/internal/cache/redis_codec.go +++ b/internal/cache/redis_codec.go @@ -58,10 +58,12 @@ func tenantTokenKey(prefix string, id tenant.ID) string { // reads and bumps these keys for itself, so two builds that lay them out // differently — a change here, or to what keyenc keeps — split them, and one // build's bumps miss the entries the other filed, which are served until -// their TTL for the whole rolling deploy. Such a change needs a -// compatibility step (bump both layouts through the transition) or a -// documented flush. The tenant token, placed verbatim, stays shared; a -// valueKey change only orphans values, which is safe to roll. +// their TTL for the whole rolling deploy. Such a change needs the new build +// to read and bump both layouts (fold the old tokens into what it files) +// until no old build is left, a later build dropping the old; or an upgrade +// that never runs two builds against the server at once. The tenant token, +// placed verbatim, stays shared; a valueKey change only orphans values, +// which is safe to roll. func tableTokenKey(prefix string, id tenant.ID, table string) string { return string(keyenc.AppendJoin([]byte(prefix+":{"+string(id)+"}:B:"), ':', table)) } From ee51e8401cac9e4e8669a6924dcd6b1931e5445a Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 04:50:24 -0400 Subject: [PATCH 15/45] test(app): assert the wired cache is a pruner before swapping it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit TestReload_PrunesCacheIndexToServedTenants replaced a.cache with a recorder by hand, so the run-time a.cache.(pruner) check against the cache wireCache actually builds was never exercised: a change that stopped wiring a pruning cache left the test green while pruning silently went dead. Assert the wired cache satisfies pruner before the swap. Also reword a stale assertion message ("a rejection releases; it orphans nothing yet") that no longer describes real wiring now that a rejection drops the tenant's cache index via Prune, and correct the CHANGELOG's claim that the Redis-compatible backend (#613) already bounds its versions with a TTL — that backend does not exist yet. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_017aS7rLrH1RKkUMem7X4ckd --- CHANGELOG.md | 2 +- internal/app/app_test.go | 5 ++++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2515caa5a..674c2df03 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -79,7 +79,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Fixed - **A write that lands while a cached read is running no longer re-homes the pre-write rows under the post-write key** (`internal/cache/{cache,local,version_manager}.go` (+ tests), `internal/testutil/cachetest` (new), `internal/api/{structured_query,pipes}.go` (+ tests), `internal/app/wire.go`, `docs/src/content/docs/{architecture,deployment}.md`, `AGENTS.md`): fixes [#382](https://github.com/Wave-RF/WaveHouse/issues/382), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `POST /v1/query` and pipe execution rebuilt the version-folded cache key after the query ran, so an insert invalidating the table mid-query filed the rows read before it under the new versions, and they were served as fresh until their TTL. The `Cache` interface now snapshots at lookup: `Lookup(ctx, tenant, sha, deps)` returns the `Entry` and a `Snapshot` of the versions it read, and `Set(ctx, snapshot, value, ttl)` stores under that snapshot, so such a fill is orphaned and the next request reads the post-write rows. The singleflight leader's snapshot is the one used; coalescing is unchanged. A `Lookup` whose dependencies name another tenant is refused (`ErrForeignDependency`). `Set` now errors only when the backend failed: a value the cache declines (larger than the pool, a non-positive TTL) is not an error. One behavior change: the tenant's version is folded into every key, a pipe result's included, so `InvalidateTenant` (a tenant back on a pool after an absence, or moved to another ClickHouse address or database) now drops that tenant's cached pipe results as well as its query results; before, a pipe result stayed until its TTL. Inserts still do not invalidate pipe results ([#343](https://github.com/Wave-RF/WaveHouse/pull/343)). A backend-agnostic conformance suite, `cachetest.Run`, pins what a hit, a miss and each kind of bump mean, and `LocalCache` runs it; the Redis-compatible backend will run the same suite. -- **The cache's version index no longer grows with every bump, and forgets a tenant no longer served** (`internal/cache/{local,version_manager}.go` (+ tests), `internal/app/wire.go` (+ tests), `docs/src/content/docs/architecture.md`, `AGENTS.md`): fixes [#262](https://github.com/Wave-RF/WaveHouse/issues/262) for the in-process cache, part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The index nested each table under its tenant's version and each scope under its table's, and never pruned, so every tenant invalidation left the tenant's whole index behind, and it grew with every tenant ever served. It now holds one version per tenant, per (tenant, table) and per (tenant, table, scope), bumped in place. A tenant invalidation drops the tenant's index and hands its next key a generation unique within the process, so nothing cached before it can match again, and a table bump drops the table's scope versions. After each settings reload the index of every tenant no longer served, removed or rejected, is dropped the same way; its cached results are orphaned with it, as they already were when such a tenant came back on a pool. No change to what is cached or served. The Redis-compatible backend (#613) bounds its versions with a TTL instead. +- **The cache's version index no longer grows with every bump, and forgets a tenant no longer served** (`internal/cache/{local,version_manager}.go` (+ tests), `internal/app/wire.go` (+ tests), `docs/src/content/docs/architecture.md`, `AGENTS.md`): fixes [#262](https://github.com/Wave-RF/WaveHouse/issues/262) for the in-process cache, part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The index nested each table under its tenant's version and each scope under its table's, and never pruned, so every tenant invalidation left the tenant's whole index behind, and it grew with every tenant ever served. It now holds one version per tenant, per (tenant, table) and per (tenant, table, scope), bumped in place. A tenant invalidation drops the tenant's index and hands its next key a generation unique within the process, so nothing cached before it can match again, and a table bump drops the table's scope versions. After each settings reload the index of every tenant no longer served, removed or rejected, is dropped the same way; its cached results are orphaned with it, as they already were when such a tenant came back on a pool. No change to what is cached or served. The Redis-compatible backend (#613) will bound its versions with a TTL instead. - **An insert invalidates a table's cached results under every tenant the directory holds** (`internal/app/wire.go` (+ tests), `internal/settings/registry.go` (+ tests), `AGENTS.md`, `docs/src/content/docs/{deployment,architecture,ingest-pipeline}.md`): until [#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 6 gives each tenant its own ClickHouse, every tenant reads the same tables, but the ingest worker — which writes every event as tenant `0`'s until story 5 — bumped only tenant `0`'s cache namespaces after an insert, so another tenant's cached query could answer stale rows for up to its TTL (an hour at most). The cache the worker invalidates through now fans each bumped namespace out to every tenant the registry knows (the new `Registry.Known`), the named one and a rejected one included — a rejected tenant comes back into service with the entries it has, so leaving it out would let a folder repaired inside a TTL serve pre-insert rows; reads are untouched, so a tenant is still never served another's cached rows. The residual, a folder removed and restored inside a TTL, is closed since #610: a tenant back on a pool after an absence has its structured-query results orphaned at once (`Cache.InvalidateTenant`). Raised by CodeRabbit on #602. - **The `?token=` strip no longer repairs a query string that does not parse** (`internal/auth/auth.go`): `bearerToken` removed a query-string token by parsing the query, deleting `token`, and re-encoding what was left — and `url.ParseQuery` skips a pair it cannot read, so the re-encoding erased that pair. A handler that parses the query strictly in order to refuse a malformed one would then see a clean query: `GET /v1/ops/pipes?tenant=acme;x=1&token=…` would have answered `200` with the default tenant's pipes. The token is read exactly as before and a query that parses is rewritten exactly as before; a query that does not parse now loses its token pairs and nothing else, byte for byte. Pinned through `api.NewRouter` with the real authenticator, since a handler-level test never runs the middleware that rewrote the URL. diff --git a/internal/app/app_test.go b/internal/app/app_test.go index 84c670703..d21f376a0 100644 --- a/internal/app/app_test.go +++ b/internal/app/app_test.go @@ -671,7 +671,7 @@ func TestReload_ReadmittedTenantCacheIsOrphaned(t *testing.T) { rewriteSettings(t, filepath.Join(root, "globex"), invalidQuery) a.tenants.Reload("test") - assert.Empty(t, mock.GetTenants(), "a rejection releases; it orphans nothing yet") + assert.Empty(t, mock.GetTenants(), "a rejection calls no InvalidateTenant (Prune drops its index)") rewriteSettings(t, filepath.Join(root, "globex"), nil) _, adopted = a.tenants.Reload("test") require.True(t, adopted) @@ -712,6 +712,9 @@ func (p *pruneRecorder) last() map[tenant.ID]bool { func TestReload_PrunesCacheIndexToServedTenants(t *testing.T) { root := writeNestedSettings(t, map[string]map[string]any{"acme": nil, "globex": nil}) a := newApp(t, testConfig(t, root), Options{}) + _, ok := a.cache.(pruner) + require.True(t, ok, "the wired cache prunes") + rec := &pruneRecorder{} a.cache = rec From 9a4888a4959b8523f8c7907947928a2384068b66 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 04:58:48 -0400 Subject: [PATCH 16/45] fix(config): cache.redis defaults in defaults(); 0 never compresses Adopt #632's convention for the cache.redis block: its seven non-zero defaults move from env-default tags into defaults(), each gets a zero case, and the configuration.mdx rows use *(empty)* for an empty default. The doc-defaults test learns to read a duration and an empty list. With the file's zeros now kept, compress_min_bytes needs no -1: 0 means never compress, the backend's own meaning, from the file and the environment alike, and wire.go passes it through unchanged. A negative value refuses boot. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_017aS7rLrH1RKkUMem7X4ckd --- CHANGELOG.md | 2 +- docs/src/content/docs/architecture.md | 2 +- docs/src/content/docs/configuration.mdx | 20 ++++++++++---------- internal/app/app_test.go | 15 +++++++++------ internal/app/wire.go | 6 +----- internal/config/cache_redis.go | 23 +++++++++++------------ internal/config/cache_redis_test.go | 19 ++++++++----------- internal/config/config.go | 14 +++++++++++--- internal/config/defaults_test.go | 14 ++++++++++++++ 9 files changed, 66 insertions(+), 49 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6f742cb0c..7d0e6ea2a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **`cache.backend: redis` shares the query cache across instances** (`internal/config/{cache_redis,backends}.go` (+ tests), `internal/app/wire.go` (+ tests), `tests/integration/shared_cache_test.go`, `scripts/orchestrator/main.go`, `tests/e2e/fixtures/config.yaml`, `.testcoverage.yml`, `deployments/compose/dependencies.yaml`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md,settings-directory.mdx,getting-started.md,pipes.mdx,api.md,development.md}`, `README.md`, `AGENTS.md`): PR E4 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config's `cache.backend` now takes `redis`, configured by a new `cache.redis` block (`WH_CACHE_REDIS_*`): `addrs` (required), `mode` (`standalone`, `cluster`, `sentinel`), `sentinel_master`, `username`, `password` (a secret — set it through the environment), `db`, `tls.{enabled,ca_file,cert_file,key_file,server_name,insecure_skip_verify}`, `key_prefix` (`wh`), `timeout` (`100ms`), `dial_timeout` (`1s`), `max_value_bytes` (1 MiB), `compress_min_bytes` (`1024`; `-1` never compresses, since the loader reads a `0` in the file as unset) and `version_ttl` (`168h`). Every instance pointed at one server shares its cached results, and an insert on any instance invalidates every instance's. A malformed block — no address, an address without a port, an unknown mode, `db` other than `0` in cluster mode, an unreadable TLS file, a TLS key set while `tls.enabled` is off — refuses boot; an unreachable server, or one that rejects the password, does not: the process boots with the cache bypassed and keeps reconnecting, logging a rejected password at `ERROR` on every attempt, so a rotated secret cannot crash-loop every instance at once. `insecure_skip_verify`, and a `cache.redis.addrs` set while `cache.backend` is `local`, are logged at `WARN` at boot. The e2e suite now runs against a Redis testcontainer with `cache.backend: redis`, so the shared backend is exercised end to end, and the e2e coverage exclude for its files is gone; the unit and integration suites keep covering `local`. An integration test boots two instances over one Redis and one ClickHouse: a result one fills is a hit for the other, and a row ingested through one is served fresh by the other on its next query, well inside the stale entry's TTL; another pauses Redis and checks queries keep succeeding from ClickHouse, then turn back to hits. `deployments/compose/dependencies.yaml` gains an optional `redis` profile for local multi-instance work. The deployment guide gains a "Multiple instances and the shared cache" section: what each instance keeps to itself, what a reader on another instance can see and when, `maxmemory-policy`, and the metrics to alert on. +- **`cache.backend: redis` shares the query cache across instances** (`internal/config/{cache_redis,backends,config}.go` (+ tests), `internal/app/wire.go` (+ tests), `tests/integration/shared_cache_test.go`, `scripts/orchestrator/main.go`, `tests/e2e/fixtures/config.yaml`, `.testcoverage.yml`, `deployments/compose/dependencies.yaml`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md,settings-directory.mdx,getting-started.md,pipes.mdx,api.md,development.md}`, `README.md`, `AGENTS.md`): PR E4 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config's `cache.backend` now takes `redis`, configured by a new `cache.redis` block (`WH_CACHE_REDIS_*`): `addrs` (required), `mode` (`standalone`, `cluster`, `sentinel`), `sentinel_master`, `username`, `password` (a secret — set it through the environment), `db`, `tls.{enabled,ca_file,cert_file,key_file,server_name,insecure_skip_verify}`, `key_prefix` (`wh`), `timeout` (`100ms`), `dial_timeout` (`1s`), `max_value_bytes` (1 MiB), `compress_min_bytes` (`1024`; `0` never compresses) and `version_ttl` (`168h`). Every instance pointed at one server shares its cached results, and an insert on any instance invalidates every instance's. A malformed block — no address, an address without a port, an unknown mode, `db` other than `0` in cluster mode, an unreadable TLS file, a TLS key set while `tls.enabled` is off — refuses boot; an unreachable server, or one that rejects the password, does not: the process boots with the cache bypassed and keeps reconnecting, logging a rejected password at `ERROR` on every attempt, so a rotated secret cannot crash-loop every instance at once. `insecure_skip_verify`, and a `cache.redis.addrs` set while `cache.backend` is `local`, are logged at `WARN` at boot. The e2e suite now runs against a Redis testcontainer with `cache.backend: redis`, so the shared backend is exercised end to end, and the e2e coverage exclude for its files is gone; the unit and integration suites keep covering `local`. An integration test boots two instances over one Redis and one ClickHouse: a result one fills is a hit for the other, and a row ingested through one is served fresh by the other on its next query, well inside the stale entry's TTL; another pauses Redis and checks queries keep succeeding from ClickHouse, then turn back to hits. `deployments/compose/dependencies.yaml` gains an optional `redis` profile for local multi-instance work. The deployment guide gains a "Multiple instances and the shared cache" section: what each instance keeps to itself, what a reader on another instance can see and when, `maxmemory-policy`, and the metrics to alert on. - **A Redis-compatible shared cache backend** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row open a circuit breaker that skips the server until a probe succeeds, and deferred invalidations are retried until they land, collapsing to one tenant-wide bump per tenant past 100,000 keys. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. `cache.backend: redis` selects it (the entry above). Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression and server-stops-answering cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. - **One conformance suite for every `mq.Broker`, and a transient broker failure is a `503`** (`internal/mq/mqtest/` (new: the suite and the embedded broker's run of it), `internal/mq/{mq,embedded}.go` (+ tests), `internal/api/ingest.go` (+ tests), `.testcoverage.yml`, `docs/src/content/docs/{api,architecture}.md`, `AGENTS.md`): the first piece of the external-NATS workstream of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `mqtest.Run` states the `Broker` contract as behavior — round trips with names that need encoding, per-tenant order, `Nak` and `AckWait` redelivery, the trace context reaching `Subscribe`, dead-lettering that keeps the topic and leaves the original unacked, per-tenant dead-letter counts, replay bounds and isolation, and exactly one `failed` report when delivery ends underneath a consumer — through the interfaces alone, so the external backend runs the same cases, with `mqtest.Caps` for the four places where its semantics legitimately differ. The embedded broker passes it; writing it turned up that a durable deleted on several tenants' queues could report on `failed` more than once, which is fixed and pinned by a test that deletes it on one queue after another. It also turned up a replay that lost its connection mid-pull passing for a caught-up one when the pull ended in a timeout; that is an error now, as the `Replayer` contract says. The interface comments now allow a delivery unit that is a partition holding several tenants, a `CreateConsumer` that finds a durable rather than creating one, a `PurgeAcked` that leaves retention to the operator, and zero dead-letter counts where there is no per-tenant queue. A new sentinel, `mq.ErrUnavailable`, is a broker that cannot be reached or does not answer in time: the ingest handler answers it with `503` + `Retry-After: 5` rather than the `500` "publish failed" it would have been. Nothing returns it yet; the external backend of #613 will. - **Process roles: the API and the background workers can run in separate processes** (`internal/config/config.go` (+ `roles_test.go`, `defaults_test.go`), `internal/config/backends.go`, `internal/app/{app,wire}.go` (+ `roles_test.go`), `internal/api/router.go` (+ tests), `tests/integration/{setup,tenants}_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md}`, `AGENTS.md`), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The boot config gains `roles` (`WH_ROLES`, default `api,ingest,sweeper`, set in `defaults()` like every boot default, so an explicit `roles: []` refuses boot) and `instance_id` (`WH_INSTANCE_ID`, default `-<8 hex>`, fresh at every boot; logged at boot, and recorded as a lease holder once a shared `coord.backend` exists). A process wires only what its roles need: `api` runs the HTTP API with schema discovery, the token verifiers, the dedupe stores and the SSE hub (all per API process); `ingest` runs the ingest worker; `sweeper` runs the sweeper under its lease. A process without `api` serves an ops-only listener on `server.port` (the probes and their aliases, `/version`, the same-port metrics path, and `POST /v1/ops/settings/reload`, which takes the operator key alone); every other route answers 404, under `/v1/ops` once the operator key has passed. Boot refuses any split over the embedded MQ, which no other process can reach, and `api` without `ingest` (or the reverse) over a local cache, which the ingest worker's invalidations would never reach. Until a shared `mq.backend` exists, every process therefore runs every role, which is the default, so nothing changes for an existing deployment. `data_dir` is probed for Pebble only in a process running `api`. A `config.Config` built without `config.Load` must now name its roles (`config.AllRoles()` for all of them): `app.New` refuses an empty set. diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index fbe6324f4..2aba0526f 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -126,7 +126,7 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ - **config.go** — Loads *boot* configuration from a YAML file with environment variable overrides (using [cleanenv](https://github.com/ilyakaznacheev/cleanenv)); every key has a `WH_`-prefixed env var. Boot config is only what can't change under a running process — the implementation each layer runs on, the process's `roles`, resource sizing, listeners, observability exporters, the settings-directory path, and the secrets (`clickhouse.password`, `cache.redis.password`, `auth.jwt_secret`, `auth.operator_key`). Everything tenant-tunable lives in the settings directory (`settings/`). Both sources are strict: `Load` refuses to boot naming every YAML key the struct doesn't declare (`strict.go`) and every `WH_*` environment variable no field binds (`check.go`), so a tunable that moved to the settings directory can't be read, ignored, and believed. Boot is the validator for this half — there is no dry-run command. See [Configuration Reference](/configuration). - **check.go** — `rejectUnboundEnv` is the environment half of the strict loader: `unboundEnv` walks the struct's `env` tags (plus the two process-level names, `WH_CONFIG` and `WH_LOG_LEVEL`) against the environment; `CheckDataDir` probes `data_dir` — run by `main` right after `Load` when `Config.NeedsDataDir` says a selected backend keeps state there, so an unusable `data_dir` refuses boot before anything dials out. It refuses an empty value (reachable through `WH_DATA_DIR=`) outright rather than probing the working directory; a path that exists and is not a directory; a dangling symlink at `data_dir` or any component above it (the walk to the nearest existing ancestor uses `Lstat`, so a failed mount is not skipped over as "does not exist"); and a directory the process cannot write to — or, when it does not exist, an unwritable nearest ancestor — probed by creating and removing one temp file. A permission denial, on the probe or on reaching the path through a parent without search permission, carries the UID-65532 hint, since a bind mount owned by root is the typical cause. - **backends.go** — the `.backend` keys: one string type per layer (`MQBackend`, `CacheBackend`, `DedupeBackend`, `CoordBackend`), each with its list of the backends this build has, and a `validate` per layer block (`checkBackend`, run by `validateBackends` in `Validate`, between `validateRoles` and `validateTopology`), which refuses a value not on the list and names the ones that are. A backend's own settings go in a `.` sub-block that is its `validate`'s case to check. `Distributed` reports whether the MQ is shared with other processes, `NeedsDataDir` whether a selected backend keeps state under `data_dir`, and `Warnings` returns what a valid configuration is still likely to get wrong — the combinations that are correct for one replica only (a shared MQ over a local cache or Pebble dedupe), a `cache.redis` block that is not read, certificate verification turned off — which `app.New` logs at `WARN`. -- **cache_redis.go** — `CacheRedisConfig`, the `cache.redis` sub-block, and its checks: an address, each `host:port`, a known mode, a sentinel's master name, `db` 0 in cluster mode, positive timeouts and sizes, a `version_ttl` of at least 2 s, and `compress_min_bytes` positive or `-1` (never: cleanenv reads a `0` in the file as unset and applies the default, so `0` cannot mean off). `CacheRedisTLS.Config` builds the `tls.Config`, reading the files; `Validate` calls it so an unreadable file refuses boot, and `internal/app` calls it again to build the connection. A TLS key set while `tls.enabled` is off is an error rather than a plaintext connection. +- **cache_redis.go** — `CacheRedisConfig`, the `cache.redis` sub-block, and its checks: an address, each `host:port`, a known mode, a sentinel's master name, `db` 0 in cluster mode, positive timeouts and sizes, a `version_ttl` of at least 2 s, and a `compress_min_bytes` that is not negative (`0` never compresses); its defaults are in `defaults()` with the rest. `CacheRedisTLS.Config` builds the `tls.Config`, reading the files; `Validate` calls it so an unreadable file refuses boot, and `internal/app` calls it again to build the connection. A TLS key set while `tls.enabled` is off is an error rather than a plaintext connection. - **config.go**, roles — `roles` (`[]Role`: `api`, `ingest`, `sweeper`; `AllRoles` by default; `Has(Role)`) picks which components `internal/app` wires, and `instance_id` names the process (`-<8 hex>` when empty, resolved in `Load`; today only logged at boot, and a distributed coordinator will record it as a lease's holder). `validateRoles` refuses an empty list, an empty entry, an unknown or a repeated role; `validateTopology` refuses a role set the backends cannot serve: any split over the embedded MQ, and a process with exactly one of `api` and `ingest` over a local cache. `NeedsDataDir` counts Pebble only for a process running `api`, and `Warnings` is empty without `api`, since only that role opens a cache it reads or a dedupe store. - **strict.go** — `rejectUnknownKeys`, the YAML half: re-reads the file as a generic tree and walks it against the struct's `yaml` tags, listing every key the struct doesn't declare. cleanenv itself is lenient by design, which is exactly wrong for boot config once keys have moved to the settings directory. - **persistence.go** — `WarnIfFreshDataDir` logs the startup `WARN` when `data_dir` is missing or empty (on a redeploy, the sign that the volume didn't persist); `LogStorageInitError` attaches the UID-65532 `permissionHint` to a NATS or Pebble open failure that looks like a permission denial — the same hint string `CheckDataDir` uses. diff --git a/docs/src/content/docs/configuration.mdx b/docs/src/content/docs/configuration.mdx index 91aa72d6b..0b5348549 100644 --- a/docs/src/content/docs/configuration.mdx +++ b/docs/src/content/docs/configuration.mdx @@ -150,23 +150,23 @@ The `redis` backend's settings, read only when `cache.backend` is `redis`. It is | YAML Key | Env Var | Default | Description | | --- | --- | ------- | ----------- | -| `cache.redis.addrs` | `WH_CACHE_REDIS_ADDRS` | — | **Required** with `backend: redis`. `host:port` of the server; several are a cluster's seed nodes or the sentinels. Comma-separated in the env var. | +| `cache.redis.addrs` | `WH_CACHE_REDIS_ADDRS` | *(empty)* | **Required** with `backend: redis`. `host:port` of the server; several are a cluster's seed nodes or the sentinels. Comma-separated in the env var. | | `cache.redis.mode` | `WH_CACHE_REDIS_MODE` | `standalone` | `standalone`, `cluster` or `sentinel`. | -| `cache.redis.sentinel_master` | `WH_CACHE_REDIS_SENTINEL_MASTER` | — | The master set name. Required with `mode: sentinel`. | -| `cache.redis.username` | `WH_CACHE_REDIS_USERNAME` | — | ACL user. Empty uses the server's `default` user. | -| `cache.redis.password` | `WH_CACHE_REDIS_PASSWORD` | — | A secret: set it through the environment (or your secret store's env injection), not in a tracked `config.yaml`. | +| `cache.redis.sentinel_master` | `WH_CACHE_REDIS_SENTINEL_MASTER` | *(empty)* | The master set name. Required with `mode: sentinel`. | +| `cache.redis.username` | `WH_CACHE_REDIS_USERNAME` | *(empty)* | ACL user. Empty uses the server's `default` user. | +| `cache.redis.password` | `WH_CACHE_REDIS_PASSWORD` | *(empty)* | A secret: set it through the environment (or your secret store's env injection), not in a tracked `config.yaml`. | | `cache.redis.db` | `WH_CACHE_REDIS_DB` | `0` | Database number (`SELECT`). Standalone and sentinel only: a cluster has only database `0`, and any other value refuses boot. | | `cache.redis.tls.enabled` | `WH_CACHE_REDIS_TLS_ENABLED` | `false` | Connect over TLS, verifying the server against the system roots or `ca_file`. Any other `tls` key set while this is off refuses boot, rather than connecting in plaintext. | -| `cache.redis.tls.ca_file` | `WH_CACHE_REDIS_TLS_CA_FILE` | — | PEM file of the authorities to trust instead of the system roots. | -| `cache.redis.tls.cert_file` | `WH_CACHE_REDIS_TLS_CERT_FILE` | — | Client certificate (PEM) for mutual TLS. Set together with `key_file`. | -| `cache.redis.tls.key_file` | `WH_CACHE_REDIS_TLS_KEY_FILE` | — | The client certificate's private key (PEM). | -| `cache.redis.tls.server_name` | `WH_CACHE_REDIS_TLS_SERVER_NAME` | — | Name to verify the server's certificate against, when it differs from the address. | +| `cache.redis.tls.ca_file` | `WH_CACHE_REDIS_TLS_CA_FILE` | *(empty)* | PEM file of the authorities to trust instead of the system roots. | +| `cache.redis.tls.cert_file` | `WH_CACHE_REDIS_TLS_CERT_FILE` | *(empty)* | Client certificate (PEM) for mutual TLS. Set together with `key_file`. | +| `cache.redis.tls.key_file` | `WH_CACHE_REDIS_TLS_KEY_FILE` | *(empty)* | The client certificate's private key (PEM). | +| `cache.redis.tls.server_name` | `WH_CACHE_REDIS_TLS_SERVER_NAME` | *(empty)* | Name to verify the server's certificate against, when it differs from the address. | | `cache.redis.tls.insecure_skip_verify` | `WH_CACHE_REDIS_TLS_INSECURE_SKIP_VERIFY` | `false` | Accept any server certificate. Logged at `WARN` at boot: whoever can intercept the connection can read and replace cached results. | | `cache.redis.key_prefix` | `WH_CACHE_REDIS_KEY_PREFIX` | `wh` | Leads every key, so several deployments can share one server, provided you trust each as much as the others: any of them can overwrite what the rest serve. No `{` or `}`. | | `cache.redis.timeout` | `WH_CACHE_REDIS_TIMEOUT` | `100ms` | Per operation. A lookup or fill that takes longer is a miss or a skipped fill, never a failed query. | | `cache.redis.dial_timeout` | `WH_CACHE_REDIS_DIAL_TIMEOUT` | `1s` | Per connection attempt. | | `cache.redis.max_value_bytes` | `WH_CACHE_REDIS_MAX_VALUE_BYTES` | `1048576` | Largest result stored, after compression (1 MiB). A larger one is returned to the caller but not cached. | -| `cache.redis.compress_min_bytes` | `WH_CACHE_REDIS_COMPRESS_MIN_BYTES` | `1024` | Results at least this large are zstd-compressed when that makes them smaller. `-1` never compresses. `0` is not "off": in the YAML file it reads as unset and takes the default, and in the env var it refuses boot. | +| `cache.redis.compress_min_bytes` | `WH_CACHE_REDIS_COMPRESS_MIN_BYTES` | `1024` | Results at least this large are zstd-compressed when that makes them smaller. `0` never compresses. | | `cache.redis.version_ttl` | `WH_CACHE_REDIS_VERSION_TTL` | `168h` | How long a table's or tenant's version token outlives its last write, so the tokens of dropped tables and removed tenants eventually expire. At least `2s`. An expired token only causes misses. | **When the server is unreachable or misbehaves, the cache is bypassed; queries are not.** A failure or a timeout makes the lookup a miss and the fill a no-op, and five in a row open a circuit breaker that skips the server entirely until a probe, every 5 s, gets an answer. Queries then go straight to ClickHouse, still coalesced per instance by `singleflight`. An invalidation the server did not take is kept and retried until it lands. `/readyz` does not depend on the cache. @@ -284,7 +284,7 @@ cache: timeout: 100ms dial_timeout: 1s max_value_bytes: 1048576 - compress_min_bytes: 1024 # -1 = never + compress_min_bytes: 1024 # 0 = never version_ttl: 168h dedupe: diff --git a/internal/app/app_test.go b/internal/app/app_test.go index 2ca4a205c..204c13711 100644 --- a/internal/app/app_test.go +++ b/internal/app/app_test.go @@ -796,9 +796,10 @@ func TestNew_RedisCacheRefusesAnUnreadableTLSFile(t *testing.T) { require.ErrorContains(t, err, "cache init: cache.redis.tls.ca_file") } -// The boot config's defaults are the backend's, and -1 is the backend's -// "never compress". Driven from Load, not a literal, so a default changed on -// one side only fails here. +// The boot config's defaults are the backend's, and a compress_min_bytes of +// 0 reaches the backend as its "never compress" rather than its default. +// Driven from Load, not a literal, so a default changed on one side only +// fails here. func TestRedisConfig_FromLoadedDefaults(t *testing.T) { t.Setenv("WH_SETTINGS_DIR", t.TempDir()) t.Setenv("WH_CACHE_BACKEND", "redis") @@ -815,11 +816,13 @@ func TestRedisConfig_FromLoadedDefaults(t *testing.T) { CompressMinBytes: cache.DefaultRedisCompressMinBytes, VersionTTL: cache.DefaultRedisVersionTTL, }, got) - loaded.Cache.Redis.CompressMinBytes = -1 - loaded.Cache.Redis.Mode = config.RedisCluster + t.Setenv("WH_CACHE_REDIS_COMPRESS_MIN_BYTES", "0") + t.Setenv("WH_CACHE_REDIS_MODE", "cluster") + loaded, err = config.Load(filepath.Join(t.TempDir(), "none.yaml")) + require.NoError(t, err) got, err = redisConfig(loaded.Cache.Redis) require.NoError(t, err) - assert.Zero(t, got.CompressMinBytes) + assert.Zero(t, got.CompressMinBytes, "the backend's never, not its default") assert.Equal(t, cache.RedisCluster, got.Mode) assert.Equal(t, cache.RedisSentinel, config.RedisSentinel) } diff --git a/internal/app/wire.go b/internal/app/wire.go index 60c9e4a9e..ea23606d0 100644 --- a/internal/app/wire.go +++ b/internal/app/wire.go @@ -664,10 +664,6 @@ func redisConfig(r config.CacheRedisConfig) (cache.RedisConfig, error) { if err != nil { return cache.RedisConfig{}, err } - compressMin := r.CompressMinBytes - if compressMin < 0 { - compressMin = 0 // the backend's "never" - } return cache.RedisConfig{ Addrs: r.Addrs, Mode: r.Mode, @@ -680,7 +676,7 @@ func redisConfig(r config.CacheRedisConfig) (cache.RedisConfig, error) { Timeout: r.Timeout, DialTimeout: r.DialTimeout, MaxValueBytes: r.MaxValueBytes, - CompressMinBytes: compressMin, + CompressMinBytes: r.CompressMinBytes, VersionTTL: r.VersionTTL, }, nil } diff --git a/internal/config/cache_redis.go b/internal/config/cache_redis.go index 3ae3114ea..997d19a3a 100644 --- a/internal/config/cache_redis.go +++ b/internal/config/cache_redis.go @@ -25,24 +25,23 @@ type CacheRedisConfig struct { // Addrs are host:port pairs: the server, or seeds for a cluster, or the // sentinels. Addrs []string `yaml:"addrs" env:"WH_CACHE_REDIS_ADDRS"` - Mode string `yaml:"mode" env:"WH_CACHE_REDIS_MODE" env-default:"standalone"` + Mode string `yaml:"mode" env:"WH_CACHE_REDIS_MODE"` SentinelMaster string `yaml:"sentinel_master" env:"WH_CACHE_REDIS_SENTINEL_MASTER"` Username string `yaml:"username" env:"WH_CACHE_REDIS_USERNAME"` Password string `yaml:"password" env:"WH_CACHE_REDIS_PASSWORD"` DB int `yaml:"db" env:"WH_CACHE_REDIS_DB"` TLS CacheRedisTLS `yaml:"tls"` // KeyPrefix leads every key, so deployments can share one server. - KeyPrefix string `yaml:"key_prefix" env:"WH_CACHE_REDIS_KEY_PREFIX" env-default:"wh"` - Timeout time.Duration `yaml:"timeout" env:"WH_CACHE_REDIS_TIMEOUT" env-default:"100ms"` - DialTimeout time.Duration `yaml:"dial_timeout" env:"WH_CACHE_REDIS_DIAL_TIMEOUT" env-default:"1s"` + KeyPrefix string `yaml:"key_prefix" env:"WH_CACHE_REDIS_KEY_PREFIX"` + Timeout time.Duration `yaml:"timeout" env:"WH_CACHE_REDIS_TIMEOUT"` + DialTimeout time.Duration `yaml:"dial_timeout" env:"WH_CACHE_REDIS_DIAL_TIMEOUT"` // MaxValueBytes is the largest value stored, after compression. - MaxValueBytes int `yaml:"max_value_bytes" env:"WH_CACHE_REDIS_MAX_VALUE_BYTES" env-default:"1048576"` - // CompressMinBytes is the smallest value zstd-compressed; -1 never - // compresses. Not 0: the loader reads a 0 in the file as unset and - // applies the default, so 0 cannot mean off. - CompressMinBytes int `yaml:"compress_min_bytes" env:"WH_CACHE_REDIS_COMPRESS_MIN_BYTES" env-default:"1024"` + MaxValueBytes int `yaml:"max_value_bytes" env:"WH_CACHE_REDIS_MAX_VALUE_BYTES"` + // CompressMinBytes is the smallest value zstd-compressed; 0 never + // compresses. + CompressMinBytes int `yaml:"compress_min_bytes" env:"WH_CACHE_REDIS_COMPRESS_MIN_BYTES"` // VersionTTL is how long a version token outlives its last bump. - VersionTTL time.Duration `yaml:"version_ttl" env:"WH_CACHE_REDIS_VERSION_TTL" env-default:"168h"` + VersionTTL time.Duration `yaml:"version_ttl" env:"WH_CACHE_REDIS_VERSION_TTL"` } // CacheRedisTLS is cache.redis.tls. The files are paths, read at boot. @@ -106,8 +105,8 @@ func (r CacheRedisConfig) validate() error { if r.MaxValueBytes <= 0 { return fmt.Errorf("cache.redis.max_value_bytes (WH_CACHE_REDIS_MAX_VALUE_BYTES) %d must be positive", r.MaxValueBytes) } - if r.CompressMinBytes == 0 || r.CompressMinBytes < -1 { - return fmt.Errorf("cache.redis.compress_min_bytes (WH_CACHE_REDIS_COMPRESS_MIN_BYTES) %d: want a positive size, or -1 to never compress", r.CompressMinBytes) + if r.CompressMinBytes < 0 { + return fmt.Errorf("cache.redis.compress_min_bytes (WH_CACHE_REDIS_COMPRESS_MIN_BYTES) %d is negative: want a size, or 0 to never compress", r.CompressMinBytes) } if _, err := r.TLS.Config(); err != nil { return err diff --git a/internal/config/cache_redis_test.go b/internal/config/cache_redis_test.go index e1b5783a8..5389715a9 100644 --- a/internal/config/cache_redis_test.go +++ b/internal/config/cache_redis_test.go @@ -60,7 +60,7 @@ func TestLoad_CacheRedisFromEnv(t *testing.T) { "WH_CACHE_REDIS_TIMEOUT": "250ms", "WH_CACHE_REDIS_DIAL_TIMEOUT": "3s", "WH_CACHE_REDIS_MAX_VALUE_BYTES": "2048", - "WH_CACHE_REDIS_COMPRESS_MIN_BYTES": "-1", + "WH_CACHE_REDIS_COMPRESS_MIN_BYTES": "0", "WH_CACHE_REDIS_VERSION_TTL": "24h", "WH_CACHE_REDIS_TLS_INSECURE_SKIP_VERIFY": "false", } { @@ -75,7 +75,7 @@ func TestLoad_CacheRedisFromEnv(t *testing.T) { Enabled: true, CAFile: caFile, CertFile: certFile, KeyFile: keyFile, ServerName: "redis.internal", }, KeyPrefix: "staging", Timeout: 250 * time.Millisecond, DialTimeout: 3 * time.Second, - MaxValueBytes: 2048, CompressMinBytes: -1, VersionTTL: 24 * time.Hour, + MaxValueBytes: 2048, CompressMinBytes: 0, VersionTTL: 24 * time.Hour, }, cfg.Cache.Redis) tc, err := cfg.Cache.Redis.TLS.Config() require.NoError(t, err) @@ -112,10 +112,8 @@ cache: assert.Equal(t, 1024, r.CompressMinBytes) } -// A 0 in the file is read as unset, so it takes the default instead of -// switching compression off: why "never" is -1. Pinned so a loader that -// starts honoring the 0 is noticed. -func TestLoad_CacheRedisCompressZeroInYAMLIsTheDefault(t *testing.T) { +// A 0 in the file is kept, and is the backend's "never compress". +func TestLoad_CacheRedisCompressZeroInYAMLIsNever(t *testing.T) { t.Parallel() path := filepath.Join(t.TempDir(), "config.yaml") require.NoError(t, os.WriteFile(path, []byte(` @@ -129,7 +127,7 @@ cache: `), 0o600)) cfg, err := Load(path) require.NoError(t, err) - assert.Equal(t, 1024, cfg.Cache.Redis.CompressMinBytes) + assert.Zero(t, cfg.Cache.Redis.CompressMinBytes) } func TestLoad_CacheRedisRefusesUnknownKeys(t *testing.T) { @@ -171,7 +169,7 @@ func TestUnboundEnv_KnowsTheCacheRedisVariables(t *testing.T) { t.Parallel() assert.Empty(t, unboundEnv([]string{ "WH_CACHE_REDIS_ADDRS=r:6379", "WH_CACHE_REDIS_PASSWORD=x", "WH_CACHE_REDIS_TLS_CA_FILE=/ca.pem", - "WH_CACHE_REDIS_VERSION_TTL=1h", "WH_CACHE_REDIS_COMPRESS_MIN_BYTES=-1", + "WH_CACHE_REDIS_VERSION_TTL=1h", "WH_CACHE_REDIS_COMPRESS_MIN_BYTES=0", })) assert.Equal(t, []string{"WH_CACHE_REDIS_ADDR"}, unboundEnv([]string{"WH_CACHE_REDIS_ADDR=r:6379"})) } @@ -203,9 +201,8 @@ func TestValidate_CacheRedis(t *testing.T) { {"negative dial timeout", func(r *CacheRedisConfig) { r.DialTimeout = -time.Second }, "cache.redis.dial_timeout"}, {"short version ttl", func(r *CacheRedisConfig) { r.VersionTTL = time.Second }, "cache.redis.version_ttl (WH_CACHE_REDIS_VERSION_TTL) 1s is under 2s"}, {"zero max value", func(r *CacheRedisConfig) { r.MaxValueBytes = 0 }, "cache.redis.max_value_bytes"}, - {"compress 0", func(r *CacheRedisConfig) { r.CompressMinBytes = 0 }, "or -1 to never compress"}, - {"compress -2", func(r *CacheRedisConfig) { r.CompressMinBytes = -2 }, "or -1 to never compress"}, - {"compress never", func(r *CacheRedisConfig) { r.CompressMinBytes = -1 }, ""}, + {"compress never", func(r *CacheRedisConfig) { r.CompressMinBytes = 0 }, ""}, + {"compress negative", func(r *CacheRedisConfig) { r.CompressMinBytes = -1 }, "-1 is negative: want a size, or 0 to never compress"}, {"tls files while off", func(r *CacheRedisConfig) { r.TLS.CAFile = caFile }, "cache.redis.tls.enabled (WH_CACHE_REDIS_TLS_ENABLED) is off"}, {"tls system roots", func(r *CacheRedisConfig) { r.TLS.Enabled = true }, ""}, {"tls full", func(r *CacheRedisConfig) { diff --git a/internal/config/config.go b/internal/config/config.go index 95bfacb15..27f4188cc 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -7,6 +7,7 @@ import ( "os" "slices" "strings" + "time" "github.com/ilyakaznacheev/cleanenv" ) @@ -255,9 +256,16 @@ func defaults() Config { Roles: AllRoles(), Server: Server{Port: 8080, ShutdownTimeout: 10}, MQ: MQ{Backend: MQEmbedded}, - Cache: Cache{Backend: CacheLocal, L1MaxCost: 64 << 20}, - Dedupe: Dedupe{Backend: DedupePebble}, - Coord: Coord{Backend: CoordLocal}, + Cache: Cache{ + Backend: CacheLocal, L1MaxCost: 64 << 20, + Redis: CacheRedisConfig{ + Mode: RedisStandalone, KeyPrefix: "wh", + Timeout: 100 * time.Millisecond, DialTimeout: time.Second, + MaxValueBytes: 1 << 20, CompressMinBytes: 1 << 10, VersionTTL: 168 * time.Hour, + }, + }, + Dedupe: Dedupe{Backend: DedupePebble}, + Coord: Coord{Backend: CoordLocal}, OTel: OTel{ Traces: OTelTraces{Enabled: true, SampleRate: 1.0}, Metrics: OTelMetrics{Enabled: true}, diff --git a/internal/config/defaults_test.go b/internal/config/defaults_test.go index 6877be4bb..1a8fba337 100644 --- a/internal/config/defaults_test.go +++ b/internal/config/defaults_test.go @@ -9,6 +9,7 @@ import ( "strconv" "strings" "testing" + "time" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" @@ -37,6 +38,15 @@ var zeroCases = []zeroCase{ {"cache.l1_max_cost", "WH_CACHE_L1_MAX_COST", int64(0), int64(64 << 20), "1024", int64(1024), func(c *Config) any { return c.Cache.L1MaxCost }}, {"prometheus.path", "WH_PROMETHEUS_PATH", "", "/metrics", "/prom", "/prom", func(c *Config) any { return c.Prometheus.Path }}, {"data_dir", "WH_DATA_DIR", "", "./data", "/var/lib/wh", "/var/lib/wh", func(c *Config) any { return c.DataDir }}, + // The cache.redis block is validated only under backend=redis, so under + // the default backend its zeros load as written. + {"cache.redis.mode", "WH_CACHE_REDIS_MODE", "", RedisStandalone, RedisCluster, RedisCluster, func(c *Config) any { return c.Cache.Redis.Mode }}, + {"cache.redis.key_prefix", "WH_CACHE_REDIS_KEY_PREFIX", "", "wh", "staging", "staging", func(c *Config) any { return c.Cache.Redis.KeyPrefix }}, + {"cache.redis.timeout", "WH_CACHE_REDIS_TIMEOUT", time.Duration(0), 100 * time.Millisecond, "250ms", 250 * time.Millisecond, func(c *Config) any { return c.Cache.Redis.Timeout }}, + {"cache.redis.dial_timeout", "WH_CACHE_REDIS_DIAL_TIMEOUT", time.Duration(0), time.Second, "500ms", 500 * time.Millisecond, func(c *Config) any { return c.Cache.Redis.DialTimeout }}, + {"cache.redis.max_value_bytes", "WH_CACHE_REDIS_MAX_VALUE_BYTES", 0, 1 << 20, "2048", 2048, func(c *Config) any { return c.Cache.Redis.MaxValueBytes }}, + {"cache.redis.compress_min_bytes", "WH_CACHE_REDIS_COMPRESS_MIN_BYTES", 0, 1 << 10, "2048", 2048, func(c *Config) any { return c.Cache.Redis.CompressMinBytes }}, + {"cache.redis.version_ttl", "WH_CACHE_REDIS_VERSION_TTL", time.Duration(0), 168 * time.Hour, "1h", time.Hour, func(c *Config) any { return c.Cache.Redis.VersionTTL }}, } // refusedZeros are the non-zero defaults whose zero Validate refuses: written @@ -296,11 +306,15 @@ func parseDocDefault(t *testing.T, key, cell string, like any) any { v, err = strconv.ParseInt(cell, 10, 64) case float64: v, err = strconv.ParseFloat(cell, 64) + case time.Duration: + v, err = time.ParseDuration(cell) default: rt := reflect.TypeOf(like) switch { case rt.Kind() == reflect.String: v = reflect.ValueOf(cell).Convert(rt).Interface() + case rt.Kind() == reflect.Slice && rt.Elem().Kind() == reflect.String && cell == "": + v = reflect.Zero(rt).Interface() // cache.redis.addrs: nil case rt.Kind() == reflect.Slice && rt.Elem().Kind() == reflect.String: // roles: a comma-separated cell parts := strings.Split(cell, ",") sv := reflect.MakeSlice(rt, len(parts), len(parts)) From 6fa1ae6eb949998c548c34bdc6e1cc623402029c Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 05:00:52 -0400 Subject: [PATCH 17/45] fix(cache): give the two-instance test its roles; redis is the shared cache app.New now wires the API, ingest and cache only for the roles a config names (#622), and refuses a Config with none, so the shared-cache integration test's instances run every role, as the suite's own app does. The refusal of api without ingest over a local cache now names cache.backend=redis, the shared cache it asks for, and the configuration and deployment pages say this build has a shared cache but still refuses every split while the queue and leases are in-process. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_017aS7rLrH1RKkUMem7X4ckd --- CHANGELOG.md | 2 +- docs/src/content/docs/configuration.mdx | 2 +- docs/src/content/docs/deployment.md | 2 +- internal/config/config.go | 2 +- internal/config/roles_test.go | 4 +++- tests/integration/shared_cache_test.go | 1 + 6 files changed, 8 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7d0e6ea2a..f8f2efab6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **`cache.backend: redis` shares the query cache across instances** (`internal/config/{cache_redis,backends,config}.go` (+ tests), `internal/app/wire.go` (+ tests), `tests/integration/shared_cache_test.go`, `scripts/orchestrator/main.go`, `tests/e2e/fixtures/config.yaml`, `.testcoverage.yml`, `deployments/compose/dependencies.yaml`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md,settings-directory.mdx,getting-started.md,pipes.mdx,api.md,development.md}`, `README.md`, `AGENTS.md`): PR E4 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config's `cache.backend` now takes `redis`, configured by a new `cache.redis` block (`WH_CACHE_REDIS_*`): `addrs` (required), `mode` (`standalone`, `cluster`, `sentinel`), `sentinel_master`, `username`, `password` (a secret — set it through the environment), `db`, `tls.{enabled,ca_file,cert_file,key_file,server_name,insecure_skip_verify}`, `key_prefix` (`wh`), `timeout` (`100ms`), `dial_timeout` (`1s`), `max_value_bytes` (1 MiB), `compress_min_bytes` (`1024`; `0` never compresses) and `version_ttl` (`168h`). Every instance pointed at one server shares its cached results, and an insert on any instance invalidates every instance's. A malformed block — no address, an address without a port, an unknown mode, `db` other than `0` in cluster mode, an unreadable TLS file, a TLS key set while `tls.enabled` is off — refuses boot; an unreachable server, or one that rejects the password, does not: the process boots with the cache bypassed and keeps reconnecting, logging a rejected password at `ERROR` on every attempt, so a rotated secret cannot crash-loop every instance at once. `insecure_skip_verify`, and a `cache.redis.addrs` set while `cache.backend` is `local`, are logged at `WARN` at boot. The e2e suite now runs against a Redis testcontainer with `cache.backend: redis`, so the shared backend is exercised end to end, and the e2e coverage exclude for its files is gone; the unit and integration suites keep covering `local`. An integration test boots two instances over one Redis and one ClickHouse: a result one fills is a hit for the other, and a row ingested through one is served fresh by the other on its next query, well inside the stale entry's TTL; another pauses Redis and checks queries keep succeeding from ClickHouse, then turn back to hits. `deployments/compose/dependencies.yaml` gains an optional `redis` profile for local multi-instance work. The deployment guide gains a "Multiple instances and the shared cache" section: what each instance keeps to itself, what a reader on another instance can see and when, `maxmemory-policy`, and the metrics to alert on. +- **`cache.backend: redis` shares the query cache across instances** (`internal/config/{cache_redis,backends,config}.go` (+ tests), `internal/app/wire.go` (+ tests), `tests/integration/shared_cache_test.go`, `scripts/orchestrator/main.go`, `tests/e2e/fixtures/config.yaml`, `.testcoverage.yml`, `deployments/compose/dependencies.yaml`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md,settings-directory.mdx,getting-started.md,pipes.mdx,api.md,development.md}`, `README.md`, `AGENTS.md`): PR E4 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config's `cache.backend` now takes `redis`, configured by a new `cache.redis` block (`WH_CACHE_REDIS_*`): `addrs` (required), `mode` (`standalone`, `cluster`, `sentinel`), `sentinel_master`, `username`, `password` (a secret — set it through the environment), `db`, `tls.{enabled,ca_file,cert_file,key_file,server_name,insecure_skip_verify}`, `key_prefix` (`wh`), `timeout` (`100ms`), `dial_timeout` (`1s`), `max_value_bytes` (1 MiB), `compress_min_bytes` (`1024`; `0` never compresses) and `version_ttl` (`168h`). Every instance pointed at one server shares its cached results, and an insert on any instance invalidates every instance's. It is also the shared cache a split of [`roles`](https://github.com/Wave-RF/WaveHouse/pull/622) needs: the refusal of `api` without `ingest`, or the reverse, over `cache.backend: local` now names `redis`, though every split is still refused while the queue is embedded. A malformed block — no address, an address without a port, an unknown mode, `db` other than `0` in cluster mode, an unreadable TLS file, a TLS key set while `tls.enabled` is off — refuses boot; an unreachable server, or one that rejects the password, does not: the process boots with the cache bypassed and keeps reconnecting, logging a rejected password at `ERROR` on every attempt, so a rotated secret cannot crash-loop every instance at once. `insecure_skip_verify`, and a `cache.redis.addrs` set while `cache.backend` is `local`, are logged at `WARN` at boot. The e2e suite now runs against a Redis testcontainer with `cache.backend: redis`, so the shared backend is exercised end to end, and the e2e coverage exclude for its files is gone; the unit and integration suites keep covering `local`. An integration test boots two instances over one Redis and one ClickHouse: a result one fills is a hit for the other, and a row ingested through one is served fresh by the other on its next query, well inside the stale entry's TTL; another pauses Redis and checks queries keep succeeding from ClickHouse, then turn back to hits. `deployments/compose/dependencies.yaml` gains an optional `redis` profile for local multi-instance work. The deployment guide gains a "Multiple instances and the shared cache" section: what each instance keeps to itself, what a reader on another instance can see and when, `maxmemory-policy`, and the metrics to alert on. - **A Redis-compatible shared cache backend** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row open a circuit breaker that skips the server until a probe succeeds, and deferred invalidations are retried until they land, collapsing to one tenant-wide bump per tenant past 100,000 keys. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. `cache.backend: redis` selects it (the entry above). Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression and server-stops-answering cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. - **One conformance suite for every `mq.Broker`, and a transient broker failure is a `503`** (`internal/mq/mqtest/` (new: the suite and the embedded broker's run of it), `internal/mq/{mq,embedded}.go` (+ tests), `internal/api/ingest.go` (+ tests), `.testcoverage.yml`, `docs/src/content/docs/{api,architecture}.md`, `AGENTS.md`): the first piece of the external-NATS workstream of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `mqtest.Run` states the `Broker` contract as behavior — round trips with names that need encoding, per-tenant order, `Nak` and `AckWait` redelivery, the trace context reaching `Subscribe`, dead-lettering that keeps the topic and leaves the original unacked, per-tenant dead-letter counts, replay bounds and isolation, and exactly one `failed` report when delivery ends underneath a consumer — through the interfaces alone, so the external backend runs the same cases, with `mqtest.Caps` for the four places where its semantics legitimately differ. The embedded broker passes it; writing it turned up that a durable deleted on several tenants' queues could report on `failed` more than once, which is fixed and pinned by a test that deletes it on one queue after another. It also turned up a replay that lost its connection mid-pull passing for a caught-up one when the pull ended in a timeout; that is an error now, as the `Replayer` contract says. The interface comments now allow a delivery unit that is a partition holding several tenants, a `CreateConsumer` that finds a durable rather than creating one, a `PurgeAcked` that leaves retention to the operator, and zero dead-letter counts where there is no per-tenant queue. A new sentinel, `mq.ErrUnavailable`, is a broker that cannot be reached or does not answer in time: the ingest handler answers it with `503` + `Retry-After: 5` rather than the `500` "publish failed" it would have been. Nothing returns it yet; the external backend of #613 will. - **Process roles: the API and the background workers can run in separate processes** (`internal/config/config.go` (+ `roles_test.go`, `defaults_test.go`), `internal/config/backends.go`, `internal/app/{app,wire}.go` (+ `roles_test.go`), `internal/api/router.go` (+ tests), `tests/integration/{setup,tenants}_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md}`, `AGENTS.md`), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The boot config gains `roles` (`WH_ROLES`, default `api,ingest,sweeper`, set in `defaults()` like every boot default, so an explicit `roles: []` refuses boot) and `instance_id` (`WH_INSTANCE_ID`, default `-<8 hex>`, fresh at every boot; logged at boot, and recorded as a lease holder once a shared `coord.backend` exists). A process wires only what its roles need: `api` runs the HTTP API with schema discovery, the token verifiers, the dedupe stores and the SSE hub (all per API process); `ingest` runs the ingest worker; `sweeper` runs the sweeper under its lease. A process without `api` serves an ops-only listener on `server.port` (the probes and their aliases, `/version`, the same-port metrics path, and `POST /v1/ops/settings/reload`, which takes the operator key alone); every other route answers 404, under `/v1/ops` once the operator key has passed. Boot refuses any split over the embedded MQ, which no other process can reach, and `api` without `ingest` (or the reverse) over a local cache, which the ingest worker's invalidations would never reach. Until a shared `mq.backend` exists, every process therefore runs every role, which is the default, so nothing changes for an existing deployment. `data_dir` is probed for Pebble only in a process running `api`. A `config.Config` built without `config.Load` must now name its roles (`config.AllRoles()` for all of them): `app.New` refuses an empty set. diff --git a/docs/src/content/docs/configuration.mdx b/docs/src/content/docs/configuration.mdx index 0b5348549..6425d6f77 100644 --- a/docs/src/content/docs/configuration.mdx +++ b/docs/src/content/docs/configuration.mdx @@ -70,7 +70,7 @@ Every process, whatever its roles, reads the settings directory and reloads it ( Boot refuses a role set the selected backends cannot serve: - **Any split with `mq.backend=embedded`.** The embedded queue lives inside its process and listens on no port, so a process without every role could not reach it. Until a shared `mq.backend` exists, every process runs every role. -- **`api` without `ingest`, or `ingest` without `api`, with `cache.backend=local`.** The ingest worker invalidates the cache the API reads, and a local cache in another process never sees that invalidation. Run `api` and `ingest` together, or choose a shared `cache.backend`. A `sweeper`-only process holds no cache, so this rule does not apply to it. +- **`api` without `ingest`, or `ingest` without `api`, with `cache.backend=local`.** The ingest worker invalidates the cache the API reads, and a local cache in another process never sees that invalidation. Run `api` and `ingest` together, or set [`cache.backend: redis`](#backends), one cache every process shares. A `sweeper`-only process holds no cache, so this rule does not apply to it. ### Server diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index 32e508a68..2acf348c9 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -347,7 +347,7 @@ By default one process runs all of WaveHouse. [`roles`](/configuration#process-r - **Ingest.** Every ingest pod consumes the same shared durable consumer and competes for its messages, so throughput scales with the pod count. The rows of one table are then split across pods: each pod writes smaller batches, and rows written by different pods do not reach ClickHouse in publish order. - **Sweeper.** The sweeper runs under a lease held in the shared `coord.backend`, so only one pod sweeps at a time. A second replica waits and takes over when the first stops. -A split needs backends that every process can reach: a shared `mq.backend`, so that every process reaches the same queue; a shared `cache.backend`, so that the ingest pods' invalidations reach the API pods' cache; and a shared `coord.backend`, so that the sweeper lease spans pods. **This build has only the in-process backends, so boot refuses any split** and names the backend to change. Until shared backends ship, run every role in one process, the default. +A split needs backends that every process can reach: a shared `mq.backend`, so that every process reaches the same queue; a shared `cache.backend` ([`redis`](#multiple-instances-and-the-shared-cache)), so that the ingest pods' invalidations reach the API pods' cache; and a shared `coord.backend`, so that the sweeper lease spans pods. **This build has a shared cache but only the in-process queue and leases, so boot refuses any split** and names the backend to change. Until a shared `mq.backend` and `coord.backend` ship, run every role in one process, the default. A pod without the `api` role serves an ops listener on `:8080`: `/livez`, `/readyz` and their aliases, `/version`, the metrics path when `prometheus.port` is `0`, and `POST /v1/ops/settings/reload`. Every other route answers 404 (under `/v1/ops`, 403 without the operator key, and 401 for a bearer token). Point the same probes at it as at an API pod. `/livez` does not wait for schema discovery there, because only the API runs it. `/readyz` checks ClickHouse in an ingest pod, and is ready once a sweeper pod has booted. Every pod reads the settings directory, so mount it in every Deployment. The reload route on the ops listener accepts only the operator key, so whatever reloads your API pods over HTTP must send the operator key to the worker pods too, or rely on `SIGHUP` (or, over a flat directory, the directory watcher) instead. diff --git a/internal/config/config.go b/internal/config/config.go index 27f4188cc..0bc70f2ab 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -218,7 +218,7 @@ func (c *Config) validateTopology() error { return fmt.Errorf("roles %s with mq.backend=embedded: the embedded MQ lives inside this process, and a process without it cannot reach its queue — run every role (%s), or set a shared mq.backend", joinRoles(c.Roles), joinRoles(allRoles)) } if c.splitsCache() && c.Cache.Backend == CacheLocal { - return fmt.Errorf("roles %s with cache.backend=local: api and ingest run in different processes, and the ingest worker's cache invalidation would never reach the API's cache — run api and ingest together, or set a shared cache.backend", joinRoles(c.Roles)) + return fmt.Errorf("roles %s with cache.backend=local: api and ingest run in different processes, and the ingest worker's cache invalidation would never reach the API's cache — run api and ingest together, or set cache.backend=redis, one cache every process shares", joinRoles(c.Roles)) } return nil } diff --git a/internal/config/roles_test.go b/internal/config/roles_test.go index 18b28267c..914815bf9 100644 --- a/internal/config/roles_test.go +++ b/internal/config/roles_test.go @@ -114,12 +114,14 @@ func TestValidate_RoleSplits(t *testing.T) { {"every role, shared queue", all, "shared", CacheLocal, ""}, {"api+ingest, shared queue", []Role{RoleAPI, RoleIngest}, "shared", CacheLocal, ""}, {"sweeper, shared queue", []Role{RoleSweeper}, "shared", CacheLocal, ""}, - {"api, local cache", []Role{RoleAPI}, "shared", CacheLocal, "roles api with cache.backend=local: api and ingest run in different processes"}, + {"api, local cache", []Role{RoleAPI}, "shared", CacheLocal, "roles api with cache.backend=local: api and ingest run in different processes, and the ingest worker's cache invalidation would never reach the API's cache — run api and ingest together, or set cache.backend=redis"}, {"ingest, local cache", []Role{RoleIngest}, "shared", CacheLocal, "roles ingest with cache.backend=local"}, {"api+sweeper, local cache", []Role{RoleAPI, RoleSweeper}, "shared", CacheLocal, "roles api,sweeper with cache.backend=local"}, {"ingest+sweeper, local cache", []Role{RoleIngest, RoleSweeper}, "shared", CacheLocal, "roles ingest,sweeper with cache.backend=local"}, {"api, shared cache", []Role{RoleAPI}, "shared", "shared", ""}, {"ingest, shared cache", []Role{RoleIngest}, "shared", "shared", ""}, + {"api, redis cache", []Role{RoleAPI}, "shared", CacheRedis, ""}, + {"ingest, redis cache", []Role{RoleIngest}, "shared", CacheRedis, ""}, } { t.Run(tc.name, func(t *testing.T) { t.Parallel() diff --git a/tests/integration/shared_cache_test.go b/tests/integration/shared_cache_test.go index b665ccb67..f4a884d0e 100644 --- a/tests/integration/shared_cache_test.go +++ b/tests/integration/shared_cache_test.go @@ -85,6 +85,7 @@ func bootRedisApp(t *testing.T, redisAddr, prefix string, timeout time.Duration) }}, Dedupe: config.Dedupe{Backend: config.DedupePebble}, Coord: config.Coord{Backend: config.CoordLocal}, + Roles: config.AllRoles(), Settings: config.Settings{Dir: settingsDir}, } a, err := app.New(ctx, app.Options{Config: cfg, Listener: ln}) From 712b1db250585dd0dfed112e76754763cdc50c8e Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 05:02:57 -0400 Subject: [PATCH 18/45] fix(config): refuse cache.redis.mode=sentinel until #656 Sentinel mode passed only the master set name: the backend does not authenticate to the sentinels, sets no topology refresh, and has no test (#656). Rather than make it selectable, validation refuses it and points at the issue; standalone and cluster are unchanged. cache.redis.sentinel_master is removed rather than kept and refused: a key that no config can use is noise in the reference, and the strict loader now reports it (and WH_CACHE_REDIS_SENTINEL_MASTER) as unknown, so nobody sets it believing it works. #656 can add it back with the sentinel credentials it also needs. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_017aS7rLrH1RKkUMem7X4ckd --- CHANGELOG.md | 2 +- docs/src/content/docs/architecture.md | 2 +- docs/src/content/docs/configuration.mdx | 11 ++++----- docs/src/content/docs/deployment.md | 2 +- internal/app/wire.go | 1 - internal/config/cache_redis.go | 30 ++++++++++++------------- internal/config/cache_redis_test.go | 17 +++++++------- 7 files changed, 30 insertions(+), 35 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f8f2efab6..ac09db1ae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **`cache.backend: redis` shares the query cache across instances** (`internal/config/{cache_redis,backends,config}.go` (+ tests), `internal/app/wire.go` (+ tests), `tests/integration/shared_cache_test.go`, `scripts/orchestrator/main.go`, `tests/e2e/fixtures/config.yaml`, `.testcoverage.yml`, `deployments/compose/dependencies.yaml`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md,settings-directory.mdx,getting-started.md,pipes.mdx,api.md,development.md}`, `README.md`, `AGENTS.md`): PR E4 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config's `cache.backend` now takes `redis`, configured by a new `cache.redis` block (`WH_CACHE_REDIS_*`): `addrs` (required), `mode` (`standalone`, `cluster`, `sentinel`), `sentinel_master`, `username`, `password` (a secret — set it through the environment), `db`, `tls.{enabled,ca_file,cert_file,key_file,server_name,insecure_skip_verify}`, `key_prefix` (`wh`), `timeout` (`100ms`), `dial_timeout` (`1s`), `max_value_bytes` (1 MiB), `compress_min_bytes` (`1024`; `0` never compresses) and `version_ttl` (`168h`). Every instance pointed at one server shares its cached results, and an insert on any instance invalidates every instance's. It is also the shared cache a split of [`roles`](https://github.com/Wave-RF/WaveHouse/pull/622) needs: the refusal of `api` without `ingest`, or the reverse, over `cache.backend: local` now names `redis`, though every split is still refused while the queue is embedded. A malformed block — no address, an address without a port, an unknown mode, `db` other than `0` in cluster mode, an unreadable TLS file, a TLS key set while `tls.enabled` is off — refuses boot; an unreachable server, or one that rejects the password, does not: the process boots with the cache bypassed and keeps reconnecting, logging a rejected password at `ERROR` on every attempt, so a rotated secret cannot crash-loop every instance at once. `insecure_skip_verify`, and a `cache.redis.addrs` set while `cache.backend` is `local`, are logged at `WARN` at boot. The e2e suite now runs against a Redis testcontainer with `cache.backend: redis`, so the shared backend is exercised end to end, and the e2e coverage exclude for its files is gone; the unit and integration suites keep covering `local`. An integration test boots two instances over one Redis and one ClickHouse: a result one fills is a hit for the other, and a row ingested through one is served fresh by the other on its next query, well inside the stale entry's TTL; another pauses Redis and checks queries keep succeeding from ClickHouse, then turn back to hits. `deployments/compose/dependencies.yaml` gains an optional `redis` profile for local multi-instance work. The deployment guide gains a "Multiple instances and the shared cache" section: what each instance keeps to itself, what a reader on another instance can see and when, `maxmemory-policy`, and the metrics to alert on. +- **`cache.backend: redis` shares the query cache across instances** (`internal/config/{cache_redis,backends,config}.go` (+ tests), `internal/app/wire.go` (+ tests), `tests/integration/shared_cache_test.go`, `scripts/orchestrator/main.go`, `tests/e2e/fixtures/config.yaml`, `.testcoverage.yml`, `deployments/compose/dependencies.yaml`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md,settings-directory.mdx,getting-started.md,pipes.mdx,api.md,development.md}`, `README.md`, `AGENTS.md`): PR E4 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config's `cache.backend` now takes `redis`, configured by a new `cache.redis` block (`WH_CACHE_REDIS_*`): `addrs` (required), `mode` (`standalone` or `cluster`; `sentinel` refuses boot until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `username`, `password` (a secret — set it through the environment), `db`, `tls.{enabled,ca_file,cert_file,key_file,server_name,insecure_skip_verify}`, `key_prefix` (`wh`), `timeout` (`100ms`), `dial_timeout` (`1s`), `max_value_bytes` (1 MiB), `compress_min_bytes` (`1024`; `0` never compresses) and `version_ttl` (`168h`). Every instance pointed at one server shares its cached results, and an insert on any instance invalidates every instance's. It is also the shared cache a split of [`roles`](https://github.com/Wave-RF/WaveHouse/pull/622) needs: the refusal of `api` without `ingest`, or the reverse, over `cache.backend: local` now names `redis`, though every split is still refused while the queue is embedded. A malformed block — no address, an address without a port, an unknown mode, `db` other than `0` in cluster mode, an unreadable TLS file, a TLS key set while `tls.enabled` is off — refuses boot; an unreachable server, or one that rejects the password, does not: the process boots with the cache bypassed and keeps reconnecting, logging a rejected password at `ERROR` on every attempt, so a rotated secret cannot crash-loop every instance at once. `insecure_skip_verify`, and a `cache.redis.addrs` set while `cache.backend` is `local`, are logged at `WARN` at boot. The e2e suite now runs against a Redis testcontainer with `cache.backend: redis`, so the shared backend is exercised end to end, and the e2e coverage exclude for its files is gone; the unit and integration suites keep covering `local`. An integration test boots two instances over one Redis and one ClickHouse: a result one fills is a hit for the other, and a row ingested through one is served fresh by the other on its next query, well inside the stale entry's TTL; another pauses Redis and checks queries keep succeeding from ClickHouse, then turn back to hits. `deployments/compose/dependencies.yaml` gains an optional `redis` profile for local multi-instance work. The deployment guide gains a "Multiple instances and the shared cache" section: what each instance keeps to itself, what a reader on another instance can see and when, `maxmemory-policy`, and the metrics to alert on. - **A Redis-compatible shared cache backend** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row open a circuit breaker that skips the server until a probe succeeds, and deferred invalidations are retried until they land, collapsing to one tenant-wide bump per tenant past 100,000 keys. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. `cache.backend: redis` selects it (the entry above). Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression and server-stops-answering cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. - **One conformance suite for every `mq.Broker`, and a transient broker failure is a `503`** (`internal/mq/mqtest/` (new: the suite and the embedded broker's run of it), `internal/mq/{mq,embedded}.go` (+ tests), `internal/api/ingest.go` (+ tests), `.testcoverage.yml`, `docs/src/content/docs/{api,architecture}.md`, `AGENTS.md`): the first piece of the external-NATS workstream of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `mqtest.Run` states the `Broker` contract as behavior — round trips with names that need encoding, per-tenant order, `Nak` and `AckWait` redelivery, the trace context reaching `Subscribe`, dead-lettering that keeps the topic and leaves the original unacked, per-tenant dead-letter counts, replay bounds and isolation, and exactly one `failed` report when delivery ends underneath a consumer — through the interfaces alone, so the external backend runs the same cases, with `mqtest.Caps` for the four places where its semantics legitimately differ. The embedded broker passes it; writing it turned up that a durable deleted on several tenants' queues could report on `failed` more than once, which is fixed and pinned by a test that deletes it on one queue after another. It also turned up a replay that lost its connection mid-pull passing for a caught-up one when the pull ended in a timeout; that is an error now, as the `Replayer` contract says. The interface comments now allow a delivery unit that is a partition holding several tenants, a `CreateConsumer` that finds a durable rather than creating one, a `PurgeAcked` that leaves retention to the operator, and zero dead-letter counts where there is no per-tenant queue. A new sentinel, `mq.ErrUnavailable`, is a broker that cannot be reached or does not answer in time: the ingest handler answers it with `503` + `Retry-After: 5` rather than the `500` "publish failed" it would have been. Nothing returns it yet; the external backend of #613 will. - **Process roles: the API and the background workers can run in separate processes** (`internal/config/config.go` (+ `roles_test.go`, `defaults_test.go`), `internal/config/backends.go`, `internal/app/{app,wire}.go` (+ `roles_test.go`), `internal/api/router.go` (+ tests), `tests/integration/{setup,tenants}_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md}`, `AGENTS.md`), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The boot config gains `roles` (`WH_ROLES`, default `api,ingest,sweeper`, set in `defaults()` like every boot default, so an explicit `roles: []` refuses boot) and `instance_id` (`WH_INSTANCE_ID`, default `-<8 hex>`, fresh at every boot; logged at boot, and recorded as a lease holder once a shared `coord.backend` exists). A process wires only what its roles need: `api` runs the HTTP API with schema discovery, the token verifiers, the dedupe stores and the SSE hub (all per API process); `ingest` runs the ingest worker; `sweeper` runs the sweeper under its lease. A process without `api` serves an ops-only listener on `server.port` (the probes and their aliases, `/version`, the same-port metrics path, and `POST /v1/ops/settings/reload`, which takes the operator key alone); every other route answers 404, under `/v1/ops` once the operator key has passed. Boot refuses any split over the embedded MQ, which no other process can reach, and `api` without `ingest` (or the reverse) over a local cache, which the ingest worker's invalidations would never reach. Until a shared `mq.backend` exists, every process therefore runs every role, which is the default, so nothing changes for an existing deployment. `data_dir` is probed for Pebble only in a process running `api`. A `config.Config` built without `config.Load` must now name its roles (`config.AllRoles()` for all of them): `app.New` refuses an empty set. diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 2aba0526f..4c627ee7c 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -126,7 +126,7 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ - **config.go** — Loads *boot* configuration from a YAML file with environment variable overrides (using [cleanenv](https://github.com/ilyakaznacheev/cleanenv)); every key has a `WH_`-prefixed env var. Boot config is only what can't change under a running process — the implementation each layer runs on, the process's `roles`, resource sizing, listeners, observability exporters, the settings-directory path, and the secrets (`clickhouse.password`, `cache.redis.password`, `auth.jwt_secret`, `auth.operator_key`). Everything tenant-tunable lives in the settings directory (`settings/`). Both sources are strict: `Load` refuses to boot naming every YAML key the struct doesn't declare (`strict.go`) and every `WH_*` environment variable no field binds (`check.go`), so a tunable that moved to the settings directory can't be read, ignored, and believed. Boot is the validator for this half — there is no dry-run command. See [Configuration Reference](/configuration). - **check.go** — `rejectUnboundEnv` is the environment half of the strict loader: `unboundEnv` walks the struct's `env` tags (plus the two process-level names, `WH_CONFIG` and `WH_LOG_LEVEL`) against the environment; `CheckDataDir` probes `data_dir` — run by `main` right after `Load` when `Config.NeedsDataDir` says a selected backend keeps state there, so an unusable `data_dir` refuses boot before anything dials out. It refuses an empty value (reachable through `WH_DATA_DIR=`) outright rather than probing the working directory; a path that exists and is not a directory; a dangling symlink at `data_dir` or any component above it (the walk to the nearest existing ancestor uses `Lstat`, so a failed mount is not skipped over as "does not exist"); and a directory the process cannot write to — or, when it does not exist, an unwritable nearest ancestor — probed by creating and removing one temp file. A permission denial, on the probe or on reaching the path through a parent without search permission, carries the UID-65532 hint, since a bind mount owned by root is the typical cause. - **backends.go** — the `.backend` keys: one string type per layer (`MQBackend`, `CacheBackend`, `DedupeBackend`, `CoordBackend`), each with its list of the backends this build has, and a `validate` per layer block (`checkBackend`, run by `validateBackends` in `Validate`, between `validateRoles` and `validateTopology`), which refuses a value not on the list and names the ones that are. A backend's own settings go in a `.` sub-block that is its `validate`'s case to check. `Distributed` reports whether the MQ is shared with other processes, `NeedsDataDir` whether a selected backend keeps state under `data_dir`, and `Warnings` returns what a valid configuration is still likely to get wrong — the combinations that are correct for one replica only (a shared MQ over a local cache or Pebble dedupe), a `cache.redis` block that is not read, certificate verification turned off — which `app.New` logs at `WARN`. -- **cache_redis.go** — `CacheRedisConfig`, the `cache.redis` sub-block, and its checks: an address, each `host:port`, a known mode, a sentinel's master name, `db` 0 in cluster mode, positive timeouts and sizes, a `version_ttl` of at least 2 s, and a `compress_min_bytes` that is not negative (`0` never compresses); its defaults are in `defaults()` with the rest. `CacheRedisTLS.Config` builds the `tls.Config`, reading the files; `Validate` calls it so an unreadable file refuses boot, and `internal/app` calls it again to build the connection. A TLS key set while `tls.enabled` is off is an error rather than a plaintext connection. +- **cache_redis.go** — `CacheRedisConfig`, the `cache.redis` sub-block, and its checks: an address, each `host:port`, a known mode (`sentinel` is refused until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `db` 0 in cluster mode, positive timeouts and sizes, a `version_ttl` of at least 2 s, and a `compress_min_bytes` that is not negative (`0` never compresses); its defaults are in `defaults()` with the rest. `CacheRedisTLS.Config` builds the `tls.Config`, reading the files; `Validate` calls it so an unreadable file refuses boot, and `internal/app` calls it again to build the connection. A TLS key set while `tls.enabled` is off is an error rather than a plaintext connection. - **config.go**, roles — `roles` (`[]Role`: `api`, `ingest`, `sweeper`; `AllRoles` by default; `Has(Role)`) picks which components `internal/app` wires, and `instance_id` names the process (`-<8 hex>` when empty, resolved in `Load`; today only logged at boot, and a distributed coordinator will record it as a lease's holder). `validateRoles` refuses an empty list, an empty entry, an unknown or a repeated role; `validateTopology` refuses a role set the backends cannot serve: any split over the embedded MQ, and a process with exactly one of `api` and `ingest` over a local cache. `NeedsDataDir` counts Pebble only for a process running `api`, and `Warnings` is empty without `api`, since only that role opens a cache it reads or a dedupe store. - **strict.go** — `rejectUnknownKeys`, the YAML half: re-reads the file as a generic tree and walks it against the struct's `yaml` tags, listing every key the struct doesn't declare. cleanenv itself is lenient by design, which is exactly wrong for boot config once keys have moved to the settings directory. - **persistence.go** — `WarnIfFreshDataDir` logs the startup `WARN` when `data_dir` is missing or empty (on a redeploy, the sign that the volume didn't persist); `LogStorageInitError` attaches the UID-65532 `permissionHint` to a NATS or Pebble open failure that looks like a permission denial — the same hint string `CheckDataDir` uses. diff --git a/docs/src/content/docs/configuration.mdx b/docs/src/content/docs/configuration.mdx index 6425d6f77..700045fc7 100644 --- a/docs/src/content/docs/configuration.mdx +++ b/docs/src/content/docs/configuration.mdx @@ -150,12 +150,11 @@ The `redis` backend's settings, read only when `cache.backend` is `redis`. It is | YAML Key | Env Var | Default | Description | | --- | --- | ------- | ----------- | -| `cache.redis.addrs` | `WH_CACHE_REDIS_ADDRS` | *(empty)* | **Required** with `backend: redis`. `host:port` of the server; several are a cluster's seed nodes or the sentinels. Comma-separated in the env var. | -| `cache.redis.mode` | `WH_CACHE_REDIS_MODE` | `standalone` | `standalone`, `cluster` or `sentinel`. | -| `cache.redis.sentinel_master` | `WH_CACHE_REDIS_SENTINEL_MASTER` | *(empty)* | The master set name. Required with `mode: sentinel`. | +| `cache.redis.addrs` | `WH_CACHE_REDIS_ADDRS` | *(empty)* | **Required** with `backend: redis`. `host:port` of the server; several are a cluster's seed nodes. Comma-separated in the env var. | +| `cache.redis.mode` | `WH_CACHE_REDIS_MODE` | `standalone` | `standalone` or `cluster`. `sentinel` refuses boot until [#656](https://github.com/Wave-RF/WaveHouse/issues/656): the cache does not yet authenticate to the sentinels or refresh their topology, so it could not be trusted to follow a failover. | | `cache.redis.username` | `WH_CACHE_REDIS_USERNAME` | *(empty)* | ACL user. Empty uses the server's `default` user. | | `cache.redis.password` | `WH_CACHE_REDIS_PASSWORD` | *(empty)* | A secret: set it through the environment (or your secret store's env injection), not in a tracked `config.yaml`. | -| `cache.redis.db` | `WH_CACHE_REDIS_DB` | `0` | Database number (`SELECT`). Standalone and sentinel only: a cluster has only database `0`, and any other value refuses boot. | +| `cache.redis.db` | `WH_CACHE_REDIS_DB` | `0` | Database number (`SELECT`). Standalone only: a cluster has only database `0`, and any other value refuses boot. | | `cache.redis.tls.enabled` | `WH_CACHE_REDIS_TLS_ENABLED` | `false` | Connect over TLS, verifying the server against the system roots or `ca_file`. Any other `tls` key set while this is off refuses boot, rather than connecting in plaintext. | | `cache.redis.tls.ca_file` | `WH_CACHE_REDIS_TLS_CA_FILE` | *(empty)* | PEM file of the authorities to trust instead of the system roots. | | `cache.redis.tls.cert_file` | `WH_CACHE_REDIS_TLS_CERT_FILE` | *(empty)* | Client certificate (PEM) for mutual TLS. Set together with `key_file`. | @@ -268,8 +267,7 @@ cache: l1_max_cost: 67108864 # the local backend's size redis: # read only with backend: redis addrs: [] # required with backend: redis, e.g. ["redis:6379"] - mode: standalone # standalone | cluster | sentinel - sentinel_master: "" + mode: standalone # standalone | cluster username: "" password: "" # a secret: set WH_CACHE_REDIS_PASSWORD instead db: 0 @@ -348,7 +346,6 @@ WH_CACHE_L1_MAX_COST=67108864 # Read only with WH_CACHE_BACKEND=redis; WH_CACHE_REDIS_ADDRS is then required. WH_CACHE_REDIS_ADDRS= WH_CACHE_REDIS_MODE=standalone -WH_CACHE_REDIS_SENTINEL_MASTER= WH_CACHE_REDIS_USERNAME= WH_CACHE_REDIS_PASSWORD= WH_CACHE_REDIS_DB=0 diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index 2acf348c9..92c0cb95b 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -427,7 +427,7 @@ The folder name is the tenant id, and each folder is a complete settings directo Several WaveHouse instances can serve one ClickHouse behind a load balancer, but most of what each one holds is its own. The message queue is embedded, so an event is inserted by the instance that took its `POST /v1/ingest`, and reaches only that instance's SSE subscribers. The dedupe store is per instance too, so an id one instance has seen is new to another. -The query-result cache is the layer that can be shared today. With the default `cache.backend: local`, each instance caches in its own memory, and an insert invalidates only the cache of the instance that made it. Every other instance keeps serving its cached results for the rows before the insert until each entry's TTL runs out, between 10 s and 1 h depending on how long the query took. With [`cache.backend: redis`](/configuration#cache), every instance reads and fills one Redis-compatible server, and an insert on any instance invalidates the cached results of every instance. +The query-result cache is the layer that can be shared today. With the default `cache.backend: local`, each instance caches in its own memory, and an insert invalidates only the cache of the instance that made it. Every other instance keeps serving its cached results for the rows before the insert until each entry's TTL runs out, between 10 s and 1 h depending on how long the query took. With [`cache.backend: redis`](/configuration#cache), every instance reads and fills one Redis-compatible server, and an insert on any instance invalidates the cached results of every instance. The server is a standalone one or a Redis Cluster; Sentinel (`mode: sentinel`) refuses boot until [#656](https://github.com/Wave-RF/WaveHouse/issues/656), since the cache does not yet authenticate to the sentinels or refresh their topology. **What another instance can see.** Ingest is already asynchronous: `/v1/ingest` answers before the batch is inserted. Once the inserting instance's worker has written the batch to ClickHouse, it replaces the table's version token in Redis, and from then on a lookup on any instance misses and reads the new rows. The cache adds no delay of its own beyond that single write. The exceptions: diff --git a/internal/app/wire.go b/internal/app/wire.go index ea23606d0..2ba112a4c 100644 --- a/internal/app/wire.go +++ b/internal/app/wire.go @@ -667,7 +667,6 @@ func redisConfig(r config.CacheRedisConfig) (cache.RedisConfig, error) { return cache.RedisConfig{ Addrs: r.Addrs, Mode: r.Mode, - SentinelMaster: r.SentinelMaster, Username: r.Username, Password: r.Password, DB: r.DB, diff --git a/internal/config/cache_redis.go b/internal/config/cache_redis.go index 997d19a3a..83ff056ca 100644 --- a/internal/config/cache_redis.go +++ b/internal/config/cache_redis.go @@ -11,7 +11,8 @@ import ( "time" ) -// Redis deployment modes for cache.redis.mode. +// Redis deployment modes for cache.redis.mode. RedisSentinel is refused +// until the backend supports it (#656). const ( RedisStandalone = "standalone" RedisCluster = "cluster" @@ -22,15 +23,13 @@ const ( // (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB) shared by every process. // Read only when that backend is selected. type CacheRedisConfig struct { - // Addrs are host:port pairs: the server, or seeds for a cluster, or the - // sentinels. - Addrs []string `yaml:"addrs" env:"WH_CACHE_REDIS_ADDRS"` - Mode string `yaml:"mode" env:"WH_CACHE_REDIS_MODE"` - SentinelMaster string `yaml:"sentinel_master" env:"WH_CACHE_REDIS_SENTINEL_MASTER"` - Username string `yaml:"username" env:"WH_CACHE_REDIS_USERNAME"` - Password string `yaml:"password" env:"WH_CACHE_REDIS_PASSWORD"` - DB int `yaml:"db" env:"WH_CACHE_REDIS_DB"` - TLS CacheRedisTLS `yaml:"tls"` + // Addrs are host:port pairs: the server, or seeds for a cluster. + Addrs []string `yaml:"addrs" env:"WH_CACHE_REDIS_ADDRS"` + Mode string `yaml:"mode" env:"WH_CACHE_REDIS_MODE"` + Username string `yaml:"username" env:"WH_CACHE_REDIS_USERNAME"` + Password string `yaml:"password" env:"WH_CACHE_REDIS_PASSWORD"` + DB int `yaml:"db" env:"WH_CACHE_REDIS_DB"` + TLS CacheRedisTLS `yaml:"tls"` // KeyPrefix leads every key, so deployments can share one server. KeyPrefix string `yaml:"key_prefix" env:"WH_CACHE_REDIS_KEY_PREFIX"` Timeout time.Duration `yaml:"timeout" env:"WH_CACHE_REDIS_TIMEOUT"` @@ -61,7 +60,7 @@ func (r CacheRedisConfig) hasAddrs() bool { func (r CacheRedisConfig) validate() error { if !r.hasAddrs() { - return errors.New("cache.backend=redis needs cache.redis.addrs (WH_CACHE_REDIS_ADDRS): the server's host:port, or a cluster's seeds, or the sentinels") + return errors.New("cache.backend=redis needs cache.redis.addrs (WH_CACHE_REDIS_ADDRS): the server's host:port, or a cluster's seeds") } for _, a := range r.Addrs { if strings.TrimSpace(a) != a { @@ -72,12 +71,11 @@ func (r CacheRedisConfig) validate() error { } } switch r.Mode { - case RedisStandalone, RedisCluster, RedisSentinel: + case RedisStandalone, RedisCluster: + case RedisSentinel: + return fmt.Errorf("cache.redis.mode (WH_CACHE_REDIS_MODE) %q is not supported yet: the cache neither authenticates to the sentinels nor refreshes their topology (https://github.com/Wave-RF/WaveHouse/issues/656); valid: %s, %s", r.Mode, RedisStandalone, RedisCluster) default: - return fmt.Errorf("cache.redis.mode (WH_CACHE_REDIS_MODE) %q: valid: %s, %s, %s", r.Mode, RedisStandalone, RedisCluster, RedisSentinel) - } - if r.Mode == RedisSentinel && r.SentinelMaster == "" { - return errors.New("cache.redis.mode=sentinel needs cache.redis.sentinel_master (WH_CACHE_REDIS_SENTINEL_MASTER), the master set name") + return fmt.Errorf("cache.redis.mode (WH_CACHE_REDIS_MODE) %q: valid: %s, %s", r.Mode, RedisStandalone, RedisCluster) } if r.DB < 0 { return fmt.Errorf("cache.redis.db (WH_CACHE_REDIS_DB) %d is negative", r.DB) diff --git a/internal/config/cache_redis_test.go b/internal/config/cache_redis_test.go index 5389715a9..d9d3adab2 100644 --- a/internal/config/cache_redis_test.go +++ b/internal/config/cache_redis_test.go @@ -45,9 +45,8 @@ func TestLoad_CacheRedisFromEnv(t *testing.T) { caFile, certFile, keyFile := writeTestPKI(t, dir) for k, v := range map[string]string{ "WH_CACHE_BACKEND": "redis", - "WH_CACHE_REDIS_ADDRS": "s1:26379,s2:26379", - "WH_CACHE_REDIS_MODE": "sentinel", - "WH_CACHE_REDIS_SENTINEL_MASTER": "mymaster", + "WH_CACHE_REDIS_ADDRS": "r1:6379,r2:6379", + "WH_CACHE_REDIS_MODE": "standalone", "WH_CACHE_REDIS_USERNAME": "wavehouse", "WH_CACHE_REDIS_PASSWORD": "s3cret", "WH_CACHE_REDIS_DB": "2", @@ -69,7 +68,7 @@ func TestLoad_CacheRedisFromEnv(t *testing.T) { cfg, err := Load("nonexistent.yaml") require.NoError(t, err) assert.Equal(t, CacheRedisConfig{ - Addrs: []string{"s1:26379", "s2:26379"}, Mode: RedisSentinel, SentinelMaster: "mymaster", + Addrs: []string{"r1:6379", "r2:6379"}, Mode: RedisStandalone, Username: "wavehouse", Password: "s3cret", DB: 2, TLS: CacheRedisTLS{ Enabled: true, CAFile: caFile, CertFile: certFile, KeyFile: keyFile, ServerName: "redis.internal", @@ -142,6 +141,7 @@ cache: addr: r:6379 near_cache: max_cost: 1 + sentinel_master: mymaster tls: ca: /x memcached: @@ -149,7 +149,7 @@ cache: `), 0o600)) _, err := Load(path) require.Error(t, err) - assert.Contains(t, err.Error(), "cache.memcached, cache.redis.addr, cache.redis.near_cache, cache.redis.tls.ca") + assert.Contains(t, err.Error(), "cache.memcached, cache.redis.addr, cache.redis.near_cache, cache.redis.sentinel_master, cache.redis.tls.ca") } // The documented env file lists WH_CACHE_REDIS_ADDRS blank: that is no @@ -172,6 +172,7 @@ func TestUnboundEnv_KnowsTheCacheRedisVariables(t *testing.T) { "WH_CACHE_REDIS_VERSION_TTL=1h", "WH_CACHE_REDIS_COMPRESS_MIN_BYTES=0", })) assert.Equal(t, []string{"WH_CACHE_REDIS_ADDR"}, unboundEnv([]string{"WH_CACHE_REDIS_ADDR=r:6379"})) + assert.Equal(t, []string{"WH_CACHE_REDIS_SENTINEL_MASTER"}, unboundEnv([]string{"WH_CACHE_REDIS_SENTINEL_MASTER=m"}), "no sentinel mode until #656") } func TestValidate_CacheRedis(t *testing.T) { @@ -189,9 +190,9 @@ func TestValidate_CacheRedis(t *testing.T) { {"no addrs", func(r *CacheRedisConfig) { r.Addrs = nil }, "cache.backend=redis needs cache.redis.addrs (WH_CACHE_REDIS_ADDRS)"}, {"addr with space", func(r *CacheRedisConfig) { r.Addrs = []string{"a:6379", " b:6379"} }, "no spaces around an address"}, {"addr without port", func(r *CacheRedisConfig) { r.Addrs = []string{"redis"} }, `cache.redis.addrs (WH_CACHE_REDIS_ADDRS) "redis": want host:port`}, - {"mode", func(r *CacheRedisConfig) { r.Mode = "replica" }, `cache.redis.mode (WH_CACHE_REDIS_MODE) "replica": valid: standalone, cluster, sentinel`}, - {"sentinel without master", func(r *CacheRedisConfig) { r.Mode = RedisSentinel }, "needs cache.redis.sentinel_master"}, - {"sentinel", func(r *CacheRedisConfig) { r.Mode, r.SentinelMaster = RedisSentinel, "m" }, ""}, + {"mode", func(r *CacheRedisConfig) { r.Mode = "replica" }, `cache.redis.mode (WH_CACHE_REDIS_MODE) "replica": valid: standalone, cluster`}, + {"sentinel refused", func(r *CacheRedisConfig) { r.Mode = RedisSentinel }, `cache.redis.mode (WH_CACHE_REDIS_MODE) "sentinel" is not supported yet: the cache neither authenticates to the sentinels nor refreshes their topology (https://github.com/Wave-RF/WaveHouse/issues/656)`}, + {"cluster", func(r *CacheRedisConfig) { r.Mode = RedisCluster }, ""}, {"cluster db", func(r *CacheRedisConfig) { r.Mode, r.DB = RedisCluster, 1 }, "a Redis cluster has only database 0"}, {"standalone db", func(r *CacheRedisConfig) { r.DB = 3 }, ""}, {"negative db", func(r *CacheRedisConfig) { r.DB = -1 }, "is negative"}, From a7ef71fccd3d13f647d21e2a9c12ff7a42363692 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 05:03:55 -0400 Subject: [PATCH 19/45] fix(config): refuse a URL-style cache.redis address without echoing it WH_CACHE_REDIS_ADDRS=redis://default:s3cret@redis:6379 failed boot with an error that quoted the address, password included, twice: once in the key's message and once in net.SplitHostPort's. An address holding "://" or "@" is now refused by its position alone, pointing at the username, password and TLS settings that take those parts. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_017aS7rLrH1RKkUMem7X4ckd --- CHANGELOG.md | 2 +- docs/src/content/docs/architecture.md | 2 +- docs/src/content/docs/configuration.mdx | 2 +- internal/config/cache_redis.go | 7 ++++++- internal/config/cache_redis_test.go | 21 +++++++++++++++++++++ 5 files changed, 30 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ac09db1ae..471ccc173 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **`cache.backend: redis` shares the query cache across instances** (`internal/config/{cache_redis,backends,config}.go` (+ tests), `internal/app/wire.go` (+ tests), `tests/integration/shared_cache_test.go`, `scripts/orchestrator/main.go`, `tests/e2e/fixtures/config.yaml`, `.testcoverage.yml`, `deployments/compose/dependencies.yaml`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md,settings-directory.mdx,getting-started.md,pipes.mdx,api.md,development.md}`, `README.md`, `AGENTS.md`): PR E4 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config's `cache.backend` now takes `redis`, configured by a new `cache.redis` block (`WH_CACHE_REDIS_*`): `addrs` (required), `mode` (`standalone` or `cluster`; `sentinel` refuses boot until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `username`, `password` (a secret — set it through the environment), `db`, `tls.{enabled,ca_file,cert_file,key_file,server_name,insecure_skip_verify}`, `key_prefix` (`wh`), `timeout` (`100ms`), `dial_timeout` (`1s`), `max_value_bytes` (1 MiB), `compress_min_bytes` (`1024`; `0` never compresses) and `version_ttl` (`168h`). Every instance pointed at one server shares its cached results, and an insert on any instance invalidates every instance's. It is also the shared cache a split of [`roles`](https://github.com/Wave-RF/WaveHouse/pull/622) needs: the refusal of `api` without `ingest`, or the reverse, over `cache.backend: local` now names `redis`, though every split is still refused while the queue is embedded. A malformed block — no address, an address without a port, an unknown mode, `db` other than `0` in cluster mode, an unreadable TLS file, a TLS key set while `tls.enabled` is off — refuses boot; an unreachable server, or one that rejects the password, does not: the process boots with the cache bypassed and keeps reconnecting, logging a rejected password at `ERROR` on every attempt, so a rotated secret cannot crash-loop every instance at once. `insecure_skip_verify`, and a `cache.redis.addrs` set while `cache.backend` is `local`, are logged at `WARN` at boot. The e2e suite now runs against a Redis testcontainer with `cache.backend: redis`, so the shared backend is exercised end to end, and the e2e coverage exclude for its files is gone; the unit and integration suites keep covering `local`. An integration test boots two instances over one Redis and one ClickHouse: a result one fills is a hit for the other, and a row ingested through one is served fresh by the other on its next query, well inside the stale entry's TTL; another pauses Redis and checks queries keep succeeding from ClickHouse, then turn back to hits. `deployments/compose/dependencies.yaml` gains an optional `redis` profile for local multi-instance work. The deployment guide gains a "Multiple instances and the shared cache" section: what each instance keeps to itself, what a reader on another instance can see and when, `maxmemory-policy`, and the metrics to alert on. +- **`cache.backend: redis` shares the query cache across instances** (`internal/config/{cache_redis,backends,config}.go` (+ tests), `internal/app/wire.go` (+ tests), `tests/integration/shared_cache_test.go`, `scripts/orchestrator/main.go`, `tests/e2e/fixtures/config.yaml`, `.testcoverage.yml`, `deployments/compose/dependencies.yaml`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md,settings-directory.mdx,getting-started.md,pipes.mdx,api.md,development.md}`, `README.md`, `AGENTS.md`): PR E4 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config's `cache.backend` now takes `redis`, configured by a new `cache.redis` block (`WH_CACHE_REDIS_*`): `addrs` (required), `mode` (`standalone` or `cluster`; `sentinel` refuses boot until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `username`, `password` (a secret — set it through the environment), `db`, `tls.{enabled,ca_file,cert_file,key_file,server_name,insecure_skip_verify}`, `key_prefix` (`wh`), `timeout` (`100ms`), `dial_timeout` (`1s`), `max_value_bytes` (1 MiB), `compress_min_bytes` (`1024`; `0` never compresses) and `version_ttl` (`168h`). Every instance pointed at one server shares its cached results, and an insert on any instance invalidates every instance's. It is also the shared cache a split of [`roles`](https://github.com/Wave-RF/WaveHouse/pull/622) needs: the refusal of `api` without `ingest`, or the reverse, over `cache.backend: local` now names `redis`, though every split is still refused while the queue is embedded. A malformed block — no address, an address without a port, a URL-style address (refused without repeating it, since it may carry a password), an unknown mode, `db` other than `0` in cluster mode, an unreadable TLS file, a TLS key set while `tls.enabled` is off — refuses boot; an unreachable server, or one that rejects the password, does not: the process boots with the cache bypassed and keeps reconnecting, logging a rejected password at `ERROR` on every attempt, so a rotated secret cannot crash-loop every instance at once. `insecure_skip_verify`, and a `cache.redis.addrs` set while `cache.backend` is `local`, are logged at `WARN` at boot. The e2e suite now runs against a Redis testcontainer with `cache.backend: redis`, so the shared backend is exercised end to end, and the e2e coverage exclude for its files is gone; the unit and integration suites keep covering `local`. An integration test boots two instances over one Redis and one ClickHouse: a result one fills is a hit for the other, and a row ingested through one is served fresh by the other on its next query, well inside the stale entry's TTL; another pauses Redis and checks queries keep succeeding from ClickHouse, then turn back to hits. `deployments/compose/dependencies.yaml` gains an optional `redis` profile for local multi-instance work. The deployment guide gains a "Multiple instances and the shared cache" section: what each instance keeps to itself, what a reader on another instance can see and when, `maxmemory-policy`, and the metrics to alert on. - **A Redis-compatible shared cache backend** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row open a circuit breaker that skips the server until a probe succeeds, and deferred invalidations are retried until they land, collapsing to one tenant-wide bump per tenant past 100,000 keys. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. `cache.backend: redis` selects it (the entry above). Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression and server-stops-answering cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. - **One conformance suite for every `mq.Broker`, and a transient broker failure is a `503`** (`internal/mq/mqtest/` (new: the suite and the embedded broker's run of it), `internal/mq/{mq,embedded}.go` (+ tests), `internal/api/ingest.go` (+ tests), `.testcoverage.yml`, `docs/src/content/docs/{api,architecture}.md`, `AGENTS.md`): the first piece of the external-NATS workstream of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `mqtest.Run` states the `Broker` contract as behavior — round trips with names that need encoding, per-tenant order, `Nak` and `AckWait` redelivery, the trace context reaching `Subscribe`, dead-lettering that keeps the topic and leaves the original unacked, per-tenant dead-letter counts, replay bounds and isolation, and exactly one `failed` report when delivery ends underneath a consumer — through the interfaces alone, so the external backend runs the same cases, with `mqtest.Caps` for the four places where its semantics legitimately differ. The embedded broker passes it; writing it turned up that a durable deleted on several tenants' queues could report on `failed` more than once, which is fixed and pinned by a test that deletes it on one queue after another. It also turned up a replay that lost its connection mid-pull passing for a caught-up one when the pull ended in a timeout; that is an error now, as the `Replayer` contract says. The interface comments now allow a delivery unit that is a partition holding several tenants, a `CreateConsumer` that finds a durable rather than creating one, a `PurgeAcked` that leaves retention to the operator, and zero dead-letter counts where there is no per-tenant queue. A new sentinel, `mq.ErrUnavailable`, is a broker that cannot be reached or does not answer in time: the ingest handler answers it with `503` + `Retry-After: 5` rather than the `500` "publish failed" it would have been. Nothing returns it yet; the external backend of #613 will. - **Process roles: the API and the background workers can run in separate processes** (`internal/config/config.go` (+ `roles_test.go`, `defaults_test.go`), `internal/config/backends.go`, `internal/app/{app,wire}.go` (+ `roles_test.go`), `internal/api/router.go` (+ tests), `tests/integration/{setup,tenants}_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md}`, `AGENTS.md`), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The boot config gains `roles` (`WH_ROLES`, default `api,ingest,sweeper`, set in `defaults()` like every boot default, so an explicit `roles: []` refuses boot) and `instance_id` (`WH_INSTANCE_ID`, default `-<8 hex>`, fresh at every boot; logged at boot, and recorded as a lease holder once a shared `coord.backend` exists). A process wires only what its roles need: `api` runs the HTTP API with schema discovery, the token verifiers, the dedupe stores and the SSE hub (all per API process); `ingest` runs the ingest worker; `sweeper` runs the sweeper under its lease. A process without `api` serves an ops-only listener on `server.port` (the probes and their aliases, `/version`, the same-port metrics path, and `POST /v1/ops/settings/reload`, which takes the operator key alone); every other route answers 404, under `/v1/ops` once the operator key has passed. Boot refuses any split over the embedded MQ, which no other process can reach, and `api` without `ingest` (or the reverse) over a local cache, which the ingest worker's invalidations would never reach. Until a shared `mq.backend` exists, every process therefore runs every role, which is the default, so nothing changes for an existing deployment. `data_dir` is probed for Pebble only in a process running `api`. A `config.Config` built without `config.Load` must now name its roles (`config.AllRoles()` for all of them): `app.New` refuses an empty set. diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 4c627ee7c..c0f83326f 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -126,7 +126,7 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ - **config.go** — Loads *boot* configuration from a YAML file with environment variable overrides (using [cleanenv](https://github.com/ilyakaznacheev/cleanenv)); every key has a `WH_`-prefixed env var. Boot config is only what can't change under a running process — the implementation each layer runs on, the process's `roles`, resource sizing, listeners, observability exporters, the settings-directory path, and the secrets (`clickhouse.password`, `cache.redis.password`, `auth.jwt_secret`, `auth.operator_key`). Everything tenant-tunable lives in the settings directory (`settings/`). Both sources are strict: `Load` refuses to boot naming every YAML key the struct doesn't declare (`strict.go`) and every `WH_*` environment variable no field binds (`check.go`), so a tunable that moved to the settings directory can't be read, ignored, and believed. Boot is the validator for this half — there is no dry-run command. See [Configuration Reference](/configuration). - **check.go** — `rejectUnboundEnv` is the environment half of the strict loader: `unboundEnv` walks the struct's `env` tags (plus the two process-level names, `WH_CONFIG` and `WH_LOG_LEVEL`) against the environment; `CheckDataDir` probes `data_dir` — run by `main` right after `Load` when `Config.NeedsDataDir` says a selected backend keeps state there, so an unusable `data_dir` refuses boot before anything dials out. It refuses an empty value (reachable through `WH_DATA_DIR=`) outright rather than probing the working directory; a path that exists and is not a directory; a dangling symlink at `data_dir` or any component above it (the walk to the nearest existing ancestor uses `Lstat`, so a failed mount is not skipped over as "does not exist"); and a directory the process cannot write to — or, when it does not exist, an unwritable nearest ancestor — probed by creating and removing one temp file. A permission denial, on the probe or on reaching the path through a parent without search permission, carries the UID-65532 hint, since a bind mount owned by root is the typical cause. - **backends.go** — the `.backend` keys: one string type per layer (`MQBackend`, `CacheBackend`, `DedupeBackend`, `CoordBackend`), each with its list of the backends this build has, and a `validate` per layer block (`checkBackend`, run by `validateBackends` in `Validate`, between `validateRoles` and `validateTopology`), which refuses a value not on the list and names the ones that are. A backend's own settings go in a `.` sub-block that is its `validate`'s case to check. `Distributed` reports whether the MQ is shared with other processes, `NeedsDataDir` whether a selected backend keeps state under `data_dir`, and `Warnings` returns what a valid configuration is still likely to get wrong — the combinations that are correct for one replica only (a shared MQ over a local cache or Pebble dedupe), a `cache.redis` block that is not read, certificate verification turned off — which `app.New` logs at `WARN`. -- **cache_redis.go** — `CacheRedisConfig`, the `cache.redis` sub-block, and its checks: an address, each `host:port`, a known mode (`sentinel` is refused until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `db` 0 in cluster mode, positive timeouts and sizes, a `version_ttl` of at least 2 s, and a `compress_min_bytes` that is not negative (`0` never compresses); its defaults are in `defaults()` with the rest. `CacheRedisTLS.Config` builds the `tls.Config`, reading the files; `Validate` calls it so an unreadable file refuses boot, and `internal/app` calls it again to build the connection. A TLS key set while `tls.enabled` is off is an error rather than a plaintext connection. +- **cache_redis.go** — `CacheRedisConfig`, the `cache.redis` sub-block, and its checks: an address, each `host:port` (a URL or `user:password@` form refused without repeating it, since it may hold a password), a known mode (`sentinel` is refused until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `db` 0 in cluster mode, positive timeouts and sizes, a `version_ttl` of at least 2 s, and a `compress_min_bytes` that is not negative (`0` never compresses); its defaults are in `defaults()` with the rest. `CacheRedisTLS.Config` builds the `tls.Config`, reading the files; `Validate` calls it so an unreadable file refuses boot, and `internal/app` calls it again to build the connection. A TLS key set while `tls.enabled` is off is an error rather than a plaintext connection. - **config.go**, roles — `roles` (`[]Role`: `api`, `ingest`, `sweeper`; `AllRoles` by default; `Has(Role)`) picks which components `internal/app` wires, and `instance_id` names the process (`-<8 hex>` when empty, resolved in `Load`; today only logged at boot, and a distributed coordinator will record it as a lease's holder). `validateRoles` refuses an empty list, an empty entry, an unknown or a repeated role; `validateTopology` refuses a role set the backends cannot serve: any split over the embedded MQ, and a process with exactly one of `api` and `ingest` over a local cache. `NeedsDataDir` counts Pebble only for a process running `api`, and `Warnings` is empty without `api`, since only that role opens a cache it reads or a dedupe store. - **strict.go** — `rejectUnknownKeys`, the YAML half: re-reads the file as a generic tree and walks it against the struct's `yaml` tags, listing every key the struct doesn't declare. cleanenv itself is lenient by design, which is exactly wrong for boot config once keys have moved to the settings directory. - **persistence.go** — `WarnIfFreshDataDir` logs the startup `WARN` when `data_dir` is missing or empty (on a redeploy, the sign that the volume didn't persist); `LogStorageInitError` attaches the UID-65532 `permissionHint` to a NATS or Pebble open failure that looks like a permission denial — the same hint string `CheckDataDir` uses. diff --git a/docs/src/content/docs/configuration.mdx b/docs/src/content/docs/configuration.mdx index 700045fc7..f15788f37 100644 --- a/docs/src/content/docs/configuration.mdx +++ b/docs/src/content/docs/configuration.mdx @@ -150,7 +150,7 @@ The `redis` backend's settings, read only when `cache.backend` is `redis`. It is | YAML Key | Env Var | Default | Description | | --- | --- | ------- | ----------- | -| `cache.redis.addrs` | `WH_CACHE_REDIS_ADDRS` | *(empty)* | **Required** with `backend: redis`. `host:port` of the server; several are a cluster's seed nodes. Comma-separated in the env var. | +| `cache.redis.addrs` | `WH_CACHE_REDIS_ADDRS` | *(empty)* | **Required** with `backend: redis`. `host:port` of the server; several are a cluster's seed nodes. Comma-separated in the env var. Not a URL: `redis://user:password@host:port` refuses boot, without repeating the value, so set the credentials through `username` and `password`, and `rediss://` through `tls.enabled`. | | `cache.redis.mode` | `WH_CACHE_REDIS_MODE` | `standalone` | `standalone` or `cluster`. `sentinel` refuses boot until [#656](https://github.com/Wave-RF/WaveHouse/issues/656): the cache does not yet authenticate to the sentinels or refresh their topology, so it could not be trusted to follow a failover. | | `cache.redis.username` | `WH_CACHE_REDIS_USERNAME` | *(empty)* | ACL user. Empty uses the server's `default` user. | | `cache.redis.password` | `WH_CACHE_REDIS_PASSWORD` | *(empty)* | A secret: set it through the environment (or your secret store's env injection), not in a tracked `config.yaml`. | diff --git a/internal/config/cache_redis.go b/internal/config/cache_redis.go index 83ff056ca..0e41b0ff1 100644 --- a/internal/config/cache_redis.go +++ b/internal/config/cache_redis.go @@ -62,7 +62,12 @@ func (r CacheRedisConfig) validate() error { if !r.hasAddrs() { return errors.New("cache.backend=redis needs cache.redis.addrs (WH_CACHE_REDIS_ADDRS): the server's host:port, or a cluster's seeds") } - for _, a := range r.Addrs { + for i, a := range r.Addrs { + // A redis:// URL, or user:pass@host, may carry a password: refuse it + // without echoing it into the boot error and the logs. + if strings.Contains(a, "://") || strings.Contains(a, "@") { + return fmt.Errorf("cache.redis.addrs (WH_CACHE_REDIS_ADDRS) entry %d is a URL or holds credentials (not echoed): give host:port, and set the user and password with WH_CACHE_REDIS_USERNAME and WH_CACHE_REDIS_PASSWORD, and TLS (rediss://) with WH_CACHE_REDIS_TLS_ENABLED", i+1) + } if strings.TrimSpace(a) != a { return fmt.Errorf("cache.redis.addrs (WH_CACHE_REDIS_ADDRS) %q: no spaces around an address", a) } diff --git a/internal/config/cache_redis_test.go b/internal/config/cache_redis_test.go index d9d3adab2..21ec15fcf 100644 --- a/internal/config/cache_redis_test.go +++ b/internal/config/cache_redis_test.go @@ -165,6 +165,27 @@ func TestLoad_CacheRedisBlankAddrs(t *testing.T) { require.ErrorContains(t, err, "cache.backend=redis needs cache.redis.addrs") } +// A URL-style address carries its password, and a boot error reaches the +// logs: the refusal names the entry, never its value. +func TestLoad_CacheRedisRefusesURLAddrsWithoutEchoingThem(t *testing.T) { + for _, addr := range []string{ + "redis://default:s3cret@redis:6379", + "rediss://default:s3cret@redis:6380", + "default:s3cret@redis:6379", + "redis://redis:6379", + } { + t.Run(addr, func(t *testing.T) { + t.Setenv("WH_CACHE_BACKEND", "redis") + t.Setenv("WH_CACHE_REDIS_ADDRS", "ok:6379,"+addr) + _, err := Load("nonexistent.yaml") + require.ErrorContains(t, err, "cache.redis.addrs (WH_CACHE_REDIS_ADDRS) entry 2 is a URL or holds credentials") + assert.Contains(t, err.Error(), "WH_CACHE_REDIS_USERNAME and WH_CACHE_REDIS_PASSWORD") + assert.NotContains(t, err.Error(), "s3cret") + assert.NotContains(t, err.Error(), addr) + }) + } +} + func TestUnboundEnv_KnowsTheCacheRedisVariables(t *testing.T) { t.Parallel() assert.Empty(t, unboundEnv([]string{ From de27b600f670ec6ae53777836fd3a1f85951eb1c Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 05:04:55 -0400 Subject: [PATCH 20/45] fix(config): cap cache.redis.dial_timeout at 2s A dial is a TCP connect and a handshake, each bounded by dial_timeout. NewRedis dials once synchronously, so boot waits out one, and the cache's Close waits out a reconnect dial in flight: rueidis.NewClient takes no context. An unbounded dial_timeout could therefore push the cache's close past the 5s release budget and leave the queue, dedupe and ClickHouse pools unreleased. At 2s the wait is at most 4s; the default (1s) is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_017aS7rLrH1RKkUMem7X4ckd --- CHANGELOG.md | 2 +- docs/src/content/docs/architecture.md | 2 +- docs/src/content/docs/configuration.mdx | 2 +- internal/config/cache_redis.go | 9 +++++++++ internal/config/cache_redis_test.go | 6 ++++-- 5 files changed, 16 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 471ccc173..1454d47ae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **`cache.backend: redis` shares the query cache across instances** (`internal/config/{cache_redis,backends,config}.go` (+ tests), `internal/app/wire.go` (+ tests), `tests/integration/shared_cache_test.go`, `scripts/orchestrator/main.go`, `tests/e2e/fixtures/config.yaml`, `.testcoverage.yml`, `deployments/compose/dependencies.yaml`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md,settings-directory.mdx,getting-started.md,pipes.mdx,api.md,development.md}`, `README.md`, `AGENTS.md`): PR E4 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config's `cache.backend` now takes `redis`, configured by a new `cache.redis` block (`WH_CACHE_REDIS_*`): `addrs` (required), `mode` (`standalone` or `cluster`; `sentinel` refuses boot until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `username`, `password` (a secret — set it through the environment), `db`, `tls.{enabled,ca_file,cert_file,key_file,server_name,insecure_skip_verify}`, `key_prefix` (`wh`), `timeout` (`100ms`), `dial_timeout` (`1s`), `max_value_bytes` (1 MiB), `compress_min_bytes` (`1024`; `0` never compresses) and `version_ttl` (`168h`). Every instance pointed at one server shares its cached results, and an insert on any instance invalidates every instance's. It is also the shared cache a split of [`roles`](https://github.com/Wave-RF/WaveHouse/pull/622) needs: the refusal of `api` without `ingest`, or the reverse, over `cache.backend: local` now names `redis`, though every split is still refused while the queue is embedded. A malformed block — no address, an address without a port, a URL-style address (refused without repeating it, since it may carry a password), an unknown mode, `db` other than `0` in cluster mode, an unreadable TLS file, a TLS key set while `tls.enabled` is off — refuses boot; an unreachable server, or one that rejects the password, does not: the process boots with the cache bypassed and keeps reconnecting, logging a rejected password at `ERROR` on every attempt, so a rotated secret cannot crash-loop every instance at once. `insecure_skip_verify`, and a `cache.redis.addrs` set while `cache.backend` is `local`, are logged at `WARN` at boot. The e2e suite now runs against a Redis testcontainer with `cache.backend: redis`, so the shared backend is exercised end to end, and the e2e coverage exclude for its files is gone; the unit and integration suites keep covering `local`. An integration test boots two instances over one Redis and one ClickHouse: a result one fills is a hit for the other, and a row ingested through one is served fresh by the other on its next query, well inside the stale entry's TTL; another pauses Redis and checks queries keep succeeding from ClickHouse, then turn back to hits. `deployments/compose/dependencies.yaml` gains an optional `redis` profile for local multi-instance work. The deployment guide gains a "Multiple instances and the shared cache" section: what each instance keeps to itself, what a reader on another instance can see and when, `maxmemory-policy`, and the metrics to alert on. +- **`cache.backend: redis` shares the query cache across instances** (`internal/config/{cache_redis,backends,config}.go` (+ tests), `internal/app/wire.go` (+ tests), `tests/integration/shared_cache_test.go`, `scripts/orchestrator/main.go`, `tests/e2e/fixtures/config.yaml`, `.testcoverage.yml`, `deployments/compose/dependencies.yaml`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md,settings-directory.mdx,getting-started.md,pipes.mdx,api.md,development.md}`, `README.md`, `AGENTS.md`): PR E4 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config's `cache.backend` now takes `redis`, configured by a new `cache.redis` block (`WH_CACHE_REDIS_*`): `addrs` (required), `mode` (`standalone` or `cluster`; `sentinel` refuses boot until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `username`, `password` (a secret — set it through the environment), `db`, `tls.{enabled,ca_file,cert_file,key_file,server_name,insecure_skip_verify}`, `key_prefix` (`wh`), `timeout` (`100ms`), `dial_timeout` (`1s`, at most `2s`: boot and shutdown each wait out a dial), `max_value_bytes` (1 MiB), `compress_min_bytes` (`1024`; `0` never compresses) and `version_ttl` (`168h`). Every instance pointed at one server shares its cached results, and an insert on any instance invalidates every instance's. It is also the shared cache a split of [`roles`](https://github.com/Wave-RF/WaveHouse/pull/622) needs: the refusal of `api` without `ingest`, or the reverse, over `cache.backend: local` now names `redis`, though every split is still refused while the queue is embedded. A malformed block — no address, an address without a port, a URL-style address (refused without repeating it, since it may carry a password), an unknown mode, `db` other than `0` in cluster mode, an unreadable TLS file, a TLS key set while `tls.enabled` is off — refuses boot; an unreachable server, or one that rejects the password, does not: the process boots with the cache bypassed and keeps reconnecting, logging a rejected password at `ERROR` on every attempt, so a rotated secret cannot crash-loop every instance at once. `insecure_skip_verify`, and a `cache.redis.addrs` set while `cache.backend` is `local`, are logged at `WARN` at boot. The e2e suite now runs against a Redis testcontainer with `cache.backend: redis`, so the shared backend is exercised end to end, and the e2e coverage exclude for its files is gone; the unit and integration suites keep covering `local`. An integration test boots two instances over one Redis and one ClickHouse: a result one fills is a hit for the other, and a row ingested through one is served fresh by the other on its next query, well inside the stale entry's TTL; another pauses Redis and checks queries keep succeeding from ClickHouse, then turn back to hits. `deployments/compose/dependencies.yaml` gains an optional `redis` profile for local multi-instance work. The deployment guide gains a "Multiple instances and the shared cache" section: what each instance keeps to itself, what a reader on another instance can see and when, `maxmemory-policy`, and the metrics to alert on. - **A Redis-compatible shared cache backend** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row open a circuit breaker that skips the server until a probe succeeds, and deferred invalidations are retried until they land, collapsing to one tenant-wide bump per tenant past 100,000 keys. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. `cache.backend: redis` selects it (the entry above). Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression and server-stops-answering cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. - **One conformance suite for every `mq.Broker`, and a transient broker failure is a `503`** (`internal/mq/mqtest/` (new: the suite and the embedded broker's run of it), `internal/mq/{mq,embedded}.go` (+ tests), `internal/api/ingest.go` (+ tests), `.testcoverage.yml`, `docs/src/content/docs/{api,architecture}.md`, `AGENTS.md`): the first piece of the external-NATS workstream of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `mqtest.Run` states the `Broker` contract as behavior — round trips with names that need encoding, per-tenant order, `Nak` and `AckWait` redelivery, the trace context reaching `Subscribe`, dead-lettering that keeps the topic and leaves the original unacked, per-tenant dead-letter counts, replay bounds and isolation, and exactly one `failed` report when delivery ends underneath a consumer — through the interfaces alone, so the external backend runs the same cases, with `mqtest.Caps` for the four places where its semantics legitimately differ. The embedded broker passes it; writing it turned up that a durable deleted on several tenants' queues could report on `failed` more than once, which is fixed and pinned by a test that deletes it on one queue after another. It also turned up a replay that lost its connection mid-pull passing for a caught-up one when the pull ended in a timeout; that is an error now, as the `Replayer` contract says. The interface comments now allow a delivery unit that is a partition holding several tenants, a `CreateConsumer` that finds a durable rather than creating one, a `PurgeAcked` that leaves retention to the operator, and zero dead-letter counts where there is no per-tenant queue. A new sentinel, `mq.ErrUnavailable`, is a broker that cannot be reached or does not answer in time: the ingest handler answers it with `503` + `Retry-After: 5` rather than the `500` "publish failed" it would have been. Nothing returns it yet; the external backend of #613 will. - **Process roles: the API and the background workers can run in separate processes** (`internal/config/config.go` (+ `roles_test.go`, `defaults_test.go`), `internal/config/backends.go`, `internal/app/{app,wire}.go` (+ `roles_test.go`), `internal/api/router.go` (+ tests), `tests/integration/{setup,tenants}_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md}`, `AGENTS.md`), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The boot config gains `roles` (`WH_ROLES`, default `api,ingest,sweeper`, set in `defaults()` like every boot default, so an explicit `roles: []` refuses boot) and `instance_id` (`WH_INSTANCE_ID`, default `-<8 hex>`, fresh at every boot; logged at boot, and recorded as a lease holder once a shared `coord.backend` exists). A process wires only what its roles need: `api` runs the HTTP API with schema discovery, the token verifiers, the dedupe stores and the SSE hub (all per API process); `ingest` runs the ingest worker; `sweeper` runs the sweeper under its lease. A process without `api` serves an ops-only listener on `server.port` (the probes and their aliases, `/version`, the same-port metrics path, and `POST /v1/ops/settings/reload`, which takes the operator key alone); every other route answers 404, under `/v1/ops` once the operator key has passed. Boot refuses any split over the embedded MQ, which no other process can reach, and `api` without `ingest` (or the reverse) over a local cache, which the ingest worker's invalidations would never reach. Until a shared `mq.backend` exists, every process therefore runs every role, which is the default, so nothing changes for an existing deployment. `data_dir` is probed for Pebble only in a process running `api`. A `config.Config` built without `config.Load` must now name its roles (`config.AllRoles()` for all of them): `app.New` refuses an empty set. diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index c0f83326f..cd44c8d1b 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -126,7 +126,7 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ - **config.go** — Loads *boot* configuration from a YAML file with environment variable overrides (using [cleanenv](https://github.com/ilyakaznacheev/cleanenv)); every key has a `WH_`-prefixed env var. Boot config is only what can't change under a running process — the implementation each layer runs on, the process's `roles`, resource sizing, listeners, observability exporters, the settings-directory path, and the secrets (`clickhouse.password`, `cache.redis.password`, `auth.jwt_secret`, `auth.operator_key`). Everything tenant-tunable lives in the settings directory (`settings/`). Both sources are strict: `Load` refuses to boot naming every YAML key the struct doesn't declare (`strict.go`) and every `WH_*` environment variable no field binds (`check.go`), so a tunable that moved to the settings directory can't be read, ignored, and believed. Boot is the validator for this half — there is no dry-run command. See [Configuration Reference](/configuration). - **check.go** — `rejectUnboundEnv` is the environment half of the strict loader: `unboundEnv` walks the struct's `env` tags (plus the two process-level names, `WH_CONFIG` and `WH_LOG_LEVEL`) against the environment; `CheckDataDir` probes `data_dir` — run by `main` right after `Load` when `Config.NeedsDataDir` says a selected backend keeps state there, so an unusable `data_dir` refuses boot before anything dials out. It refuses an empty value (reachable through `WH_DATA_DIR=`) outright rather than probing the working directory; a path that exists and is not a directory; a dangling symlink at `data_dir` or any component above it (the walk to the nearest existing ancestor uses `Lstat`, so a failed mount is not skipped over as "does not exist"); and a directory the process cannot write to — or, when it does not exist, an unwritable nearest ancestor — probed by creating and removing one temp file. A permission denial, on the probe or on reaching the path through a parent without search permission, carries the UID-65532 hint, since a bind mount owned by root is the typical cause. - **backends.go** — the `.backend` keys: one string type per layer (`MQBackend`, `CacheBackend`, `DedupeBackend`, `CoordBackend`), each with its list of the backends this build has, and a `validate` per layer block (`checkBackend`, run by `validateBackends` in `Validate`, between `validateRoles` and `validateTopology`), which refuses a value not on the list and names the ones that are. A backend's own settings go in a `.` sub-block that is its `validate`'s case to check. `Distributed` reports whether the MQ is shared with other processes, `NeedsDataDir` whether a selected backend keeps state under `data_dir`, and `Warnings` returns what a valid configuration is still likely to get wrong — the combinations that are correct for one replica only (a shared MQ over a local cache or Pebble dedupe), a `cache.redis` block that is not read, certificate verification turned off — which `app.New` logs at `WARN`. -- **cache_redis.go** — `CacheRedisConfig`, the `cache.redis` sub-block, and its checks: an address, each `host:port` (a URL or `user:password@` form refused without repeating it, since it may hold a password), a known mode (`sentinel` is refused until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `db` 0 in cluster mode, positive timeouts and sizes, a `version_ttl` of at least 2 s, and a `compress_min_bytes` that is not negative (`0` never compresses); its defaults are in `defaults()` with the rest. `CacheRedisTLS.Config` builds the `tls.Config`, reading the files; `Validate` calls it so an unreadable file refuses boot, and `internal/app` calls it again to build the connection. A TLS key set while `tls.enabled` is off is an error rather than a plaintext connection. +- **cache_redis.go** — `CacheRedisConfig`, the `cache.redis` sub-block, and its checks: an address, each `host:port` (a URL or `user:password@` form refused without repeating it, since it may hold a password), a known mode (`sentinel` is refused until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `db` 0 in cluster mode, positive timeouts and sizes, a `dial_timeout` of at most 2 s (boot and `Close` each wait out a dial, a connect and a handshake bounded by it), a `version_ttl` of at least 2 s, and a `compress_min_bytes` that is not negative (`0` never compresses); its defaults are in `defaults()` with the rest. `CacheRedisTLS.Config` builds the `tls.Config`, reading the files; `Validate` calls it so an unreadable file refuses boot, and `internal/app` calls it again to build the connection. A TLS key set while `tls.enabled` is off is an error rather than a plaintext connection. - **config.go**, roles — `roles` (`[]Role`: `api`, `ingest`, `sweeper`; `AllRoles` by default; `Has(Role)`) picks which components `internal/app` wires, and `instance_id` names the process (`-<8 hex>` when empty, resolved in `Load`; today only logged at boot, and a distributed coordinator will record it as a lease's holder). `validateRoles` refuses an empty list, an empty entry, an unknown or a repeated role; `validateTopology` refuses a role set the backends cannot serve: any split over the embedded MQ, and a process with exactly one of `api` and `ingest` over a local cache. `NeedsDataDir` counts Pebble only for a process running `api`, and `Warnings` is empty without `api`, since only that role opens a cache it reads or a dedupe store. - **strict.go** — `rejectUnknownKeys`, the YAML half: re-reads the file as a generic tree and walks it against the struct's `yaml` tags, listing every key the struct doesn't declare. cleanenv itself is lenient by design, which is exactly wrong for boot config once keys have moved to the settings directory. - **persistence.go** — `WarnIfFreshDataDir` logs the startup `WARN` when `data_dir` is missing or empty (on a redeploy, the sign that the volume didn't persist); `LogStorageInitError` attaches the UID-65532 `permissionHint` to a NATS or Pebble open failure that looks like a permission denial — the same hint string `CheckDataDir` uses. diff --git a/docs/src/content/docs/configuration.mdx b/docs/src/content/docs/configuration.mdx index f15788f37..e00fdaec0 100644 --- a/docs/src/content/docs/configuration.mdx +++ b/docs/src/content/docs/configuration.mdx @@ -163,7 +163,7 @@ The `redis` backend's settings, read only when `cache.backend` is `redis`. It is | `cache.redis.tls.insecure_skip_verify` | `WH_CACHE_REDIS_TLS_INSECURE_SKIP_VERIFY` | `false` | Accept any server certificate. Logged at `WARN` at boot: whoever can intercept the connection can read and replace cached results. | | `cache.redis.key_prefix` | `WH_CACHE_REDIS_KEY_PREFIX` | `wh` | Leads every key, so several deployments can share one server, provided you trust each as much as the others: any of them can overwrite what the rest serve. No `{` or `}`. | | `cache.redis.timeout` | `WH_CACHE_REDIS_TIMEOUT` | `100ms` | Per operation. A lookup or fill that takes longer is a miss or a skipped fill, never a failed query. | -| `cache.redis.dial_timeout` | `WH_CACHE_REDIS_DIAL_TIMEOUT` | `1s` | Per connection attempt. | +| `cache.redis.dial_timeout` | `WH_CACHE_REDIS_DIAL_TIMEOUT` | `1s` | Per connection attempt, at most `2s`. An attempt is a connect and a handshake, each bounded by this, and both boot and shutdown wait out one in flight, so the cap keeps a stop inside the release's fixed 5s budget (see [Stopping](/deployment#stopping)). | | `cache.redis.max_value_bytes` | `WH_CACHE_REDIS_MAX_VALUE_BYTES` | `1048576` | Largest result stored, after compression (1 MiB). A larger one is returned to the caller but not cached. | | `cache.redis.compress_min_bytes` | `WH_CACHE_REDIS_COMPRESS_MIN_BYTES` | `1024` | Results at least this large are zstd-compressed when that makes them smaller. `0` never compresses. | | `cache.redis.version_ttl` | `WH_CACHE_REDIS_VERSION_TTL` | `168h` | How long a table's or tenant's version token outlives its last write, so the tokens of dropped tables and removed tenants eventually expire. At least `2s`. An expired token only causes misses. | diff --git a/internal/config/cache_redis.go b/internal/config/cache_redis.go index 0e41b0ff1..920630157 100644 --- a/internal/config/cache_redis.go +++ b/internal/config/cache_redis.go @@ -19,6 +19,12 @@ const ( RedisSentinel = "sentinel" ) +// maxRedisDialTimeout caps cache.redis.dial_timeout. A dial is a connect +// and a handshake, each bounded by it, and both boot and the cache's Close +// wait out one in flight: at 2s that is at most 4s, inside the 5s budget +// Close shares with the stores released after the cache. +const maxRedisDialTimeout = 2 * time.Second + // CacheRedisConfig configures cache.backend=redis: one Redis-compatible server // (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB) shared by every process. // Read only when that backend is selected. @@ -102,6 +108,9 @@ func (r CacheRedisConfig) validate() error { return fmt.Errorf("%s %s must be positive", d.key, d.v) } } + if r.DialTimeout > maxRedisDialTimeout { + return fmt.Errorf("cache.redis.dial_timeout (WH_CACHE_REDIS_DIAL_TIMEOUT) %s is over %s: boot and shutdown each wait out a dial, up to twice this", r.DialTimeout, maxRedisDialTimeout) + } if r.VersionTTL < 2*time.Second { return fmt.Errorf("cache.redis.version_ttl (WH_CACHE_REDIS_VERSION_TTL) %s is under 2s", r.VersionTTL) } diff --git a/internal/config/cache_redis_test.go b/internal/config/cache_redis_test.go index 21ec15fcf..7f91a0499 100644 --- a/internal/config/cache_redis_test.go +++ b/internal/config/cache_redis_test.go @@ -57,7 +57,7 @@ func TestLoad_CacheRedisFromEnv(t *testing.T) { "WH_CACHE_REDIS_TLS_SERVER_NAME": "redis.internal", "WH_CACHE_REDIS_KEY_PREFIX": "staging", "WH_CACHE_REDIS_TIMEOUT": "250ms", - "WH_CACHE_REDIS_DIAL_TIMEOUT": "3s", + "WH_CACHE_REDIS_DIAL_TIMEOUT": "2s", "WH_CACHE_REDIS_MAX_VALUE_BYTES": "2048", "WH_CACHE_REDIS_COMPRESS_MIN_BYTES": "0", "WH_CACHE_REDIS_VERSION_TTL": "24h", @@ -73,7 +73,7 @@ func TestLoad_CacheRedisFromEnv(t *testing.T) { TLS: CacheRedisTLS{ Enabled: true, CAFile: caFile, CertFile: certFile, KeyFile: keyFile, ServerName: "redis.internal", }, - KeyPrefix: "staging", Timeout: 250 * time.Millisecond, DialTimeout: 3 * time.Second, + KeyPrefix: "staging", Timeout: 250 * time.Millisecond, DialTimeout: 2 * time.Second, MaxValueBytes: 2048, CompressMinBytes: 0, VersionTTL: 24 * time.Hour, }, cfg.Cache.Redis) tc, err := cfg.Cache.Redis.TLS.Config() @@ -221,6 +221,8 @@ func TestValidate_CacheRedis(t *testing.T) { {"brace prefix", func(r *CacheRedisConfig) { r.KeyPrefix = "a{b}" }, "hash-tag brace"}, {"zero timeout", func(r *CacheRedisConfig) { r.Timeout = 0 }, "cache.redis.timeout (WH_CACHE_REDIS_TIMEOUT) 0s must be positive"}, {"negative dial timeout", func(r *CacheRedisConfig) { r.DialTimeout = -time.Second }, "cache.redis.dial_timeout"}, + {"dial timeout at the cap", func(r *CacheRedisConfig) { r.DialTimeout = 2 * time.Second }, ""}, + {"dial timeout over the cap", func(r *CacheRedisConfig) { r.DialTimeout = 2*time.Second + time.Millisecond }, "cache.redis.dial_timeout (WH_CACHE_REDIS_DIAL_TIMEOUT) 2.001s is over 2s"}, {"short version ttl", func(r *CacheRedisConfig) { r.VersionTTL = time.Second }, "cache.redis.version_ttl (WH_CACHE_REDIS_VERSION_TTL) 1s is under 2s"}, {"zero max value", func(r *CacheRedisConfig) { r.MaxValueBytes = 0 }, "cache.redis.max_value_bytes"}, {"compress never", func(r *CacheRedisConfig) { r.CompressMinBytes = 0 }, ""}, From 8189783eb2ab4972753c27357c3bedfd8351690a Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 05:08:18 -0400 Subject: [PATCH 21/45] test(cache): the local backend's insert-invalidates lifecycle, end to end With e2e on Redis, no test above unit level showed an ingest invalidating LocalCache, the default backend. The integration suite's own app runs cache.backend=local, so it gets the shared-cache test's twin: a fill is a hit, an ingested row is first served as a miss well inside the stale entry's TTL, and the refill is a hit again. With LocalCache.Invalidate made a no-op it fails: the row landed only ~10.0s after the fill, every round. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_017aS7rLrH1RKkUMem7X4ckd --- CHANGELOG.md | 2 +- tests/integration/shared_cache_test.go | 42 ++++++++++++++++++++++++++ 2 files changed, 43 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1454d47ae..48eb10147 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **`cache.backend: redis` shares the query cache across instances** (`internal/config/{cache_redis,backends,config}.go` (+ tests), `internal/app/wire.go` (+ tests), `tests/integration/shared_cache_test.go`, `scripts/orchestrator/main.go`, `tests/e2e/fixtures/config.yaml`, `.testcoverage.yml`, `deployments/compose/dependencies.yaml`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md,settings-directory.mdx,getting-started.md,pipes.mdx,api.md,development.md}`, `README.md`, `AGENTS.md`): PR E4 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config's `cache.backend` now takes `redis`, configured by a new `cache.redis` block (`WH_CACHE_REDIS_*`): `addrs` (required), `mode` (`standalone` or `cluster`; `sentinel` refuses boot until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `username`, `password` (a secret — set it through the environment), `db`, `tls.{enabled,ca_file,cert_file,key_file,server_name,insecure_skip_verify}`, `key_prefix` (`wh`), `timeout` (`100ms`), `dial_timeout` (`1s`, at most `2s`: boot and shutdown each wait out a dial), `max_value_bytes` (1 MiB), `compress_min_bytes` (`1024`; `0` never compresses) and `version_ttl` (`168h`). Every instance pointed at one server shares its cached results, and an insert on any instance invalidates every instance's. It is also the shared cache a split of [`roles`](https://github.com/Wave-RF/WaveHouse/pull/622) needs: the refusal of `api` without `ingest`, or the reverse, over `cache.backend: local` now names `redis`, though every split is still refused while the queue is embedded. A malformed block — no address, an address without a port, a URL-style address (refused without repeating it, since it may carry a password), an unknown mode, `db` other than `0` in cluster mode, an unreadable TLS file, a TLS key set while `tls.enabled` is off — refuses boot; an unreachable server, or one that rejects the password, does not: the process boots with the cache bypassed and keeps reconnecting, logging a rejected password at `ERROR` on every attempt, so a rotated secret cannot crash-loop every instance at once. `insecure_skip_verify`, and a `cache.redis.addrs` set while `cache.backend` is `local`, are logged at `WARN` at boot. The e2e suite now runs against a Redis testcontainer with `cache.backend: redis`, so the shared backend is exercised end to end, and the e2e coverage exclude for its files is gone; the unit and integration suites keep covering `local`. An integration test boots two instances over one Redis and one ClickHouse: a result one fills is a hit for the other, and a row ingested through one is served fresh by the other on its next query, well inside the stale entry's TTL; another pauses Redis and checks queries keep succeeding from ClickHouse, then turn back to hits. `deployments/compose/dependencies.yaml` gains an optional `redis` profile for local multi-instance work. The deployment guide gains a "Multiple instances and the shared cache" section: what each instance keeps to itself, what a reader on another instance can see and when, `maxmemory-policy`, and the metrics to alert on. +- **`cache.backend: redis` shares the query cache across instances** (`internal/config/{cache_redis,backends,config}.go` (+ tests), `internal/app/wire.go` (+ tests), `tests/integration/shared_cache_test.go`, `scripts/orchestrator/main.go`, `tests/e2e/fixtures/config.yaml`, `.testcoverage.yml`, `deployments/compose/dependencies.yaml`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md,settings-directory.mdx,getting-started.md,pipes.mdx,api.md,development.md}`, `README.md`, `AGENTS.md`): PR E4 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config's `cache.backend` now takes `redis`, configured by a new `cache.redis` block (`WH_CACHE_REDIS_*`): `addrs` (required), `mode` (`standalone` or `cluster`; `sentinel` refuses boot until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `username`, `password` (a secret — set it through the environment), `db`, `tls.{enabled,ca_file,cert_file,key_file,server_name,insecure_skip_verify}`, `key_prefix` (`wh`), `timeout` (`100ms`), `dial_timeout` (`1s`, at most `2s`: boot and shutdown each wait out a dial), `max_value_bytes` (1 MiB), `compress_min_bytes` (`1024`; `0` never compresses) and `version_ttl` (`168h`). Every instance pointed at one server shares its cached results, and an insert on any instance invalidates every instance's. It is also the shared cache a split of [`roles`](https://github.com/Wave-RF/WaveHouse/pull/622) needs: the refusal of `api` without `ingest`, or the reverse, over `cache.backend: local` now names `redis`, though every split is still refused while the queue is embedded. A malformed block — no address, an address without a port, a URL-style address (refused without repeating it, since it may carry a password), an unknown mode, `db` other than `0` in cluster mode, an unreadable TLS file, a TLS key set while `tls.enabled` is off — refuses boot; an unreachable server, or one that rejects the password, does not: the process boots with the cache bypassed and keeps reconnecting, logging a rejected password at `ERROR` on every attempt, so a rotated secret cannot crash-loop every instance at once. `insecure_skip_verify`, and a `cache.redis.addrs` set while `cache.backend` is `local`, are logged at `WARN` at boot. The e2e suite now runs against a Redis testcontainer with `cache.backend: redis`, so the shared backend is exercised end to end, and the e2e coverage exclude for its files is gone; the unit and integration suites keep covering `local`. An integration test boots two instances over one Redis and one ClickHouse: a result one fills is a hit for the other, and a row ingested through one is served fresh by the other on its next query, well inside the stale entry's TTL; another pauses Redis and checks queries keep succeeding from ClickHouse, then turn back to hits; a third runs the first one's hit, insert and fresh-miss lifecycle on the suite's own `cache.backend: local` app, since e2e no longer exercises that backend. `deployments/compose/dependencies.yaml` gains an optional `redis` profile for local multi-instance work. The deployment guide gains a "Multiple instances and the shared cache" section: what each instance keeps to itself, what a reader on another instance can see and when, `maxmemory-policy`, and the metrics to alert on. - **A Redis-compatible shared cache backend** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row open a circuit breaker that skips the server until a probe succeeds, and deferred invalidations are retried until they land, collapsing to one tenant-wide bump per tenant past 100,000 keys. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. `cache.backend: redis` selects it (the entry above). Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression and server-stops-answering cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. - **One conformance suite for every `mq.Broker`, and a transient broker failure is a `503`** (`internal/mq/mqtest/` (new: the suite and the embedded broker's run of it), `internal/mq/{mq,embedded}.go` (+ tests), `internal/api/ingest.go` (+ tests), `.testcoverage.yml`, `docs/src/content/docs/{api,architecture}.md`, `AGENTS.md`): the first piece of the external-NATS workstream of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `mqtest.Run` states the `Broker` contract as behavior — round trips with names that need encoding, per-tenant order, `Nak` and `AckWait` redelivery, the trace context reaching `Subscribe`, dead-lettering that keeps the topic and leaves the original unacked, per-tenant dead-letter counts, replay bounds and isolation, and exactly one `failed` report when delivery ends underneath a consumer — through the interfaces alone, so the external backend runs the same cases, with `mqtest.Caps` for the four places where its semantics legitimately differ. The embedded broker passes it; writing it turned up that a durable deleted on several tenants' queues could report on `failed` more than once, which is fixed and pinned by a test that deletes it on one queue after another. It also turned up a replay that lost its connection mid-pull passing for a caught-up one when the pull ended in a timeout; that is an error now, as the `Replayer` contract says. The interface comments now allow a delivery unit that is a partition holding several tenants, a `CreateConsumer` that finds a durable rather than creating one, a `PurgeAcked` that leaves retention to the operator, and zero dead-letter counts where there is no per-tenant queue. A new sentinel, `mq.ErrUnavailable`, is a broker that cannot be reached or does not answer in time: the ingest handler answers it with `503` + `Retry-After: 5` rather than the `500` "publish failed" it would have been. Nothing returns it yet; the external backend of #613 will. - **Process roles: the API and the background workers can run in separate processes** (`internal/config/config.go` (+ `roles_test.go`, `defaults_test.go`), `internal/config/backends.go`, `internal/app/{app,wire}.go` (+ `roles_test.go`), `internal/api/router.go` (+ tests), `tests/integration/{setup,tenants}_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md}`, `AGENTS.md`), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The boot config gains `roles` (`WH_ROLES`, default `api,ingest,sweeper`, set in `defaults()` like every boot default, so an explicit `roles: []` refuses boot) and `instance_id` (`WH_INSTANCE_ID`, default `-<8 hex>`, fresh at every boot; logged at boot, and recorded as a lease holder once a shared `coord.backend` exists). A process wires only what its roles need: `api` runs the HTTP API with schema discovery, the token verifiers, the dedupe stores and the SSE hub (all per API process); `ingest` runs the ingest worker; `sweeper` runs the sweeper under its lease. A process without `api` serves an ops-only listener on `server.port` (the probes and their aliases, `/version`, the same-port metrics path, and `POST /v1/ops/settings/reload`, which takes the operator key alone); every other route answers 404, under `/v1/ops` once the operator key has passed. Boot refuses any split over the embedded MQ, which no other process can reach, and `api` without `ingest` (or the reverse) over a local cache, which the ingest worker's invalidations would never reach. Until a shared `mq.backend` exists, every process therefore runs every role, which is the default, so nothing changes for an existing deployment. `data_dir` is probed for Pebble only in a process running `api`. A `config.Config` built without `config.Load` must now name its roles (`config.AllRoles()` for all of them): `app.New` refuses an empty set. diff --git a/tests/integration/shared_cache_test.go b/tests/integration/shared_cache_test.go index f4a884d0e..e9857b725 100644 --- a/tests/integration/shared_cache_test.go +++ b/tests/integration/shared_cache_test.go @@ -216,6 +216,48 @@ func TestSharedCache_IngestOnOneInstanceInvalidatesAnother(t *testing.T) { } } +// The same lifecycle on the default backend, against the suite's own app +// (cache.backend=local), which e2e no longer runs: a fill is a hit, and an +// ingest invalidates it, so the first answer carrying the new row is a miss +// well inside the stale entry's TTL, and the refill is a hit again. +func TestLocalCache_IngestInvalidates(t *testing.T) { + base := env(t).baseURL + // As above: a round whose row lands past the TTL floor proves nothing, + // and runs again on a fresh table, so a fresh fill. + for round := 1; ; round++ { + table := createTable(t, "user_id String, value Float64", "ORDER BY user_id") + status, xc, body := structuredQuery(t, base, table) + require.Equal(t, http.StatusOK, status, body) + require.Equal(t, "MISS", xc) + filled := time.Now() + status, xc, _ = structuredQuery(t, base, table) + require.Equal(t, http.StatusOK, status) + require.Equal(t, "HIT", xc) + + ingestRow(t, base, table, "u1") + var seenAt time.Time + require.Eventually(t, func() bool { + var err error + status, xc, body, err = tryStructuredQuery(base, table) + if err == nil && status == http.StatusOK && strings.Contains(body, "u1") { + seenAt = time.Now() + return true + } + return false + }, 30*time.Second, 100*time.Millisecond, "the ingested row is never served") + if seenAt.Sub(filled) < minCacheTTL-time.Second { + assert.Equal(t, "MISS", xc, "the first answer carrying the new row is a refill") + status, xc, body = structuredQuery(t, base, table) + require.Equal(t, http.StatusOK, status) + assert.Equal(t, "HIT", xc, "the refill is cached again") + assert.Contains(t, body, "u1") + return + } + require.Less(t, round, 3, "the new row was served only once the stale entry could have expired, in every round: the ingest never invalidated it, or ingest is too slow here to tell") + t.Logf("round %d: row landed %s after the fill, past the TTL floor; retrying", round, seenAt.Sub(filled)) + } +} + // A Redis that stops answering costs queries nothing but the cache: they // keep succeeding, straight from ClickHouse, each a miss; an ingest made // meanwhile is visible at once. Once it answers again, the cache serves hits. From 5e8476271d03db7fc5ca51aba68e3a48db5353c4 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 05:10:05 -0400 Subject: [PATCH 22/45] fix(ingest): log an invalidation that did not land at WARN With cache.backend=redis a bump the server does not take is deferred and retried, so a Redis outage logged an ERROR for every batch the worker inserted. It is a WARN now; wavehouse_cache_invalidations_pending is the signal to alert on. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_017aS7rLrH1RKkUMem7X4ckd --- CHANGELOG.md | 2 +- internal/ingest/worker.go | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 48eb10147..27ee2eaf7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **`cache.backend: redis` shares the query cache across instances** (`internal/config/{cache_redis,backends,config}.go` (+ tests), `internal/app/wire.go` (+ tests), `tests/integration/shared_cache_test.go`, `scripts/orchestrator/main.go`, `tests/e2e/fixtures/config.yaml`, `.testcoverage.yml`, `deployments/compose/dependencies.yaml`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md,settings-directory.mdx,getting-started.md,pipes.mdx,api.md,development.md}`, `README.md`, `AGENTS.md`): PR E4 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config's `cache.backend` now takes `redis`, configured by a new `cache.redis` block (`WH_CACHE_REDIS_*`): `addrs` (required), `mode` (`standalone` or `cluster`; `sentinel` refuses boot until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `username`, `password` (a secret — set it through the environment), `db`, `tls.{enabled,ca_file,cert_file,key_file,server_name,insecure_skip_verify}`, `key_prefix` (`wh`), `timeout` (`100ms`), `dial_timeout` (`1s`, at most `2s`: boot and shutdown each wait out a dial), `max_value_bytes` (1 MiB), `compress_min_bytes` (`1024`; `0` never compresses) and `version_ttl` (`168h`). Every instance pointed at one server shares its cached results, and an insert on any instance invalidates every instance's. It is also the shared cache a split of [`roles`](https://github.com/Wave-RF/WaveHouse/pull/622) needs: the refusal of `api` without `ingest`, or the reverse, over `cache.backend: local` now names `redis`, though every split is still refused while the queue is embedded. A malformed block — no address, an address without a port, a URL-style address (refused without repeating it, since it may carry a password), an unknown mode, `db` other than `0` in cluster mode, an unreadable TLS file, a TLS key set while `tls.enabled` is off — refuses boot; an unreachable server, or one that rejects the password, does not: the process boots with the cache bypassed and keeps reconnecting, logging a rejected password at `ERROR` on every attempt, so a rotated secret cannot crash-loop every instance at once. `insecure_skip_verify`, and a `cache.redis.addrs` set while `cache.backend` is `local`, are logged at `WARN` at boot. The e2e suite now runs against a Redis testcontainer with `cache.backend: redis`, so the shared backend is exercised end to end, and the e2e coverage exclude for its files is gone; the unit and integration suites keep covering `local`. An integration test boots two instances over one Redis and one ClickHouse: a result one fills is a hit for the other, and a row ingested through one is served fresh by the other on its next query, well inside the stale entry's TTL; another pauses Redis and checks queries keep succeeding from ClickHouse, then turn back to hits; a third runs the first one's hit, insert and fresh-miss lifecycle on the suite's own `cache.backend: local` app, since e2e no longer exercises that backend. `deployments/compose/dependencies.yaml` gains an optional `redis` profile for local multi-instance work. The deployment guide gains a "Multiple instances and the shared cache" section: what each instance keeps to itself, what a reader on another instance can see and when, `maxmemory-policy`, and the metrics to alert on. +- **`cache.backend: redis` shares the query cache across instances** (`internal/config/{cache_redis,backends,config}.go` (+ tests), `internal/app/wire.go` (+ tests), `internal/ingest/worker.go`, `tests/integration/shared_cache_test.go`, `scripts/orchestrator/main.go`, `tests/e2e/fixtures/config.yaml`, `.testcoverage.yml`, `deployments/compose/dependencies.yaml`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md,settings-directory.mdx,getting-started.md,pipes.mdx,api.md,development.md}`, `README.md`, `AGENTS.md`): PR E4 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config's `cache.backend` now takes `redis`, configured by a new `cache.redis` block (`WH_CACHE_REDIS_*`): `addrs` (required), `mode` (`standalone` or `cluster`; `sentinel` refuses boot until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `username`, `password` (a secret — set it through the environment), `db`, `tls.{enabled,ca_file,cert_file,key_file,server_name,insecure_skip_verify}`, `key_prefix` (`wh`), `timeout` (`100ms`), `dial_timeout` (`1s`, at most `2s`: boot and shutdown each wait out a dial), `max_value_bytes` (1 MiB), `compress_min_bytes` (`1024`; `0` never compresses) and `version_ttl` (`168h`). Every instance pointed at one server shares its cached results, and an insert on any instance invalidates every instance's. It is also the shared cache a split of [`roles`](https://github.com/Wave-RF/WaveHouse/pull/622) needs: the refusal of `api` without `ingest`, or the reverse, over `cache.backend: local` now names `redis`, though every split is still refused while the queue is embedded. A malformed block — no address, an address without a port, a URL-style address (refused without repeating it, since it may carry a password), an unknown mode, `db` other than `0` in cluster mode, an unreadable TLS file, a TLS key set while `tls.enabled` is off — refuses boot; an unreachable server, or one that rejects the password, does not: the process boots with the cache bypassed and keeps reconnecting, logging a rejected password at `ERROR` on every attempt, so a rotated secret cannot crash-loop every instance at once. `insecure_skip_verify`, and a `cache.redis.addrs` set while `cache.backend` is `local`, are logged at `WARN` at boot. The ingest worker's log of an invalidation that did not land drops from `ERROR` to `WARN`, since the shared backend defers and retries it: an outage would otherwise log an `ERROR` for every batch. The e2e suite now runs against a Redis testcontainer with `cache.backend: redis`, so the shared backend is exercised end to end, and the e2e coverage exclude for its files is gone; the unit and integration suites keep covering `local`. An integration test boots two instances over one Redis and one ClickHouse: a result one fills is a hit for the other, and a row ingested through one is served fresh by the other on its next query, well inside the stale entry's TTL; another pauses Redis and checks queries keep succeeding from ClickHouse, then turn back to hits; a third runs the first one's hit, insert and fresh-miss lifecycle on the suite's own `cache.backend: local` app, since e2e no longer exercises that backend. `deployments/compose/dependencies.yaml` gains an optional `redis` profile for local multi-instance work. The deployment guide gains a "Multiple instances and the shared cache" section: what each instance keeps to itself, what a reader on another instance can see and when, `maxmemory-policy`, and the metrics to alert on. - **A Redis-compatible shared cache backend** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row open a circuit breaker that skips the server until a probe succeeds, and deferred invalidations are retried until they land, collapsing to one tenant-wide bump per tenant past 100,000 keys. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. `cache.backend: redis` selects it (the entry above). Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression and server-stops-answering cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. - **One conformance suite for every `mq.Broker`, and a transient broker failure is a `503`** (`internal/mq/mqtest/` (new: the suite and the embedded broker's run of it), `internal/mq/{mq,embedded}.go` (+ tests), `internal/api/ingest.go` (+ tests), `.testcoverage.yml`, `docs/src/content/docs/{api,architecture}.md`, `AGENTS.md`): the first piece of the external-NATS workstream of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `mqtest.Run` states the `Broker` contract as behavior — round trips with names that need encoding, per-tenant order, `Nak` and `AckWait` redelivery, the trace context reaching `Subscribe`, dead-lettering that keeps the topic and leaves the original unacked, per-tenant dead-letter counts, replay bounds and isolation, and exactly one `failed` report when delivery ends underneath a consumer — through the interfaces alone, so the external backend runs the same cases, with `mqtest.Caps` for the four places where its semantics legitimately differ. The embedded broker passes it; writing it turned up that a durable deleted on several tenants' queues could report on `failed` more than once, which is fixed and pinned by a test that deletes it on one queue after another. It also turned up a replay that lost its connection mid-pull passing for a caught-up one when the pull ended in a timeout; that is an error now, as the `Replayer` contract says. The interface comments now allow a delivery unit that is a partition holding several tenants, a `CreateConsumer` that finds a durable rather than creating one, a `PurgeAcked` that leaves retention to the operator, and zero dead-letter counts where there is no per-tenant queue. A new sentinel, `mq.ErrUnavailable`, is a broker that cannot be reached or does not answer in time: the ingest handler answers it with `503` + `Retry-After: 5` rather than the `500` "publish failed" it would have been. Nothing returns it yet; the external backend of #613 will. - **Process roles: the API and the background workers can run in separate processes** (`internal/config/config.go` (+ `roles_test.go`, `defaults_test.go`), `internal/config/backends.go`, `internal/app/{app,wire}.go` (+ `roles_test.go`), `internal/api/router.go` (+ tests), `tests/integration/{setup,tenants}_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md}`, `AGENTS.md`), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The boot config gains `roles` (`WH_ROLES`, default `api,ingest,sweeper`, set in `defaults()` like every boot default, so an explicit `roles: []` refuses boot) and `instance_id` (`WH_INSTANCE_ID`, default `-<8 hex>`, fresh at every boot; logged at boot, and recorded as a lease holder once a shared `coord.backend` exists). A process wires only what its roles need: `api` runs the HTTP API with schema discovery, the token verifiers, the dedupe stores and the SSE hub (all per API process); `ingest` runs the ingest worker; `sweeper` runs the sweeper under its lease. A process without `api` serves an ops-only listener on `server.port` (the probes and their aliases, `/version`, the same-port metrics path, and `POST /v1/ops/settings/reload`, which takes the operator key alone); every other route answers 404, under `/v1/ops` once the operator key has passed. Boot refuses any split over the embedded MQ, which no other process can reach, and `api` without `ingest` (or the reverse) over a local cache, which the ingest worker's invalidations would never reach. Until a shared `mq.backend` exists, every process therefore runs every role, which is the default, so nothing changes for an existing deployment. `data_dir` is probed for Pebble only in a process running `api`. A `config.Config` built without `config.Load` must now name its roles (`config.AllRoles()` for all of them): `app.New` refuses an empty set. diff --git a/internal/ingest/worker.go b/internal/ingest/worker.go index 70517cd2e..529b8485e 100644 --- a/internal/ingest/worker.go +++ b/internal/ingest/worker.go @@ -910,7 +910,9 @@ func (w *IngestWorker) invalidate(ctx context.Context, id tenant.ID, tableName s } invCtx := trace.ContextWithSpanContext(context.WithoutCancel(ctx), trace.SpanContextFromContext(ctx)) if _, err := w.cache.Invalidate(invCtx, namespaces); err != nil { - slog.ErrorContext(invCtx, "failed to invalidate cache after insert - your cache is holding stale data now!", "tenant", id, "table", tableName, "error", err) + // WARN, not ERROR: a shared backend defers and retries the bump, and + // an outage would otherwise log an ERROR for every batch. + slog.WarnContext(invCtx, "cache invalidation after insert did not land; the table's cached results may be stale until it does", "tenant", id, "table", tableName, "error", err) } } From 0696702c730393a9a18795cadc8b5d0532744243 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 05:18:13 -0400 Subject: [PATCH 23/45] fix(cache): owed bumps hold lookups; refused writes open the breaker A process that owes a version bump no longer serves what the bump would orphan: a lookup reading a token key still pending is a bypass with a zero snapshot, so it neither hits nor fills. An atomic count keeps the check free while nothing is owed, and the tenant token a pending set collapses to past PendingMax is one of every lookup's keys, so the collapse holds the whole tenant. An error reply saying the server takes no writes (READONLY from a demoted primary, OOM under noeviction, MASTERDOWN, NOREPLICAS, MISCONF, LOADING, BUSY, CLUSTERDOWN) opens the breaker at once instead of counting as the server being up. The probe is now a write, so a server that answers but refuses writes stays bypassed, and only the probe's success closes an open breaker. This is the READONLY/OOM half of #656. The first deferral wakes the drain at once, and while the breaker is open the drain wakes when the probe is due rather than at its own backoff (up to 10 s), so a process that makes no lookups recovers as soon as one that does. Invalidate sends its bumps in batches of 1,000, as the drain does. The zstd encoder window drops to 1 MiB: its pool kept about 121 MiB at 14 procs with 240 KB values, now about 36 MiB, at the same speed and ratio. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_017aS7rLrH1RKkUMem7X4ckd --- CHANGELOG.md | 2 +- docs/src/content/docs/architecture.md | 8 +- internal/cache/breaker.go | 37 ++++- internal/cache/breaker_test.go | 38 ++++++ internal/cache/export_test.go | 3 + internal/cache/metrics.go | 4 +- internal/cache/pending.go | 32 +++-- internal/cache/pending_test.go | 26 ++++ internal/cache/redis.go | 132 +++++++++++++----- internal/cache/redis_codec.go | 10 +- internal/cache/redis_codec_test.go | 35 +++++ internal/cache/redis_integration_test.go | 164 +++++++++++++++++++++++ internal/cache/redis_test.go | 45 +++++++ 13 files changed, 479 insertions(+), 57 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d02ae4ad3..b21658065 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **A Redis-compatible shared cache backend, built but not yet selectable** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag, with the table and scope escaped into the key (`internal/keyenc`) so no two names share a token; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row open a circuit breaker that skips the server until a probe succeeds, and deferred invalidations are retried until they land, collapsing to one tenant-wide bump per tenant past 100,000 keys. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. Nothing selects it yet: the `cache.backend` switch and the `cache.redis.*` settings arrive with the wiring (E4), so every deployment still runs `LocalCache`. Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression and server-stops-answering cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. +- **A Redis-compatible shared cache backend, built but not yet selectable** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag, with the table and scope escaped into the key (`internal/keyenc`) so no two names share a token; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row, or one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like), open a circuit breaker that skips the server until a probe write succeeds, and deferred invalidations are retried until they land — the first at once, and at the probe's cadence while the breaker is open — collapsing to one tenant-wide bump per tenant past 100,000 keys; until one lands, the process that owes it bypasses the lookups it would orphan. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. Nothing selects it yet: the `cache.backend` switch and the `cache.redis.*` settings arrive with the wiring (E4), so every deployment still runs `LocalCache`. Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression, server-stops-answering, refused-writes (a demoted primary, a full `noeviction` server) and owed-invalidation cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. - **A tenant removed or rejected at runtime has its open streams ended** (`internal/stream/{hub,subscriber,bucket}.go` (+ tests), `internal/api/stream.go` (+ tests), `internal/api/router.go`, `internal/settings/{registry,tree}.go` (+ tests), `internal/ingest/worker.go` (+ tests), `internal/app/wire.go` (+ tests), `clients/ts/src/stream/sse.ts`, `docs/src/content/docs/{api,deployment,architecture,ingest-pipeline}.md`, `docs/src/content/docs/settings-directory.mdx`, `AGENTS.md`): story 3 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)). A `GET /v1/stream` used to outlive its tenant: the hub read no policy for it and withheld every row while the keepalive wheel held the connection open, so the client could not tell it from a quiet table. After every reload the hub now evicts the subscribers of each tenant no longer served, removed or rejected alike (`Hub.Prune`, through the close-once `Subscriber.Evict`), and the handler ends the stream: a gap-fill in progress included, and a stream `TenantMW` admitted just before the reload but registered just after it, which the handler checks for as it registers. The client's reconnect gets `404` (removed) or `503` (rejected); the SDK stops on the first and retries the second, resuming from `Last-Event-ID` once the folder is back — except in a browser going cross-origin where tenant `0` is not served or its CORS list does not admit the page, which cannot read either refusal (both are decorated from tenant `0`'s list) and re-dials as after a dropped connection. A flat directory never stops serving tenant `0`, so nothing changes there. Removing a tenant is deleting its folder, then reloading the whole directory, the last folder included: a server started with tenant folders reads the emptied directory as no folder left, where it read as a change of shape and the reload was rejected whole, leaving that tenant served; at boot an empty directory still reads as the four files, missing. Reloading the deleted folder by name leaves its tenant rejected. `GET /v1/health` keeps resolving a tenant, deliberately: its `404` tells a caller with no token no more than every tenant route's does, since they all answer before authenticating, and resolving answers a served tenant's ping from that tenant's own CORS list. A batch queued for a tenant with no ClickHouse connection — one no longer served, or one no pool could be opened for (such as by the connection ceiling) — skips the row-by-row retry, which no row of it could pass, and meets its DLQ switch once, whole, logged once per batch rather than twice per row; a tenant no longer served reads the switch as on, so its queued rows are parked under its own subject rather than dropped, inserted into another tenant's ClickHouse, or left unacked, redelivered for as long as the tenant is away and holding its queue's ack floor, which the purge waits on. Over a nested directory `/livez` no longer keeps naming a tenant that stopped being served before any tenant completed a first discovery: the diagnostic goes back to `no tenant has completed a first discovery yet`. - **One ClickHouse pool per tuple and one schema registry per tenant** (`internal/chconn/chconn.go` (+ tests), `internal/discovery/discovery.go` (+ tests), `internal/app/discoveries.go` (new), `internal/app/{app,wire}.go` (+ tests), `internal/api/{schema,ingest,structured_query,pipes,query,health,errors,clickhouse_exec}.go` (+ tests), `internal/stream/hub.go`, `internal/ingest/worker.go`, `internal/cache/{cache,local,version_manager}.go` (+ tests), `internal/testutil/{testutil,mocks}.go`, `tests/integration/{setup,tenants,boot_resilience,query_limits}_test.go`, `clients/ts/src/{schema,table,sql,client,types}.ts` (+ tests), `tests/e2e/sdk/admin.test.ts`, `docs/src/content/docs/{api,deployment,architecture,ingest-pipeline}.md`, `docs/src/content/docs/{settings-directory,configuration,access-control,reverse-proxy}.mdx`, `docs/src/content/docs/sdk/{admin,reference,queries}.md`, `AGENTS.md`): the second slice of story 6 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)), with no behavior change for a settings directory that holds the four files beyond the three noted at the end. The process opens one native pool per distinct `clickhouse.addr` / `database` / `username` / password / `tls` tuple among the served tenants (`chconn.Identity`, `chconn.Pools`), shared by the tenants naming it and sized to their largest `max_open_conns` and `max_idle_conns`; `http_port`, `http_scheme`, `headers` and `query_timeout` stay each tenant's own. Every reload reconciles the pools: a new tuple opens (never dials), a tenant whose tuple changed is repointed, a tuple no tenant names closes after the longest `query_timeout` among the tenants it had, and a pool whose largest ask changed is resized with the same grace. The boot config's `clickhouse.max_total_conns` now bounds the open pools together: boot is refused naming the sum and the ceiling; at a reload a resize above it is refused with the pool kept at its size, and a tuple that cannot be opened — the ceiling, a certificate file that cannot be read, or options the driver refuses — leaves its tenants on the pool they had (the keep-previous-wiring rule of the connection ceiling) or on none when they had none; both are logged and the next reload retries. A tenant on no pool fails closed: `503` with `Retry-After: 30` on `POST /v1/query`, `GET/POST /v1/pipes/{name}`, `POST /v1/ops/query` and `POST /v1/ops/schema/refresh`, ahead of the cache. Each served tenant gets a `discovery.SchemaRegistry` of its own over its pool, kept fresh by its own loop — the boot retry until the first success, then `schema.refresh_interval` with the first refresh at a random point within the interval so tenants adopted together do not refresh together — created and stopped from the settings reload and stopped under `App.Close`; `wavehouse_schema_refresh_failures_total{tenant}` counts a loop's failed attempts. `GET /v1/ops/schema`, `POST /v1/ops/schema/refresh` and `POST /v1/ops/query` take the strict `?tenant=` the pipe reads take (absent is tenant `0`, `400` malformed, `404` unknown, `503` rejected); the SDK sends it as the `tenant` option of `wh.schema.list()`, `wh.schema.refresh()`, `wh.from(t).schema()` and `wh.sql()`. Over a nested directory `/livez` (and `/v1/health`) is `503` with the latest discovery failure, naming its tenant, while no tenant has completed a first discovery, then `200` for the rest of the process lifetime; `/readyz` pings every open pool at once, is ready at the first answer, and names every pool that did not answer when none does — one tenant's ClickHouse outage is its log line and counter, never a probe failure. The ingest worker inserts each batch into its own tenant's ClickHouse — the tenant the message's topic names, through that tenant's HTTP wiring (`chconn.Pools.Target`); a tenant on no pool takes the failure path an unreachable ClickHouse takes — and its cache invalidation fans out to the tenants on the same address and database as the batch's tenant, whatever their user or `tls` block (they read the same tables), rather than to every known tenant, and a tenant adopted after an absence — rejected or removed, so out of that fan-out — or moved to another address or database has its cached results orphaned in one step (`Cache.InvalidateTenant`, a tenant generation in every version key), so a repaired folder never serves query rows cached before the inserts it missed (since #614 this drops the tenant's cached pipe results too; no insert invalidates a pipe result, which names no table). Three changes reach the single-tenant directory too: a table lookup before the first discovery is a `503` with `Retry-After: 5` (`schema not loaded yet`) rather than a `404`, on `POST /v1/ingest`, `POST /v1/query` and `GET /v1/ops/schema` (the list included, where `[]` would read as no tables); the first periodic refresh fires at a random point within the interval rather than a full interval after boot; and a reload that moves `clickhouse.addr` or `clickhouse.database` now orphans the results cached before it, where they were served until their TTL. Until [#529](https://github.com/Wave-RF/WaveHouse/issues/529) every tenant's user authenticates with `WH_CH_PASSWORD`, so the tuple is in effect the address, database, username and `tls` block. - **One token verifier per tenant, built off the boot and reload paths** (`internal/auth/auth.go` (+ tests), `internal/api/router.go` (+ tests), `internal/app/{app,wire}.go` (+ tests), `go.mod`, `docs/src/content/docs/sdk/{reference,streaming}.md`, `docs/src/content/docs/{architecture,deployment,api}.md`, `docs/src/content/docs/{settings-directory,configuration}.mdx`, `SECURITY.md`): story 9 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)). Over a nested settings directory each tenant's folder now wires that tenant's verifier (`auth.jwks_url`, `auth.role_claim`), so a JWKS-issued token verifies only under the tenants whose `jwks_url` names its identity provider's key set — under any other tenant's header it is refused as invalid — where before one verifier, tenant `0`'s, accepted a token under any header. `auth.Config` is now the boot-config half alone (the HMAC secret and the operator key, shared by every tenant) and the new `auth.Wiring` a tenant's half; `NewAuthenticator` builds no verifier, `Reconfigure(id, wiring)` gives a tenant one — swapped atomically when its wiring changed, kept when it did not — `Prune` drops the verifiers of the tenants a reload stopped serving, rejected or removed alike (no work runs for a tenant that is not served; a folder adopted again gets a fresh verifier), and `Close` stops every JWKS refresh as a component of `App.Close`. This holds on every reload because the registry's `AfterAdopt` hooks run after every reload it applied, empty list included, so a per-tenant reload that rejects a folder drops its verifier then rather than at the next adoption. The middleware reads the request tenant's verifier through an injected `auth.TenantSource` — the store `api.TenantMW` resolved names its tenant with `settings.Store.Tenant()`, one read of the context — a tenant-exempt route (the ops tree) verifies as tenant `0`, and a tenant with no verifier fails closed. The operator key stamps the request tenant's `admin_role`, read from that tenant's policy, rather than tenant `0`'s. A JWKS key set is fetched on its own goroutine: the verifier is in place at once and *pending* until a set has been stored — the first fetch retried with backoff from one second to a minute, a stored set then kept fresh by the library hourly and, rate-limited, on an unknown key id — so neither boot nor a reload (which holds the registry's lock) waits on the endpoint, and **an unreachable JWKS no longer refuses boot** — it logs (`jwks refresh failed; no token validates until it succeeds`) and that tenant alone is affected. A token checked against a pending verifier is a new outcome, `auth.ErrVerifierPending`: every `/v1` route answers it `503 {"error": "token verifier not ready: …"}` with `Retry-After: 30` (`api.refuseUnverifiable`) rather than evaluate the request under the `default_role`, which could accept its data under a lesser role while another pod holding the keys would have served it as its own; a request without a token, and the operator key, are unaffected. Every fetch goes through one client that caps the response at 1 MiB, refusing a larger one as unreachable with `jwks response exceeds 1048576 bytes` as the logged cause. Nothing changes for a flat directory beyond that boot rule: its one tenant gets exactly the verifier it had. The boot warning for the secretless posture (no `auth.jwt_secret` and no `jwks_url`, whose token refusal landed in [#607](https://github.com/Wave-RF/WaveHouse/pull/607)) is now per tenant — each served tenant with no `jwks_url` while the boot secret is unset — where one line that any JWKS tenant silenced used to stand for the whole directory. diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index e18ace9c1..f124f8d7c 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -113,11 +113,11 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ - **cache.go** — `Cache` interface: `Lookup`, `Set`, `Invalidate`, `InvalidateTenant`, `Close`, plus `QueryTimeToTTL`, which sets a result's TTL from how long its query took (10 s floor, 1 h ceiling). Every entry is one tenant's: `Lookup` takes the tenant, the caller's query key — `:query:`, built by the two cached handlers in `api/` (`queryCacheKey`, with the tenant read off the request's store — `settings.Store.Tenant`), which use it as their [singleflight](https://pkg.go.dev/golang.org/x/sync/singleflight) key too — and the `Namespace`s the result depends on — one for a structured query, none yet for a pipe (a pipe's table dependencies are [#343](https://github.com/Wave-RF/WaveHouse/pull/343)) — each of that tenant (another tenant's is `ErrForeignDependency`) and naming a table and scope by their raw names, which the cache escapes where it builds a key. `Lookup` returns the `Entry` (a nil value is a miss) and a `Snapshot` of the versions it read; on a miss the handler runs the query and passes that snapshot to `Set`, so a result is filed under the versions read *before* its query ran, and a write that lands while it runs orphans the fill rather than re-homing pre-write rows under the post-write versions ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)). The singleflight leader's snapshot is the one used. `Set` returns an error only when the backend failed; a value the cache declines — larger than it keeps, a non-positive TTL, a zero snapshot — is not one. What a hit, a miss and a bump mean is pinned by the conformance suite every backend runs, `internal/testutil/cachetest`. - **local.go** — `LocalCache`, the in-process L1 on [Ristretto](https://github.com/dgraph-io/ristretto): one pool shared by every tenant (a heavier tenant holds more of it), sized by the boot config's `cache.l1_max_cost`. - **version_manager.go** — `VersionManager`, the invalidation index behind `Invalidate` and `InvalidateTenant`: a namespace key is `..
.
.`, its fields joined with `keyenc.Join` so a dot in a table or scope name is escaped rather than read as a separator, and an entry's key, `|.||…` with the caller's query key escaped whole (its `:` become `%3A`), folds in the tenant's version and each dependency's namespace key and namespace version, so bumping a table (a scopeless write) or one scope — scope is reserved and empty today, so every write is the whole-table bump — orphans every dependent entry without touching the pool. The tenant leads every key ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8): the same table under two tenants is two namespaces, so a bump through `Invalidate` under one tenant never touches — and a read under one tenant is never served — the other's results, and the flat directory's single tenant simply carries the `0` prefix. `BumpTenant` (behind `InvalidateTenant`) advances the tenant version that leads every namespace key of one tenant, orphaning its every namespace and every cached query in one step (the tenant version is folded into every query key, so a pipe result with no dependencies is orphaned too) — a table no bump ever keyed included, which an enumeration of the index would miss — for a tenant back on a pool after an absence from the fan-out, or moved to another address or database (story 6). The index is per tenant; the cross-tenant invalidation an insert into a shared table needs is not the index's but the wiring's: `internal/app` hands the ingest worker a cache (`sharedTables`) that repeats each bump under every tenant on the same ClickHouse address and database. -- **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — only `GET`, `SET`, `MGET` and `PING`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot; the table and scope are escaped with `keyenc` after the fixed prefix, so a `:` in a name is never read as the separator. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, the hash over their escaped forms, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, and a token key holding anything but a token (a string of another length, a list, a hash) is replaced, so a token lost to eviction, expiry or a restart is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing. Nothing selects this backend yet — the boot config and wiring arrive with [#613](https://github.com/Wave-RF/WaveHouse/issues/613)'s E4. +- **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — only `GET`, `SET` and `MGET`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot; the table and scope are escaped with `keyenc` after the fixed prefix, so a `:` in a name is never read as the separator. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, the hash over their escaped forms, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, and a token key holding anything but a token (a string of another length, a list, a hash) is replaced, so a token lost to eviction, expiry or a restart is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. While this process owes a bump on any of a lookup's tokens, that lookup is a bypass that files nothing: the bump would orphan whatever it found (other processes, which cannot know, serve those entries until it lands). `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing. Nothing selects this backend yet — the boot config and wiring arrive with [#613](https://github.com/Wave-RF/WaveHouse/issues/613)'s E4. - **redis_codec.go** — the key schema (`tokenKeys`, `bumpKeys`, `valueKey`, which take a `Namespace`'s raw names and escape them) and the value frame: format, flags, expiry, the token list, the payload, zstd-compressed from `CompressMinBytes` when that is smaller. A stored value is capped at `MaxValueBytes` and a decoded one at eight times that, which refuses a zip bomb planted in a shared server; an unknown format, as a newer process writes during a rolling upgrade, is a miss. -- **breaker.go** — the circuit breaker: `BreakerThreshold` (5) consecutive failures open it, and while open every operation skips the server at once; after `BreakerOpenFor` (5 s) one background `PING` decides whether it closes. Only a transport failure or a timeout counts against the server: any reply, an error reply or one the backend cannot use included, counts as a success, for operations and the probe alike, and a caller that gave up first counts as nothing. -- **pending.go** — invalidations the server did not take, kept per token key (repeats coalesce) and retried by a background loop with backoff from 100 ms to 10 s until they land. Landing late is still correct: a fresh token orphans the pre-write entries and any fill made meanwhile. Past `PendingMax` (100,000) keys the set collapses to one tenant bump per affected tenant. `Close` makes one last attempt at the bumps still owed, past the breaker (an open one is why they are owed); what that attempt cannot deliver is lost, and the entries they would orphan are served until their TTL — the same failure as a worker stopping between an insert and its invalidation. -- **metrics.go** — the shared backend's instruments (meter `wavehouse-cache`, `backend="redis"`, no tenant attribute): `wavehouse_cache_lookups_total` by `result` (`hit`, `miss`, `stale` — filed under since-bumped tokens, `bypass` — server skipped, `error`), `wavehouse_cache_op_duration_seconds` by `op` (`lookup`, `set`, `invalidate`), `wavehouse_cache_breaker_open` (1 while bypassed, including before the first connection), `wavehouse_cache_invalidations_total` by `result` (`ok`, `deferred`), `wavehouse_cache_invalidations_pending` (bumps owed: entries they would orphan may be served stale meanwhile), `wavehouse_cache_value_bytes` (stored size), `wavehouse_cache_oversize_total` and `wavehouse_cache_set_failures_total` by `reason` (`oom`, `timeout`, `other`). +- **breaker.go** — the circuit breaker: `BreakerThreshold` (5) consecutive failures open it, and while open every operation skips the server at once; after `BreakerOpenFor` (5 s) one background write (`SET :probe`) decides whether it closes, and only that probe's success closes it. A transport failure or a timeout counts against the server; an error reply saying it takes no writes right now — `READONLY` (a demoted primary), `MASTERDOWN`, `OOM` (full under `maxmemory-policy noeviction`, Redis's default, so give the cache an evicting policy), `NOREPLICAS`, `MISCONF`, `LOADING`, `BUSY`, `CLUSTERDOWN` — opens it at once, whatever the threshold, since no bump can land. Any other reply, an error reply about one key (`WRONGTYPE`, `NOPERM`, `TRYAGAIN`) or one the backend cannot use included, counts as a success, and a caller that gave up first counts as nothing. +- **pending.go** — invalidations the server did not take, kept per token key (repeats coalesce) and retried by a background loop until they land: the first one owed at once, then with backoff from 100 ms to 10 s, and while the breaker is open at each probe, which the loop starts when due, so a process that makes no lookups (ingest only) recovers as soon as one that does. `Invalidate` sends its bumps 1,000 to a round trip, as the retry does. Landing late is still correct: a fresh token orphans the pre-write entries and any fill made meanwhile. Past `PendingMax` (100,000) keys the set collapses to one tenant bump per affected tenant. `Close` makes one last attempt at the bumps still owed, past the breaker (an open one is why they are owed); what that attempt cannot deliver is lost, and the entries they would orphan are served until their TTL — the same failure as a worker stopping between an insert and its invalidation. +- **metrics.go** — the shared backend's instruments (meter `wavehouse-cache`, `backend="redis"`, no tenant attribute): `wavehouse_cache_lookups_total` by `result` (`hit`, `miss`, `stale` — filed under since-bumped tokens, `bypass` — server skipped, or held by a bump this process owes, `error`), `wavehouse_cache_op_duration_seconds` by `op` (`lookup`, `set`, `invalidate`), `wavehouse_cache_breaker_open` (1 while bypassed, including before the first connection), `wavehouse_cache_invalidations_total` by `result` (`ok`, `deferred`), `wavehouse_cache_invalidations_pending` (bumps owed: entries they would orphan may be served stale meanwhile), `wavehouse_cache_value_bytes` (stored size), `wavehouse_cache_oversize_total` and `wavehouse_cache_set_failures_total` by `reason` (`oom`, `timeout`, `other`). ### `config/` — Configuration diff --git a/internal/cache/breaker.go b/internal/cache/breaker.go index c90ab3a91..a33c94dae 100644 --- a/internal/cache/breaker.go +++ b/internal/cache/breaker.go @@ -6,9 +6,10 @@ import ( ) // breaker stops a failing cache server from costing every request its full -// timeout: after threshold consecutive failures it opens, and while open -// callers skip the server entirely. Once openFor has passed, one caller is -// told to probe; the probe's outcome closes the breaker or reopens it. +// timeout: after threshold consecutive failures it opens — at once, for a +// reply refusing the work — and while open callers skip the server +// entirely. Once openFor has passed, one caller is told to probe; the +// probe's outcome closes the breaker or reopens it. type breaker struct { threshold int openFor time.Duration @@ -40,10 +41,15 @@ func (b *breaker) allow() (ok, probe bool) { return false, true } -// success records a call the server answered, and closes the breaker. +// success records a call the server answered. Only a probe's closes an +// open breaker: any other set out before it opened, and a server that +// answers reads may still be refusing writes. func (b *breaker) success() { b.mu.Lock() defer b.mu.Unlock() + if b.open && !b.probing { + return + } b.failures, b.open, b.probing = 0, false, false } @@ -58,8 +64,31 @@ func (b *breaker) failure() { } } +// trip opens the breaker at once, for a reply that says the server cannot +// do the work: one is as conclusive as any number. +func (b *breaker) trip() { + b.mu.Lock() + defer b.mu.Unlock() + b.open, b.openedAt, b.probing = true, b.now(), false +} + func (b *breaker) isOpen() bool { b.mu.Lock() defer b.mu.Unlock() return b.open } + +// untilProbe reports how long until an open breaker is due its probe, and +// whether it is open. While a probe runs it reports openFor: the probe ends +// by closing the breaker or by opening it afresh. +func (b *breaker) untilProbe() (time.Duration, bool) { + b.mu.Lock() + defer b.mu.Unlock() + switch { + case !b.open: + return 0, false + case b.probing: + return b.openFor, true + } + return b.openedAt.Add(b.openFor).Sub(b.now()), true +} diff --git a/internal/cache/breaker_test.go b/internal/cache/breaker_test.go index e6dc2296c..3b2109d96 100644 --- a/internal/cache/breaker_test.go +++ b/internal/cache/breaker_test.go @@ -5,6 +5,7 @@ import ( "time" "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" ) type fakeClock struct{ t time.Time } @@ -57,3 +58,40 @@ func TestBreaker(t *testing.T) { assert.True(t, ok) assert.False(t, probe) } + +// A reply refusing the work opens the breaker at once, and only the probe's +// success closes it: a call that set out before it opened proves nothing. +func TestBreaker_TripAndProbeSchedule(t *testing.T) { + t.Parallel() + clock := &fakeClock{t: time.Unix(0, 0)} + b := newBreaker(100, 5*time.Second, clock.now) + _, open := b.untilProbe() + assert.False(t, open) + + b.trip() + assert.True(t, b.isOpen(), "no threshold for a refusal") + b.success() + assert.True(t, b.isOpen(), "a success that is not the probe's leaves it open") + d, open := b.untilProbe() + assert.True(t, open) + assert.Equal(t, 5*time.Second, d) + + clock.t = clock.t.Add(2 * time.Second) + d, _ = b.untilProbe() + assert.Equal(t, 3*time.Second, d) + + clock.t = clock.t.Add(3 * time.Second) + _, probe := b.allow() + require.True(t, probe) + d, _ = b.untilProbe() + assert.Equal(t, 5*time.Second, d, "while the probe runs, wait out a whole period") + b.trip() // the probe was refused too + d, _ = b.untilProbe() + assert.Equal(t, 5*time.Second, d) + + clock.t = clock.t.Add(5 * time.Second) + _, probe = b.allow() + require.True(t, probe) + b.success() + assert.False(t, b.isOpen()) +} diff --git a/internal/cache/export_test.go b/internal/cache/export_test.go index 7c392c13e..aaf712f7f 100644 --- a/internal/cache/export_test.go +++ b/internal/cache/export_test.go @@ -8,5 +8,8 @@ func Pending(r *RedisCache) int { return r.pending.len() } // Bypassed reports whether r is skipping the server. func Bypassed(r *RedisCache) bool { return r.bypassed() } +// ZeroSnapshot reports whether s files nothing. +func ZeroSnapshot(s Snapshot) bool { return s.key == "" && s.tokens == nil } + // DecodedFactor is how many times MaxValueBytes a value may decompress to. const DecodedFactor = decodedFactor diff --git a/internal/cache/metrics.go b/internal/cache/metrics.go index cf76eb7b3..34539f13b 100644 --- a/internal/cache/metrics.go +++ b/internal/cache/metrics.go @@ -15,7 +15,7 @@ const ( resultHit = "hit" resultMiss = "miss" // nothing stored resultStale = "stale" // stored under versions since bumped - resultBypass = "bypass" // server skipped: breaker open or not yet connected + resultBypass = "bypass" // server skipped: breaker open, not yet connected, or a bump this process owes would orphan the entry resultError = "error" // server failed or timed out ) @@ -40,7 +40,7 @@ func newMetrics(backend string, breakerOpen func() bool, pending func() int) (*m m := &metrics{backend: attribute.String("backend", backend)} var errs [9]error m.lookups, errs[0] = meter.Int64Counter("wavehouse_cache_lookups_total", - metric.WithDescription("Shared-cache lookups by result: hit, miss, stale (stored under since-bumped versions), bypass (server skipped), error")) + metric.WithDescription("Shared-cache lookups by result: hit, miss, stale (stored under since-bumped versions), bypass (server skipped, or held by an invalidation this process has yet to deliver), error")) m.duration, errs[1] = meter.Float64Histogram("wavehouse_cache_op_duration_seconds", metric.WithDescription("Shared-cache round-trip time by op: lookup, set, invalidate"), metric.WithUnit("s"), metric.WithExplicitBucketBoundaries(.0001, .00025, .0005, .001, .0025, .005, .01, .025, .05, .1, .25)) diff --git a/internal/cache/pending.go b/internal/cache/pending.go index c19b21786..481ab9ac7 100644 --- a/internal/cache/pending.go +++ b/internal/cache/pending.go @@ -2,6 +2,7 @@ package cache import ( "sync" + "sync/atomic" "github.com/Wave-RF/WaveHouse/internal/tenant" ) @@ -18,6 +19,7 @@ type pendingBumps struct { mu sync.Mutex keys map[string]pendingKey gen uint64 + n atomic.Int64 // len(keys), read without mu on every lookup } type pendingKey struct { @@ -37,14 +39,14 @@ func (p *pendingBumps) add(id tenant.ID, keys ...string) { for _, k := range keys { p.keys[k] = pendingKey{tenant: id, gen: p.gen} } - if len(p.keys) <= p.max { - return - } - collapsed := make(map[string]pendingKey, len(p.keys)) - for _, pk := range p.keys { - collapsed[tenantTokenKey(p.prefix, pk.tenant)] = pendingKey{tenant: pk.tenant, gen: p.gen} + if len(p.keys) > p.max { + collapsed := make(map[string]pendingKey, len(p.keys)) + for _, pk := range p.keys { + collapsed[tenantTokenKey(p.prefix, pk.tenant)] = pendingKey{tenant: pk.tenant, gen: p.gen} + } + p.keys = collapsed } - p.keys = collapsed + p.n.Store(int64(len(p.keys))) } // snapshot returns the keys owed a bump with the generation each was added @@ -69,10 +71,22 @@ func (p *pendingBumps) done(landed map[string]uint64) { delete(p.keys, k) } } + p.n.Store(int64(len(p.keys))) } -func (p *pendingBumps) len() int { +// owesAny reports whether any of keys is owed a bump. +func (p *pendingBumps) owesAny(keys []string) bool { + if p.n.Load() == 0 { + return false + } p.mu.Lock() defer p.mu.Unlock() - return len(p.keys) + for _, k := range keys { + if _, ok := p.keys[k]; ok { + return true + } + } + return false } + +func (p *pendingBumps) len() int { return int(p.n.Load()) } diff --git a/internal/cache/pending_test.go b/internal/cache/pending_test.go index cc8f331da..1c8390896 100644 --- a/internal/cache/pending_test.go +++ b/internal/cache/pending_test.go @@ -43,3 +43,29 @@ func TestPendingBumps_OverflowCollapsesToTenants(t *testing.T) { assert.Contains(t, got, "wh:{acme}:T") assert.Contains(t, got, "wh:{globex}:T") } + +// A lookup is held by a bump owed on any of its token keys, including the +// tenant token a set past its maximum collapses to. +func TestPendingBumps_OwesAny(t *testing.T) { + t.Parallel() + events := tokenKeys("wh", "acme", []Namespace{{Tenant: "acme", Table: "events", Scope: "org_1"}}) + orders := tokenKeys("wh", "acme", []Namespace{{Tenant: "acme", Table: "orders"}}) + globex := tokenKeys("wh", "globex", []Namespace{{Tenant: "globex", Table: "events"}}) + p := newPendingBumps("wh", 3) + assert.False(t, p.owesAny(events)) + + p.add("acme", bumpKeys("wh", Namespace{Tenant: "acme", Table: "events", Scope: "org_1"})...) + assert.True(t, p.owesAny(events)) + assert.False(t, p.owesAny(orders), "another table's lookups are not held") + assert.False(t, p.owesAny(globex)) + + p.add("acme", "wh:{acme}:B:a", "wh:{acme}:B:b") // past the maximum + assert.Equal(t, map[string]uint64{"wh:{acme}:T": 2}, p.snapshot()) + assert.True(t, p.owesAny(orders), "collapsed to the tenant token, which every lookup of the tenant reads") + assert.True(t, p.owesAny(tokenKeys("wh", "acme", nil))) + assert.False(t, p.owesAny(globex)) + + p.done(p.snapshot()) + assert.False(t, p.owesAny(events)) + assert.Zero(t, p.len()) +} diff --git a/internal/cache/redis.go b/internal/cache/redis.go index 01c6a6c41..ec24893d1 100644 --- a/internal/cache/redis.go +++ b/internal/cache/redis.go @@ -7,7 +7,9 @@ import ( "errors" "fmt" "log/slog" + "maps" "net" + "slices" "strings" "sync" "sync/atomic" @@ -163,8 +165,8 @@ func (c RedisConfig) clientOption() rueidis.ClientOption { } // RedisCache is a Cache shared by every process pointed at one Redis — -// or Valkey, Dragonfly, ElastiCache, MemoryDB: it uses only GET, SET, MGET -// and PING, no scripts and no client tracking. +// or Valkey, Dragonfly, ElastiCache, MemoryDB: it uses only GET, SET and +// MGET, no scripts and no client tracking. // // Versions are random tokens, one per tenant, per table and per scope, // under the tenant's hash tag; a bump sets a fresh one. A value carries the @@ -291,10 +293,12 @@ func (r *RedisCache) conn() rueidis.Client { return *cp } +// probe decides whether an open breaker closes. It writes: a server that +// answers but refuses writes (refusesWork) would take no bump either. func (r *RedisCache) probe(c rueidis.Client) { ctx, cancel := context.WithTimeout(r.ctx, r.cfg.Timeout) defer cancel() - r.record(r.ctx, c.Do(ctx, c.B().Ping().Build()).Error()) + r.record(r.ctx, c.Do(ctx, c.B().Set().Key(r.cfg.KeyPrefix+":probe").Value("1").Ex(time.Minute).Build()).Error()) if r.breaker.isOpen() { return } @@ -308,14 +312,23 @@ func (r *RedisCache) bypassed() bool { } // record feeds an operation's outcome to the breaker. A reply from the -// server, even an error reply or one this code cannot use, shows it is up; a caller that gave up first -// shows nothing about it. +// server, even an error reply or one this code cannot use, shows it is up — +// unless it refuses the work outright, which opens the breaker at once. A +// caller that gave up first shows nothing about it. func (r *RedisCache) record(parent context.Context, err error) { if err == nil || rueidis.IsRedisNil(err) { r.breaker.success() return } - if _, ok := rueidis.IsRedisErr(err); ok || errors.Is(err, errMalformedReply) { + if re, ok := rueidis.IsRedisErr(err); ok { + if refusesWork(re.Error()) { + r.breaker.trip() + } else { + r.breaker.success() + } + return + } + if errors.Is(err, errMalformedReply) { r.breaker.success() return } @@ -325,6 +338,23 @@ func (r *RedisCache) record(parent context.Context, err error) { r.breaker.failure() } +// refusesWork reports whether an error reply says the server takes no +// writes from anyone right now, so no bump can land: a replica (READONLY, +// or MASTERDOWN, which refuses reads too), memory full under noeviction +// (OOM), writes stopped by min-replicas-to-write (NOREPLICAS) or a failed +// snapshot (MISCONF), a dataset still loading (LOADING), a script holding +// the server (BUSY), a cluster not serving the slot (CLUSTERDOWN). Replies +// about one key or one moment — WRONGTYPE, NOPERM, TRYAGAIN during a slot +// migration — are not. +func refusesWork(msg string) bool { + code, _, _ := strings.Cut(msg, " ") + switch code { + case "READONLY", "MASTERDOWN", "OOM", "NOREPLICAS", "MISCONF", "LOADING", "BUSY", "CLUSTERDOWN": + return true + } + return false +} + // Lookup reads the tokens deps fold and the entry for sha in one pipelined // round trip. A token that does not exist yet is created, never read as a // value, so its first use is a miss. @@ -338,6 +368,16 @@ func (r *RedisCache) Lookup(ctx context.Context, id tenant.ID, sha string, deps if len(keys) > maxTokenKeys { return Entry{}, Snapshot{}, fmt.Errorf("cache: %d dependencies is more than a value can record", len(deps)) } + // A bump this process owes would orphan what a lookup reading its key + // finds, so that lookup is a bypass: no hit, and a zero snapshot, so no + // fill either. A lookup's token keys are exactly those whose bumps orphan + // its entry, and include the tenant token a set past PendingMax collapses + // to. Other processes cannot know what this one owes, and serve those + // entries until the bump lands. + if r.pending.owesAny(keys) { + r.metrics.lookup(resultBypass) + return Entry{}, Snapshot{}, nil + } c := r.conn() if c == nil { r.metrics.lookup(resultBypass) @@ -540,20 +580,19 @@ func (r *RedisCache) bump(ctx context.Context, owner map[string]tenant.ID) error return fmt.Errorf("%w: %d invalidations deferred", errBypassed, len(owner)) } defer r.metrics.op("invalidate", time.Now()) - opCtx, cancel := context.WithTimeout(ctx, r.cfg.Timeout) - defer cancel() - keys := make([]string, 0, len(owner)) - cmds := make(rueidis.Commands, 0, len(owner)) - for k := range owner { - keys = append(keys, k) - cmds = append(cmds, r.bumpCmd(c, k)) - } failed := map[string]tenant.ID{} var firstErr error - for i, rr := range c.DoMulti(opCtx, cmds...) { - if err := rr.Error(); err != nil { - failed[keys[i]] = owner[keys[i]] - firstErr = cmpOr(firstErr, err) + // In batches, so a wide fan-out is several round trips each within the + // timeout; past a failure the rest are deferred unsent. + for batch := range slices.Chunk(slices.Collect(maps.Keys(owner)), drainBatch) { + var landed []bool + if firstErr == nil { + landed, firstErr = r.sendBumps(ctx, c, batch) + } + for i, k := range batch { + if landed == nil || !landed[i] { + failed[k] = owner[k] + } } } r.record(ctx, firstErr) @@ -570,11 +609,36 @@ func (r *RedisCache) bumpCmd(c rueidis.Client, key string) rueidis.Completed { return c.B().Set().Key(key).Value(rueidis.BinaryString(tok)).Ex(jitter(r.cfg.VersionTTL, tok)).Build() } +// sendBumps sets a fresh token under each key in one pipeline, within the +// op timeout, reporting which landed and the first error. +func (r *RedisCache) sendBumps(ctx context.Context, c rueidis.Client, keys []string) (landed []bool, firstErr error) { + cmds := make(rueidis.Commands, 0, len(keys)) + for _, k := range keys { + cmds = append(cmds, r.bumpCmd(c, k)) + } + opCtx, cancel := context.WithTimeout(ctx, r.cfg.Timeout) + defer cancel() + landed = make([]bool, len(keys)) + for i, rr := range c.DoMulti(opCtx, cmds...) { + err := rr.Error() + landed[i] = err == nil + firstErr = cmpOr(firstErr, err) + } + return landed, firstErr +} + func (r *RedisCache) deferBumps(owner map[string]tenant.ID) { + first := r.pending.len() == 0 for k, id := range owner { r.pending.add(id, k) } r.metrics.invalidated("deferred", len(owner)) + // The first bump owed wakes the drain now, not at its next tick: it is + // retried at once, or, past an open breaker, when the probe is due. + // Later ones join the retry already backing off. + if first { + r.nudge() + } } // nudge wakes the drain loop now, rather than at its next tick. @@ -600,7 +664,7 @@ func (r *RedisCache) drainLoop() { } wait := drainIdle if r.breaker.isOpen() { - r.conn() // starts the probe when due, so an idle process recovers too + r.conn() // starts the probe when due } if r.pending.len() > 0 { if r.drain(r.ctx, r.conn) { @@ -609,6 +673,12 @@ func (r *RedisCache) drainLoop() { wait, backoff = backoff, min(backoff*2, drainMaxBackoff) } } + // Lookups start the probe that closes an open breaker; a process + // with none (ingest only) has this loop, which wakes when the probe + // is due rather than at the drain's backoff, so it recovers as soon. + if d, open := r.breaker.untilProbe(); open { + wait = min(wait, d) + } timer.Reset(wait) } } @@ -621,32 +691,22 @@ func (r *RedisCache) drain(ctx context.Context, conn func() rueidis.Client) bool for k := range owed { keys = append(keys, k) } - for len(keys) > 0 { - batch := keys[:min(drainBatch, len(keys))] - keys = keys[len(batch):] + for batch := range slices.Chunk(keys, drainBatch) { c := conn() if c == nil { return false } - cmds := make(rueidis.Commands, 0, len(batch)) - for _, k := range batch { - cmds = append(cmds, r.bumpCmd(c, k)) - } - opCtx, cancel := context.WithTimeout(ctx, r.cfg.Timeout) + sent, err := r.sendBumps(ctx, c, batch) landed := map[string]uint64{} - var firstErr error - for i, rr := range c.DoMulti(opCtx, cmds...) { - if err := rr.Error(); err != nil { - firstErr = cmpOr(firstErr, err) - continue + for i, k := range batch { + if sent[i] { + landed[k] = owed[k] } - landed[batch[i]] = owed[batch[i]] } - cancel() - r.record(ctx, firstErr) + r.record(ctx, err) r.pending.done(landed) r.metrics.invalidated("ok", len(landed)) - if firstErr != nil { + if err != nil { return false } } diff --git a/internal/cache/redis_codec.go b/internal/cache/redis_codec.go index 96cddb452..bb9a853ef 100644 --- a/internal/cache/redis_codec.go +++ b/internal/cache/redis_codec.go @@ -26,6 +26,8 @@ const tokenLen = 8 // stored-size limit, refusing a zip bomb planted in a shared server. const decodedFactor = 8 +const encoderWindow = 1 << 20 + // Value layout: format, flags, expires-at (unix ms), token count, tokens, // payload. Big-endian. const ( @@ -122,6 +124,12 @@ func valueKey(prefix string, id tenant.ID, sha string, deps []Namespace) string // codec compresses and frames values. Its zstd encoder and decoder are safe // for concurrent EncodeAll/DecodeAll. +// +// The encoder keeps one window-sized history per concurrent caller for the +// life of the process; 1 MiB, not SpeedFastest's 4, cuts that about +// threefold at no measurable cost in speed or ratio on row payloads. The +// decoder takes any window up to maxDecoded, so a value written with a +// larger one still reads. type codec struct { enc *zstd.Encoder dec *zstd.Decoder @@ -130,7 +138,7 @@ type codec struct { } func newCodec(compressMin, maxDecoded int) (*codec, error) { - enc, err := zstd.NewWriter(nil, zstd.WithEncoderLevel(zstd.SpeedFastest)) + enc, err := zstd.NewWriter(nil, zstd.WithEncoderLevel(zstd.SpeedFastest), zstd.WithWindowSize(encoderWindow)) if err != nil { return nil, err } diff --git a/internal/cache/redis_codec_test.go b/internal/cache/redis_codec_test.go index c956a6097..a61d84dd4 100644 --- a/internal/cache/redis_codec_test.go +++ b/internal/cache/redis_codec_test.go @@ -6,6 +6,7 @@ import ( "testing" "time" + "github.com/klauspost/compress/zstd" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) @@ -192,6 +193,40 @@ func TestCodec_RefusesBadValues(t *testing.T) { } } +// The encoder's window bounds the history it keeps per concurrent caller, +// and a value an encoder with a wider one wrote (an earlier build's) still +// decodes. +func TestCodec_EncoderWindow(t *testing.T) { + t.Parallel() + window := func(t *testing.T, b []byte) uint64 { + t.Helper() + var h zstd.Header + require.NoError(t, h.Decode(b[headerLen:])) + if h.SingleSegment { + return h.FrameContentSize + } + return h.WindowSize + } + c := newTestCodec(t, 1, 8<<20) + wide, err := zstd.NewWriter(nil, zstd.WithEncoderLevel(zstd.SpeedFastest)) + require.NoError(t, err) + t.Cleanup(func() { _ = wide.Close() }) + earlier := &codec{enc: wide, compressMin: 1} + + for _, n := range []int{2 << 20, 6 << 20} { + payload := bytes.Repeat([]byte(`{"user_id":"u-1","event":"click","value":42.5},`), n/48) + b := c.encode(nil, time.Now(), payload) + require.Equal(t, byte(flagZstd), b[1]) + assert.LessOrEqual(t, window(t, b), uint64(encoderWindow), n) + + b = earlier.encode(nil, time.Now(), payload) + require.Greater(t, window(t, b), uint64(encoderWindow), n) + _, _, got, err := c.decode(b) + require.NoError(t, err, n) + assert.Equal(t, payload, got, n) + } +} + func TestJitter(t *testing.T) { t.Parallel() d := 100 * time.Second diff --git a/internal/cache/redis_integration_test.go b/internal/cache/redis_integration_test.go index 422e372e7..882ed6a52 100644 --- a/internal/cache/redis_integration_test.go +++ b/internal/cache/redis_integration_test.go @@ -7,6 +7,7 @@ import ( "context" "fmt" "net" + "slices" "strconv" "strings" "sync/atomic" @@ -495,3 +496,166 @@ func TestRedis_CloseDeliversPastAnOpenBreaker(t *testing.T) { require.NoError(t, err) assert.Nil(t, e.Value, "the bump Close delivered orphans the fill") } + +// command runs cmd on s through c, for the test to reconfigure the server. +func command(t *testing.T, c rueidis.Client, cmd ...string) { + t.Helper() + require.NoError(t, c.Do(context.Background(), c.B().Arbitrary(cmd[0]).Args(cmd[1:]...).Build()).Error(), "%q", cmd) +} + +// A bump this process owes holds the lookups it would orphan — a bypass +// that files nothing — until it lands, with the breaker closed and every +// other lookup served. The ACL lets the process read the tokens but not +// replace them, so the bump stays owed until the test grants the write. +func TestRedis_OwedBumpHoldsItsLookups(t *testing.T) { + t.Parallel() + ctx := context.Background() + s := startRedis(t) + r := raw(t, s) + prefix := uniquePrefix() + command(t, r, "ACL", "SETUSER", "limited", "on", ">pw", "+@all", "%R~*", "%W~"+prefix+":q:*") + + events := []cache.Namespace{{Tenant: "acme", Table: "events"}} + orders := []cache.Namespace{{Tenant: "acme", Table: "orders"}} + seed := open(t, s, prefix) + for _, deps := range [][]cache.Namespace{events, orders, nil} { + _, snap, err := seed.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + require.NoError(t, seed.Set(ctx, snap, []byte("pre-write rows"), time.Minute)) + } + a := open(t, s, prefix, func(c *cache.RedisConfig) { c.Username, c.Password, c.BreakerThreshold = "limited", "pw", 1000 }) + lookup := func(deps []cache.Namespace) (string, cache.Snapshot) { + t.Helper() + e, snap, err := a.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + return string(e.Value), snap + } + got, _ := lookup(events) + require.Equal(t, "pre-write rows", got) + + _, err := a.Invalidate(ctx, events) + require.ErrorContains(t, err, "NOPERM") + require.Equal(t, 1, cache.Pending(a)) + require.False(t, cache.Bypassed(a), "a refused key is not a refusing server") + + got, snap := lookup(events) + assert.Empty(t, got, "the owed bump would orphan it") + assert.True(t, cache.ZeroSnapshot(snap), "and a fill under the token it replaces would be orphaned too") + for _, deps := range [][]cache.Namespace{orders, nil} { + got, _ = lookup(deps) + assert.Equal(t, "pre-write rows", got, "%v: lookups the bump does not orphan are served", deps) + } + + command(t, r, "ACL", "SETUSER", "limited", "~*") + deadline := time.Now().Add(10 * time.Second) + for cache.Pending(a) > 0 { + require.True(t, time.Now().Before(deadline), "the bump lands once the server takes it") + got, _ = lookup(events) + require.NotEqual(t, "pre-write rows", got, "served before the owed bump landed") + time.Sleep(time.Millisecond) + } + got, _ = lookup(events) + assert.Empty(t, got, "the landed bump orphaned the pre-write rows") +} + +// A server that answers but refuses writes — a primary demoted to a replica +// (READONLY), memory full under noeviction (OOM) — takes no bump, so its +// first refusal opens the breaker whatever the threshold, and the bump stays +// owed. The probe writes, so it keeps the breaker open until the server +// takes writes again; then the owed bump lands before anything it would +// orphan is served, and a process that only invalidates, whose drain loop +// is all that probes for it, recovers at the probe's cadence too. +func TestRedis_RefusedWrites(t *testing.T) { + t.Parallel() + for _, tt := range []struct { + name string + refuse, restore [][]string + reply string + }{ + { + "demoted to a replica", + [][]string{{"REPLICAOF", "127.0.0.1", "1"}}, // nothing listens: it stays a replica, serving reads + [][]string{{"REPLICAOF", "NO", "ONE"}}, + "READONLY", + }, + { + "full under noeviction", + [][]string{{"CONFIG", "SET", "maxmemory-policy", "noeviction"}, {"CONFIG", "SET", "maxmemory", "1"}}, + [][]string{{"CONFIG", "SET", "maxmemory", "0"}}, + "OOM", + }, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + ctx := context.Background() + s := startRedis(t) + r := raw(t, s) + prefix := uniquePrefix() + const openFor = 200 * time.Millisecond + tune := func(c *cache.RedisConfig) { c.BreakerThreshold, c.BreakerOpenFor = 1000, openFor } + a := open(t, s, prefix, tune) + ingest := open(t, s, prefix, tune) + reader := open(t, s, prefix, tune) + events := []cache.Namespace{{Tenant: "acme", Table: "events"}} + orders := []cache.Namespace{{Tenant: "acme", Table: "orders"}} + for _, deps := range [][]cache.Namespace{events, orders} { + _, snap, err := a.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + require.NoError(t, a.Set(ctx, snap, []byte("pre-write rows"), time.Minute)) + } + lookup := func(c *cache.RedisCache, deps []cache.Namespace) string { + t.Helper() + e, _, err := c.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + return string(e.Value) + } + + for _, cmd := range tt.refuse { + command(t, r, cmd...) + } + _, err := a.Invalidate(ctx, events) + require.ErrorContains(t, err, tt.reply) + assert.Equal(t, 1, cache.Pending(a)) + assert.True(t, cache.Bypassed(a), "the first refusal opens the breaker") + wide := slices.Clone(orders) // past one batch: the rest go unsent after the refusal, and drain in two + for i := range 1500 { + wide = append(wide, cache.Namespace{Tenant: "acme", Table: fmt.Sprintf("t%d", i)}) + } + _, err = ingest.Invalidate(ctx, wide) + require.ErrorContains(t, err, tt.reply) + assert.True(t, cache.Bypassed(ingest)) + assert.Equal(t, len(wide), cache.Pending(ingest)) + _, snap, err := reader.Lookup(ctx, "acme", "another query", orders) + require.NoError(t, err) + require.ErrorContains(t, reader.Set(ctx, snap, []byte("rows"), time.Minute), tt.reply) + assert.True(t, cache.Bypassed(reader), "a refused fill opens the breaker too") + + // Long enough for many probes to be refused, and for a drain + // backing off unchecked to be seconds from its next attempt. + time.Sleep(3500 * time.Millisecond) + assert.True(t, cache.Bypassed(reader), "owing nothing, it is held open by probes that write and are refused") + assert.True(t, cache.Bypassed(a)) + assert.Empty(t, lookup(a, events)) + assert.Equal(t, 1, cache.Pending(a)) + assert.Equal(t, len(wide), cache.Pending(ingest)) + + for _, cmd := range tt.restore { + command(t, r, cmd...) + } + restored := time.Now() + for cache.Pending(a) > 0 { + require.Less(t, time.Since(restored), 10*time.Second, "the bump lands once the server takes writes") + require.NotEqual(t, "pre-write rows", lookup(a, events), "served before the owed bump landed") + time.Sleep(time.Millisecond) + } + require.Eventually(t, func() bool { return cache.Pending(ingest) == 0 }, 10*time.Second, 5*time.Millisecond) + assert.Less(t, time.Since(restored), openFor+time.Second, "an ingest-only process probes when due, not at the drain's backoff") + require.Eventually(t, func() bool { return !cache.Bypassed(reader) }, 10*time.Second, 5*time.Millisecond) + + fresh := open(t, s, prefix) + for _, deps := range [][]cache.Namespace{events, orders} { + assert.Empty(t, lookup(fresh, deps), "%v: the landed bumps orphaned the pre-write rows", deps) + } + }) + } +} diff --git a/internal/cache/redis_test.go b/internal/cache/redis_test.go index 9aafb4f12..3fe67a2f5 100644 --- a/internal/cache/redis_test.go +++ b/internal/cache/redis_test.go @@ -15,6 +15,8 @@ import ( "go.opentelemetry.io/otel/attribute" sdkmetric "go.opentelemetry.io/otel/sdk/metric" "go.opentelemetry.io/otel/sdk/metric/metricdata" + + "github.com/Wave-RF/WaveHouse/internal/tenant" ) func TestRedisConfig_Validation(t *testing.T) { @@ -149,6 +151,49 @@ func TestRedis_Record(t *testing.T) { assert.True(t, r.breaker.isOpen()) } +func TestRefusesWork(t *testing.T) { + t.Parallel() + for _, msg := range []string{ + "READONLY You can't write against a read only replica.", + "OOM command not allowed when used memory > 'maxmemory'.", + "MASTERDOWN Link with MASTER is down and replica-serve-stale-data is set to 'no'.", + "NOREPLICAS Not enough good replicas to write.", + "MISCONF Errors writing to the AOF file: No space left on device", + "LOADING Redis is loading the dataset in memory", + "BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE.", + "CLUSTERDOWN The cluster is down", + } { + assert.True(t, refusesWork(msg), msg) + } + for _, msg := range []string{ + "WRONGTYPE Operation against a key holding the wrong kind of value", + "NOPERM this user has no permissions to access one of the keys used as arguments", + "TRYAGAIN Multiple keys request during rehashing of slot", + "BUSYKEY Target key name already exists.", + "ERR unknown command", + "", + } { + assert.False(t, refusesWork(msg), msg) + } +} + +// The first bump owed wakes the drain at once; later ones ride the retry +// already under way rather than resetting its backoff. +func TestRedis_FirstDeferralWakesTheDrain(t *testing.T) { + t.Parallel() + r := &RedisCache{breaker: newBreaker(1, time.Hour, time.Now), pending: newPendingBumps("wh", 10), wake: make(chan struct{}, 1)} + var err error + r.metrics, err = newMetrics("redis", r.bypassed, r.pending.len) + require.NoError(t, err) + t.Cleanup(r.metrics.close) + + r.deferBumps(map[string]tenant.ID{"wh:{acme}:B:events": "acme"}) + assert.Len(t, r.wake, 1) + <-r.wake + r.deferBumps(map[string]tenant.ID{"wh:{acme}:B:orders": "acme"}) + assert.Empty(t, r.wake) +} + func cancelledCtx() context.Context { ctx, cancel := context.WithCancel(context.Background()) cancel() From c6df13ad35f5311c6cb8f0fd3010c339ac5f111a Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 05:21:09 -0400 Subject: [PATCH 24/45] fix(cache): bound a cluster client's topology read like a dial rueidis reads the cluster topology after the handshake under ConnWriteTimeout, 10 s when unset. A node that answered the handshake and then went quiet held NewClient for 10 s (measured), and with it NewRedis's first dial and a Close waiting on the dial loop; a refused or silent node already failed within DialTimeout. ConnWriteTimeout is now the larger of DialTimeout and the op Timeout: bounded like a dial, and never cutting a connection an operation may still wait on. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_017aS7rLrH1RKkUMem7X4ckd --- docs/src/content/docs/architecture.md | 2 +- internal/cache/export_test.go | 8 ++++++ internal/cache/redis.go | 5 ++++ internal/cache/redis_integration_test.go | 34 ++++++++++++++++++++++++ internal/cache/redis_test.go | 4 +++ 5 files changed, 52 insertions(+), 1 deletion(-) diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index f124f8d7c..d9c08f81e 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -113,7 +113,7 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ - **cache.go** — `Cache` interface: `Lookup`, `Set`, `Invalidate`, `InvalidateTenant`, `Close`, plus `QueryTimeToTTL`, which sets a result's TTL from how long its query took (10 s floor, 1 h ceiling). Every entry is one tenant's: `Lookup` takes the tenant, the caller's query key — `:query:`, built by the two cached handlers in `api/` (`queryCacheKey`, with the tenant read off the request's store — `settings.Store.Tenant`), which use it as their [singleflight](https://pkg.go.dev/golang.org/x/sync/singleflight) key too — and the `Namespace`s the result depends on — one for a structured query, none yet for a pipe (a pipe's table dependencies are [#343](https://github.com/Wave-RF/WaveHouse/pull/343)) — each of that tenant (another tenant's is `ErrForeignDependency`) and naming a table and scope by their raw names, which the cache escapes where it builds a key. `Lookup` returns the `Entry` (a nil value is a miss) and a `Snapshot` of the versions it read; on a miss the handler runs the query and passes that snapshot to `Set`, so a result is filed under the versions read *before* its query ran, and a write that lands while it runs orphans the fill rather than re-homing pre-write rows under the post-write versions ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)). The singleflight leader's snapshot is the one used. `Set` returns an error only when the backend failed; a value the cache declines — larger than it keeps, a non-positive TTL, a zero snapshot — is not one. What a hit, a miss and a bump mean is pinned by the conformance suite every backend runs, `internal/testutil/cachetest`. - **local.go** — `LocalCache`, the in-process L1 on [Ristretto](https://github.com/dgraph-io/ristretto): one pool shared by every tenant (a heavier tenant holds more of it), sized by the boot config's `cache.l1_max_cost`. - **version_manager.go** — `VersionManager`, the invalidation index behind `Invalidate` and `InvalidateTenant`: a namespace key is `..
.
.`, its fields joined with `keyenc.Join` so a dot in a table or scope name is escaped rather than read as a separator, and an entry's key, `|.||…` with the caller's query key escaped whole (its `:` become `%3A`), folds in the tenant's version and each dependency's namespace key and namespace version, so bumping a table (a scopeless write) or one scope — scope is reserved and empty today, so every write is the whole-table bump — orphans every dependent entry without touching the pool. The tenant leads every key ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8): the same table under two tenants is two namespaces, so a bump through `Invalidate` under one tenant never touches — and a read under one tenant is never served — the other's results, and the flat directory's single tenant simply carries the `0` prefix. `BumpTenant` (behind `InvalidateTenant`) advances the tenant version that leads every namespace key of one tenant, orphaning its every namespace and every cached query in one step (the tenant version is folded into every query key, so a pipe result with no dependencies is orphaned too) — a table no bump ever keyed included, which an enumeration of the index would miss — for a tenant back on a pool after an absence from the fan-out, or moved to another address or database (story 6). The index is per tenant; the cross-tenant invalidation an insert into a shared table needs is not the index's but the wiring's: `internal/app` hands the ingest worker a cache (`sharedTables`) that repeats each bump under every tenant on the same ClickHouse address and database. -- **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — only `GET`, `SET` and `MGET`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot; the table and scope are escaped with `keyenc` after the fixed prefix, so a `:` in a name is never read as the separator. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, the hash over their escaped forms, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, and a token key holding anything but a token (a string of another length, a list, a hash) is replaced, so a token lost to eviction, expiry or a restart is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. While this process owes a bump on any of a lookup's tokens, that lookup is a bypass that files nothing: the bump would orphan whatever it found (other processes, which cannot know, serve those entries until it lands). `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing. Nothing selects this backend yet — the boot config and wiring arrive with [#613](https://github.com/Wave-RF/WaveHouse/issues/613)'s E4. +- **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — only `GET`, `SET` and `MGET`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot; the table and scope are escaped with `keyenc` after the fixed prefix, so a `:` in a name is never read as the separator. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, the hash over their escaped forms, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, and a token key holding anything but a token (a string of another length, a list, a hash) is replaced, so a token lost to eviction, expiry or a restart is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. While this process owes a bump on any of a lookup's tokens, that lookup is a bypass that files nothing: the bump would orphan whatever it found (other processes, which cannot know, serve those entries until it lands). `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing, each attempt bounded by `DialTimeout` (1 s), and a cluster client's topology read after the handshake by the larger of it and `Timeout`, which is also how long a connection waits on a silent server before it is redialed. Nothing selects this backend yet — the boot config and wiring arrive with [#613](https://github.com/Wave-RF/WaveHouse/issues/613)'s E4. - **redis_codec.go** — the key schema (`tokenKeys`, `bumpKeys`, `valueKey`, which take a `Namespace`'s raw names and escape them) and the value frame: format, flags, expiry, the token list, the payload, zstd-compressed from `CompressMinBytes` when that is smaller. A stored value is capped at `MaxValueBytes` and a decoded one at eight times that, which refuses a zip bomb planted in a shared server; an unknown format, as a newer process writes during a rolling upgrade, is a miss. - **breaker.go** — the circuit breaker: `BreakerThreshold` (5) consecutive failures open it, and while open every operation skips the server at once; after `BreakerOpenFor` (5 s) one background write (`SET :probe`) decides whether it closes, and only that probe's success closes it. A transport failure or a timeout counts against the server; an error reply saying it takes no writes right now — `READONLY` (a demoted primary), `MASTERDOWN`, `OOM` (full under `maxmemory-policy noeviction`, Redis's default, so give the cache an evicting policy), `NOREPLICAS`, `MISCONF`, `LOADING`, `BUSY`, `CLUSTERDOWN` — opens it at once, whatever the threshold, since no bump can land. Any other reply, an error reply about one key (`WRONGTYPE`, `NOPERM`, `TRYAGAIN`) or one the backend cannot use included, counts as a success, and a caller that gave up first counts as nothing. - **pending.go** — invalidations the server did not take, kept per token key (repeats coalesce) and retried by a background loop until they land: the first one owed at once, then with backoff from 100 ms to 10 s, and while the breaker is open at each probe, which the loop starts when due, so a process that makes no lookups (ingest only) recovers as soon as one that does. `Invalidate` sends its bumps 1,000 to a round trip, as the retry does. Landing late is still correct: a fresh token orphans the pre-write entries and any fill made meanwhile. Past `PendingMax` (100,000) keys the set collapses to one tenant bump per affected tenant. `Close` makes one last attempt at the bumps still owed, past the breaker (an open one is why they are owed); what that attempt cannot deliver is lost, and the entries they would orphan are served until their TTL — the same failure as a worker stopping between an insert and its invalidation. diff --git a/internal/cache/export_test.go b/internal/cache/export_test.go index aaf712f7f..9501f6be5 100644 --- a/internal/cache/export_test.go +++ b/internal/cache/export_test.go @@ -1,7 +1,15 @@ package cache +import "github.com/redis/rueidis" + // Hooks for the integration tests in package cache_test. +// ClientOption is the rueidis option a RedisCache built from c dials with. +func ClientOption(c RedisConfig) (rueidis.ClientOption, error) { + c, err := c.withDefaults() + return c.clientOption(), err +} + // Pending reports how many token bumps r still owes the server. func Pending(r *RedisCache) int { return r.pending.len() } diff --git a/internal/cache/redis.go b/internal/cache/redis.go index ec24893d1..b8bd5e286 100644 --- a/internal/cache/redis.go +++ b/internal/cache/redis.go @@ -158,6 +158,11 @@ func (c RedisConfig) clientOption() rueidis.ClientOption { DisableCache: true, // no client-side caching until the near-cache (E5) ForceSingleClient: c.Mode == RedisStandalone, } + // How long a connection waits on a silent server, 10 s unset. A cluster + // client reads the topology under it after the handshake, which boot and + // Close would wait out; never under Timeout, so no connection is cut + // while an operation may still wait on it. + opt.ConnWriteTimeout = max(c.DialTimeout, c.Timeout) if c.Mode == RedisSentinel { opt.Sentinel = rueidis.SentinelOption{MasterSet: c.SentinelMaster, TLSConfig: c.TLS, Dialer: opt.Dialer} } diff --git a/internal/cache/redis_integration_test.go b/internal/cache/redis_integration_test.go index 882ed6a52..a6f0aae23 100644 --- a/internal/cache/redis_integration_test.go +++ b/internal/cache/redis_integration_test.go @@ -5,6 +5,7 @@ package cache_test import ( "bytes" "context" + "crypto/tls" "fmt" "net" "slices" @@ -497,6 +498,39 @@ func TestRedis_CloseDeliversPastAnOpenBreaker(t *testing.T) { assert.Nil(t, e.Value, "the bump Close delivered orphans the fill") } +// A cluster client reads the topology after the handshake; a node that +// answers the handshake and then goes quiet held that read, and so boot and +// Close, for rueidis's 10 s default. It is bounded like a dial now. Any +// server serves: what matters is that the read goes unanswered. +func TestRedis_ClusterTopologyReadIsBounded(t *testing.T) { + t.Parallel() + s := startRedis(t) + opt, err := cache.ClientOption(cache.RedisConfig{Addrs: []string{s.addr}, Mode: cache.RedisCluster, DialTimeout: 300 * time.Millisecond}) + require.NoError(t, err) + opt.DialCtxFn = func(ctx context.Context, addr string, d *net.Dialer, _ *tls.Config) (net.Conn, error) { + c, err := d.DialContext(ctx, "tcp", addr) + return unanswered{c}, err + } + start := time.Now() + c, err := rueidis.NewClient(opt) + if err == nil { + c.Close() + } + require.Error(t, err, "nothing answered the topology read") + assert.Less(t, time.Since(start), 3*time.Second) +} + +// unanswered drops CLUSTER commands unsent, as a node gone quiet would +// leave them unanswered. +type unanswered struct{ net.Conn } + +func (c unanswered) Write(b []byte) (int, error) { + if bytes.Contains(b, []byte("CLUSTER")) { + return len(b), nil + } + return c.Conn.Write(b) +} + // command runs cmd on s through c, for the test to reconfigure the server. func command(t *testing.T, c rueidis.Client, cmd ...string) { t.Helper() diff --git a/internal/cache/redis_test.go b/internal/cache/redis_test.go index 3fe67a2f5..17da785a6 100644 --- a/internal/cache/redis_test.go +++ b/internal/cache/redis_test.go @@ -60,6 +60,10 @@ func TestRedisConfig_Defaults(t *testing.T) { assert.Equal(t, DefaultRedisVersionTTL, c.VersionTTL) assert.Zero(t, c.CompressMinBytes, "0 means never compress, not the default") assert.True(t, c.clientOption().ForceSingleClient) + assert.Equal(t, DefaultRedisDialTimeout, c.clientOption().ConnWriteTimeout) + slow := c + slow.Timeout = 5 * time.Second + assert.Equal(t, slow.Timeout, slow.clientOption().ConnWriteTimeout, "never under the op timeout") c.Mode, c.SentinelMaster = RedisSentinel, "mymaster" assert.Equal(t, "mymaster", c.clientOption().Sentinel.MasterSet) From 3c19a857298460889966245076d8eae4eb5197fb Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 05:24:43 -0400 Subject: [PATCH 25/45] test(cache): count Redis values so the zero-snapshot case runs The conformance suite's zero-snapshot case needs Options.Entries and skipped for the Redis backend. It now counts the values under the case's own key prefix, scanning every node, plus the empty key a fill under the zero snapshot would land on, so the case runs on Redis, Valkey, Dragonfly and the Redis Cluster node. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_017aS7rLrH1RKkUMem7X4ckd --- internal/cache/export_test.go | 3 +++ internal/cache/redis_integration_test.go | 28 ++++++++++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/internal/cache/export_test.go b/internal/cache/export_test.go index 8b4a503c8..0ef1f0b61 100644 --- a/internal/cache/export_test.go +++ b/internal/cache/export_test.go @@ -16,6 +16,9 @@ func Pending(r *RedisCache) int { return r.pending.len() } // Bypassed reports whether r is skipping the server. func Bypassed(r *RedisCache) bool { return r.bypassed() } +// KeyPrefix is the prefix every key r writes leads with. +func KeyPrefix(r *RedisCache) string { return r.cfg.KeyPrefix } + // ZeroSnapshot reports whether s files nothing. func ZeroSnapshot(s Snapshot) bool { return s.key == "" && s.tokens == nil } diff --git a/internal/cache/redis_integration_test.go b/internal/cache/redis_integration_test.go index a6f0aae23..68c3e71a2 100644 --- a/internal/cache/redis_integration_test.go +++ b/internal/cache/redis_integration_test.go @@ -222,6 +222,7 @@ func TestRedis_Conformance(t *testing.T) { p := uniquePrefix() return open(t, s, p), open(t, s, p) }, + Entries: valueCount(raw(t, s)), }) t.Run("cross-tenant invalidation spans slots", func(t *testing.T) { t.Parallel() @@ -235,6 +236,33 @@ func TestRedis_Conformance(t *testing.T) { } } +// valueCount counts the values under a cache's own key prefix, scanning +// every node r knows (the test cluster's one node has no replica), and the +// empty key, where a fill under the zero snapshot's empty key would land; +// -1 is a failed read. +func valueCount(r rueidis.Client) func(cache.Cache) int { + return func(c cache.Cache) int { + match := cache.KeyPrefix(c.(*cache.RedisCache)) + ":q:*" + n, err := r.Do(context.Background(), r.B().Exists().Key("").Build()).AsInt64() + if err != nil { + return -1 + } + for _, node := range r.Nodes() { + for cursor := uint64(0); ; { + e, err := node.Do(context.Background(), node.B().Scan().Cursor(cursor).Match(match).Count(1000).Build()).AsScanEntry() + if err != nil { + return -1 + } + if n += int64(len(e.Elements)); e.Cursor == 0 { + break + } + cursor = e.Cursor + } + } + return int(n) + } +} + // The ingest worker's shared-tables fan-out bumps one table under several // tenants in one call: tokens in as many slots, one pipeline. func testCrossTenantInvalidate(t *testing.T, c *cache.RedisCache) { From c427697ed6a716cccadd678172c64d0d1ad33e3b Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 05:45:45 -0400 Subject: [PATCH 26/45] fix(cache): replace connections so a failover reaches the new primary After a failover behind a stable address (a DNS name, a proxy), the connections a process holds stay on the demoted node, which answers but refuses writes, and rueidis does not redial on READONLY. The refusal opened the breaker, and the write probe kept it open for the life of the process, with its owed bumps never delivered. Each connection is now replaced after a minute (rueidis retries what was in flight), which re-resolves the address, so the process reaches the new primary and delivers what it owes within about that long. A test fails over from a primary to its replica behind a forwarder that moves new connections only. Review fixes: the CHANGELOG no longer says the cache.backend switch arrives with E4 (it exists and accepts only local), AGENTS.md scopes "every key leads with the tenant" to LocalCache, Invalidate's godoc says batches, and the refused-writes test holds the refusal until an unchecked drain would sit at its 10 s cap, bounding recovery at the probe period plus 3 s. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_017aS7rLrH1RKkUMem7X4ckd --- AGENTS.md | 2 +- CHANGELOG.md | 2 +- docs/src/content/docs/architecture.md | 2 +- internal/cache/export_test.go | 10 ++- internal/cache/redis.go | 13 +++- internal/cache/redis_integration_test.go | 99 +++++++++++++++++++++++- internal/cache/redis_test.go | 1 + 7 files changed, 121 insertions(+), 8 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 9f7d9bcea..c4ee04283 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -31,7 +31,7 @@ Twenty internal packages under `internal/` (plus `internal/testutil/` for shared - **`api/`** — Chi HTTP router, JWT/JWKS middleware (from `auth/`), ingest/query/structured-query/SSE/schema/DLQ/pipes handlers; `ch_errors.go` (`writeCHError`) is the one mapping from a failed ClickHouse query to status, `code` and `retryable` - **`app/`** — the process wiring: `New` builds every component from the boot config and the settings directory (each one wired in one place — what it opens, what it loops, what it releases — with the settings registry handed to its wiring function whole, the injection point of the per-tenant registry of #583: store-keyed getters for the handlers, `perTenant` for the async paths (with the tenant each message's `mq.Topic` names for the stream hub and the ingest worker), the `chconn.Pools` and the per-tenant `discoveries` reconciled from `AfterAdopt`, `shortestKeepalive` for the one setting folded over every tenant served, `gapWindows` handing the sweeper each tenant's own gap window (a rejected tenant's as its folder last had it, unbounded for one rejected since boot) and the `mq.max_bytes_gb` reconcile each served tenant's byte budget, and `defaultPolicy` for the one setting that still follows tenant `0`, a flat directory's ops-gate admin role; the auth verifiers are per tenant, reconfigured (rebuilt only on changed wiring) and pruned from `AfterAdopt`, and the same hook's `Hub.Prune` ends the open streams of a tenant no longer served), `Run` drives the long-lived ones under one `errgroup` until the context is cancelled or one fails, `Close` releases them in reverse order. `New` wires only what the process's `roles` need (discovery, dedupe, auth verifiers, the hub bridge and keepalive per API process; the ingest worker per ingest process; the sweeper under its lease through `elected`); a process without `api` serves `api.NewOpsRouter` — probes, `/version`, metrics, and the settings reload behind the operator key alone. `cmd/wavehouse` and `tests/integration` both boot through it - **`auth/`** — JWT auth middleware: HMAC **or** JWKS verification with `alg` pinned to the active verifier, role extraction from a configurable claim path; always runs, never rejects (bad token → empty role + stashed reason). One verifier per tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 9): `Authenticator` keys them by `tenant.ID` — the request store's `settings.Store.Tenant()`, through an injected `TenantSource`; `tenant.Default` on the tenant-exempt routes — built from each tenant's `auth` block by `Reconfigure`, dropped by `Prune` once the tenant stops being served (rejected or removed), released by `Close`; the secrets (`Config`) are boot-level and shared. A JWKS key set is fetched off the boot and reload paths: until one has been stored the verifier is pending and a token-bearing request gets `503` + `Retry-After` from `api.refuseUnverifiable` (`auth.ErrVerifierPending`), never a `default_role` evaluation; refresh is library-managed (Eric, 2026-09-22), response capped at 1 MiB; the operator key's admin role is the request tenant's -- **`cache/`** — `Cache` interface → `LocalCache` (Ristretto: one pool for every tenant) + `VersionManager` (the invalidation index), and `RedisCache`, the Redis-compatible shared backend (random version tokens under the tenant's hash tag, one-round-trip lookups, bypass on failure behind a circuit breaker, deferred invalidations retried; built and tested, not yet selectable by config — [#613](https://github.com/Wave-RF/WaveHouse/issues/613) E4). Every key leads with the tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8) — `:query:` for the caller's query key and its singleflight (escaped whole as the lead field of the stored key, `|.||…`), `..
.
.` for a namespace, each field escaped by `keyenc` where the key is built (a `Namespace` carries the raw table and scope, so no caller escapes) — so no cached read or coalesced flight crosses tenants, a bump through `Invalidate` names one tenant's namespaces and no other's, and `InvalidateTenant` advances the tenant version folded into every namespace and query key of one tenant, orphaning its every cached result in one step, pipe results included (no insert reaches a pipe result until [#343](https://github.com/Wave-RF/WaveHouse/pull/343)); `Lookup` returns a `Snapshot` of the versions it read, taken before the handler chooses any input a bump invalidates — the tenant's connection included — and `Set` files the fill under it, so a write landing mid-query, or a reload moving the tenant to another address or database after the request took its connection, orphans the fill ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)), and every backend runs the conformance suite `internal/testutil/cachetest`; the one crossing is the wiring's, above the package: `internal/app` hands the ingest worker the cache through `sharedTables`, which repeats each of the worker's bumps under every tenant on the same ClickHouse address and database (`chconn.Pools.SharingTables`, whatever their user or tls block — they read the same tables), and orphans the whole cache of a tenant back on a pool after an absence, since it was out of that fan-out while away, or moved to another address or database, since it now reads other tables (story 6) +- **`cache/`** — `Cache` interface → `LocalCache` (Ristretto: one pool for every tenant) + `VersionManager` (the invalidation index), and `RedisCache`, the Redis-compatible shared backend (random version tokens under the tenant's hash tag, one-round-trip lookups, bypass on failure behind a circuit breaker, deferred invalidations retried; built and tested, not yet selectable by config — [#613](https://github.com/Wave-RF/WaveHouse/issues/613) E4). Every key carries the tenant (in `RedisCache`, right after the key prefix); in `LocalCache` and the version index it leads ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8) — `:query:` for the caller's query key and its singleflight (escaped whole as the lead field of the stored key, `|.||…`), `..
.
.` for a namespace, each field escaped by `keyenc` where the key is built (a `Namespace` carries the raw table and scope, so no caller escapes) — so no cached read or coalesced flight crosses tenants, a bump through `Invalidate` names one tenant's namespaces and no other's, and `InvalidateTenant` advances the tenant version folded into every namespace and query key of one tenant, orphaning its every cached result in one step, pipe results included (no insert reaches a pipe result until [#343](https://github.com/Wave-RF/WaveHouse/pull/343)); `Lookup` returns a `Snapshot` of the versions it read, taken before the handler chooses any input a bump invalidates — the tenant's connection included — and `Set` files the fill under it, so a write landing mid-query, or a reload moving the tenant to another address or database after the request took its connection, orphans the fill ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)), and every backend runs the conformance suite `internal/testutil/cachetest`; the one crossing is the wiring's, above the package: `internal/app` hands the ingest worker the cache through `sharedTables`, which repeats each of the worker's bumps under every tenant on the same ClickHouse address and database (`chconn.Pools.SharingTables`, whatever their user or tls block — they read the same tables), and orphans the whole cache of a tenant back on a pool after an absence, since it was out of that fan-out while away, or moved to another address or database, since it now reads other tables (story 6) - **`chconn/`** — `Pools`, one `Manager` (a `driver.Conn`) per distinct `Identity{Addr, Database, Username, Password, TLS}` tuple among the served tenants, reconciled from the settings registry's `AfterAdopt` after every reload ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 6): tenants naming one tuple share its pool, sized to their largest `max_open_conns`/`max_idle_conns`; a tenant whose tuple changed is repointed; a tuple no tenant names is released after the longest `query_timeout` among the tenants it had (never dials; a resize swaps the connection with the same grace). The boot config's `clickhouse.max_total_conns` bounds the open pools' `max_open_conns` together: boot refuses naming sum and ceiling; at a reload a resize above it keeps the pool's size, and a tuple that cannot be opened (the ceiling, an unreadable certificate, or options the driver refuses) leaves its tenants on the pool they had or on none — logged, retried by the next reload. Every consumer resolves its tenant's pool per call: `For` (nil for a tenant on no pool, a `503`), `Target` (the tenant's own HTTP wiring over its pool's TLS config), `SharingTables`, `Ping` (every pool at once, ready at the first answer). `HTTPClients` keeps one `http.Client` per TLS config. `Classify` (`errclass.go`) says what a failed ClickHouse request means for the request — `Unavailable`, `Denied`, `Rejected` (any unlisted exception code: the server read it and refused it), or `Unknown` (no code, no recognizable transport failure) — over the driver's error types and the HTTP interface's `HTTPError`; the ingest worker and the query handlers (`api/ch_errors.go` `writeCHError`, [#403](https://github.com/Wave-RF/WaveHouse/issues/403), [#271](https://github.com/Wave-RF/WaveHouse/issues/271)) both use it - **`chsql/`** — dependency-free ClickHouse SQL helpers shared by `query`/`policy` (avoids an import cycle): `QuoteIdent` (backtick-quote every identifier) + `BindUnsafe` (reject names with a literal `?`) - **`config/`** — YAML + env var config loading (cleanenv); strict on both sides (undeclared YAML key, unbound `WH_*` variable) and probes `data_dir` writability when a selected backend keeps state there (`NeedsDataDir`); `backends.go` holds each layer's `.backend` (only the in-process value today); `config.go` holds `roles` (`Has(Role)`) and `instance_id`, and `Validate` refuses a role split the backends cannot serve (any split over the embedded MQ; `api` without `ingest`, or the reverse, over a local cache) — boot is the validator, there is no dry run diff --git a/CHANGELOG.md b/CHANGELOG.md index b3553aebc..4cc140370 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **A Redis-compatible shared cache backend, built but not yet selectable** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag, with the table and scope escaped into the key (`internal/keyenc`) so no two names share a token; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row, or one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like), open a circuit breaker that skips the server until a probe write succeeds, and deferred invalidations are retried until they land — the first at once, and at the probe's cadence while the breaker is open — collapsing to one tenant-wide bump per tenant past 100,000 keys; until one lands, the process that owes it bypasses the lookups it would orphan. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. Nothing selects it yet: the `cache.backend` switch and the `cache.redis.*` settings arrive with the wiring (E4), so every deployment still runs `LocalCache`. Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression, server-stops-answering, refused-writes (a demoted primary, a full `noeviction` server) and owed-invalidation cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. +- **A Redis-compatible shared cache backend, built but not yet selectable** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/cache.go`, `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`, `CONTRIBUTING.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag, with the table and scope escaped into the key (`internal/keyenc`) so no two names share a token; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row, or one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like), open a circuit breaker that skips the server until a probe write succeeds (connections are replaced every minute, so one that outlived a failover behind a stable address reaches the new primary), and deferred invalidations are retried until they land — the first at once, and at the probe's cadence while the breaker is open — collapsing to one tenant-wide bump per tenant past 100,000 keys; until one lands, the process that owes it bypasses the lookups it would orphan. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. Nothing selects it yet: `cache.backend` accepts only `local` until the wiring (E4) adds `redis` and the `cache.redis.*` settings, so every deployment still runs `LocalCache`. Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression, server-stops-answering, refused-writes (a demoted primary, a full `noeviction` server), failover-behind-a-stable-address and owed-invalidation cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. - **One conformance suite for every `mq.Broker`, and a transient broker failure is a `503`** (`internal/mq/mqtest/` (new: the suite and the embedded broker's run of it), `internal/mq/{mq,embedded}.go` (+ tests), `internal/api/ingest.go` (+ tests), `.testcoverage.yml`, `docs/src/content/docs/{api,architecture}.md`, `AGENTS.md`): the first piece of the external-NATS workstream of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `mqtest.Run` states the `Broker` contract as behavior — round trips with names that need encoding, per-tenant order, `Nak` and `AckWait` redelivery, the trace context reaching `Subscribe`, dead-lettering that keeps the topic and leaves the original unacked, per-tenant dead-letter counts, replay bounds and isolation, and exactly one `failed` report when delivery ends underneath a consumer — through the interfaces alone, so the external backend runs the same cases, with `mqtest.Caps` for the four places where its semantics legitimately differ. The embedded broker passes it; writing it turned up that a durable deleted on several tenants' queues could report on `failed` more than once, which is fixed and pinned by a test that deletes it on one queue after another. It also turned up a replay that lost its connection mid-pull passing for a caught-up one when the pull ended in a timeout; that is an error now, as the `Replayer` contract says. The interface comments now allow a delivery unit that is a partition holding several tenants, a `CreateConsumer` that finds a durable rather than creating one, a `PurgeAcked` that leaves retention to the operator, and zero dead-letter counts where there is no per-tenant queue. A new sentinel, `mq.ErrUnavailable`, is a broker that cannot be reached or does not answer in time: the ingest handler answers it with `503` + `Retry-After: 5` rather than the `500` "publish failed" it would have been. Nothing returns it yet; the external backend of #613 will. - **Process roles: the API and the background workers can run in separate processes** (`internal/config/config.go` (+ `roles_test.go`, `defaults_test.go`), `internal/config/backends.go`, `internal/app/{app,wire}.go` (+ `roles_test.go`), `internal/api/router.go` (+ tests), `tests/integration/{setup,tenants}_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md}`, `AGENTS.md`), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The boot config gains `roles` (`WH_ROLES`, default `api,ingest,sweeper`, set in `defaults()` like every boot default, so an explicit `roles: []` refuses boot) and `instance_id` (`WH_INSTANCE_ID`, default `-<8 hex>`, fresh at every boot; logged at boot, and recorded as a lease holder once a shared `coord.backend` exists). A process wires only what its roles need: `api` runs the HTTP API with schema discovery, the token verifiers, the dedupe stores and the SSE hub (all per API process); `ingest` runs the ingest worker; `sweeper` runs the sweeper under its lease. A process without `api` serves an ops-only listener on `server.port` (the probes and their aliases, `/version`, the same-port metrics path, and `POST /v1/ops/settings/reload`, which takes the operator key alone); every other route answers 404, under `/v1/ops` once the operator key has passed. Boot refuses any split over the embedded MQ, which no other process can reach, and `api` without `ingest` (or the reverse) over a local cache, which the ingest worker's invalidations would never reach. Until a shared `mq.backend` exists, every process therefore runs every role, which is the default, so nothing changes for an existing deployment. `data_dir` is probed for Pebble only in a process running `api`. A `config.Config` built without `config.Load` must now name its roles (`config.AllRoles()` for all of them): `app.New` refuses an empty set. - **Leases for work that must run in one process at a time, and the sweeper runs under one** (`internal/coord/` (new: `coord.go`, `local.go`, `elect.go`, `coordtest/`, + tests), `internal/app/{app,wire}.go` (+ tests), `internal/config/backends.go`, `internal/ingest/sweeper.go`, `.github/labeler.yml`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development,ingest-pipeline}.md`, `docs/src/content/docs/configuration.mdx`, `config.yaml`, `AGENTS.md`): part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `coord.Coordinator` hands out named leases (`TryAcquire` → a `Term` with a strictly increasing fencing `Token`, a `Done` channel and `Resign`; `ErrHeld` while another holder's term is live), and `coord.RunElected` runs a loop only while its process holds the lease, resigning when the loop returns and campaigning again every 2s. `coord.Local` is the in-process implementation, and `coordtest.Conformance` is the suite every implementation runs — the NATS KV backend that lets several replicas share one queue comes next. The sweeper now runs through `RunElected` under the `sweeper` lease; with the in-process coordinator the one process always holds it, so nothing changes for a single-process deployment beyond one `coord: elected` log line at startup. `coord.backend` now selects the coordinator (`local`, the only value), so a `config.Config` built without `config.Load` must name it as well as the other three layers' backends. diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 1d6a90356..ac449529b 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -117,7 +117,7 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ - **version_manager.go** — `VersionManager`, the invalidation index behind `Invalidate` and `InvalidateTenant`: a namespace key is `..
.
.`, its fields joined with `keyenc.Join` so a dot in a table or scope name is escaped rather than read as a separator, and an entry's key, `|.||…` with the caller's query key escaped whole (its `:` become `%3A`), folds in the tenant's version and each dependency's namespace key and namespace version, so bumping a table (a scopeless write) or one scope — scope is reserved and empty today, so every write is the whole-table bump — orphans every dependent entry without touching the pool. The tenant leads every key ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8): the same table under two tenants is two namespaces, so a bump through `Invalidate` under one tenant never touches — and a read under one tenant is never served — the other's results, and the flat directory's single tenant simply carries the `0` prefix. `BumpTenant` (behind `InvalidateTenant`) advances the tenant version that leads every namespace key of one tenant, orphaning its every namespace and every cached query in one step (the tenant version is folded into every query key, so a pipe result with no dependencies is orphaned too) — a table no bump ever keyed included, which an enumeration of the index would miss — for a tenant back on a pool after an absence from the fan-out, or moved to another address or database (story 6). The index is per tenant; the cross-tenant invalidation an insert into a shared table needs is not the index's but the wiring's: `internal/app` hands the ingest worker a cache (`sharedTables`) that repeats each bump under every tenant on the same ClickHouse address and database. - **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — only `GET`, `SET` and `MGET`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot; the table and scope are escaped with `keyenc` after the fixed prefix, so a `:` in a name is never read as the separator. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, the hash over their escaped forms, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, and a token key holding anything but a token (a string of another length, a list, a hash) is replaced, so a token lost to eviction, expiry or a restart is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. While this process owes a bump on any of a lookup's tokens, that lookup is a bypass that files nothing: the bump would orphan whatever it found (other processes, which cannot know, serve those entries until it lands). `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing, each attempt bounded by `DialTimeout` (1 s), and a cluster client's topology read after the handshake by the larger of it and `Timeout`, which is also how long a connection waits on a silent server before it is redialed. Nothing selects this backend yet — the boot config and wiring arrive with [#613](https://github.com/Wave-RF/WaveHouse/issues/613)'s E4. - **redis_codec.go** — the key schema (`tokenKeys`, `bumpKeys`, `valueKey`, which take a `Namespace`'s raw names and escape them) and the value frame: format, flags, expiry, the token list, the payload, zstd-compressed from `CompressMinBytes` when that is smaller. A stored value is capped at `MaxValueBytes` and a decoded one at eight times that, which refuses a zip bomb planted in a shared server; an unknown format, as a newer process writes during a rolling upgrade, is a miss. -- **breaker.go** — the circuit breaker: `BreakerThreshold` (5) consecutive failures open it, and while open every operation skips the server at once; after `BreakerOpenFor` (5 s) one background write (`SET :probe`) decides whether it closes, and only that probe's success closes it. A transport failure or a timeout counts against the server; an error reply saying it takes no writes right now — `READONLY` (a demoted primary), `MASTERDOWN`, `OOM` (full under `maxmemory-policy noeviction`, Redis's default, so give the cache an evicting policy), `NOREPLICAS`, `MISCONF`, `LOADING`, `BUSY`, `CLUSTERDOWN` — opens it at once, whatever the threshold, since no bump can land. Any other reply, an error reply about one key (`WRONGTYPE`, `NOPERM`, `TRYAGAIN`) or one the backend cannot use included, counts as a success, and a caller that gave up first counts as nothing. +- **breaker.go** — the circuit breaker: `BreakerThreshold` (5) consecutive failures open it, and while open every operation skips the server at once; after `BreakerOpenFor` (5 s) one background write (`SET :probe`) decides whether it closes, and only that probe's success closes it. Every connection is replaced after a minute (rueidis retries what was in flight), because one that outlives a failover behind a stable address stays on the demoted node, which answers but refuses writes; the replacement re-resolves the address, so a bypassed process reaches the new primary, and delivers the bumps it owes, within about that long. A transport failure or a timeout counts against the server; an error reply saying it takes no writes right now — `READONLY` (a demoted primary), `MASTERDOWN`, `OOM` (full under `maxmemory-policy noeviction`, Redis's default, so give the cache an evicting policy), `NOREPLICAS`, `MISCONF`, `LOADING`, `BUSY`, `CLUSTERDOWN` — opens it at once, whatever the threshold, since no bump can land. Any other reply, an error reply about one key (`WRONGTYPE`, `NOPERM`, `TRYAGAIN`) or one the backend cannot use included, counts as a success, and a caller that gave up first counts as nothing. - **pending.go** — invalidations the server did not take, kept per token key (repeats coalesce) and retried by a background loop until they land: the first one owed at once, then with backoff from 100 ms to 10 s, and while the breaker is open at each probe, which the loop starts when due, so a process that makes no lookups (ingest only) recovers as soon as one that does. `Invalidate` sends its bumps 1,000 to a round trip, as the retry does. Landing late is still correct: a fresh token orphans the pre-write entries and any fill made meanwhile. Past `PendingMax` (100,000) keys the set collapses to one tenant bump per affected tenant. `Close` makes one last attempt at the bumps still owed, past the breaker (an open one is why they are owed); what that attempt cannot deliver is lost, and the entries they would orphan are served until their TTL — the same failure as a worker stopping between an insert and its invalidation. - **metrics.go** — the shared backend's instruments (meter `wavehouse-cache`, `backend="redis"`, no tenant attribute): `wavehouse_cache_lookups_total` by `result` (`hit`, `miss`, `stale` — filed under since-bumped tokens, `bypass` — server skipped, or held by a bump this process owes, `error`), `wavehouse_cache_op_duration_seconds` by `op` (`lookup`, `set`, `invalidate`), `wavehouse_cache_breaker_open` (1 while bypassed, including before the first connection), `wavehouse_cache_invalidations_total` by `result` (`ok`, `deferred`), `wavehouse_cache_invalidations_pending` (bumps owed: entries they would orphan may be served stale meanwhile), `wavehouse_cache_value_bytes` (stored size), `wavehouse_cache_oversize_total` and `wavehouse_cache_set_failures_total` by `reason` (`oom`, `timeout`, `other`). diff --git a/internal/cache/export_test.go b/internal/cache/export_test.go index 0ef1f0b61..70b8544d2 100644 --- a/internal/cache/export_test.go +++ b/internal/cache/export_test.go @@ -1,6 +1,10 @@ package cache -import "github.com/redis/rueidis" +import ( + "time" + + "github.com/redis/rueidis" +) // Hooks for the integration tests in package cache_test. @@ -16,6 +20,10 @@ func Pending(r *RedisCache) int { return r.pending.len() } // Bypassed reports whether r is skipping the server. func Bypassed(r *RedisCache) bool { return r.bypassed() } +// SetConnLifetime shortens how long c's connections live before they are +// replaced, for a failover test. +func SetConnLifetime(c *RedisConfig, d time.Duration) { c.connLifetime = d } + // KeyPrefix is the prefix every key r writes leads with. func KeyPrefix(r *RedisCache) string { return r.cfg.KeyPrefix } diff --git a/internal/cache/redis.go b/internal/cache/redis.go index b8bd5e286..9d6733d4e 100644 --- a/internal/cache/redis.go +++ b/internal/cache/redis.go @@ -38,6 +38,7 @@ const ( DefaultRedisPendingMax = 100_000 defaultBreakerThreshold = 5 defaultBreakerOpenFor = 5 * time.Second + defaultConnLifetime = time.Minute ) const ( @@ -77,6 +78,8 @@ type RedisConfig struct { BreakerThreshold int // consecutive failures that open the breaker BreakerOpenFor time.Duration // how long it stays open before a probe + + connLifetime time.Duration // defaultConnLifetime; tests shorten it } func (c RedisConfig) withDefaults() (RedisConfig, error) { @@ -135,6 +138,7 @@ func (c RedisConfig) withDefaults() (RedisConfig, error) { c.PendingMax = cmpOr(c.PendingMax, DefaultRedisPendingMax) c.BreakerThreshold = cmpOr(c.BreakerThreshold, defaultBreakerThreshold) c.BreakerOpenFor = cmpOr(c.BreakerOpenFor, defaultBreakerOpenFor) + c.connLifetime = cmpOr(c.connLifetime, defaultConnLifetime) return c, nil } @@ -163,6 +167,13 @@ func (c RedisConfig) clientOption() rueidis.ClientOption { // Close would wait out; never under Timeout, so no connection is cut // while an operation may still wait on it. opt.ConnWriteTimeout = max(c.DialTimeout, c.Timeout) + // A connection outlives a failover behind a stable address: the demoted + // node still answers, refusing writes, and rueidis does not redial on + // READONLY. Replacing each connection this often (rueidis retries what + // was in flight) re-resolves the address, so a process the refusals + // bypass reaches the new primary, and delivers its owed bumps, within + // about this long. + opt.ConnLifetime = c.connLifetime if c.Mode == RedisSentinel { opt.Sentinel = rueidis.SentinelOption{MasterSet: c.SentinelMaster, TLSConfig: c.TLS, Dialer: opt.Dialer} } @@ -558,7 +569,7 @@ func setFailureReason(err error) string { } // Invalidate sets a fresh token for every token the namespaces' writes -// reach, in one pipelined round trip. Bumps the server does not take are +// reach, in pipelined batches of drainBatch, one round trip each. Bumps the server does not take are // kept and retried until it does, and reported as an error meanwhile. func (r *RedisCache) Invalidate(ctx context.Context, namespaces []Namespace) (uint64, error) { owner := map[string]tenant.ID{} diff --git a/internal/cache/redis_integration_test.go b/internal/cache/redis_integration_test.go index 68c3e71a2..3169e9230 100644 --- a/internal/cache/redis_integration_test.go +++ b/internal/cache/redis_integration_test.go @@ -7,6 +7,7 @@ import ( "context" "crypto/tls" "fmt" + "io" "net" "slices" "strconv" @@ -559,6 +560,98 @@ func (c unanswered) Write(b []byte) (int, error) { return c.Conn.Write(b) } +// forward proxies each connection it accepts to the address target holds +// at that moment, so a switch moves new connections only, as a stable DNS +// name or a proxy does after a failover. It returns the address to dial. +func forward(t *testing.T, target *atomic.Pointer[string]) string { + t.Helper() + var lc net.ListenConfig + ln, err := lc.Listen(context.Background(), "tcp", "127.0.0.1:0") + require.NoError(t, err) + t.Cleanup(func() { _ = ln.Close() }) + go func() { + for { + c, err := ln.Accept() + if err != nil { + return + } + go func() { + defer func() { _ = c.Close() }() + var d net.Dialer + u, err := d.DialContext(context.Background(), "tcp", *target.Load()) + if err != nil { + return + } + defer func() { _ = u.Close() }() + go func() { _, _ = io.Copy(u, c); _ = u.Close() }() + _, _ = io.Copy(c, u) + }() + } + }() + return ln.Addr().String() +} + +// A failover behind a stable address: the connections the process holds +// stay on the demoted node, which answers but refuses writes, while new ones +// reach the node promoted in its place. The refusals bypass the cache, and +// replacing connections carries it to the new primary, where the owed bump +// lands before anything it would orphan is served. +func TestRedis_FailoverBehindAStableAddress(t *testing.T) { + t.Parallel() + ctx := context.Background() + start := func() *server { // no delay before a full sync + return startStandalone(t, redisImage, "redis-server", "--save", "", "--appendonly", "no", "--repl-diskless-sync-delay", "0") + } + primary, replica := start(), start() + rPrimary, rReplica := raw(t, primary), raw(t, replica) + ip := func(s *server) string { + t.Helper() + ip, err := s.ctr.ContainerIP(ctx) + require.NoError(t, err) + return ip + } + command(t, rReplica, "REPLICAOF", ip(primary), "6379") + require.Eventually(t, func() bool { + info, err := rReplica.Do(ctx, rReplica.B().Info().Section("replication").Build()).ToString() + return err == nil && strings.Contains(info, "master_link_status:up") + }, 30*time.Second, 50*time.Millisecond, "the replica syncs") + + var target atomic.Pointer[string] + target.Store(&primary.addr) + stable := &server{addr: forward(t, &target), mode: cache.RedisStandalone} + prefix := uniquePrefix() + a := open(t, stable, prefix, func(c *cache.RedisConfig) { + c.BreakerThreshold, c.BreakerOpenFor = 1000, 200*time.Millisecond + cache.SetConnLifetime(c, time.Second) + }) + deps := []cache.Namespace{{Tenant: "acme", Table: "events"}} + _, snap, err := a.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + require.NoError(t, a.Set(ctx, snap, []byte("pre-write rows"), time.Minute)) + acked, err := rPrimary.Do(ctx, rPrimary.B().Wait().Numreplicas(1).Timeout(5000).Build()).AsInt64() + require.NoError(t, err) + require.Equal(t, int64(1), acked, "the replica holds the fill") + + command(t, rReplica, "REPLICAOF", "NO", "ONE") + command(t, rPrimary, "REPLICAOF", ip(replica), "6379") + _, err = a.Invalidate(ctx, deps) + require.ErrorContains(t, err, "READONLY") + require.True(t, cache.Bypassed(a)) + target.Store(&replica.addr) + + deadline := time.Now().Add(15 * time.Second) + for cache.Pending(a) > 0 { + require.True(t, time.Now().Before(deadline), "the owed bump reaches the new primary") + e, _, err := a.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + require.NotEqual(t, "pre-write rows", string(e.Value), "served before the owed bump landed") + time.Sleep(10 * time.Millisecond) + } + e, _, err := open(t, stable, prefix).Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + assert.Nil(t, e.Value, "the bump landed where every process now reads") +} + // command runs cmd on s through c, for the test to reconfigure the server. func command(t *testing.T, c rueidis.Client, cmd ...string) { t.Helper() @@ -693,8 +786,8 @@ func TestRedis_RefusedWrites(t *testing.T) { assert.True(t, cache.Bypassed(reader), "a refused fill opens the breaker too") // Long enough for many probes to be refused, and for a drain - // backing off unchecked to be seconds from its next attempt. - time.Sleep(3500 * time.Millisecond) + // backing off unchecked to be at its 10 s cap. + time.Sleep(7 * time.Second) assert.True(t, cache.Bypassed(reader), "owing nothing, it is held open by probes that write and are refused") assert.True(t, cache.Bypassed(a)) assert.Empty(t, lookup(a, events)) @@ -711,7 +804,7 @@ func TestRedis_RefusedWrites(t *testing.T) { time.Sleep(time.Millisecond) } require.Eventually(t, func() bool { return cache.Pending(ingest) == 0 }, 10*time.Second, 5*time.Millisecond) - assert.Less(t, time.Since(restored), openFor+time.Second, "an ingest-only process probes when due, not at the drain's backoff") + assert.Less(t, time.Since(restored), openFor+3*time.Second, "an ingest-only process probes when due, not at the drain's backoff") require.Eventually(t, func() bool { return !cache.Bypassed(reader) }, 10*time.Second, 5*time.Millisecond) fresh := open(t, s, prefix) diff --git a/internal/cache/redis_test.go b/internal/cache/redis_test.go index 17da785a6..a9ccc3711 100644 --- a/internal/cache/redis_test.go +++ b/internal/cache/redis_test.go @@ -61,6 +61,7 @@ func TestRedisConfig_Defaults(t *testing.T) { assert.Zero(t, c.CompressMinBytes, "0 means never compress, not the default") assert.True(t, c.clientOption().ForceSingleClient) assert.Equal(t, DefaultRedisDialTimeout, c.clientOption().ConnWriteTimeout) + assert.Equal(t, defaultConnLifetime, c.clientOption().ConnLifetime) slow := c slow.Timeout = 5 * time.Second assert.Equal(t, slow.Timeout, slow.clientOption().ConnWriteTimeout, "never under the op timeout") From 4df73f995fd14139088e310b2175faf27eebc69d Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 05:46:54 -0400 Subject: [PATCH 27/45] docs(cache): clarify which bumps a no-index tenant absorbs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The version_manager.go bullet read "a bump of a tenant with no index is a no-op" right after introducing BumpTenant, so it looked scoped to that one call. It actually covers any bump — table, scope or tenant — against a tenant with no index, which is what lets Prune and a departed tenant's index stay gone (neither the sharedTables fan-out nor an insert still in flight for a just-pruned tenant can revive it). Found by pre-push review of #621 (origin/feat/cache-snapshot...HEAD). Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_017aS7rLrH1RKkUMem7X4ckd --- docs/src/content/docs/architecture.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 4690761df..7aede5bad 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -114,7 +114,7 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ - **cache.go** — `Cache` interface: `Lookup`, `Set`, `Invalidate`, `InvalidateTenant`, `Close`, plus `QueryTimeToTTL`, which sets a result's TTL from how long its query took (10 s floor, 1 h ceiling). Every entry is one tenant's: `Lookup` takes the tenant, the caller's query key — `:query:`, built by the two cached handlers in `api/` (`queryCacheKey`, with the tenant read off the request's store — `settings.Store.Tenant`), which use it as their [singleflight](https://pkg.go.dev/golang.org/x/sync/singleflight) key too — and the `Namespace`s the result depends on — one for a structured query, none yet for a pipe (a pipe's table dependencies are [#343](https://github.com/Wave-RF/WaveHouse/pull/343)) — each of that tenant (another tenant's is `ErrForeignDependency`) and naming a table and scope by their raw names, which the cache escapes where it builds a key. `Lookup` returns the `Entry` (a nil value is a miss) and a `Snapshot` of the versions it read; on a miss the handler runs the query and passes that snapshot to `Set`, so a result is filed under the versions read *before* its query ran, and a write that lands while it runs orphans the fill rather than re-homing pre-write rows under the post-write versions ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)). The snapshot is taken before any input a bump invalidates is chosen, the tenant's connection included: a reload that moves the tenant to another address or database runs `Pools.Reconcile` and then `InvalidateTenant` (a repoint that keeps both, such as a username or `tls` change, reads the same tables and bumps nothing), so a request that took the old pool files the old database's rows under a version that bump orphans, whether its `Set` lands before the bump or after. The singleflight leader's snapshot is the one used. `Set` returns an error only when the backend failed; a value the cache declines — larger than it keeps, a non-positive TTL, a zero snapshot — is not one. What a hit, a miss and a bump mean is pinned by the conformance suite every backend runs, `internal/testutil/cachetest`. - **local.go** — `LocalCache`, the in-process L1 on [Ristretto](https://github.com/dgraph-io/ristretto): one pool shared by every tenant (a heavier tenant holds more of it), sized by the boot config's `cache.l1_max_cost`. -- **version_manager.go** — `VersionManager`, the invalidation index behind `Invalidate` and `InvalidateTenant`: one version per tenant, per (tenant, table) and per (tenant, table, scope), each keyed by its name alone and bumped in place, so the index holds one entry per live tenant, table and scope however often each is bumped ([#262](https://github.com/Wave-RF/WaveHouse/issues/262)). A query key folds the tenant's version and, for each dependency, its tenant's, table's and scope's, so bumping a table (a scopeless write) orphans every scope of it, and bumping one scope orphans that scope and the whole-table view — scope is reserved and empty today, so every write is the whole-table bump — all without touching the pool. Every field — the caller's query key, the tenant id, and each dependency's table and scope — is escaped and joined by `internal/keyenc` where the key is built, so a dot, a space or a `%` in a name is never read as a separator: each dependency renders as `..
.
..`, and the whole entry key is `|.||…`. The tenant leads every key ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8): the same table under two tenants is two namespaces, so a bump through `Invalidate` under one tenant never touches — and a read under one tenant is never served — the other's results. A tenant's version is a *generation*, unique within the process and handed out by the first key built for the tenant; `BumpTenant` (behind `InvalidateTenant`) drops the tenant's whole index, so the next key gets a fresh generation no cached entry folds, orphaning every cached result of the tenant in one step — a pipe result with no dependencies, and a table no bump ever keyed, included — for a tenant back on a pool after an absence from the fan-out, or moved to another address or database (story 6). `LocalCache.Prune` does the same for every tenant no longer served, which `internal/app` runs after each settings reload, so a tenant removed or rejected stops holding its index. A table bump drops the table's scope versions with it, since every key they were folded into also folds the old table version; and a bump of a tenant with no index is a no-op, since no key folds its next generation yet. The index is per tenant; the cross-tenant invalidation an insert into a shared table needs is not the index's but the wiring's: `internal/app` hands the ingest worker a cache (`sharedTables`) that repeats each bump under every tenant on the same ClickHouse address and database. +- **version_manager.go** — `VersionManager`, the invalidation index behind `Invalidate` and `InvalidateTenant`: one version per tenant, per (tenant, table) and per (tenant, table, scope), each keyed by its name alone and bumped in place, so the index holds one entry per live tenant, table and scope however often each is bumped ([#262](https://github.com/Wave-RF/WaveHouse/issues/262)). A query key folds the tenant's version and, for each dependency, its tenant's, table's and scope's, so bumping a table (a scopeless write) orphans every scope of it, and bumping one scope orphans that scope and the whole-table view — scope is reserved and empty today, so every write is the whole-table bump — all without touching the pool. Every field — the caller's query key, the tenant id, and each dependency's table and scope — is escaped and joined by `internal/keyenc` where the key is built, so a dot, a space or a `%` in a name is never read as a separator: each dependency renders as `..
.
..`, and the whole entry key is `|.||…`. The tenant leads every key ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8): the same table under two tenants is two namespaces, so a bump through `Invalidate` under one tenant never touches — and a read under one tenant is never served — the other's results. A tenant's version is a *generation*, unique within the process and handed out by the first key built for the tenant; `BumpTenant` (behind `InvalidateTenant`) drops the tenant's whole index, so the next key gets a fresh generation no cached entry folds, orphaning every cached result of the tenant in one step — a pipe result with no dependencies, and a table no bump ever keyed, included — for a tenant back on a pool after an absence from the fan-out, or moved to another address or database (story 6). `LocalCache.Prune` does the same for every tenant no longer served, which `internal/app` runs after each settings reload, so a tenant removed or rejected stops holding its index. A table bump drops the table's scope versions with it, since every key they were folded into also folds the old table version; and any bump (of a table, a scope or the tenant) under a tenant with no index is a no-op that records nothing, since the next key built for it gets a fresh generation no cached entry folds — so neither the `sharedTables` fan-out nor an insert still in flight for a tenant just pruned brings its index back. The index is per tenant; the cross-tenant invalidation an insert into a shared table needs is not the index's but the wiring's: `internal/app` hands the ingest worker a cache (`sharedTables`) that repeats each bump under every tenant on the same ClickHouse address and database. ### `config/` — Configuration From 95575f27a49982cddd1c1febd72028ff669a901b Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 05:54:37 -0400 Subject: [PATCH 28/45] docs(cache): CHANGELOG says #262 is part-fixed, not fixed The #262 entry said "fixes #262 for the in-process cache", which would close the issue on merge (delete_branch_on_merge + squash_merge_commit_message: PR_BODY carry the PR body's own "Fixes #262" into main's history the same way). #262 has a residual left open on purpose: per-table scope cardinality isn't capped, since scope stays empty until #235 populates it. Reworded to "part of #262" and named the residual, matching the PR body and the issue comment that records the trigger. Found by pre-push review of #621 (origin/feat/cache-snapshot...HEAD). Co-Authored-By: Claude Sonnet 5 --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a89a0d2be..6631b3c4a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -89,7 +89,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), - **An unavailable ClickHouse is retried with backoff instead of dead-lettering every row** (`internal/chconn/errclass.go` (new, + tests), `internal/ingest/{worker,backoff}.go` (`backoff.go` new, + tests), `internal/mq/{mq,embedded}.go`, `internal/testutil/mocks.go`, `tests/integration/ingest_outage_test.go` (new), `AGENTS.md`, `README.md`, `docs/src/content/docs/{ingest-pipeline,architecture,api,deployment,why-wavehouse}.md`, `docs/src/content/docs/{settings-directory,index,access-control}.mdx`): workstream A of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). A failed batch insert used to go through row-by-row isolation whatever the failure, so a ClickHouse that was down, overloaded or read-only failed every row twice and parked the whole batch on the DLQ. `chconn.Classify` now classes the failure first — `Rejected` (any ClickHouse exception code outside the availability and credential lists: the server read the row and refused it), `Unavailable` (connection refused/reset, timeouts, `TOO_MANY_SIMULTANEOUS_QUERIES`, `SERVER_OVERLOADED`, `MEMORY_LIMIT_EXCEEDED`, `TOO_MANY_PARTS`, `READONLY`, `TABLE_IS_READ_ONLY`, `KEEPER_EXCEPTION`, …), `Denied` (`AUTHENTICATION_FAILED`, `ACCESS_DENIED`, …) or `Unknown` (no code, no recognizable transport failure). Only `Rejected` is isolated and dead-lettered as before, and a multi-row batch refused with `TOO_MANY_PARTS` or `MEMORY_LIMIT_EXCEEDED` is split row by row first (`chconn.Splittable`), because a batch spanning too many partitions or too much memory can fail where each of its rows inserts; every other class hands the batch back to the queue with a delayed nak (`mq.Message.NakWithDelay`, new) under a jittered 1 s → 30 s backoff shared by every table on the same ClickHouse pool (a failure of one table — read-only, too many parts or mutations, a grant missing on it, `chconn.TableScoped` — backs off that table alone), which turns rows away without a request while it runs and probes once per window, and ClickHouse going away mid-isolation stops isolation and retries the rows it had not settled. Counted by the new `wavehouse_ingest_retries_total{table, reason}`; logged at `WARN` when an outage starts and at most every 30 s during it. A long outage now shows as a growing ingest stream and, at `mq.max_bytes_gb`, ingest `503`s — not as a full DLQ; a lasting failure of one table holds back its tenant's other tables once its waiting rows reach `maxAckPending`. Retried rows come back out of arrival order, which matters only to a `ReplacingMergeTree` without a version column or a `CollapsingMergeTree`. - **Schema discovery's retry loop jitters its backoff** (`internal/discovery/discovery.go` (+ tests), `internal/app/wire.go`, `internal/api/errors.go`, `AGENTS.md`, `docs/src/content/docs/{architecture,api,deployment}.md`): `RetryRefresh` slept exactly `2s * 2^n` capped at 60s, so instances retrying against one recovering ClickHouse fired in lockstep, every 60s on the same second. Each sleep is now drawn uniformly from below the backoff (full jitter), spreading the retries over the whole window and halving the mean wait — so a failing tenant's retries, their log lines and `wavehouse_schema_refresh_failures_total` come about twice as often ([#141](https://github.com/Wave-RF/WaveHouse/issues/141)). - **A write that lands while a cached read is running no longer re-homes the pre-write rows under the post-write key** (`internal/cache/{cache,local,version_manager}.go` (+ tests), `internal/testutil/cachetest` (new), `internal/api/{structured_query,pipes}.go` (+ tests), `internal/ingest/worker.go` (+ tests), `internal/query/ident.go` (removed, + tests), `internal/app/wire.go`, `docs/src/content/docs/{api,architecture,deployment}.md`, `AGENTS.md`): fixes [#382](https://github.com/Wave-RF/WaveHouse/issues/382), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `POST /v1/query` rebuilt the version-folded cache key after the query ran, so an insert invalidating the table mid-query filed the rows read before it under the new versions, and they were served as fresh until their TTL (a pipe result's key folded no version, so pipes were unaffected; with the tenant's version in every key they now take the same snapshot). The `Cache` interface now snapshots at lookup: `Lookup(ctx, tenant, sha, deps)` returns the `Entry` and a `Snapshot` of the versions it read, and `Set(ctx, snapshot, value, ttl)` stores under that snapshot, so such a fill is orphaned and the next request reads the post-write rows. The singleflight leader's snapshot is the one used; coalescing is unchanged. The snapshot is taken before any input a bump invalidates is chosen, the tenant's ClickHouse connection included: both handlers now look up before they resolve the tenant's pool, so a reload that moves the tenant to another address or database after a request took the old pool orphans that request's fill instead of filing the old database's rows as fresh under the new tenant version. A tenant on no pool is still a `503` before a cached result is served or a query runs. A `Lookup` whose dependencies name another tenant is refused (`ErrForeignDependency`). `Set` now errors only when the backend failed: a value the cache declines (larger than the pool, a non-positive TTL) is not an error. One behavior change: the tenant's version is folded into every key, a pipe result's included, so `InvalidateTenant` (a tenant back on a pool after an absence, or moved to another ClickHouse address or database) now drops that tenant's cached pipe results as well as its query results; before, a pipe result stayed until its TTL. Inserts still do not invalidate pipe results ([#343](https://github.com/Wave-RF/WaveHouse/pull/343)). A backend-agnostic conformance suite, `cachetest.Run`, pins what a hit, a miss and each kind of bump mean, and `LocalCache` runs it; the Redis-compatible backend will run the same suite. The cache now escapes table and scope names itself: a `Namespace` carries them raw and the version index builds its keys with `internal/keyenc`, so neither the structured-query read nor the ingest worker's invalidation escapes them (`query.SafeEncodeToken` is gone) and no name reaches a key unescaped. The suite pins that a name holding a dot, a space or a `%` is read and bumped under one key, and that names which would run together unescaped (`a.0.b` against `a` with scope `b.0.`) stay two entries. The version index's own entries are unaffected by the escaping; only the rendered key changes: it now carries the tenant's version and escapes the caller's query key whole. All of them live in the process, so nothing stored is orphaned. -- **The cache's version index no longer grows with every bump, and forgets a tenant no longer served** (`internal/cache/{local,version_manager}.go` (+ tests), `internal/app/wire.go` (+ tests), `docs/src/content/docs/architecture.md`, `AGENTS.md`): fixes [#262](https://github.com/Wave-RF/WaveHouse/issues/262) for the in-process cache, part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The index nested each table under its tenant's version and each scope under its table's, and never pruned, so every tenant invalidation left the tenant's whole index behind, and it grew with every tenant ever served. It now holds one version per tenant, per (tenant, table) and per (tenant, table, scope), bumped in place. A tenant invalidation drops the tenant's index and hands its next key a generation unique within the process, so nothing cached before it can match again, and a table bump drops the table's scope versions. After each settings reload the index of every tenant no longer served, removed or rejected, is dropped the same way; its cached results are orphaned with it, as they already were when such a tenant came back on a pool. No change to what is cached or served. The Redis-compatible backend (#613) will bound its versions with a TTL instead. +- **The cache's version index no longer grows with every bump, and forgets a tenant no longer served** (`internal/cache/{local,version_manager}.go` (+ tests), `internal/app/wire.go` (+ tests), `docs/src/content/docs/architecture.md`, `AGENTS.md`): part of [#262](https://github.com/Wave-RF/WaveHouse/issues/262) (growth across bumps and a departed tenant's memory; per-table scope cardinality is left open, see the issue) and of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The index nested each table under its tenant's version and each scope under its table's, and never pruned, so every tenant invalidation left the tenant's whole index behind, and it grew with every tenant ever served. It now holds one version per tenant, per (tenant, table) and per (tenant, table, scope), bumped in place. A tenant invalidation drops the tenant's index and hands its next key a generation unique within the process, so nothing cached before it can match again, and a table bump drops the table's scope versions. After each settings reload the index of every tenant no longer served, removed or rejected, is dropped the same way; its cached results are orphaned with it, as they already were when such a tenant came back on a pool. No change to what is cached or served. The Redis-compatible backend (#613) will bound its versions with a TTL instead. - **An explicit `false`, `0` or `""` in `config.yaml` is no longer replaced by the key's default** (`internal/config/config.go`, `internal/config/defaults_test.go` (new), `docs/src/content/docs/configuration.mdx`, `AGENTS.md`): [#631](https://github.com/Wave-RF/WaveHouse/issues/631). Defaults lived in cleanenv `env-default` tags, which cleanenv applies after the YAML decode to any field still at its zero value, so it could not tell a key the file set to its zero value from one the file left out. `otel.traces.enabled: false`, `otel.metrics.enabled: false` and `otel.logs.enabled: false` came back `true`; `otel.traces.sample_rate: 0` and `otel.logs.sample_rate: 0` came back `1.0`; `server.shutdown_timeout: 0` came back `10`; `cache.l1_max_cost: 0`, `prometheus.path: ""` and `data_dir: ""` came back as their defaults; `server.port: 0` came back `8080`. All of it was silent. Defaults now live in one Go function, `defaults()`, which `Load` starts from before decoding the file and then applying `WH_*` variables, so the order is env > YAML > default and a key the file sets always wins. **Behaviour change if your file relied on the bug:** a zero you wrote now takes effect. A file that says `sample_rate: 0` now exports no traces (or no DEBUG/INFO logs), where it silently exported everything; a signal set `enabled: false` is now off; `shutdown_timeout: 0` now skips the drain. `cache.l1_max_cost: 0`, `server.port: 0`, and `data_dir: ""` now refuse boot (`cache init: MaxCost can't be zero`, `server.port 0 out of range`, `data_dir (WH_DATA_DIR) is required`) instead of running on the default; an empty `prometheus.path` refuses boot when `prometheus.enabled` is true. Delete the key to get the default back. Env vars are unchanged: they already honoured an explicit zero. New tests load through `config.Load` for every affected key (a YAML zero is kept, an absent key gets the default, env wins in both directions), refuse an `env-default` tag on any field, and pin each documented default in `configuration.mdx` to `defaults()`. - **An embedded queue store that cannot be created fails boot at once, naming the cause** (`internal/mq/embedded.go` (+ tests)): part of [#617](https://github.com/Wave-RF/WaveHouse/issues/617). A regular file at `/nats`, or a `nats` directory that could not be created there, failed JetStream in the background, so boot waited out the server's 5s readiness check and reported only `nats server not ready`. `NewEmbedded` now creates the directory first (at `0700`, as the server does) and refuses boot with the mkdir error. An existing but unwritable `nats` directory still takes the old path. - **An insert invalidates a table's cached results under every tenant the directory holds** (`internal/app/wire.go` (+ tests), `internal/settings/registry.go` (+ tests), `AGENTS.md`, `docs/src/content/docs/{deployment,architecture,ingest-pipeline}.md`): until [#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 6 gives each tenant its own ClickHouse, every tenant reads the same tables, but the ingest worker — which writes every event as tenant `0`'s until story 5 — bumped only tenant `0`'s cache namespaces after an insert, so another tenant's cached query could answer stale rows for up to its TTL (an hour at most). The cache the worker invalidates through now fans each bumped namespace out to every tenant the registry knows (the new `Registry.Known`), the named one and a rejected one included — a rejected tenant comes back into service with the entries it has, so leaving it out would let a folder repaired inside a TTL serve pre-insert rows; reads are untouched, so a tenant is still never served another's cached rows. The residual, a folder removed and restored inside a TTL, is closed since #610: a tenant back on a pool after an absence has its cached results orphaned at once (`Cache.InvalidateTenant`). Raised by CodeRabbit on #602. From 47528116cf4e6cb0e02c55f27a2e68af2c4bd520 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 06:07:58 -0400 Subject: [PATCH 29/45] docs(cache): cache.backend exists; Redis key shapes; failover wording architecture.md no longer says the boot config arrives with E4: cache.backend exists and accepts only local until E4 adds redis. AGENTS.md gives both Redis key shapes (a token's tenant is a hash tag, a value's follows :q:), and the CHANGELOG's failover sentence says the process, not the old connection, reaches the new primary, within about a minute. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 2 +- CHANGELOG.md | 2 +- docs/src/content/docs/architecture.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index c4ee04283..81882f018 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -31,7 +31,7 @@ Twenty internal packages under `internal/` (plus `internal/testutil/` for shared - **`api/`** — Chi HTTP router, JWT/JWKS middleware (from `auth/`), ingest/query/structured-query/SSE/schema/DLQ/pipes handlers; `ch_errors.go` (`writeCHError`) is the one mapping from a failed ClickHouse query to status, `code` and `retryable` - **`app/`** — the process wiring: `New` builds every component from the boot config and the settings directory (each one wired in one place — what it opens, what it loops, what it releases — with the settings registry handed to its wiring function whole, the injection point of the per-tenant registry of #583: store-keyed getters for the handlers, `perTenant` for the async paths (with the tenant each message's `mq.Topic` names for the stream hub and the ingest worker), the `chconn.Pools` and the per-tenant `discoveries` reconciled from `AfterAdopt`, `shortestKeepalive` for the one setting folded over every tenant served, `gapWindows` handing the sweeper each tenant's own gap window (a rejected tenant's as its folder last had it, unbounded for one rejected since boot) and the `mq.max_bytes_gb` reconcile each served tenant's byte budget, and `defaultPolicy` for the one setting that still follows tenant `0`, a flat directory's ops-gate admin role; the auth verifiers are per tenant, reconfigured (rebuilt only on changed wiring) and pruned from `AfterAdopt`, and the same hook's `Hub.Prune` ends the open streams of a tenant no longer served), `Run` drives the long-lived ones under one `errgroup` until the context is cancelled or one fails, `Close` releases them in reverse order. `New` wires only what the process's `roles` need (discovery, dedupe, auth verifiers, the hub bridge and keepalive per API process; the ingest worker per ingest process; the sweeper under its lease through `elected`); a process without `api` serves `api.NewOpsRouter` — probes, `/version`, metrics, and the settings reload behind the operator key alone. `cmd/wavehouse` and `tests/integration` both boot through it - **`auth/`** — JWT auth middleware: HMAC **or** JWKS verification with `alg` pinned to the active verifier, role extraction from a configurable claim path; always runs, never rejects (bad token → empty role + stashed reason). One verifier per tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 9): `Authenticator` keys them by `tenant.ID` — the request store's `settings.Store.Tenant()`, through an injected `TenantSource`; `tenant.Default` on the tenant-exempt routes — built from each tenant's `auth` block by `Reconfigure`, dropped by `Prune` once the tenant stops being served (rejected or removed), released by `Close`; the secrets (`Config`) are boot-level and shared. A JWKS key set is fetched off the boot and reload paths: until one has been stored the verifier is pending and a token-bearing request gets `503` + `Retry-After` from `api.refuseUnverifiable` (`auth.ErrVerifierPending`), never a `default_role` evaluation; refresh is library-managed (Eric, 2026-09-22), response capped at 1 MiB; the operator key's admin role is the request tenant's -- **`cache/`** — `Cache` interface → `LocalCache` (Ristretto: one pool for every tenant) + `VersionManager` (the invalidation index), and `RedisCache`, the Redis-compatible shared backend (random version tokens under the tenant's hash tag, one-round-trip lookups, bypass on failure behind a circuit breaker, deferred invalidations retried; built and tested, not yet selectable by config — [#613](https://github.com/Wave-RF/WaveHouse/issues/613) E4). Every key carries the tenant (in `RedisCache`, right after the key prefix); in `LocalCache` and the version index it leads ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8) — `:query:` for the caller's query key and its singleflight (escaped whole as the lead field of the stored key, `|.||…`), `..
.
.` for a namespace, each field escaped by `keyenc` where the key is built (a `Namespace` carries the raw table and scope, so no caller escapes) — so no cached read or coalesced flight crosses tenants, a bump through `Invalidate` names one tenant's namespaces and no other's, and `InvalidateTenant` advances the tenant version folded into every namespace and query key of one tenant, orphaning its every cached result in one step, pipe results included (no insert reaches a pipe result until [#343](https://github.com/Wave-RF/WaveHouse/pull/343)); `Lookup` returns a `Snapshot` of the versions it read, taken before the handler chooses any input a bump invalidates — the tenant's connection included — and `Set` files the fill under it, so a write landing mid-query, or a reload moving the tenant to another address or database after the request took its connection, orphans the fill ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)), and every backend runs the conformance suite `internal/testutil/cachetest`; the one crossing is the wiring's, above the package: `internal/app` hands the ingest worker the cache through `sharedTables`, which repeats each of the worker's bumps under every tenant on the same ClickHouse address and database (`chconn.Pools.SharingTables`, whatever their user or tls block — they read the same tables), and orphans the whole cache of a tenant back on a pool after an absence, since it was out of that fan-out while away, or moved to another address or database, since it now reads other tables (story 6) +- **`cache/`** — `Cache` interface → `LocalCache` (Ristretto: one pool for every tenant) + `VersionManager` (the invalidation index), and `RedisCache`, the Redis-compatible shared backend (random version tokens under the tenant's hash tag, one-round-trip lookups, bypass on failure behind a circuit breaker, deferred invalidations retried; built and tested, not yet selectable by config — [#613](https://github.com/Wave-RF/WaveHouse/issues/613) E4). Every key carries the tenant (in `RedisCache`, after the key prefix: `:{}:…` for a version token, `:q::…` for a value); in `LocalCache` and the version index it leads ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8) — `:query:` for the caller's query key and its singleflight (escaped whole as the lead field of the stored key, `|.||…`), `..
.
.` for a namespace, each field escaped by `keyenc` where the key is built (a `Namespace` carries the raw table and scope, so no caller escapes) — so no cached read or coalesced flight crosses tenants, a bump through `Invalidate` names one tenant's namespaces and no other's, and `InvalidateTenant` advances the tenant version folded into every namespace and query key of one tenant, orphaning its every cached result in one step, pipe results included (no insert reaches a pipe result until [#343](https://github.com/Wave-RF/WaveHouse/pull/343)); `Lookup` returns a `Snapshot` of the versions it read, taken before the handler chooses any input a bump invalidates — the tenant's connection included — and `Set` files the fill under it, so a write landing mid-query, or a reload moving the tenant to another address or database after the request took its connection, orphans the fill ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)), and every backend runs the conformance suite `internal/testutil/cachetest`; the one crossing is the wiring's, above the package: `internal/app` hands the ingest worker the cache through `sharedTables`, which repeats each of the worker's bumps under every tenant on the same ClickHouse address and database (`chconn.Pools.SharingTables`, whatever their user or tls block — they read the same tables), and orphans the whole cache of a tenant back on a pool after an absence, since it was out of that fan-out while away, or moved to another address or database, since it now reads other tables (story 6) - **`chconn/`** — `Pools`, one `Manager` (a `driver.Conn`) per distinct `Identity{Addr, Database, Username, Password, TLS}` tuple among the served tenants, reconciled from the settings registry's `AfterAdopt` after every reload ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 6): tenants naming one tuple share its pool, sized to their largest `max_open_conns`/`max_idle_conns`; a tenant whose tuple changed is repointed; a tuple no tenant names is released after the longest `query_timeout` among the tenants it had (never dials; a resize swaps the connection with the same grace). The boot config's `clickhouse.max_total_conns` bounds the open pools' `max_open_conns` together: boot refuses naming sum and ceiling; at a reload a resize above it keeps the pool's size, and a tuple that cannot be opened (the ceiling, an unreadable certificate, or options the driver refuses) leaves its tenants on the pool they had or on none — logged, retried by the next reload. Every consumer resolves its tenant's pool per call: `For` (nil for a tenant on no pool, a `503`), `Target` (the tenant's own HTTP wiring over its pool's TLS config), `SharingTables`, `Ping` (every pool at once, ready at the first answer). `HTTPClients` keeps one `http.Client` per TLS config. `Classify` (`errclass.go`) says what a failed ClickHouse request means for the request — `Unavailable`, `Denied`, `Rejected` (any unlisted exception code: the server read it and refused it), or `Unknown` (no code, no recognizable transport failure) — over the driver's error types and the HTTP interface's `HTTPError`; the ingest worker and the query handlers (`api/ch_errors.go` `writeCHError`, [#403](https://github.com/Wave-RF/WaveHouse/issues/403), [#271](https://github.com/Wave-RF/WaveHouse/issues/271)) both use it - **`chsql/`** — dependency-free ClickHouse SQL helpers shared by `query`/`policy` (avoids an import cycle): `QuoteIdent` (backtick-quote every identifier) + `BindUnsafe` (reject names with a literal `?`) - **`config/`** — YAML + env var config loading (cleanenv); strict on both sides (undeclared YAML key, unbound `WH_*` variable) and probes `data_dir` writability when a selected backend keeps state there (`NeedsDataDir`); `backends.go` holds each layer's `.backend` (only the in-process value today); `config.go` holds `roles` (`Has(Role)`) and `instance_id`, and `Validate` refuses a role split the backends cannot serve (any split over the embedded MQ; `api` without `ingest`, or the reverse, over a local cache) — boot is the validator, there is no dry run diff --git a/CHANGELOG.md b/CHANGELOG.md index 4cc140370..2521ef894 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **A Redis-compatible shared cache backend, built but not yet selectable** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/cache.go`, `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`, `CONTRIBUTING.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag, with the table and scope escaped into the key (`internal/keyenc`) so no two names share a token; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row, or one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like), open a circuit breaker that skips the server until a probe write succeeds (connections are replaced every minute, so one that outlived a failover behind a stable address reaches the new primary), and deferred invalidations are retried until they land — the first at once, and at the probe's cadence while the breaker is open — collapsing to one tenant-wide bump per tenant past 100,000 keys; until one lands, the process that owes it bypasses the lookups it would orphan. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. Nothing selects it yet: `cache.backend` accepts only `local` until the wiring (E4) adds `redis` and the `cache.redis.*` settings, so every deployment still runs `LocalCache`. Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression, server-stops-answering, refused-writes (a demoted primary, a full `noeviction` server), failover-behind-a-stable-address and owed-invalidation cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. +- **A Redis-compatible shared cache backend, built but not yet selectable** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/cache.go`, `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`, `CONTRIBUTING.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag, with the table and scope escaped into the key (`internal/keyenc`) so no two names share a token; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row, or one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like), open a circuit breaker that skips the server until a probe write succeeds (connections are replaced every minute, so after a failover behind a stable address the process reaches the new primary, and delivers the bumps it owes, within about a minute), and deferred invalidations are retried until they land — the first at once, and at the probe's cadence while the breaker is open — collapsing to one tenant-wide bump per tenant past 100,000 keys; until one lands, the process that owes it bypasses the lookups it would orphan. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. Nothing selects it yet: `cache.backend` accepts only `local` until the wiring (E4) adds `redis` and the `cache.redis.*` settings, so every deployment still runs `LocalCache`. Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression, server-stops-answering, refused-writes (a demoted primary, a full `noeviction` server), failover-behind-a-stable-address and owed-invalidation cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. - **One conformance suite for every `mq.Broker`, and a transient broker failure is a `503`** (`internal/mq/mqtest/` (new: the suite and the embedded broker's run of it), `internal/mq/{mq,embedded}.go` (+ tests), `internal/api/ingest.go` (+ tests), `.testcoverage.yml`, `docs/src/content/docs/{api,architecture}.md`, `AGENTS.md`): the first piece of the external-NATS workstream of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `mqtest.Run` states the `Broker` contract as behavior — round trips with names that need encoding, per-tenant order, `Nak` and `AckWait` redelivery, the trace context reaching `Subscribe`, dead-lettering that keeps the topic and leaves the original unacked, per-tenant dead-letter counts, replay bounds and isolation, and exactly one `failed` report when delivery ends underneath a consumer — through the interfaces alone, so the external backend runs the same cases, with `mqtest.Caps` for the four places where its semantics legitimately differ. The embedded broker passes it; writing it turned up that a durable deleted on several tenants' queues could report on `failed` more than once, which is fixed and pinned by a test that deletes it on one queue after another. It also turned up a replay that lost its connection mid-pull passing for a caught-up one when the pull ended in a timeout; that is an error now, as the `Replayer` contract says. The interface comments now allow a delivery unit that is a partition holding several tenants, a `CreateConsumer` that finds a durable rather than creating one, a `PurgeAcked` that leaves retention to the operator, and zero dead-letter counts where there is no per-tenant queue. A new sentinel, `mq.ErrUnavailable`, is a broker that cannot be reached or does not answer in time: the ingest handler answers it with `503` + `Retry-After: 5` rather than the `500` "publish failed" it would have been. Nothing returns it yet; the external backend of #613 will. - **Process roles: the API and the background workers can run in separate processes** (`internal/config/config.go` (+ `roles_test.go`, `defaults_test.go`), `internal/config/backends.go`, `internal/app/{app,wire}.go` (+ `roles_test.go`), `internal/api/router.go` (+ tests), `tests/integration/{setup,tenants}_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md}`, `AGENTS.md`), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The boot config gains `roles` (`WH_ROLES`, default `api,ingest,sweeper`, set in `defaults()` like every boot default, so an explicit `roles: []` refuses boot) and `instance_id` (`WH_INSTANCE_ID`, default `-<8 hex>`, fresh at every boot; logged at boot, and recorded as a lease holder once a shared `coord.backend` exists). A process wires only what its roles need: `api` runs the HTTP API with schema discovery, the token verifiers, the dedupe stores and the SSE hub (all per API process); `ingest` runs the ingest worker; `sweeper` runs the sweeper under its lease. A process without `api` serves an ops-only listener on `server.port` (the probes and their aliases, `/version`, the same-port metrics path, and `POST /v1/ops/settings/reload`, which takes the operator key alone); every other route answers 404, under `/v1/ops` once the operator key has passed. Boot refuses any split over the embedded MQ, which no other process can reach, and `api` without `ingest` (or the reverse) over a local cache, which the ingest worker's invalidations would never reach. Until a shared `mq.backend` exists, every process therefore runs every role, which is the default, so nothing changes for an existing deployment. `data_dir` is probed for Pebble only in a process running `api`. A `config.Config` built without `config.Load` must now name its roles (`config.AllRoles()` for all of them): `app.New` refuses an empty set. - **Leases for work that must run in one process at a time, and the sweeper runs under one** (`internal/coord/` (new: `coord.go`, `local.go`, `elect.go`, `coordtest/`, + tests), `internal/app/{app,wire}.go` (+ tests), `internal/config/backends.go`, `internal/ingest/sweeper.go`, `.github/labeler.yml`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development,ingest-pipeline}.md`, `docs/src/content/docs/configuration.mdx`, `config.yaml`, `AGENTS.md`): part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `coord.Coordinator` hands out named leases (`TryAcquire` → a `Term` with a strictly increasing fencing `Token`, a `Done` channel and `Resign`; `ErrHeld` while another holder's term is live), and `coord.RunElected` runs a loop only while its process holds the lease, resigning when the loop returns and campaigning again every 2s. `coord.Local` is the in-process implementation, and `coordtest.Conformance` is the suite every implementation runs — the NATS KV backend that lets several replicas share one queue comes next. The sweeper now runs through `RunElected` under the `sweeper` lease; with the in-process coordinator the one process always holds it, so nothing changes for a single-process deployment beyond one `coord: elected` log line at startup. `coord.backend` now selects the coordinator (`local`, the only value), so a `config.Config` built without `config.Load` must name it as well as the other three layers' backends. diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index ac449529b..dfd1b1400 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -115,7 +115,7 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ - **cache.go** — `Cache` interface: `Lookup`, `Set`, `Invalidate`, `InvalidateTenant`, `Close`, plus `QueryTimeToTTL`, which sets a result's TTL from how long its query took (10 s floor, 1 h ceiling). Every entry is one tenant's: `Lookup` takes the tenant, the caller's query key — `:query:`, built by the two cached handlers in `api/` (`queryCacheKey`, with the tenant read off the request's store — `settings.Store.Tenant`), which use it as their [singleflight](https://pkg.go.dev/golang.org/x/sync/singleflight) key too — and the `Namespace`s the result depends on — one for a structured query, none yet for a pipe (a pipe's table dependencies are [#343](https://github.com/Wave-RF/WaveHouse/pull/343)) — each of that tenant (another tenant's is `ErrForeignDependency`) and naming a table and scope by their raw names, which the cache escapes where it builds a key. `Lookup` returns the `Entry` (a nil value is a miss) and a `Snapshot` of the versions it read; on a miss the handler runs the query and passes that snapshot to `Set`, so a result is filed under the versions read *before* its query ran, and a write that lands while it runs orphans the fill rather than re-homing pre-write rows under the post-write versions ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)). The snapshot is taken before any input a bump invalidates is chosen, the tenant's connection included: a reload that moves the tenant to another address or database runs `Pools.Reconcile` and then `InvalidateTenant` (a repoint that keeps both, such as a username or `tls` change, reads the same tables and bumps nothing), so a request that took the old pool files the old database's rows under a version that bump orphans, whether its `Set` lands before the bump or after. The singleflight leader's snapshot is the one used. `Set` returns an error only when the backend failed; a value the cache declines — larger than it keeps, a non-positive TTL, a zero snapshot — is not one. What a hit, a miss and a bump mean is pinned by the conformance suite every backend runs, `internal/testutil/cachetest`. - **local.go** — `LocalCache`, the in-process L1 on [Ristretto](https://github.com/dgraph-io/ristretto): one pool shared by every tenant (a heavier tenant holds more of it), sized by the boot config's `cache.l1_max_cost`. - **version_manager.go** — `VersionManager`, the invalidation index behind `Invalidate` and `InvalidateTenant`: a namespace key is `..
.
.`, its fields joined with `keyenc.Join` so a dot in a table or scope name is escaped rather than read as a separator, and an entry's key, `|.||…` with the caller's query key escaped whole (its `:` become `%3A`), folds in the tenant's version and each dependency's namespace key and namespace version, so bumping a table (a scopeless write) or one scope — scope is reserved and empty today, so every write is the whole-table bump — orphans every dependent entry without touching the pool. The tenant leads every key ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8): the same table under two tenants is two namespaces, so a bump through `Invalidate` under one tenant never touches — and a read under one tenant is never served — the other's results, and the flat directory's single tenant simply carries the `0` prefix. `BumpTenant` (behind `InvalidateTenant`) advances the tenant version that leads every namespace key of one tenant, orphaning its every namespace and every cached query in one step (the tenant version is folded into every query key, so a pipe result with no dependencies is orphaned too) — a table no bump ever keyed included, which an enumeration of the index would miss — for a tenant back on a pool after an absence from the fan-out, or moved to another address or database (story 6). The index is per tenant; the cross-tenant invalidation an insert into a shared table needs is not the index's but the wiring's: `internal/app` hands the ingest worker a cache (`sharedTables`) that repeats each bump under every tenant on the same ClickHouse address and database. -- **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — only `GET`, `SET` and `MGET`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot; the table and scope are escaped with `keyenc` after the fixed prefix, so a `:` in a name is never read as the separator. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, the hash over their escaped forms, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, and a token key holding anything but a token (a string of another length, a list, a hash) is replaced, so a token lost to eviction, expiry or a restart is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. While this process owes a bump on any of a lookup's tokens, that lookup is a bypass that files nothing: the bump would orphan whatever it found (other processes, which cannot know, serve those entries until it lands). `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing, each attempt bounded by `DialTimeout` (1 s), and a cluster client's topology read after the handshake by the larger of it and `Timeout`, which is also how long a connection waits on a silent server before it is redialed. Nothing selects this backend yet — the boot config and wiring arrive with [#613](https://github.com/Wave-RF/WaveHouse/issues/613)'s E4. +- **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — only `GET`, `SET` and `MGET`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot; the table and scope are escaped with `keyenc` after the fixed prefix, so a `:` in a name is never read as the separator. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, the hash over their escaped forms, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, and a token key holding anything but a token (a string of another length, a list, a hash) is replaced, so a token lost to eviction, expiry or a restart is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. While this process owes a bump on any of a lookup's tokens, that lookup is a bypass that files nothing: the bump would orphan whatever it found (other processes, which cannot know, serve those entries until it lands). `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing, each attempt bounded by `DialTimeout` (1 s), and a cluster client's topology read after the handshake by the larger of it and `Timeout`, which is also how long a connection waits on a silent server before it is redialed. Nothing selects this backend yet: `cache.backend` accepts only `local` until [#613](https://github.com/Wave-RF/WaveHouse/issues/613)'s E4 adds `redis`, the `cache.redis.*` settings and the wiring. - **redis_codec.go** — the key schema (`tokenKeys`, `bumpKeys`, `valueKey`, which take a `Namespace`'s raw names and escape them) and the value frame: format, flags, expiry, the token list, the payload, zstd-compressed from `CompressMinBytes` when that is smaller. A stored value is capped at `MaxValueBytes` and a decoded one at eight times that, which refuses a zip bomb planted in a shared server; an unknown format, as a newer process writes during a rolling upgrade, is a miss. - **breaker.go** — the circuit breaker: `BreakerThreshold` (5) consecutive failures open it, and while open every operation skips the server at once; after `BreakerOpenFor` (5 s) one background write (`SET :probe`) decides whether it closes, and only that probe's success closes it. Every connection is replaced after a minute (rueidis retries what was in flight), because one that outlives a failover behind a stable address stays on the demoted node, which answers but refuses writes; the replacement re-resolves the address, so a bypassed process reaches the new primary, and delivers the bumps it owes, within about that long. A transport failure or a timeout counts against the server; an error reply saying it takes no writes right now — `READONLY` (a demoted primary), `MASTERDOWN`, `OOM` (full under `maxmemory-policy noeviction`, Redis's default, so give the cache an evicting policy), `NOREPLICAS`, `MISCONF`, `LOADING`, `BUSY`, `CLUSTERDOWN` — opens it at once, whatever the threshold, since no bump can land. Any other reply, an error reply about one key (`WRONGTYPE`, `NOPERM`, `TRYAGAIN`) or one the backend cannot use included, counts as a success, and a caller that gave up first counts as nothing. - **pending.go** — invalidations the server did not take, kept per token key (repeats coalesce) and retried by a background loop until they land: the first one owed at once, then with backoff from 100 ms to 10 s, and while the breaker is open at each probe, which the loop starts when due, so a process that makes no lookups (ingest only) recovers as soon as one that does. `Invalidate` sends its bumps 1,000 to a round trip, as the retry does. Landing late is still correct: a fresh token orphans the pre-write entries and any fill made meanwhile. Past `PendingMax` (100,000) keys the set collapses to one tenant bump per affected tenant. `Close` makes one last attempt at the bumps still owed, past the breaker (an open one is why they are owed); what that attempt cannot deliver is lost, and the entries they would orphan are served until their TTL — the same failure as a worker stopping between an insert and its invalidation. From 1f24da5fff5b087112cc34355cbc12d7a8185a75 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 06:16:03 -0400 Subject: [PATCH 30/45] docs(cache): a reconnect must fit in the operation timeout rueidis dials a replacement connection lazily, under the timeout of the operation that needs it, not DialTimeout. Recovery after a failover within about a minute therefore assumes a fresh connection's dial and handshake, TLS included, fit in Timeout; say so, and that Timeout should be sized above it. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 2 +- docs/src/content/docs/architecture.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2521ef894..563dd2e76 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **A Redis-compatible shared cache backend, built but not yet selectable** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/cache.go`, `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`, `CONTRIBUTING.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag, with the table and scope escaped into the key (`internal/keyenc`) so no two names share a token; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row, or one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like), open a circuit breaker that skips the server until a probe write succeeds (connections are replaced every minute, so after a failover behind a stable address the process reaches the new primary, and delivers the bumps it owes, within about a minute), and deferred invalidations are retried until they land — the first at once, and at the probe's cadence while the breaker is open — collapsing to one tenant-wide bump per tenant past 100,000 keys; until one lands, the process that owes it bypasses the lookups it would orphan. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. Nothing selects it yet: `cache.backend` accepts only `local` until the wiring (E4) adds `redis` and the `cache.redis.*` settings, so every deployment still runs `LocalCache`. Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression, server-stops-answering, refused-writes (a demoted primary, a full `noeviction` server), failover-behind-a-stable-address and owed-invalidation cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. +- **A Redis-compatible shared cache backend, built but not yet selectable** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/cache.go`, `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`, `CONTRIBUTING.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag, with the table and scope escaped into the key (`internal/keyenc`) so no two names share a token; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row, or one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like), open a circuit breaker that skips the server until a probe write succeeds (connections are replaced every minute, so after a failover behind a stable address the process reaches the new primary, and delivers the bumps it owes, within about a minute, provided a fresh connection's dial and handshake fit in the per-operation timeout), and deferred invalidations are retried until they land — the first at once, and at the probe's cadence while the breaker is open — collapsing to one tenant-wide bump per tenant past 100,000 keys; until one lands, the process that owes it bypasses the lookups it would orphan. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. Nothing selects it yet: `cache.backend` accepts only `local` until the wiring (E4) adds `redis` and the `cache.redis.*` settings, so every deployment still runs `LocalCache`. Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression, server-stops-answering, refused-writes (a demoted primary, a full `noeviction` server), failover-behind-a-stable-address and owed-invalidation cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. - **One conformance suite for every `mq.Broker`, and a transient broker failure is a `503`** (`internal/mq/mqtest/` (new: the suite and the embedded broker's run of it), `internal/mq/{mq,embedded}.go` (+ tests), `internal/api/ingest.go` (+ tests), `.testcoverage.yml`, `docs/src/content/docs/{api,architecture}.md`, `AGENTS.md`): the first piece of the external-NATS workstream of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `mqtest.Run` states the `Broker` contract as behavior — round trips with names that need encoding, per-tenant order, `Nak` and `AckWait` redelivery, the trace context reaching `Subscribe`, dead-lettering that keeps the topic and leaves the original unacked, per-tenant dead-letter counts, replay bounds and isolation, and exactly one `failed` report when delivery ends underneath a consumer — through the interfaces alone, so the external backend runs the same cases, with `mqtest.Caps` for the four places where its semantics legitimately differ. The embedded broker passes it; writing it turned up that a durable deleted on several tenants' queues could report on `failed` more than once, which is fixed and pinned by a test that deletes it on one queue after another. It also turned up a replay that lost its connection mid-pull passing for a caught-up one when the pull ended in a timeout; that is an error now, as the `Replayer` contract says. The interface comments now allow a delivery unit that is a partition holding several tenants, a `CreateConsumer` that finds a durable rather than creating one, a `PurgeAcked` that leaves retention to the operator, and zero dead-letter counts where there is no per-tenant queue. A new sentinel, `mq.ErrUnavailable`, is a broker that cannot be reached or does not answer in time: the ingest handler answers it with `503` + `Retry-After: 5` rather than the `500` "publish failed" it would have been. Nothing returns it yet; the external backend of #613 will. - **Process roles: the API and the background workers can run in separate processes** (`internal/config/config.go` (+ `roles_test.go`, `defaults_test.go`), `internal/config/backends.go`, `internal/app/{app,wire}.go` (+ `roles_test.go`), `internal/api/router.go` (+ tests), `tests/integration/{setup,tenants}_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md}`, `AGENTS.md`), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The boot config gains `roles` (`WH_ROLES`, default `api,ingest,sweeper`, set in `defaults()` like every boot default, so an explicit `roles: []` refuses boot) and `instance_id` (`WH_INSTANCE_ID`, default `-<8 hex>`, fresh at every boot; logged at boot, and recorded as a lease holder once a shared `coord.backend` exists). A process wires only what its roles need: `api` runs the HTTP API with schema discovery, the token verifiers, the dedupe stores and the SSE hub (all per API process); `ingest` runs the ingest worker; `sweeper` runs the sweeper under its lease. A process without `api` serves an ops-only listener on `server.port` (the probes and their aliases, `/version`, the same-port metrics path, and `POST /v1/ops/settings/reload`, which takes the operator key alone); every other route answers 404, under `/v1/ops` once the operator key has passed. Boot refuses any split over the embedded MQ, which no other process can reach, and `api` without `ingest` (or the reverse) over a local cache, which the ingest worker's invalidations would never reach. Until a shared `mq.backend` exists, every process therefore runs every role, which is the default, so nothing changes for an existing deployment. `data_dir` is probed for Pebble only in a process running `api`. A `config.Config` built without `config.Load` must now name its roles (`config.AllRoles()` for all of them): `app.New` refuses an empty set. - **Leases for work that must run in one process at a time, and the sweeper runs under one** (`internal/coord/` (new: `coord.go`, `local.go`, `elect.go`, `coordtest/`, + tests), `internal/app/{app,wire}.go` (+ tests), `internal/config/backends.go`, `internal/ingest/sweeper.go`, `.github/labeler.yml`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development,ingest-pipeline}.md`, `docs/src/content/docs/configuration.mdx`, `config.yaml`, `AGENTS.md`): part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `coord.Coordinator` hands out named leases (`TryAcquire` → a `Term` with a strictly increasing fencing `Token`, a `Done` channel and `Resign`; `ErrHeld` while another holder's term is live), and `coord.RunElected` runs a loop only while its process holds the lease, resigning when the loop returns and campaigning again every 2s. `coord.Local` is the in-process implementation, and `coordtest.Conformance` is the suite every implementation runs — the NATS KV backend that lets several replicas share one queue comes next. The sweeper now runs through `RunElected` under the `sweeper` lease; with the in-process coordinator the one process always holds it, so nothing changes for a single-process deployment beyond one `coord: elected` log line at startup. `coord.backend` now selects the coordinator (`local`, the only value), so a `config.Config` built without `config.Load` must name it as well as the other three layers' backends. diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index dfd1b1400..51239a095 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -117,7 +117,7 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ - **version_manager.go** — `VersionManager`, the invalidation index behind `Invalidate` and `InvalidateTenant`: a namespace key is `..
.
.`, its fields joined with `keyenc.Join` so a dot in a table or scope name is escaped rather than read as a separator, and an entry's key, `|.||…` with the caller's query key escaped whole (its `:` become `%3A`), folds in the tenant's version and each dependency's namespace key and namespace version, so bumping a table (a scopeless write) or one scope — scope is reserved and empty today, so every write is the whole-table bump — orphans every dependent entry without touching the pool. The tenant leads every key ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8): the same table under two tenants is two namespaces, so a bump through `Invalidate` under one tenant never touches — and a read under one tenant is never served — the other's results, and the flat directory's single tenant simply carries the `0` prefix. `BumpTenant` (behind `InvalidateTenant`) advances the tenant version that leads every namespace key of one tenant, orphaning its every namespace and every cached query in one step (the tenant version is folded into every query key, so a pipe result with no dependencies is orphaned too) — a table no bump ever keyed included, which an enumeration of the index would miss — for a tenant back on a pool after an absence from the fan-out, or moved to another address or database (story 6). The index is per tenant; the cross-tenant invalidation an insert into a shared table needs is not the index's but the wiring's: `internal/app` hands the ingest worker a cache (`sharedTables`) that repeats each bump under every tenant on the same ClickHouse address and database. - **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — only `GET`, `SET` and `MGET`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot; the table and scope are escaped with `keyenc` after the fixed prefix, so a `:` in a name is never read as the separator. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, the hash over their escaped forms, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, and a token key holding anything but a token (a string of another length, a list, a hash) is replaced, so a token lost to eviction, expiry or a restart is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. While this process owes a bump on any of a lookup's tokens, that lookup is a bypass that files nothing: the bump would orphan whatever it found (other processes, which cannot know, serve those entries until it lands). `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing, each attempt bounded by `DialTimeout` (1 s), and a cluster client's topology read after the handshake by the larger of it and `Timeout`, which is also how long a connection waits on a silent server before it is redialed. Nothing selects this backend yet: `cache.backend` accepts only `local` until [#613](https://github.com/Wave-RF/WaveHouse/issues/613)'s E4 adds `redis`, the `cache.redis.*` settings and the wiring. - **redis_codec.go** — the key schema (`tokenKeys`, `bumpKeys`, `valueKey`, which take a `Namespace`'s raw names and escape them) and the value frame: format, flags, expiry, the token list, the payload, zstd-compressed from `CompressMinBytes` when that is smaller. A stored value is capped at `MaxValueBytes` and a decoded one at eight times that, which refuses a zip bomb planted in a shared server; an unknown format, as a newer process writes during a rolling upgrade, is a miss. -- **breaker.go** — the circuit breaker: `BreakerThreshold` (5) consecutive failures open it, and while open every operation skips the server at once; after `BreakerOpenFor` (5 s) one background write (`SET :probe`) decides whether it closes, and only that probe's success closes it. Every connection is replaced after a minute (rueidis retries what was in flight), because one that outlives a failover behind a stable address stays on the demoted node, which answers but refuses writes; the replacement re-resolves the address, so a bypassed process reaches the new primary, and delivers the bumps it owes, within about that long. A transport failure or a timeout counts against the server; an error reply saying it takes no writes right now — `READONLY` (a demoted primary), `MASTERDOWN`, `OOM` (full under `maxmemory-policy noeviction`, Redis's default, so give the cache an evicting policy), `NOREPLICAS`, `MISCONF`, `LOADING`, `BUSY`, `CLUSTERDOWN` — opens it at once, whatever the threshold, since no bump can land. Any other reply, an error reply about one key (`WRONGTYPE`, `NOPERM`, `TRYAGAIN`) or one the backend cannot use included, counts as a success, and a caller that gave up first counts as nothing. +- **breaker.go** — the circuit breaker: `BreakerThreshold` (5) consecutive failures open it, and while open every operation skips the server at once; after `BreakerOpenFor` (5 s) one background write (`SET :probe`) decides whether it closes, and only that probe's success closes it. Every connection is replaced after a minute (rueidis retries what was in flight), because one that outlives a failover behind a stable address stays on the demoted node, which answers but refuses writes; the replacement re-resolves the address, so a bypassed process reaches the new primary, and delivers the bumps it owes, within about that long. rueidis dials a replacement lazily, under the `Timeout` of the operation that needs it rather than `DialTimeout`, so this holds only where a fresh connection's dial and handshake (TLS included) fit in `Timeout`; size `Timeout` above that, or every reconnect, at the lifetime or after a drop, fails and the next one starts over. A transport failure or a timeout counts against the server; an error reply saying it takes no writes right now — `READONLY` (a demoted primary), `MASTERDOWN`, `OOM` (full under `maxmemory-policy noeviction`, Redis's default, so give the cache an evicting policy), `NOREPLICAS`, `MISCONF`, `LOADING`, `BUSY`, `CLUSTERDOWN` — opens it at once, whatever the threshold, since no bump can land. Any other reply, an error reply about one key (`WRONGTYPE`, `NOPERM`, `TRYAGAIN`) or one the backend cannot use included, counts as a success, and a caller that gave up first counts as nothing. - **pending.go** — invalidations the server did not take, kept per token key (repeats coalesce) and retried by a background loop until they land: the first one owed at once, then with backoff from 100 ms to 10 s, and while the breaker is open at each probe, which the loop starts when due, so a process that makes no lookups (ingest only) recovers as soon as one that does. `Invalidate` sends its bumps 1,000 to a round trip, as the retry does. Landing late is still correct: a fresh token orphans the pre-write entries and any fill made meanwhile. Past `PendingMax` (100,000) keys the set collapses to one tenant bump per affected tenant. `Close` makes one last attempt at the bumps still owed, past the breaker (an open one is why they are owed); what that attempt cannot deliver is lost, and the entries they would orphan are served until their TTL — the same failure as a worker stopping between an insert and its invalidation. - **metrics.go** — the shared backend's instruments (meter `wavehouse-cache`, `backend="redis"`, no tenant attribute): `wavehouse_cache_lookups_total` by `result` (`hit`, `miss`, `stale` — filed under since-bumped tokens, `bypass` — server skipped, or held by a bump this process owes, `error`), `wavehouse_cache_op_duration_seconds` by `op` (`lookup`, `set`, `invalidate`), `wavehouse_cache_breaker_open` (1 while bypassed, including before the first connection), `wavehouse_cache_invalidations_total` by `result` (`ok`, `deferred`), `wavehouse_cache_invalidations_pending` (bumps owed: entries they would orphan may be served stale meanwhile), `wavehouse_cache_value_bytes` (stored size), `wavehouse_cache_oversize_total` and `wavehouse_cache_set_failures_total` by `reason` (`oom`, `timeout`, `other`). From 5b31ce6d97bc0b176e172219e2ee9cf63a7c4db0 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 06:23:37 -0400 Subject: [PATCH 31/45] fix(config): cap cache.redis timeout and dial_timeout at 1s Since #626 bounds a cluster client's topology read by the larger of the two, a dial held by boot or Close is a connect and a handshake, each up to dial_timeout, plus that read. At a 2s dial cap it could reach 6s, past the 5s release budget. Both caps are now 1s, so at most 3s. The shared-cache docs now describe #626's behavior: a refused write opens the breaker, the owing instance bypasses the lookups it would orphan, and connections are replaced every minute after a failover behind a stable address. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 2 +- docs/src/content/docs/architecture.md | 2 +- docs/src/content/docs/configuration.mdx | 6 +++--- docs/src/content/docs/deployment.md | 6 +++--- internal/config/cache_redis.go | 23 ++++++++++++++++------- internal/config/cache_redis_test.go | 9 +++++---- tests/integration/shared_cache_test.go | 6 +++--- 7 files changed, 32 insertions(+), 22 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ae2b6ac50..51ed88656 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **`cache.backend: redis` shares the query cache across instances** (`internal/config/{cache_redis,backends,config}.go` (+ tests), `internal/app/wire.go` (+ tests), `internal/ingest/worker.go`, `tests/integration/shared_cache_test.go`, `scripts/orchestrator/main.go`, `tests/e2e/fixtures/config.yaml`, `.testcoverage.yml`, `deployments/compose/dependencies.yaml`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md,settings-directory.mdx,getting-started.md,pipes.mdx,api.md,development.md}`, `README.md`, `AGENTS.md`): PR E4 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config's `cache.backend` now takes `redis`, configured by a new `cache.redis` block (`WH_CACHE_REDIS_*`): `addrs` (required), `mode` (`standalone` or `cluster`; `sentinel` refuses boot until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `username`, `password` (a secret — set it through the environment), `db`, `tls.{enabled,ca_file,cert_file,key_file,server_name,insecure_skip_verify}`, `key_prefix` (`wh`), `timeout` (`100ms`), `dial_timeout` (`1s`, at most `2s`: boot and shutdown each wait out a dial), `max_value_bytes` (1 MiB), `compress_min_bytes` (`1024`; `0` never compresses) and `version_ttl` (`168h`). Every instance pointed at one server shares its cached results, and an insert on any instance invalidates every instance's. It is also the shared cache a split of [`roles`](https://github.com/Wave-RF/WaveHouse/pull/622) needs: the refusal of `api` without `ingest`, or the reverse, over `cache.backend: local` now names `redis`, though every split is still refused while the queue is embedded. A malformed block — no address, an address without a port, a URL-style address (refused without repeating it, since it may carry a password), an unknown mode, `db` other than `0` in cluster mode, an unreadable TLS file, a TLS key set while `tls.enabled` is off — refuses boot; an unreachable server, or one that rejects the password, does not: the process boots with the cache bypassed and keeps reconnecting, logging a rejected password at `ERROR` on every attempt, so a rotated secret cannot crash-loop every instance at once. `insecure_skip_verify`, and a `cache.redis.addrs` set while `cache.backend` is `local`, are logged at `WARN` at boot. The ingest worker's log of an invalidation that did not land drops from `ERROR` to `WARN`, since the shared backend defers and retries it: an outage would otherwise log an `ERROR` for every batch. The e2e suite now runs against a Redis testcontainer with `cache.backend: redis`, so the shared backend is exercised end to end, and the e2e coverage exclude for its files is gone; the unit and integration suites keep covering `local`. An integration test boots two instances over one Redis and one ClickHouse: a result one fills is a hit for the other, and a row ingested through one is served fresh by the other on its next query, well inside the stale entry's TTL; another pauses Redis and checks queries keep succeeding from ClickHouse, then turn back to hits; a third runs the first one's hit, insert and fresh-miss lifecycle on the suite's own `cache.backend: local` app, since e2e no longer exercises that backend. `deployments/compose/dependencies.yaml` gains an optional `redis` profile for local multi-instance work. The deployment guide gains a "Multiple instances and the shared cache" section: what each instance keeps to itself, what a reader on another instance can see and when, `maxmemory-policy`, and the metrics to alert on. +- **`cache.backend: redis` shares the query cache across instances** (`internal/config/{cache_redis,backends,config}.go` (+ tests), `internal/app/wire.go` (+ tests), `internal/ingest/worker.go`, `tests/integration/shared_cache_test.go`, `scripts/orchestrator/main.go`, `tests/e2e/fixtures/config.yaml`, `.testcoverage.yml`, `deployments/compose/dependencies.yaml`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md,settings-directory.mdx,getting-started.md,pipes.mdx,api.md,development.md}`, `README.md`, `AGENTS.md`): PR E4 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config's `cache.backend` now takes `redis`, configured by a new `cache.redis` block (`WH_CACHE_REDIS_*`): `addrs` (required), `mode` (`standalone` or `cluster`; `sentinel` refuses boot until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `username`, `password` (a secret — set it through the environment), `db`, `tls.{enabled,ca_file,cert_file,key_file,server_name,insecure_skip_verify}`, `key_prefix` (`wh`), `timeout` (`100ms`) and `dial_timeout` (`1s`), each at most `1s` since boot and shutdown each wait out a connection attempt they bound, `max_value_bytes` (1 MiB), `compress_min_bytes` (`1024`; `0` never compresses) and `version_ttl` (`168h`). Every instance pointed at one server shares its cached results, and an insert on any instance invalidates every instance's. It is also the shared cache a split of [`roles`](https://github.com/Wave-RF/WaveHouse/pull/622) needs: the refusal of `api` without `ingest`, or the reverse, over `cache.backend: local` now names `redis`, though every split is still refused while the queue is embedded. A malformed block — no address, an address without a port, a URL-style address (refused without repeating it, since it may carry a password), an unknown mode, `db` other than `0` in cluster mode, an unreadable TLS file, a TLS key set while `tls.enabled` is off — refuses boot; an unreachable server, or one that rejects the password, does not: the process boots with the cache bypassed and keeps reconnecting, logging a rejected password at `ERROR` on every attempt, so a rotated secret cannot crash-loop every instance at once. `insecure_skip_verify`, and a `cache.redis.addrs` set while `cache.backend` is `local`, are logged at `WARN` at boot. The ingest worker's log of an invalidation that did not land drops from `ERROR` to `WARN`, since the shared backend defers and retries it: an outage would otherwise log an `ERROR` for every batch. The e2e suite now runs against a Redis testcontainer with `cache.backend: redis`, so the shared backend is exercised end to end, and the e2e coverage exclude for its files is gone; the unit and integration suites keep covering `local`. An integration test boots two instances over one Redis and one ClickHouse: a result one fills is a hit for the other, and a row ingested through one is served fresh by the other on its next query, well inside the stale entry's TTL; another pauses Redis and checks queries keep succeeding from ClickHouse, then turn back to hits; a third runs the first one's hit, insert and fresh-miss lifecycle on the suite's own `cache.backend: local` app, since e2e no longer exercises that backend. `deployments/compose/dependencies.yaml` gains an optional `redis` profile for local multi-instance work. The deployment guide gains a "Multiple instances and the shared cache" section: what each instance keeps to itself, what a reader on another instance can see and when, `maxmemory-policy`, and the metrics to alert on. - **A Redis-compatible shared cache backend** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/cache.go`, `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`, `CONTRIBUTING.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag, with the table and scope escaped into the key (`internal/keyenc`) so no two names share a token; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row, or one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like), open a circuit breaker that skips the server until a probe write succeeds (connections are replaced every minute, so after a failover behind a stable address the process reaches the new primary, and delivers the bumps it owes, within about a minute, provided a fresh connection's dial and handshake fit in the per-operation timeout), and deferred invalidations are retried until they land — the first at once, and at the probe's cadence while the breaker is open — collapsing to one tenant-wide bump per tenant past 100,000 keys; until one lands, the process that owes it bypasses the lookups it would orphan. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. `cache.backend: redis` selects it (the entry above). Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression, server-stops-answering, refused-writes (a demoted primary, a full `noeviction` server), failover-behind-a-stable-address and owed-invalidation cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. - **One conformance suite for every `mq.Broker`, and a transient broker failure is a `503`** (`internal/mq/mqtest/` (new: the suite and the embedded broker's run of it), `internal/mq/{mq,embedded}.go` (+ tests), `internal/api/ingest.go` (+ tests), `.testcoverage.yml`, `docs/src/content/docs/{api,architecture}.md`, `AGENTS.md`): the first piece of the external-NATS workstream of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `mqtest.Run` states the `Broker` contract as behavior — round trips with names that need encoding, per-tenant order, `Nak` and `AckWait` redelivery, the trace context reaching `Subscribe`, dead-lettering that keeps the topic and leaves the original unacked, per-tenant dead-letter counts, replay bounds and isolation, and exactly one `failed` report when delivery ends underneath a consumer — through the interfaces alone, so the external backend runs the same cases, with `mqtest.Caps` for the four places where its semantics legitimately differ. The embedded broker passes it; writing it turned up that a durable deleted on several tenants' queues could report on `failed` more than once, which is fixed and pinned by a test that deletes it on one queue after another. It also turned up a replay that lost its connection mid-pull passing for a caught-up one when the pull ended in a timeout; that is an error now, as the `Replayer` contract says. The interface comments now allow a delivery unit that is a partition holding several tenants, a `CreateConsumer` that finds a durable rather than creating one, a `PurgeAcked` that leaves retention to the operator, and zero dead-letter counts where there is no per-tenant queue. A new sentinel, `mq.ErrUnavailable`, is a broker that cannot be reached or does not answer in time: the ingest handler answers it with `503` + `Retry-After: 5` rather than the `500` "publish failed" it would have been. Nothing returns it yet; the external backend of #613 will. - **Process roles: the API and the background workers can run in separate processes** (`internal/config/config.go` (+ `roles_test.go`, `defaults_test.go`), `internal/config/backends.go`, `internal/app/{app,wire}.go` (+ `roles_test.go`), `internal/api/router.go` (+ tests), `tests/integration/{setup,tenants}_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md}`, `AGENTS.md`), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The boot config gains `roles` (`WH_ROLES`, default `api,ingest,sweeper`, set in `defaults()` like every boot default, so an explicit `roles: []` refuses boot) and `instance_id` (`WH_INSTANCE_ID`, default `-<8 hex>`, fresh at every boot; logged at boot, and recorded as a lease holder once a shared `coord.backend` exists). A process wires only what its roles need: `api` runs the HTTP API with schema discovery, the token verifiers, the dedupe stores and the SSE hub (all per API process); `ingest` runs the ingest worker; `sweeper` runs the sweeper under its lease. A process without `api` serves an ops-only listener on `server.port` (the probes and their aliases, `/version`, the same-port metrics path, and `POST /v1/ops/settings/reload`, which takes the operator key alone); every other route answers 404, under `/v1/ops` once the operator key has passed. Boot refuses any split over the embedded MQ, which no other process can reach, and `api` without `ingest` (or the reverse) over a local cache, which the ingest worker's invalidations would never reach. Until a shared `mq.backend` exists, every process therefore runs every role, which is the default, so nothing changes for an existing deployment. `data_dir` is probed for Pebble only in a process running `api`. A `config.Config` built without `config.Load` must now name its roles (`config.AllRoles()` for all of them): `app.New` refuses an empty set. diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index fdab8bbe1..d3e479627 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -126,7 +126,7 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ - **config.go** — Loads *boot* configuration from a YAML file with environment variable overrides (using [cleanenv](https://github.com/ilyakaznacheev/cleanenv)); every key has a `WH_`-prefixed env var. Boot config is only what can't change under a running process — the implementation each layer runs on, the process's `roles`, resource sizing, listeners, observability exporters, the settings-directory path, and the secrets (`clickhouse.password`, `cache.redis.password`, `auth.jwt_secret`, `auth.operator_key`). Everything tenant-tunable lives in the settings directory (`settings/`). Both sources are strict: `Load` refuses to boot naming every YAML key the struct doesn't declare (`strict.go`) and every `WH_*` environment variable no field binds (`check.go`), so a tunable that moved to the settings directory can't be read, ignored, and believed. Boot is the validator for this half — there is no dry-run command. See [Configuration Reference](/configuration). - **check.go** — `rejectUnboundEnv` is the environment half of the strict loader: `unboundEnv` walks the struct's `env` tags (plus the two process-level names, `WH_CONFIG` and `WH_LOG_LEVEL`) against the environment; `CheckDataDir` probes `data_dir` — run by `main` right after `Load` when `Config.NeedsDataDir` says a selected backend keeps state there, so an unusable `data_dir` refuses boot before anything dials out. It refuses an empty value (reachable through `WH_DATA_DIR=`) outright rather than probing the working directory; a path that exists and is not a directory; a dangling symlink at `data_dir` or any component above it (the walk to the nearest existing ancestor uses `Lstat`, so a failed mount is not skipped over as "does not exist"); and a directory the process cannot write to — or, when it does not exist, an unwritable nearest ancestor — probed by creating and removing one temp file. A permission denial, on the probe or on reaching the path through a parent without search permission, carries the UID-65532 hint, since a bind mount owned by root is the typical cause. - **backends.go** — the `.backend` keys: one string type per layer (`MQBackend`, `CacheBackend`, `DedupeBackend`, `CoordBackend`), each with its list of the backends this build has, and a `validate` per layer block (`checkBackend`, run by `validateBackends` in `Validate`, between `validateRoles` and `validateTopology`), which refuses a value not on the list and names the ones that are. A backend's own settings go in a `.` sub-block that is its `validate`'s case to check. `Distributed` reports whether the MQ is shared with other processes, `NeedsDataDir` whether a selected backend keeps state under `data_dir`, and `Warnings` returns what a valid configuration is still likely to get wrong — the combinations that are correct for one replica only (a shared MQ over a local cache or Pebble dedupe), a `cache.redis` block that is not read, certificate verification turned off — which `app.New` logs at `WARN`. -- **cache_redis.go** — `CacheRedisConfig`, the `cache.redis` sub-block, and its checks: an address, each `host:port` (a URL or `user:password@` form refused without repeating it, since it may hold a password), a known mode (`sentinel` is refused until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `db` 0 in cluster mode, positive timeouts and sizes, a `dial_timeout` of at most 2 s (boot and `Close` each wait out a dial, a connect and a handshake bounded by it), a `version_ttl` of at least 2 s, and a `compress_min_bytes` that is not negative (`0` never compresses); its defaults are in `defaults()` with the rest. `CacheRedisTLS.Config` builds the `tls.Config`, reading the files; `Validate` calls it so an unreadable file refuses boot, and `internal/app` calls it again to build the connection. A TLS key set while `tls.enabled` is off is an error rather than a plaintext connection. +- **cache_redis.go** — `CacheRedisConfig`, the `cache.redis` sub-block, and its checks: an address, each `host:port` (a URL or `user:password@` form refused without repeating it, since it may hold a password), a known mode (`sentinel` is refused until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `db` 0 in cluster mode, positive timeouts and sizes, a `timeout` and `dial_timeout` of at most 1 s each (boot and `Close` each wait out a dial: a connect and a handshake bounded by `dial_timeout`, and a cluster's topology read bounded by the larger of the two), a `version_ttl` of at least 2 s, and a `compress_min_bytes` that is not negative (`0` never compresses); its defaults are in `defaults()` with the rest. `CacheRedisTLS.Config` builds the `tls.Config`, reading the files; `Validate` calls it so an unreadable file refuses boot, and `internal/app` calls it again to build the connection. A TLS key set while `tls.enabled` is off is an error rather than a plaintext connection. - **config.go**, roles — `roles` (`[]Role`: `api`, `ingest`, `sweeper`; `AllRoles` by default; `Has(Role)`) picks which components `internal/app` wires, and `instance_id` names the process (`-<8 hex>` when empty, resolved in `Load`; today only logged at boot, and a distributed coordinator will record it as a lease's holder). `validateRoles` refuses an empty list, an empty entry, an unknown or a repeated role; `validateTopology` refuses a role set the backends cannot serve: any split over the embedded MQ, and a process with exactly one of `api` and `ingest` over a local cache. `NeedsDataDir` counts Pebble only for a process running `api`, and `Warnings` is empty without `api`, since only that role opens a cache it reads or a dedupe store. - **strict.go** — `rejectUnknownKeys`, the YAML half: re-reads the file as a generic tree and walks it against the struct's `yaml` tags, listing every key the struct doesn't declare. cleanenv itself is lenient by design, which is exactly wrong for boot config once keys have moved to the settings directory. - **persistence.go** — `WarnIfFreshDataDir` logs the startup `WARN` when `data_dir` is missing or empty (on a redeploy, the sign that the volume didn't persist); `LogStorageInitError` attaches the UID-65532 `permissionHint` to a NATS or Pebble open failure that looks like a permission denial — the same hint string `CheckDataDir` uses. diff --git a/docs/src/content/docs/configuration.mdx b/docs/src/content/docs/configuration.mdx index d55936e55..509088aee 100644 --- a/docs/src/content/docs/configuration.mdx +++ b/docs/src/content/docs/configuration.mdx @@ -162,13 +162,13 @@ The `redis` backend's settings, read only when `cache.backend` is `redis`. It is | `cache.redis.tls.server_name` | `WH_CACHE_REDIS_TLS_SERVER_NAME` | *(empty)* | Name to verify the server's certificate against, when it differs from the address. | | `cache.redis.tls.insecure_skip_verify` | `WH_CACHE_REDIS_TLS_INSECURE_SKIP_VERIFY` | `false` | Accept any server certificate. Logged at `WARN` at boot: whoever can intercept the connection can read and replace cached results. | | `cache.redis.key_prefix` | `WH_CACHE_REDIS_KEY_PREFIX` | `wh` | Leads every key, so several deployments can share one server, provided you trust each as much as the others: any of them can overwrite what the rest serve. No `{` or `}`. | -| `cache.redis.timeout` | `WH_CACHE_REDIS_TIMEOUT` | `100ms` | Per operation. A lookup or fill that takes longer is a miss or a skipped fill, never a failed query. | -| `cache.redis.dial_timeout` | `WH_CACHE_REDIS_DIAL_TIMEOUT` | `1s` | Per connection attempt, at most `2s`. An attempt is a connect and a handshake, each bounded by this, and both boot and shutdown wait out one in flight, so the cap keeps a stop inside the release's fixed 5s budget (see [Stopping](/deployment#stopping)). | +| `cache.redis.timeout` | `WH_CACHE_REDIS_TIMEOUT` | `100ms` | Per operation, at most `1s`. A lookup or fill that takes longer is a miss or a skipped fill, never a failed query. | +| `cache.redis.dial_timeout` | `WH_CACHE_REDIS_DIAL_TIMEOUT` | `1s` | Per connection attempt, at most `1s`. An attempt is a connect and a handshake, each bounded by this, and in `cluster` mode a topology read bounded by the larger of this and `timeout`. Boot and shutdown each wait out one in flight, so the two caps keep that under 3s, inside a stop's fixed 5s release budget (see [Stopping](/deployment#stopping)). | | `cache.redis.max_value_bytes` | `WH_CACHE_REDIS_MAX_VALUE_BYTES` | `1048576` | Largest result stored, after compression (1 MiB). A larger one is returned to the caller but not cached. | | `cache.redis.compress_min_bytes` | `WH_CACHE_REDIS_COMPRESS_MIN_BYTES` | `1024` | Results at least this large are zstd-compressed when that makes them smaller. `0` never compresses. | | `cache.redis.version_ttl` | `WH_CACHE_REDIS_VERSION_TTL` | `168h` | How long a table's or tenant's version token outlives its last write, so the tokens of dropped tables and removed tenants eventually expire. At least `2s`. An expired token only causes misses. | -**When the server is unreachable or misbehaves, the cache is bypassed; queries are not.** A failure or a timeout makes the lookup a miss and the fill a no-op, and five in a row open a circuit breaker that skips the server entirely until a probe, every 5 s, gets an answer. Queries then go straight to ClickHouse, still coalesced per instance by `singleflight`. An invalidation the server did not take is kept and retried until it lands. `/readyz` does not depend on the cache. +**When the server is unreachable or misbehaves, the cache is bypassed; queries are not.** A failure or a timeout makes the lookup a miss and the fill a no-op. Five in a row, or one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like), open a circuit breaker that skips the server entirely until a probe write, every 5 s, succeeds. Queries then go straight to ClickHouse, still coalesced per instance by `singleflight`. An invalidation the server did not take is kept and retried until it lands, and until then the instance that owes it bypasses the lookups it would orphan. `/readyz` does not depend on the cache. **Boot does not wait for the server.** A malformed block (an address without a port, `mode: cluster` with `db` other than `0`, an unreadable or unparsable TLS file) refuses boot. A server that cannot be reached, or that refuses the credentials, does not: the process boots with the cache bypassed and keeps reconnecting, with backoff up to 30 s. A rejected password (`WRONGPASS`, `NOAUTH`) is logged at `ERROR` on every attempt; any other failure at `WARN`. This is deliberate: a rotated Redis password must not crash-loop every instance at once. Watch `wavehouse_cache_breaker_open`, which reads `1` while the cache is bypassed, including before the first connection. diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index fc366405d..1e14399b2 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -432,12 +432,12 @@ The query-result cache is the layer that can be shared today. With the default ` **What another instance can see.** Ingest is already asynchronous: `/v1/ingest` answers before the batch is inserted. Once the inserting instance's worker has written the batch to ClickHouse, it replaces the table's version token in Redis, and from then on a lookup on any instance misses and reads the new rows. The cache adds no delay of its own beyond that single write. The exceptions: - **The server is unreachable from the inserting instance.** The invalidation is kept and retried until it lands (`wavehouse_cache_invalidations_pending` counts what is owed). Meanwhile other instances that can still reach the server keep serving the older results, for as long as the outage lasts and at most until each entry's TTL. An instance that stops while invalidations are still owed loses them, with the same bound. The same thing happens today when a process stops between an insert and its invalidation. -- **A failover to a replica that had not yet received the latest token writes** can bring back entries filed under the older tokens, bounded by the replication lag at the moment of failover and those entries' TTL. WaveHouse never reads from replicas. -- **The server is full and `maxmemory-policy` is `noeviction`.** It refuses the token writes, so invalidations are kept and retried, and until one lands every instance serves the results from before the insert, up to their TTL. +- **A failover to a replica that had not yet received the latest token writes** can bring back entries filed under the older tokens, bounded by the replication lag at the moment of failover and those entries' TTL. WaveHouse never reads from replicas. Behind a stable address (a managed primary endpoint), an instance still connected to the demoted node has its writes refused, which bypasses its cache; connections are replaced every minute, so it reaches the new primary and delivers the invalidations it owes within about that long. +- **The server is full and `maxmemory-policy` is `noeviction`.** It refuses the token writes. The inserting instance keeps its invalidations and retries them, bypassing its cache meanwhile, but every other instance serves the results from before the insert until one lands, up to their TTL. - **A pipe that writes** (an `INSERT` in `pipes.json`) has its result cached like a read, so a repeated identical call is answered from the cache and the write does not run again ([#386](https://github.com/Wave-RF/WaveHouse/issues/386)). With a shared cache that holds on every instance, until the entry's TTL. - **Admin writes through `POST /v1/ops/query`** do not invalidate the cache ([#394](https://github.com/Wave-RF/WaveHouse/issues/394)). With a shared cache, the stale results they leave are served by every instance, not only one. -**Sizing the server.** Every key WaveHouse writes has a TTL, and a version token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses, never bring back an entry it had invalidated. So set `maxmemory` and let the server evict: `maxmemory-policy allkeys-lru` (or `allkeys-lfu`, `volatile-lru`, `volatile-lfu`). Under `noeviction`, a full server refuses the writes. Fills then fail (counted by `wavehouse_cache_set_failures_total{reason="oom"}`), only lookups whose version tokens already exist keep working, and invalidations are kept and retried, so the pre-insert results above stay served: avoid `noeviction`. A stored result is capped at `cache.redis.max_value_bytes` (1 MiB compressed). A tenant's version tokens share one hash tag, so each lookup reads them in one `MGET` in cluster mode as well. The results themselves carry no hash tag and spread across shards. Persistence is not needed: an empty server after a restart is a cold cache, not a wrong one. +**Sizing the server.** Every key WaveHouse writes has a TTL, and a version token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses, never bring back an entry it had invalidated. So set `maxmemory` and let the server evict: `maxmemory-policy allkeys-lru` (or `allkeys-lfu`, `volatile-lru`, `volatile-lfu`). Under `noeviction`, a full server refuses the writes: each refusal (a fill's is counted by `wavehouse_cache_set_failures_total{reason="oom"}`) bypasses the cache of the instance that got it, and invalidations are kept and retried, so the pre-insert results above stay served by the others: avoid `noeviction`. A stored result is capped at `cache.redis.max_value_bytes` (1 MiB compressed). A tenant's version tokens share one hash tag, so each lookup reads them in one `MGET` in cluster mode as well. The results themselves carry no hash tag and spread across shards. Persistence is not needed: an empty server after a restart is a cold cache, not a wrong one. **The server is inside the trust boundary.** A cached result is served after the access policy has filtered it, so whoever can write to the server can change what any caller reads. Keep it on a private network, require a password or ACL user (`WH_CACHE_REDIS_PASSWORD`), use TLS across links you do not trust, and share it only with deployments you trust as much as this one. diff --git a/internal/config/cache_redis.go b/internal/config/cache_redis.go index 920630157..df8174353 100644 --- a/internal/config/cache_redis.go +++ b/internal/config/cache_redis.go @@ -19,11 +19,12 @@ const ( RedisSentinel = "sentinel" ) -// maxRedisDialTimeout caps cache.redis.dial_timeout. A dial is a connect -// and a handshake, each bounded by it, and both boot and the cache's Close -// wait out one in flight: at 2s that is at most 4s, inside the 5s budget -// Close shares with the stores released after the cache. -const maxRedisDialTimeout = 2 * time.Second +// maxRedisTimeout caps cache.redis.timeout and cache.redis.dial_timeout. +// Boot and the cache's Close each wait out a dial in flight: a connect and +// a handshake, each bounded by dial_timeout, then for a cluster a topology +// read bounded by the larger of the two. At the caps that is at most 3s, +// inside the 5s budget Close shares with the stores released after it. +const maxRedisTimeout = time.Second // CacheRedisConfig configures cache.backend=redis: one Redis-compatible server // (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB) shared by every process. @@ -108,8 +109,16 @@ func (r CacheRedisConfig) validate() error { return fmt.Errorf("%s %s must be positive", d.key, d.v) } } - if r.DialTimeout > maxRedisDialTimeout { - return fmt.Errorf("cache.redis.dial_timeout (WH_CACHE_REDIS_DIAL_TIMEOUT) %s is over %s: boot and shutdown each wait out a dial, up to twice this", r.DialTimeout, maxRedisDialTimeout) + for _, d := range []struct { + key string + v time.Duration + }{ + {"cache.redis.timeout (WH_CACHE_REDIS_TIMEOUT)", r.Timeout}, + {"cache.redis.dial_timeout (WH_CACHE_REDIS_DIAL_TIMEOUT)", r.DialTimeout}, + } { + if d.v > maxRedisTimeout { + return fmt.Errorf("%s %s is over %s: boot and shutdown each wait out a connection attempt, which both bound", d.key, d.v, maxRedisTimeout) + } } if r.VersionTTL < 2*time.Second { return fmt.Errorf("cache.redis.version_ttl (WH_CACHE_REDIS_VERSION_TTL) %s is under 2s", r.VersionTTL) diff --git a/internal/config/cache_redis_test.go b/internal/config/cache_redis_test.go index 7f91a0499..264ed31d9 100644 --- a/internal/config/cache_redis_test.go +++ b/internal/config/cache_redis_test.go @@ -57,7 +57,7 @@ func TestLoad_CacheRedisFromEnv(t *testing.T) { "WH_CACHE_REDIS_TLS_SERVER_NAME": "redis.internal", "WH_CACHE_REDIS_KEY_PREFIX": "staging", "WH_CACHE_REDIS_TIMEOUT": "250ms", - "WH_CACHE_REDIS_DIAL_TIMEOUT": "2s", + "WH_CACHE_REDIS_DIAL_TIMEOUT": "500ms", "WH_CACHE_REDIS_MAX_VALUE_BYTES": "2048", "WH_CACHE_REDIS_COMPRESS_MIN_BYTES": "0", "WH_CACHE_REDIS_VERSION_TTL": "24h", @@ -73,7 +73,7 @@ func TestLoad_CacheRedisFromEnv(t *testing.T) { TLS: CacheRedisTLS{ Enabled: true, CAFile: caFile, CertFile: certFile, KeyFile: keyFile, ServerName: "redis.internal", }, - KeyPrefix: "staging", Timeout: 250 * time.Millisecond, DialTimeout: 2 * time.Second, + KeyPrefix: "staging", Timeout: 250 * time.Millisecond, DialTimeout: 500 * time.Millisecond, MaxValueBytes: 2048, CompressMinBytes: 0, VersionTTL: 24 * time.Hour, }, cfg.Cache.Redis) tc, err := cfg.Cache.Redis.TLS.Config() @@ -221,8 +221,9 @@ func TestValidate_CacheRedis(t *testing.T) { {"brace prefix", func(r *CacheRedisConfig) { r.KeyPrefix = "a{b}" }, "hash-tag brace"}, {"zero timeout", func(r *CacheRedisConfig) { r.Timeout = 0 }, "cache.redis.timeout (WH_CACHE_REDIS_TIMEOUT) 0s must be positive"}, {"negative dial timeout", func(r *CacheRedisConfig) { r.DialTimeout = -time.Second }, "cache.redis.dial_timeout"}, - {"dial timeout at the cap", func(r *CacheRedisConfig) { r.DialTimeout = 2 * time.Second }, ""}, - {"dial timeout over the cap", func(r *CacheRedisConfig) { r.DialTimeout = 2*time.Second + time.Millisecond }, "cache.redis.dial_timeout (WH_CACHE_REDIS_DIAL_TIMEOUT) 2.001s is over 2s"}, + {"timeouts at the cap", func(r *CacheRedisConfig) { r.Timeout, r.DialTimeout = time.Second, time.Second }, ""}, + {"timeout over the cap", func(r *CacheRedisConfig) { r.Timeout = time.Second + time.Millisecond }, "cache.redis.timeout (WH_CACHE_REDIS_TIMEOUT) 1.001s is over 1s"}, + {"dial timeout over the cap", func(r *CacheRedisConfig) { r.DialTimeout = time.Second + time.Millisecond }, "cache.redis.dial_timeout (WH_CACHE_REDIS_DIAL_TIMEOUT) 1.001s is over 1s"}, {"short version ttl", func(r *CacheRedisConfig) { r.VersionTTL = time.Second }, "cache.redis.version_ttl (WH_CACHE_REDIS_VERSION_TTL) 1s is under 2s"}, {"zero max value", func(r *CacheRedisConfig) { r.MaxValueBytes = 0 }, "cache.redis.max_value_bytes"}, {"compress never", func(r *CacheRedisConfig) { r.CompressMinBytes = 0 }, ""}, diff --git a/tests/integration/shared_cache_test.go b/tests/integration/shared_cache_test.go index e9857b725..f120365c3 100644 --- a/tests/integration/shared_cache_test.go +++ b/tests/integration/shared_cache_test.go @@ -80,7 +80,7 @@ func bootRedisApp(t *testing.T, redisAddr, prefix string, timeout time.Duration) MQ: config.MQ{Backend: config.MQEmbedded}, Cache: config.Cache{Backend: config.CacheRedis, Redis: config.CacheRedisConfig{ Addrs: []string{redisAddr}, Mode: config.RedisStandalone, KeyPrefix: prefix, - Timeout: timeout, DialTimeout: 2 * time.Second, + Timeout: timeout, DialTimeout: time.Second, MaxValueBytes: 1 << 20, CompressMinBytes: 1 << 10, VersionTTL: time.Hour, }}, Dedupe: config.Dedupe{Backend: config.DedupePebble}, @@ -149,8 +149,8 @@ func TestSharedCache_IngestOnOneInstanceInvalidatesAnother(t *testing.T) { table := createTable(t, "user_id String, value Float64", "ORDER BY user_id") _, redisAddr := startRedis(t) prefix := fmt.Sprintf("it%d", cachePrefixes.Add(1)) - a := bootRedisApp(t, redisAddr, prefix, 2*time.Second) - b := bootRedisApp(t, redisAddr, prefix, 2*time.Second) + a := bootRedisApp(t, redisAddr, prefix, time.Second) + b := bootRedisApp(t, redisAddr, prefix, time.Second) rc, err := rueidis.NewClient(rueidis.ClientOption{InitAddress: []string{redisAddr}, DisableCache: true, ForceSingleClient: true}) require.NoError(t, err) From 41ebc664b561d06123093141b738a9b66ff96318 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 06:36:31 -0400 Subject: [PATCH 32/45] =?UTF-8?q?fix(cache):=20review=20round=201=20?= =?UTF-8?q?=E2=80=94=20one=20standalone=20address,=20e2e=20stack=20docs,?= =?UTF-8?q?=20e2e=20gate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Standalone mode dials only the first address, so a second one was silently ignored: it now refuses boot. A port must be 1-65535. - The manual e2e boot command needs a Redis now that the fixture sets cache.backend: redis; the e2e stack is described as ClickHouse and Redis wherever it was ClickHouse only. - The e2e gate (59.9%) excludes cache_redis.go's rejection paths and pending.go's outage-only retries, as it excludes internal/settings; unit and integration cover both, and the merged total still counts them. - The integration tests read the TTL floor from cache.QueryTimeToTTL. - A failed InvalidateTenant on reconcile logs at WARN, like the worker. - The overview pages mention the shared cache. Co-Authored-By: Claude Opus 5.5 (1M context) --- .github/workflows/README.md | 2 +- .github/workflows/ci.yml | 4 ++-- .testcoverage.yml | 6 ++++++ AGENTS.md | 6 +++--- CHANGELOG.md | 2 +- docs/src/content/docs/architecture.md | 2 +- docs/src/content/docs/configuration.mdx | 2 +- docs/src/content/docs/development.md | 11 ++++++----- docs/src/content/docs/index.mdx | 2 +- docs/src/content/docs/sdk/reference.md | 4 ++-- docs/src/content/docs/why-wavehouse.md | 2 +- internal/app/app_test.go | 6 ++++-- internal/app/wire.go | 2 +- internal/config/cache_redis.go | 20 +++++++++++--------- internal/config/cache_redis_test.go | 9 +++++++-- tests/integration/shared_cache_test.go | 3 ++- 16 files changed, 50 insertions(+), 33 deletions(-) diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 575a83893..470f82f8b 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -44,7 +44,7 @@ Break one of these knowingly or not at all. 2. **A dedicated `coverage` job applies the consolidated gate, polling — not `needs`-ing — the suites.** Each suite (`unit`, `integration`, `e2e`) runs with `COV_DEFER=1` and uploads a `coverage-` fragment; the `coverage` job runs `make cov` (merge + every threshold gate) over all three — exactly like local `make ci`'s final step. Keeping it a separate job (not folded into e2e's tail) decouples the gate result from the e2e suite's pass/fail. Crucially it is `needs: changes` **only, not the suites**: a `needs` edge is a *scheduling* barrier — GitHub won't pick up a runner, check out, restore caches, or `pnpm install` until the needed jobs finish — so needing the suites would serialize this job's ~50s of setup onto the critical path after the last suite, for nothing (the setup doesn't depend on their results). Instead it starts at run creation, runs its setup in parallel with the suites, and blocks only at the merge by polling for the three fragments with [`scripts/ci/wait-artifact.sh`](../../scripts/ci/wait-artifact.sh) (fails fast if a producer concluded without producing). Tail on the critical path: ~10s, not ~50s. **The aggregator and `docs-deploy` must keep `coverage` *and* every suite in their `needs`** — the suites directly (a suite failure must red the gate even though `coverage` no longer needs them), and `coverage` (else a coverage-gate failure wouldn't block merge or a prod deploy). -3. **e2e builds its own inputs and mirrors local `make test-e2e`.** It compiles the SDK dist + cover binary itself (`make -j test-e2e`, warm per-suffix cache) rather than waiting on a builder job, and runs the suite exactly as a developer does — one orchestrator, one ClickHouse testcontainer, sequential files. The ClickHouse image pulls in the background while caches restore (also in the integration job). +3. **e2e builds its own inputs and mirrors local `make test-e2e`.** It compiles the SDK dist + cover binary itself (`make -j test-e2e`, warm per-suffix cache) rather than waiting on a builder job, and runs the suite exactly as a developer does — one orchestrator, one ClickHouse and one Redis testcontainer, sequential files. The ClickHouse image pulls in the background while caches restore (also in the integration job). 4. **One change classifier, split into a pure core + a CI wrapper.** The pure allowlist — file list on stdin ⇒ `code`/`docs` — lives in [`scripts/classify-paths.sh`](../../scripts/classify-paths.sh), dependency-free and unit-tested by [`scripts/classify-paths.test.sh`](../../scripts/classify-paths.test.sh) (`make test-classify-paths`, a `verify` leaf) so the allowlists can't silently regress. The `changes` job runs the thin wrapper [`scripts/ci/classify-changes.sh`](../../scripts/ci/classify-changes.sh), which adds the CI-only policy (API file-list fetch + fail-closed: pushes, dispatches, API hiccups ⇒ `code=true`) on top. Keeping the core pure means the local git hooks can share it (`git diff --name-only | scripts/classify-paths.sh`). The `code`/`docs` outputs gate the suites and docs jobs — gate on these, never on workflow-level `paths:` filters, which would orphan the required check (invariant 1). diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f47619274..fa572e414 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -280,8 +280,8 @@ jobs: with: go-cache-suffix: "-e2e-cov" # `-j` builds the prereqs (build-ts ∥ build-cover) concurrently, - # then runs the orchestrator: ClickHouse testcontainer + the cover - # binary + the SDK vitest suite. + # then runs the orchestrator: ClickHouse and Redis testcontainers + + # the cover binary + the SDK vitest suite. - name: Build SDK dist + cover binary, run E2E suite run: make -j "$(nproc)" test-e2e COV_DEFER=1 - name: Upload coverage fragment diff --git a/.testcoverage.yml b/.testcoverage.yml index 84f028afe..451014a32 100644 --- a/.testcoverage.yml +++ b/.testcoverage.yml @@ -84,3 +84,9 @@ exclude: # never reaches them. The unit suite and the integration suite's main # app (cache.backend=local) cover them; the merged total still counts them. - ^internal/cache/(local|version_manager)\.go$ + # What e2e's Redis never makes it run: the cache.redis block's rejection + # paths (boot adopts a valid fixture, as with internal/settings above) + # and the retry of invalidations the server did not take, which needs an + # outage. The unit and integration suites cover both. + - ^internal/config/cache_redis\.go$ + - ^internal/cache/pending\.go$ diff --git a/AGENTS.md b/AGENTS.md index 8feaab737..a04c4a8be 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -152,7 +152,7 @@ If `make ci` passes locally, your commit has crossed the same gates CI will run ### Running `make ci` (for agents) -`make ci` is **self-contained**: the integration suite (`tests/integration/`) and the E2E orchestrator (`scripts/orchestrator/`) each boot ClickHouse via **testcontainers on random host ports**, and the shared cache backend's integration tests (`internal/cache/`) start their own Redis, Valkey, Dragonfly and one-node Redis Cluster containers the same way. The only prerequisite is a running **Docker daemon** — do **not** `make deps-up` or start ClickHouse first (`deps-up` is for `make dev` only). +`make ci` is **self-contained**: the integration suite (`tests/integration/`) and the E2E orchestrator (`scripts/orchestrator/`) each boot ClickHouse and a Redis via **testcontainers on random host ports**, and the shared cache backend's integration tests (`internal/cache/`) start their own Redis, Valkey, Dragonfly and one-node Redis Cluster containers the same way. The only prerequisite is a running **Docker daemon** — do **not** `make deps-up` or start ClickHouse first (`deps-up` is for `make dev` only). Run it via the **background Bash tool** (`run_in_background: true`) and wait for the completion notification; the harness re-invokes you on exit, so polling the log with `tail` only burns context: @@ -449,8 +449,8 @@ internal/stream/ → SSE fan-out (event Hub: project once per role, Subsc internal/tenant/ → Tenant id (type, grammar, reserved default, request header name) internal/testutil/ → Shared test helpers (mocks, JWT + schema helpers; logtest/ captures or silences the default logger; cachetest/ is the conformance suite every cache.Cache backend runs) tests/ → Integration & E2E tests -tests/integration/ → Go integration tests (//go:build integration; ClickHouse testcontainer). A package tested against its own external server keeps them beside it: internal/cache/redis_integration_test.go (Redis, Valkey, Dragonfly, Redis Cluster testcontainers) -tests/e2e/ → E2E test stack (scripts/orchestrator boots a ClickHouse testcontainer + the wavehouse-cov binary) +tests/integration/ → Go integration tests (//go:build integration; ClickHouse testcontainer, and Redis for shared_cache_test.go). A package tested against its own external server keeps them beside it: internal/cache/redis_integration_test.go (Redis, Valkey, Dragonfly, Redis Cluster testcontainers) +tests/e2e/ → E2E test stack (scripts/orchestrator boots ClickHouse and Redis testcontainers + the wavehouse-cov binary) tests/e2e/fixtures/ → Idempotent ClickHouse DDL scripts for test tables tests/e2e/sdk/ → E2E integration tests via TypeScript SDK (Vitest) deployments/compose/ → Docker Compose files (standalone.yaml, dependencies.yaml) diff --git a/CHANGELOG.md b/CHANGELOG.md index 51ed88656..c4e6e2934 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **`cache.backend: redis` shares the query cache across instances** (`internal/config/{cache_redis,backends,config}.go` (+ tests), `internal/app/wire.go` (+ tests), `internal/ingest/worker.go`, `tests/integration/shared_cache_test.go`, `scripts/orchestrator/main.go`, `tests/e2e/fixtures/config.yaml`, `.testcoverage.yml`, `deployments/compose/dependencies.yaml`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md,settings-directory.mdx,getting-started.md,pipes.mdx,api.md,development.md}`, `README.md`, `AGENTS.md`): PR E4 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config's `cache.backend` now takes `redis`, configured by a new `cache.redis` block (`WH_CACHE_REDIS_*`): `addrs` (required), `mode` (`standalone` or `cluster`; `sentinel` refuses boot until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `username`, `password` (a secret — set it through the environment), `db`, `tls.{enabled,ca_file,cert_file,key_file,server_name,insecure_skip_verify}`, `key_prefix` (`wh`), `timeout` (`100ms`) and `dial_timeout` (`1s`), each at most `1s` since boot and shutdown each wait out a connection attempt they bound, `max_value_bytes` (1 MiB), `compress_min_bytes` (`1024`; `0` never compresses) and `version_ttl` (`168h`). Every instance pointed at one server shares its cached results, and an insert on any instance invalidates every instance's. It is also the shared cache a split of [`roles`](https://github.com/Wave-RF/WaveHouse/pull/622) needs: the refusal of `api` without `ingest`, or the reverse, over `cache.backend: local` now names `redis`, though every split is still refused while the queue is embedded. A malformed block — no address, an address without a port, a URL-style address (refused without repeating it, since it may carry a password), an unknown mode, `db` other than `0` in cluster mode, an unreadable TLS file, a TLS key set while `tls.enabled` is off — refuses boot; an unreachable server, or one that rejects the password, does not: the process boots with the cache bypassed and keeps reconnecting, logging a rejected password at `ERROR` on every attempt, so a rotated secret cannot crash-loop every instance at once. `insecure_skip_verify`, and a `cache.redis.addrs` set while `cache.backend` is `local`, are logged at `WARN` at boot. The ingest worker's log of an invalidation that did not land drops from `ERROR` to `WARN`, since the shared backend defers and retries it: an outage would otherwise log an `ERROR` for every batch. The e2e suite now runs against a Redis testcontainer with `cache.backend: redis`, so the shared backend is exercised end to end, and the e2e coverage exclude for its files is gone; the unit and integration suites keep covering `local`. An integration test boots two instances over one Redis and one ClickHouse: a result one fills is a hit for the other, and a row ingested through one is served fresh by the other on its next query, well inside the stale entry's TTL; another pauses Redis and checks queries keep succeeding from ClickHouse, then turn back to hits; a third runs the first one's hit, insert and fresh-miss lifecycle on the suite's own `cache.backend: local` app, since e2e no longer exercises that backend. `deployments/compose/dependencies.yaml` gains an optional `redis` profile for local multi-instance work. The deployment guide gains a "Multiple instances and the shared cache" section: what each instance keeps to itself, what a reader on another instance can see and when, `maxmemory-policy`, and the metrics to alert on. +- **`cache.backend: redis` shares the query cache across instances** (`internal/config/{cache_redis,backends,config}.go` (+ tests), `internal/app/wire.go` (+ tests), `internal/ingest/worker.go`, `tests/integration/shared_cache_test.go`, `scripts/orchestrator/main.go`, `tests/e2e/fixtures/config.yaml`, `.testcoverage.yml`, `deployments/compose/dependencies.yaml`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md,settings-directory.mdx,getting-started.md,pipes.mdx,api.md,development.md}`, `README.md`, `AGENTS.md`): PR E4 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config's `cache.backend` now takes `redis`, configured by a new `cache.redis` block (`WH_CACHE_REDIS_*`): `addrs` (required), `mode` (`standalone` or `cluster`; `sentinel` refuses boot until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `username`, `password` (a secret — set it through the environment), `db`, `tls.{enabled,ca_file,cert_file,key_file,server_name,insecure_skip_verify}`, `key_prefix` (`wh`), `timeout` (`100ms`) and `dial_timeout` (`1s`), each at most `1s` since boot and shutdown each wait out a connection attempt they bound, `max_value_bytes` (1 MiB), `compress_min_bytes` (`1024`; `0` never compresses) and `version_ttl` (`168h`). Every instance pointed at one server shares its cached results, and an insert on any instance invalidates every instance's. It is also the shared cache a split of [`roles`](https://github.com/Wave-RF/WaveHouse/pull/622) needs: the refusal of `api` without `ingest`, or the reverse, over `cache.backend: local` now names `redis`, though every split is still refused while the queue is embedded. A malformed block — no address, an address without a valid port, more than one address in `standalone` mode, a URL-style address (refused without repeating it, since it may carry a password), an unknown mode, `db` other than `0` in cluster mode, an unreadable TLS file, a TLS key set while `tls.enabled` is off — refuses boot; an unreachable server, or one that rejects the password, does not: the process boots with the cache bypassed and keeps reconnecting, logging a rejected password at `ERROR` on every attempt, so a rotated secret cannot crash-loop every instance at once. `insecure_skip_verify`, and a `cache.redis.addrs` set while `cache.backend` is `local`, are logged at `WARN` at boot. The ingest worker's log of an invalidation that did not land drops from `ERROR` to `WARN`, since the shared backend defers and retries it: an outage would otherwise log an `ERROR` for every batch. The e2e suite now runs against a Redis testcontainer with `cache.backend: redis`, so the shared backend is exercised end to end, and the e2e coverage exclude for its files is gone; the unit and integration suites keep covering `local`. An integration test boots two instances over one Redis and one ClickHouse: a result one fills is a hit for the other, and a row ingested through one is served fresh by the other on its next query, well inside the stale entry's TTL; another pauses Redis and checks queries keep succeeding from ClickHouse, then turn back to hits; a third runs the first one's hit, insert and fresh-miss lifecycle on the suite's own `cache.backend: local` app, since e2e no longer exercises that backend. `deployments/compose/dependencies.yaml` gains an optional `redis` profile for local multi-instance work. The deployment guide gains a "Multiple instances and the shared cache" section: what each instance keeps to itself, what a reader on another instance can see and when, `maxmemory-policy`, and the metrics to alert on. - **A Redis-compatible shared cache backend** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/cache.go`, `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`, `CONTRIBUTING.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag, with the table and scope escaped into the key (`internal/keyenc`) so no two names share a token; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row, or one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like), open a circuit breaker that skips the server until a probe write succeeds (connections are replaced every minute, so after a failover behind a stable address the process reaches the new primary, and delivers the bumps it owes, within about a minute, provided a fresh connection's dial and handshake fit in the per-operation timeout), and deferred invalidations are retried until they land — the first at once, and at the probe's cadence while the breaker is open — collapsing to one tenant-wide bump per tenant past 100,000 keys; until one lands, the process that owes it bypasses the lookups it would orphan. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. `cache.backend: redis` selects it (the entry above). Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression, server-stops-answering, refused-writes (a demoted primary, a full `noeviction` server), failover-behind-a-stable-address and owed-invalidation cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. - **One conformance suite for every `mq.Broker`, and a transient broker failure is a `503`** (`internal/mq/mqtest/` (new: the suite and the embedded broker's run of it), `internal/mq/{mq,embedded}.go` (+ tests), `internal/api/ingest.go` (+ tests), `.testcoverage.yml`, `docs/src/content/docs/{api,architecture}.md`, `AGENTS.md`): the first piece of the external-NATS workstream of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `mqtest.Run` states the `Broker` contract as behavior — round trips with names that need encoding, per-tenant order, `Nak` and `AckWait` redelivery, the trace context reaching `Subscribe`, dead-lettering that keeps the topic and leaves the original unacked, per-tenant dead-letter counts, replay bounds and isolation, and exactly one `failed` report when delivery ends underneath a consumer — through the interfaces alone, so the external backend runs the same cases, with `mqtest.Caps` for the four places where its semantics legitimately differ. The embedded broker passes it; writing it turned up that a durable deleted on several tenants' queues could report on `failed` more than once, which is fixed and pinned by a test that deletes it on one queue after another. It also turned up a replay that lost its connection mid-pull passing for a caught-up one when the pull ended in a timeout; that is an error now, as the `Replayer` contract says. The interface comments now allow a delivery unit that is a partition holding several tenants, a `CreateConsumer` that finds a durable rather than creating one, a `PurgeAcked` that leaves retention to the operator, and zero dead-letter counts where there is no per-tenant queue. A new sentinel, `mq.ErrUnavailable`, is a broker that cannot be reached or does not answer in time: the ingest handler answers it with `503` + `Retry-After: 5` rather than the `500` "publish failed" it would have been. Nothing returns it yet; the external backend of #613 will. - **Process roles: the API and the background workers can run in separate processes** (`internal/config/config.go` (+ `roles_test.go`, `defaults_test.go`), `internal/config/backends.go`, `internal/app/{app,wire}.go` (+ `roles_test.go`), `internal/api/router.go` (+ tests), `tests/integration/{setup,tenants}_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md}`, `AGENTS.md`), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The boot config gains `roles` (`WH_ROLES`, default `api,ingest,sweeper`, set in `defaults()` like every boot default, so an explicit `roles: []` refuses boot) and `instance_id` (`WH_INSTANCE_ID`, default `-<8 hex>`, fresh at every boot; logged at boot, and recorded as a lease holder once a shared `coord.backend` exists). A process wires only what its roles need: `api` runs the HTTP API with schema discovery, the token verifiers, the dedupe stores and the SSE hub (all per API process); `ingest` runs the ingest worker; `sweeper` runs the sweeper under its lease. A process without `api` serves an ops-only listener on `server.port` (the probes and their aliases, `/version`, the same-port metrics path, and `POST /v1/ops/settings/reload`, which takes the operator key alone); every other route answers 404, under `/v1/ops` once the operator key has passed. Boot refuses any split over the embedded MQ, which no other process can reach, and `api` without `ingest` (or the reverse) over a local cache, which the ingest worker's invalidations would never reach. Until a shared `mq.backend` exists, every process therefore runs every role, which is the default, so nothing changes for an existing deployment. `data_dir` is probed for Pebble only in a process running `api`. A `config.Config` built without `config.Load` must now name its roles (`config.AllRoles()` for all of them): `app.New` refuses an empty set. diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index d3e479627..7511a2a9b 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -126,7 +126,7 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ - **config.go** — Loads *boot* configuration from a YAML file with environment variable overrides (using [cleanenv](https://github.com/ilyakaznacheev/cleanenv)); every key has a `WH_`-prefixed env var. Boot config is only what can't change under a running process — the implementation each layer runs on, the process's `roles`, resource sizing, listeners, observability exporters, the settings-directory path, and the secrets (`clickhouse.password`, `cache.redis.password`, `auth.jwt_secret`, `auth.operator_key`). Everything tenant-tunable lives in the settings directory (`settings/`). Both sources are strict: `Load` refuses to boot naming every YAML key the struct doesn't declare (`strict.go`) and every `WH_*` environment variable no field binds (`check.go`), so a tunable that moved to the settings directory can't be read, ignored, and believed. Boot is the validator for this half — there is no dry-run command. See [Configuration Reference](/configuration). - **check.go** — `rejectUnboundEnv` is the environment half of the strict loader: `unboundEnv` walks the struct's `env` tags (plus the two process-level names, `WH_CONFIG` and `WH_LOG_LEVEL`) against the environment; `CheckDataDir` probes `data_dir` — run by `main` right after `Load` when `Config.NeedsDataDir` says a selected backend keeps state there, so an unusable `data_dir` refuses boot before anything dials out. It refuses an empty value (reachable through `WH_DATA_DIR=`) outright rather than probing the working directory; a path that exists and is not a directory; a dangling symlink at `data_dir` or any component above it (the walk to the nearest existing ancestor uses `Lstat`, so a failed mount is not skipped over as "does not exist"); and a directory the process cannot write to — or, when it does not exist, an unwritable nearest ancestor — probed by creating and removing one temp file. A permission denial, on the probe or on reaching the path through a parent without search permission, carries the UID-65532 hint, since a bind mount owned by root is the typical cause. - **backends.go** — the `.backend` keys: one string type per layer (`MQBackend`, `CacheBackend`, `DedupeBackend`, `CoordBackend`), each with its list of the backends this build has, and a `validate` per layer block (`checkBackend`, run by `validateBackends` in `Validate`, between `validateRoles` and `validateTopology`), which refuses a value not on the list and names the ones that are. A backend's own settings go in a `.` sub-block that is its `validate`'s case to check. `Distributed` reports whether the MQ is shared with other processes, `NeedsDataDir` whether a selected backend keeps state under `data_dir`, and `Warnings` returns what a valid configuration is still likely to get wrong — the combinations that are correct for one replica only (a shared MQ over a local cache or Pebble dedupe), a `cache.redis` block that is not read, certificate verification turned off — which `app.New` logs at `WARN`. -- **cache_redis.go** — `CacheRedisConfig`, the `cache.redis` sub-block, and its checks: an address, each `host:port` (a URL or `user:password@` form refused without repeating it, since it may hold a password), a known mode (`sentinel` is refused until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `db` 0 in cluster mode, positive timeouts and sizes, a `timeout` and `dial_timeout` of at most 1 s each (boot and `Close` each wait out a dial: a connect and a handshake bounded by `dial_timeout`, and a cluster's topology read bounded by the larger of the two), a `version_ttl` of at least 2 s, and a `compress_min_bytes` that is not negative (`0` never compresses); its defaults are in `defaults()` with the rest. `CacheRedisTLS.Config` builds the `tls.Config`, reading the files; `Validate` calls it so an unreadable file refuses boot, and `internal/app` calls it again to build the connection. A TLS key set while `tls.enabled` is off is an error rather than a plaintext connection. +- **cache_redis.go** — `CacheRedisConfig`, the `cache.redis` sub-block, and its checks: an address (exactly one in `standalone` mode, which dials only the first), each `host:port` with a port from 1 to 65535 (a URL or `user:password@` form refused without repeating it, since it may hold a password), a known mode (`sentinel` is refused until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `db` 0 in cluster mode, positive timeouts and sizes, a `timeout` and `dial_timeout` of at most 1 s each (boot and `Close` each wait out a dial: a connect and a handshake bounded by `dial_timeout`, and a cluster's topology read bounded by the larger of the two), a `version_ttl` of at least 2 s, and a `compress_min_bytes` that is not negative (`0` never compresses); its defaults are in `defaults()` with the rest. `CacheRedisTLS.Config` builds the `tls.Config`, reading the files; `Validate` calls it so an unreadable file refuses boot, and `internal/app` calls it again to build the connection. A TLS key set while `tls.enabled` is off is an error rather than a plaintext connection. - **config.go**, roles — `roles` (`[]Role`: `api`, `ingest`, `sweeper`; `AllRoles` by default; `Has(Role)`) picks which components `internal/app` wires, and `instance_id` names the process (`-<8 hex>` when empty, resolved in `Load`; today only logged at boot, and a distributed coordinator will record it as a lease's holder). `validateRoles` refuses an empty list, an empty entry, an unknown or a repeated role; `validateTopology` refuses a role set the backends cannot serve: any split over the embedded MQ, and a process with exactly one of `api` and `ingest` over a local cache. `NeedsDataDir` counts Pebble only for a process running `api`, and `Warnings` is empty without `api`, since only that role opens a cache it reads or a dedupe store. - **strict.go** — `rejectUnknownKeys`, the YAML half: re-reads the file as a generic tree and walks it against the struct's `yaml` tags, listing every key the struct doesn't declare. cleanenv itself is lenient by design, which is exactly wrong for boot config once keys have moved to the settings directory. - **persistence.go** — `WarnIfFreshDataDir` logs the startup `WARN` when `data_dir` is missing or empty (on a redeploy, the sign that the volume didn't persist); `LogStorageInitError` attaches the UID-65532 `permissionHint` to a NATS or Pebble open failure that looks like a permission denial — the same hint string `CheckDataDir` uses. diff --git a/docs/src/content/docs/configuration.mdx b/docs/src/content/docs/configuration.mdx index 509088aee..d625b2e46 100644 --- a/docs/src/content/docs/configuration.mdx +++ b/docs/src/content/docs/configuration.mdx @@ -150,7 +150,7 @@ The `redis` backend's settings, read only when `cache.backend` is `redis`. It is | YAML Key | Env Var | Default | Description | | --- | --- | ------- | ----------- | -| `cache.redis.addrs` | `WH_CACHE_REDIS_ADDRS` | *(empty)* | **Required** with `backend: redis`. `host:port` of the server; several are a cluster's seed nodes. Comma-separated in the env var. Not a URL: `redis://user:password@host:port` refuses boot, without repeating the value, so set the credentials through `username` and `password`, and `rediss://` through `tls.enabled`. | +| `cache.redis.addrs` | `WH_CACHE_REDIS_ADDRS` | *(empty)* | **Required** with `backend: redis`. `host:port` of the server: exactly one in `standalone` mode, where a second would be ignored and so refuses boot; several are a cluster's seed nodes. Comma-separated in the env var. Not a URL: `redis://user:password@host:port` refuses boot, without repeating the value, so set the credentials through `username` and `password`, and `rediss://` through `tls.enabled`. | | `cache.redis.mode` | `WH_CACHE_REDIS_MODE` | `standalone` | `standalone` or `cluster`. `sentinel` refuses boot until [#656](https://github.com/Wave-RF/WaveHouse/issues/656): the cache does not yet authenticate to the sentinels or refresh their topology, so it could not be trusted to follow a failover. | | `cache.redis.username` | `WH_CACHE_REDIS_USERNAME` | *(empty)* | ACL user. Empty uses the server's `default` user. | | `cache.redis.password` | `WH_CACHE_REDIS_PASSWORD` | *(empty)* | A secret: set it through the environment (or your secret store's env injection), not in a tracked `config.yaml`. | diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index 7b3ec0e65..92785bda9 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -16,7 +16,7 @@ You need these on your `PATH` before any `make` recipe will work end-to-end: | **Go** | 1.26+ (matches `go.mod`) | Compiles `cmd/wavehouse`; also runs the pinned `tool` deps (`gotestsum`, `gofumpt`, `goimports`, `govulncheck`, `deadcode`, `gsa`, `goda`) via `go tool` | [go.dev/dl](https://go.dev/dl/) | | **GNU Make** | **4.0+** | The Makefile uses `--output-sync=target` (Make 4 only) and bash-pinned recipes. macOS ships with BSD Make 3.81, which **will not work** | macOS: `brew install make` then use `gmake` or put `$(brew --prefix make)/libexec/gnubin` on your PATH. Linux: usually already installed | | **bash** | 4+ recommended | Recipes are pinned to `bash`; the helper scripts under `scripts/` use `set -euo pipefail` and bash arrays | macOS default is bash 3.2 (works for current recipes, but `brew install bash` is safer); Linux distros ship 4+ | -| **Docker** *(or Podman)* | Engine 20.10+ with the Compose **v2** plugin (`docker compose`, no hyphen) | Compose stacks under `deployments/compose/`; the E2E and integration suites boot ClickHouse via testcontainers (no compose file), and the integration suite also starts Redis, Valkey, Dragonfly (pulled from `docker.dragonflydb.io`) and a one-node Redis Cluster for the shared cache backend | [Docker Desktop](https://docs.docker.com/get-docker/), [colima](https://github.com/abiosoft/colima), or [Podman](https://podman.io) with `podman-compose` / the `podman compose` plugin. The testcontainers Go library also honors `DOCKER_HOST` for rootless Podman setups | +| **Docker** *(or Podman)* | Engine 20.10+ with the Compose **v2** plugin (`docker compose`, no hyphen) | Compose stacks under `deployments/compose/`; the E2E and integration suites boot ClickHouse and a Redis via testcontainers (no compose file), and the integration suite also starts Valkey, Valkey, Dragonfly (pulled from `docker.dragonflydb.io`) and a one-node Redis Cluster for the shared cache backend | [Docker Desktop](https://docs.docker.com/get-docker/), [colima](https://github.com/abiosoft/colima), or [Podman](https://podman.io) with `podman-compose` / the `podman compose` plugin. The testcontainers Go library also honors `DOCKER_HOST` for rootless Podman setups | | **Node.js** | 22 LTS — pinned via `.nvmrc` at the repo root | Runtime for pnpm and the Vitest suites. Pinned to match CI (`setup-node` uses 22) and to avoid Node-major surprises; older Vitest versions in this repo were known to crash on Node 26 with a V8 heap-allocation abort | [nodejs.org](https://nodejs.org/) or `nvm use` / `fnm use` / `volta` (all read `.nvmrc`) | | **pnpm** | 11.21+ (pinned via `packageManager` in the root `package.json`) | Package manager for the TypeScript SDK, E2E test harness, and docs site (managed as a single pnpm workspace from the repo root); `make build-ts`, `make test-ts`, `make test-e2e`, `make build-docs`, `make dev-docs`, `make preview-docs` all shell out to `pnpm` | `corepack enable && corepack prepare pnpm@11.21.0 --activate` (recommended), or `npm i -g pnpm` | | **git** + **curl** | any recent | `git` for source + version metadata in builds; `curl` is used by the Makefile to fetch the pinned `golangci-lint` binary into `.bin/` | usually preinstalled | @@ -362,7 +362,7 @@ The primary E2E integration test suite lives in `tests/e2e/sdk/`. It uses the Ty **Architecture**: -- `scripts/orchestrator` — the E2E entrypoint behind `make test-e2e`: it starts a clean ClickHouse **testcontainer** per run, launches the `wavehouse-cov` binary on a random free port, runs the SDK suite against it, then SIGINTs the binary to flush coverage. No Compose file is involved. CI runs the exact same path. +- `scripts/orchestrator` — the E2E entrypoint behind `make test-e2e`: it starts a clean ClickHouse **testcontainer** and a Redis one (the fixture's shared cache, `cache.backend: redis`) per run, launches the `wavehouse-cov` binary on a random free port, runs the SDK suite against it, then SIGINTs the binary to flush coverage. No Compose file is involved. CI runs the exact same path. - `tests/e2e/sdk/setup.ts` — `globalSetup`. Probes the `CLICKHOUSE_URL` / `WAVEHOUSE_URL` the orchestrator injects, creates the per-suite tables, refreshes the schema, and writes the baseline policy into the run's settings directory (adopted via `POST /v1/ops/settings/reload` — files are the only write path). It starts nothing itself and fails fast if either URL isn't up. It also prints the active Node/undici version, warning when the local Node major differs from `.nvmrc` — a runtime-specific transport bug is otherwise indistinguishable from a code failure (see [#440](https://github.com/Wave-RF/WaveHouse/issues/440)). - `tests/e2e/sdk/helpers.ts` — JWT factories, typed client constructors, async wait helpers, direct ClickHouse query helper. @@ -375,10 +375,11 @@ make test-e2e `make test-e2e` builds `bin/wavehouse-cov` (coverage-instrumented) and runs the orchestrator under `scripts/orchestrator/` to wire ClickHouse + the cover binary into the suite. covdata flushes on SIGINT into `tmp/coverage/e2e/data/`. -The orchestrator always provisions its own stack — a fresh ClickHouse testcontainer plus `wavehouse-cov` on a random free port — so a running `make dev` on `:8080` is neither detected nor reused, and the two don't collide. To run vitest against a stack you manage yourself, start the server from the **repo root** with the E2E fixture config: +The orchestrator always provisions its own stack — fresh ClickHouse and Redis testcontainers plus `wavehouse-cov` on a random free port — so a running `make dev` on `:8080` is neither detected nor reused, and the two don't collide. To run vitest against a stack you manage yourself, start a Redis for the fixture's `cache.backend: redis`, then the server from the **repo root** with the E2E fixture config: ```bash -WH_CONFIG=tests/e2e/fixtures/config.yaml go run ./cmd/wavehouse +docker compose -f deployments/compose/dependencies.yaml --profile redis up -d +WH_CONFIG=tests/e2e/fixtures/config.yaml WH_CACHE_REDIS_ADDRS=localhost:6379 go run ./cmd/wavehouse ``` The fixture matters: the suite signs its tokens with its `sdk-dev-secret` and depends on its dedupe, DLQ, and 5s schema-refresh settings. Point the suite at a default `make dev` server (`jwt_secret: change-me-in-production`) and setup's schema calls are rejected, then global setup dies 30s later on a misleading `schema not refreshed within 30s`. The repo root matters too — the fixture's `settings.dir` is relative to the working directory. The fixture's settings directory (policy, pipes, and tunables) points at ClickHouse on `localhost:9000`; if yours isn't there, edit `clickhouse.addr` / `http_port` in `tests/e2e/fixtures/settings/config.json` (the orchestrator patches them itself for its testcontainer). @@ -474,7 +475,7 @@ WaveHouse/ │ └── testutil/ # Shared test helpers and mocks ├── tests/ # Integration & E2E tests │ ├── integration/ # Go integration tests (//go:build integration) -│ └── e2e/ # E2E suite (orchestrator + ClickHouse testcontainer) +│ └── e2e/ # E2E suite (orchestrator + ClickHouse and Redis testcontainers) │ ├── fixtures/ # ClickHouse DDL + config and settings-directory fixtures │ └── sdk/ # E2E specs driven through the TypeScript SDK (Vitest) ├── clients/ # Client SDKs diff --git a/docs/src/content/docs/index.mdx b/docs/src/content/docs/index.mdx index 3a6143ca5..205a6f845 100644 --- a/docs/src/content/docs/index.mdx +++ b/docs/src/content/docs/index.mdx @@ -97,7 +97,7 @@ If you're building user-facing analytics, **WaveHouse is like Supabase for Click Every event is broadcast to SSE subscribers **before** it's flushed to ClickHouse. Gap-fill from JetStream history for late-connecting clients. - Ristretto cache plus Go `singleflight` coalesces identical concurrent queries — dashboards survive thundering herds without an extra cache tier to operate. + Ristretto cache plus Go `singleflight` coalesces identical concurrent queries — dashboards survive thundering herds without an extra cache tier to operate. Running several instances? Share one Redis-compatible cache with `cache.backend: redis`. Per-table, per-role column and row-level policies with JWT claim templating, defined in the hot-reloadable settings directory. diff --git a/docs/src/content/docs/sdk/reference.md b/docs/src/content/docs/sdk/reference.md index fa6983da4..91df43af6 100644 --- a/docs/src/content/docs/sdk/reference.md +++ b/docs/src/content/docs/sdk/reference.md @@ -184,8 +184,8 @@ export interface ClicksRow { The SDK doubles as the E2E integration test harness. Tests in `tests/e2e/sdk/` exercise the full pipeline (ingest → ClickHouse → query) through the SDK, validating both the backend and the client library in one pass. ```bash -# Run all E2E tests: the orchestrator boots a ClickHouse testcontainer + -# the wavehouse-cov binary, then runs the SDK suite +# Run all E2E tests: the orchestrator boots ClickHouse and Redis +# testcontainers + the wavehouse-cov binary, then runs the SDK suite make test-e2e ``` diff --git a/docs/src/content/docs/why-wavehouse.md b/docs/src/content/docs/why-wavehouse.md index 766d62144..883d19284 100644 --- a/docs/src/content/docs/why-wavehouse.md +++ b/docs/src/content/docs/why-wavehouse.md @@ -152,7 +152,7 @@ flowchart TB | ---------- | --------- | --------- | | Durable ingest buffer | Kafka / Redpanda cluster (3+ brokers, Zookeeper/KRaft) | Embedded NATS JetStream | | Batch consumer | Custom Go/Rust/Java service you write and operate | Built in | -| Query cache | Redis + singleflight middleware you write | Built in (Ristretto + singleflight) | +| Query cache | Redis + singleflight middleware you write | Built in (Ristretto + singleflight; or one Redis shared by every instance, `cache.backend: redis`) | | Real-time push | WebSocket service + bridge from Kafka | Built in (`/v1/stream`) | | Schema validation | Custom code in ingest API | Built in (discovers `system.columns`) | | Row/column access control | Custom middleware or a dedicated service | Built in (Hasura-style, JWT-driven) | diff --git a/internal/app/app_test.go b/internal/app/app_test.go index 4b7d400b5..7a23330ed 100644 --- a/internal/app/app_test.go +++ b/internal/app/app_test.go @@ -806,14 +806,14 @@ func TestNew_RedisCacheRefusesAnUnreadableTLSFile(t *testing.T) { func TestRedisConfig_FromLoadedDefaults(t *testing.T) { t.Setenv("WH_SETTINGS_DIR", t.TempDir()) t.Setenv("WH_CACHE_BACKEND", "redis") - t.Setenv("WH_CACHE_REDIS_ADDRS", "a:6379,b:6379") + t.Setenv("WH_CACHE_REDIS_ADDRS", "a:6379") t.Setenv("WH_CACHE_REDIS_PASSWORD", "pw") loaded, err := config.Load(filepath.Join(t.TempDir(), "none.yaml")) require.NoError(t, err) got, err := redisConfig(loaded.Cache.Redis) require.NoError(t, err) assert.Equal(t, cache.RedisConfig{ - Addrs: []string{"a:6379", "b:6379"}, Mode: cache.RedisStandalone, Password: "pw", + Addrs: []string{"a:6379"}, Mode: cache.RedisStandalone, Password: "pw", KeyPrefix: cache.DefaultRedisKeyPrefix, Timeout: cache.DefaultRedisTimeout, DialTimeout: cache.DefaultRedisDialTimeout, MaxValueBytes: cache.DefaultRedisMaxValueBytes, CompressMinBytes: cache.DefaultRedisCompressMinBytes, VersionTTL: cache.DefaultRedisVersionTTL, @@ -821,12 +821,14 @@ func TestRedisConfig_FromLoadedDefaults(t *testing.T) { t.Setenv("WH_CACHE_REDIS_COMPRESS_MIN_BYTES", "0") t.Setenv("WH_CACHE_REDIS_MODE", "cluster") + t.Setenv("WH_CACHE_REDIS_ADDRS", "a:6379,b:6379") loaded, err = config.Load(filepath.Join(t.TempDir(), "none.yaml")) require.NoError(t, err) got, err = redisConfig(loaded.Cache.Redis) require.NoError(t, err) assert.Zero(t, got.CompressMinBytes, "the backend's never, not its default") assert.Equal(t, cache.RedisCluster, got.Mode) + assert.Equal(t, []string{"a:6379", "b:6379"}, got.Addrs, "a cluster's seeds") assert.Equal(t, cache.RedisSentinel, config.RedisSentinel) } diff --git a/internal/app/wire.go b/internal/app/wire.go index a7c82a972..fd1eb2e12 100644 --- a/internal/app/wire.go +++ b/internal/app/wire.go @@ -322,7 +322,7 @@ func (a *App) wireClickHouse() error { // cached is stale, so all of it is orphaned at once. for _, id := range stale { if err := a.cache.InvalidateTenant(a.stopCtx, id); err != nil { - slog.Error("cache invalidation of a stale tenant failed; it may serve stale rows until they expire", "tenant", id, "error", err) + slog.Warn("cache invalidation of a stale tenant did not land; it may serve stale rows until it does", "tenant", id, "error", err) } } }) diff --git a/internal/config/cache_redis.go b/internal/config/cache_redis.go index df8174353..be065266f 100644 --- a/internal/config/cache_redis.go +++ b/internal/config/cache_redis.go @@ -7,6 +7,7 @@ import ( "fmt" "net" "os" + "strconv" "strings" "time" ) @@ -78,9 +79,13 @@ func (r CacheRedisConfig) validate() error { if strings.TrimSpace(a) != a { return fmt.Errorf("cache.redis.addrs (WH_CACHE_REDIS_ADDRS) %q: no spaces around an address", a) } - if _, _, err := net.SplitHostPort(a); err != nil { + _, port, err := net.SplitHostPort(a) + if err != nil { return fmt.Errorf("cache.redis.addrs (WH_CACHE_REDIS_ADDRS) %q: want host:port: %w", a, err) } + if n, err := strconv.Atoi(port); err != nil || n < 1 || n > 65535 { + return fmt.Errorf("cache.redis.addrs (WH_CACHE_REDIS_ADDRS) %q: want host:port with a port from 1 to 65535", a) + } } switch r.Mode { case RedisStandalone, RedisCluster: @@ -89,6 +94,11 @@ func (r CacheRedisConfig) validate() error { default: return fmt.Errorf("cache.redis.mode (WH_CACHE_REDIS_MODE) %q: valid: %s, %s", r.Mode, RedisStandalone, RedisCluster) } + // Standalone dials the first address only, so a second one (a replica, + // say) would be silently ignored rather than failed over to. + if r.Mode == RedisStandalone && len(r.Addrs) > 1 { + return fmt.Errorf("cache.redis.addrs (WH_CACHE_REDIS_ADDRS) has %d addresses: mode standalone connects to one server; several are a cluster's seeds (mode cluster)", len(r.Addrs)) + } if r.DB < 0 { return fmt.Errorf("cache.redis.db (WH_CACHE_REDIS_DB) %d is negative", r.DB) } @@ -108,14 +118,6 @@ func (r CacheRedisConfig) validate() error { if d.v <= 0 { return fmt.Errorf("%s %s must be positive", d.key, d.v) } - } - for _, d := range []struct { - key string - v time.Duration - }{ - {"cache.redis.timeout (WH_CACHE_REDIS_TIMEOUT)", r.Timeout}, - {"cache.redis.dial_timeout (WH_CACHE_REDIS_DIAL_TIMEOUT)", r.DialTimeout}, - } { if d.v > maxRedisTimeout { return fmt.Errorf("%s %s is over %s: boot and shutdown each wait out a connection attempt, which both bound", d.key, d.v, maxRedisTimeout) } diff --git a/internal/config/cache_redis_test.go b/internal/config/cache_redis_test.go index 264ed31d9..3c1f7c05d 100644 --- a/internal/config/cache_redis_test.go +++ b/internal/config/cache_redis_test.go @@ -45,7 +45,7 @@ func TestLoad_CacheRedisFromEnv(t *testing.T) { caFile, certFile, keyFile := writeTestPKI(t, dir) for k, v := range map[string]string{ "WH_CACHE_BACKEND": "redis", - "WH_CACHE_REDIS_ADDRS": "r1:6379,r2:6379", + "WH_CACHE_REDIS_ADDRS": "r1:6379", "WH_CACHE_REDIS_MODE": "standalone", "WH_CACHE_REDIS_USERNAME": "wavehouse", "WH_CACHE_REDIS_PASSWORD": "s3cret", @@ -68,7 +68,7 @@ func TestLoad_CacheRedisFromEnv(t *testing.T) { cfg, err := Load("nonexistent.yaml") require.NoError(t, err) assert.Equal(t, CacheRedisConfig{ - Addrs: []string{"r1:6379", "r2:6379"}, Mode: RedisStandalone, + Addrs: []string{"r1:6379"}, Mode: RedisStandalone, Username: "wavehouse", Password: "s3cret", DB: 2, TLS: CacheRedisTLS{ Enabled: true, CAFile: caFile, CertFile: certFile, KeyFile: keyFile, ServerName: "redis.internal", @@ -211,6 +211,11 @@ func TestValidate_CacheRedis(t *testing.T) { {"no addrs", func(r *CacheRedisConfig) { r.Addrs = nil }, "cache.backend=redis needs cache.redis.addrs (WH_CACHE_REDIS_ADDRS)"}, {"addr with space", func(r *CacheRedisConfig) { r.Addrs = []string{"a:6379", " b:6379"} }, "no spaces around an address"}, {"addr without port", func(r *CacheRedisConfig) { r.Addrs = []string{"redis"} }, `cache.redis.addrs (WH_CACHE_REDIS_ADDRS) "redis": want host:port`}, + {"addr empty port", func(r *CacheRedisConfig) { r.Addrs = []string{"redis:"} }, `"redis:": want host:port with a port from 1 to 65535`}, + {"addr port not a number", func(r *CacheRedisConfig) { r.Addrs = []string{"redis:637x"} }, "a port from 1 to 65535"}, + {"addr port out of range", func(r *CacheRedisConfig) { r.Addrs = []string{"redis:99999"} }, "a port from 1 to 65535"}, + {"standalone with two addrs", func(r *CacheRedisConfig) { r.Addrs = []string{"a:6379", "b:6379"} }, "has 2 addresses: mode standalone connects to one server"}, + {"cluster with two seeds", func(r *CacheRedisConfig) { r.Mode, r.Addrs = RedisCluster, []string{"a:6379", "b:6379"} }, ""}, {"mode", func(r *CacheRedisConfig) { r.Mode = "replica" }, `cache.redis.mode (WH_CACHE_REDIS_MODE) "replica": valid: standalone, cluster`}, {"sentinel refused", func(r *CacheRedisConfig) { r.Mode = RedisSentinel }, `cache.redis.mode (WH_CACHE_REDIS_MODE) "sentinel" is not supported yet: the cache neither authenticates to the sentinels nor refreshes their topology (https://github.com/Wave-RF/WaveHouse/issues/656)`}, {"cluster", func(r *CacheRedisConfig) { r.Mode = RedisCluster }, ""}, diff --git a/tests/integration/shared_cache_test.go b/tests/integration/shared_cache_test.go index f120365c3..6ace9a764 100644 --- a/tests/integration/shared_cache_test.go +++ b/tests/integration/shared_cache_test.go @@ -23,6 +23,7 @@ import ( "github.com/testcontainers/testcontainers-go/wait" "github.com/Wave-RF/WaveHouse/internal/app" + "github.com/Wave-RF/WaveHouse/internal/cache" "github.com/Wave-RF/WaveHouse/internal/config" ) @@ -31,7 +32,7 @@ const redisImage = "redis:8.10.2-alpine" // minCacheTTL is cache.QueryTimeToTTL's floor: a fill made less than this // long ago cannot have expired, so a miss inside it is an invalidation. -const minCacheTTL = 10 * time.Second +var minCacheTTL = cache.QueryTimeToTTL(0) var cachePrefixes atomic.Uint64 From 64f09e8968194757cab996583a31a046bb4d6418 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 06:42:03 -0400 Subject: [PATCH 33/45] docs(dev): list Redis among the cache backend's integration servers Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/src/content/docs/development.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index 92785bda9..581df72d1 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -16,7 +16,7 @@ You need these on your `PATH` before any `make` recipe will work end-to-end: | **Go** | 1.26+ (matches `go.mod`) | Compiles `cmd/wavehouse`; also runs the pinned `tool` deps (`gotestsum`, `gofumpt`, `goimports`, `govulncheck`, `deadcode`, `gsa`, `goda`) via `go tool` | [go.dev/dl](https://go.dev/dl/) | | **GNU Make** | **4.0+** | The Makefile uses `--output-sync=target` (Make 4 only) and bash-pinned recipes. macOS ships with BSD Make 3.81, which **will not work** | macOS: `brew install make` then use `gmake` or put `$(brew --prefix make)/libexec/gnubin` on your PATH. Linux: usually already installed | | **bash** | 4+ recommended | Recipes are pinned to `bash`; the helper scripts under `scripts/` use `set -euo pipefail` and bash arrays | macOS default is bash 3.2 (works for current recipes, but `brew install bash` is safer); Linux distros ship 4+ | -| **Docker** *(or Podman)* | Engine 20.10+ with the Compose **v2** plugin (`docker compose`, no hyphen) | Compose stacks under `deployments/compose/`; the E2E and integration suites boot ClickHouse and a Redis via testcontainers (no compose file), and the integration suite also starts Valkey, Valkey, Dragonfly (pulled from `docker.dragonflydb.io`) and a one-node Redis Cluster for the shared cache backend | [Docker Desktop](https://docs.docker.com/get-docker/), [colima](https://github.com/abiosoft/colima), or [Podman](https://podman.io) with `podman-compose` / the `podman compose` plugin. The testcontainers Go library also honors `DOCKER_HOST` for rootless Podman setups | +| **Docker** *(or Podman)* | Engine 20.10+ with the Compose **v2** plugin (`docker compose`, no hyphen) | Compose stacks under `deployments/compose/`; the E2E and integration suites boot ClickHouse and a Redis via testcontainers (no compose file), and the integration suite also runs the shared cache backend against Redis, Valkey, Dragonfly (pulled from `docker.dragonflydb.io`) and a one-node Redis Cluster | [Docker Desktop](https://docs.docker.com/get-docker/), [colima](https://github.com/abiosoft/colima), or [Podman](https://podman.io) with `podman-compose` / the `podman compose` plugin. The testcontainers Go library also honors `DOCKER_HOST` for rootless Podman setups | | **Node.js** | 22 LTS — pinned via `.nvmrc` at the repo root | Runtime for pnpm and the Vitest suites. Pinned to match CI (`setup-node` uses 22) and to avoid Node-major surprises; older Vitest versions in this repo were known to crash on Node 26 with a V8 heap-allocation abort | [nodejs.org](https://nodejs.org/) or `nvm use` / `fnm use` / `volta` (all read `.nvmrc`) | | **pnpm** | 11.21+ (pinned via `packageManager` in the root `package.json`) | Package manager for the TypeScript SDK, E2E test harness, and docs site (managed as a single pnpm workspace from the repo root); `make build-ts`, `make test-ts`, `make test-e2e`, `make build-docs`, `make dev-docs`, `make preview-docs` all shell out to `pnpm` | `corepack enable && corepack prepare pnpm@11.21.0 --activate` (recommended), or `npm i -g pnpm` | | **git** + **curl** | any recent | `git` for source + version metadata in builds; `curl` is used by the Makefile to fetch the pinned `golangci-lint` binary into `.bin/` | usually preinstalled | From 558bb23835691a2ca43c4b3b7afb85ce3d755972 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 08:30:43 -0400 Subject: [PATCH 34/45] fix(cache): pin the no-index bump test and fix a "query key" mixup MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit TestVersionManager_BumpWithoutIndex asserted only on vm.size() after its final BumpTenant call, which unconditionally deletes the tenant's index regardless of what the earlier no-index bumps did — so it could not fail against a tableLocked that wrongly creates an index instead of returning nil. Assert after each bump instead, and add the pruned-tenant case a write racing Prune must not revive. Also reword two docs passages that overloaded or misstated a term: architecture.md used "query key" for both the caller's input and the rendered entry key in the same paragraph; AGENTS.md's cache bullet read as if the index maps were keyed by escaped names; CHANGELOG.md's #382 bullet said the version index builds its keys with internal/keyenc, contradicting its own closing sentence that the index's entries are unaffected by the escaping. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_017aS7rLrH1RKkUMem7X4ckd --- AGENTS.md | 2 +- CHANGELOG.md | 2 +- docs/src/content/docs/architecture.md | 2 +- internal/cache/version_manager_test.go | 18 +++++++++++++++++- 4 files changed, 20 insertions(+), 4 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index a504ffd11..a74dc6bb8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -31,7 +31,7 @@ Twenty internal packages under `internal/` (plus `internal/testutil/` for shared - **`api/`** — Chi HTTP router, JWT/JWKS middleware (from `auth/`), ingest/query/structured-query/SSE/schema/DLQ/pipes handlers; `ch_errors.go` (`writeCHError`) is the one mapping from a failed ClickHouse query to status, `code` and `retryable` - **`app/`** — the process wiring: `New` builds every component from the boot config and the settings directory (each one wired in one place — what it opens, what it loops, what it releases — with the settings registry handed to its wiring function whole, the injection point of the per-tenant registry of #583: store-keyed getters for the handlers, `perTenant` for the async paths (with the tenant each message's `mq.Topic` names for the stream hub and the ingest worker), the `chconn.Pools` and the per-tenant `discoveries` reconciled from `AfterAdopt`, `shortestKeepalive` for the one setting folded over every tenant served, `gapWindows` handing the sweeper each tenant's own gap window (a rejected tenant's as its folder last had it, unbounded for one rejected since boot) and the `mq.max_bytes_gb` reconcile each served tenant's byte budget, and `defaultPolicy` for the one setting that still follows tenant `0`, a flat directory's ops-gate admin role; the auth verifiers are per tenant, reconfigured (rebuilt only on changed wiring) and pruned from `AfterAdopt`, the same hook's `Hub.Prune` ends the open streams of a tenant no longer served, and `wireCache`'s hook drops, through `LocalCache.Prune`, the cache version index of a tenant no longer served ([#262](https://github.com/Wave-RF/WaveHouse/issues/262))), `Run` drives the long-lived ones under one `errgroup` until the context is cancelled or one fails, `Close` releases them in reverse order. `New` wires only what the process's `roles` need (discovery, dedupe, auth verifiers, the hub bridge and keepalive per API process; the ingest worker per ingest process; the sweeper under its lease through `elected`); a process without `api` serves `api.NewOpsRouter` — probes, `/version`, metrics, and the settings reload behind the operator key alone. `cmd/wavehouse` and `tests/integration` both boot through it - **`auth/`** — JWT auth middleware: HMAC **or** JWKS verification with `alg` pinned to the active verifier, role extraction from a configurable claim path; always runs, never rejects (bad token → empty role + stashed reason). One verifier per tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 9): `Authenticator` keys them by `tenant.ID` — the request store's `settings.Store.Tenant()`, through an injected `TenantSource`; `tenant.Default` on the tenant-exempt routes — built from each tenant's `auth` block by `Reconfigure`, dropped by `Prune` once the tenant stops being served (rejected or removed), released by `Close`; the secrets (`Config`) are boot-level and shared. A JWKS key set is fetched off the boot and reload paths: until one has been stored the verifier is pending and a token-bearing request gets `503` + `Retry-After` from `api.refuseUnverifiable` (`auth.ErrVerifierPending`), never a `default_role` evaluation; refresh is library-managed (Eric, 2026-09-22), response capped at 1 MiB; the operator key's admin role is the request tenant's -- **`cache/`** — `Cache` interface → `LocalCache` (Ristretto: one pool for every tenant) + `VersionManager` (the invalidation index). Every key leads with the tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8) — `:query:` for a result and its singleflight, a version per tenant, per (tenant, table) and per (tenant, table, scope), keyed by name and bumped in place (one entry per live namespace however often it is bumped, [#262](https://github.com/Wave-RF/WaveHouse/issues/262)), each raw table and scope name escaped by `keyenc` where the key is built (a `Namespace` carries them raw, so no caller escapes) — so no cached read or coalesced flight crosses tenants, a bump through `Invalidate` names one tenant's namespaces and no other's, and `InvalidateTenant` drops the tenant's index so its next key gets a process-unique generation, orphaning its every cached result in one step, pipe results included (no insert reaches a pipe result until [#343](https://github.com/Wave-RF/WaveHouse/pull/343)); `Lookup` returns a `Snapshot` of the versions it read, taken before the handler chooses any input a bump invalidates — the tenant's connection included — and `Set` files the fill under it, so a write landing mid-query, or a reload moving the tenant to another address or database after the request took its connection, orphans the fill ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)), and every backend runs the conformance suite `internal/testutil/cachetest`; the one crossing is the wiring's, above the package: `internal/app` hands the ingest worker the cache through `sharedTables`, which repeats each of the worker's bumps under every tenant on the same ClickHouse address and database (`chconn.Pools.SharingTables`, whatever their user or tls block — they read the same tables), and orphans the whole cache of a tenant back on a pool after an absence, since it was out of that fan-out while away, or moved to another address or database, since it now reads other tables (story 6) +- **`cache/`** — `Cache` interface → `LocalCache` (Ristretto: one pool for every tenant) + `VersionManager` (the invalidation index). Every key leads with the tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8) — `:query:` for the caller's query key and its singleflight, escaped whole as the lead field of the stored key `|.||…`, where each raw table and scope name is escaped by `keyenc` (a `Namespace` carries them raw, so no caller escapes); the index holds a version per tenant, per (tenant, table) and per (tenant, table, scope), keyed by raw name and bumped in place (one entry per live namespace however often it is bumped, [#262](https://github.com/Wave-RF/WaveHouse/issues/262)) — so no cached read or coalesced flight crosses tenants, a bump through `Invalidate` names one tenant's namespaces and no other's, and `InvalidateTenant` drops the tenant's index so its next key gets a process-unique generation, orphaning its every cached result in one step, pipe results included (no insert reaches a pipe result until [#343](https://github.com/Wave-RF/WaveHouse/pull/343)); `Lookup` returns a `Snapshot` of the versions it read, taken before the handler chooses any input a bump invalidates — the tenant's connection included — and `Set` files the fill under it, so a write landing mid-query, or a reload moving the tenant to another address or database after the request took its connection, orphans the fill ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)), and every backend runs the conformance suite `internal/testutil/cachetest`; the one crossing is the wiring's, above the package: `internal/app` hands the ingest worker the cache through `sharedTables`, which repeats each of the worker's bumps under every tenant on the same ClickHouse address and database (`chconn.Pools.SharingTables`, whatever their user or tls block — they read the same tables), and orphans the whole cache of a tenant back on a pool after an absence, since it was out of that fan-out while away, or moved to another address or database, since it now reads other tables (story 6) - **`chconn/`** — `Pools`, one `Manager` (a `driver.Conn`) per distinct `Identity{Addr, Database, Username, Password, TLS}` tuple among the served tenants, reconciled from the settings registry's `AfterAdopt` after every reload ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 6): tenants naming one tuple share its pool, sized to their largest `max_open_conns`/`max_idle_conns`; a tenant whose tuple changed is repointed; a tuple no tenant names is released after the longest `query_timeout` among the tenants it had (never dials; a resize swaps the connection with the same grace). The boot config's `clickhouse.max_total_conns` bounds the open pools' `max_open_conns` together: boot refuses naming sum and ceiling; at a reload a resize above it keeps the pool's size, and a tuple that cannot be opened (the ceiling, an unreadable certificate, or options the driver refuses) leaves its tenants on the pool they had or on none — logged, retried by the next reload. Every consumer resolves its tenant's pool per call: `For` (nil for a tenant on no pool, a `503`), `Target` (the tenant's own HTTP wiring over its pool's TLS config), `SharingTables`, `Ping` (every pool at once, ready at the first answer). `HTTPClients` keeps one `http.Client` per TLS config. `Classify` (`errclass.go`) says what a failed ClickHouse request means for the request — `Unavailable`, `Denied`, `Rejected` (any unlisted exception code: the server read it and refused it), or `Unknown` (no code, no recognizable transport failure) — over the driver's error types and the HTTP interface's `HTTPError`; the ingest worker and the query handlers (`api/ch_errors.go` `writeCHError`, [#403](https://github.com/Wave-RF/WaveHouse/issues/403), [#271](https://github.com/Wave-RF/WaveHouse/issues/271)) both use it - **`chsql/`** — dependency-free ClickHouse SQL helpers shared by `query`/`policy` (avoids an import cycle): `QuoteIdent` (backtick-quote every identifier) + `BindUnsafe` (reject names with a literal `?`) - **`config/`** — YAML + env var config loading (cleanenv); strict on both sides (undeclared YAML key, unbound `WH_*` variable) and probes `data_dir` writability when a selected backend keeps state there (`NeedsDataDir`); `backends.go` holds each layer's `.backend` (only the in-process value today); `config.go` holds `roles` (`Has(Role)`) and `instance_id`, and `Validate` refuses a role split the backends cannot serve (any split over the embedded MQ; `api` without `ingest`, or the reverse, over a local cache) — boot is the validator, there is no dry run diff --git a/CHANGELOG.md b/CHANGELOG.md index 6631b3c4a..14de673f7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -88,7 +88,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), - **A failed ClickHouse query answers by what went wrong, not a flat `500`/`502`** (`internal/api/ch_errors.go` (new, + tests), `internal/api/{errors,query,structured_query,pipes,schema,ch_settings}.go`, `internal/chconn/errclass.go` (`HTTPStatus` exported), `clients/ts/src/errors.ts` (+ tests), `tests/integration/query_errors_test.go` (new), `tests/integration/query_limits_test.go`, `internal/app/app_test.go`, `tests/e2e/sdk/{admin,query}.test.ts`, `AGENTS.md`, `docs/src/content/docs/{api,architecture}.md`, `docs/src/content/docs/{access-control,configuration}.mdx`, `docs/src/content/docs/sdk/{reference.md,index.mdx}`): fixes [#403](https://github.com/Wave-RF/WaveHouse/issues/403) and [#271](https://github.com/Wave-RF/WaveHouse/issues/271), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). ClickHouse answers a syntax error, a missing grant and an overloaded server alike with HTTP `500`, so `/v1/ops/query` turned a bad statement into a `502` and `/v1/query` and pipes into a `500` the SDK retried. All three now class the failure with `chconn.Classify` through one helper, `writeCHError`: a statement ClickHouse refused is `400 clickhouse.rejected`; a query over a rows/bytes limit, the role's own memory cap, or its time cap where that is no longer than `query_timeout` is `400 clickhouse.limit_exceeded`; `ACCESS_DENIED` is `403 clickhouse.access_denied`; credentials, user or database refused, or a redirect or `4xx` with no exception code from whatever fronts ClickHouse, is `502 clickhouse.misconfigured`; ClickHouse down, unreachable or overloaded is `503 clickhouse.unavailable` with `Retry-After: 5`; a failure with no verdict stays `500` (`502` on the proxy) as `clickhouse.unknown`. The error envelope gains `code` and `retryable` next to `error` on these responses — additive. A role with `max_execution_time` now queries with no context deadline and a cancel two seconds past the cap instead: clickhouse-go overwrote the cap's `max_execution_time` with deadline+5s for any deadline over 1s, so an overrun came back as a bare deadline, indistinguishable from waiting for a pooled connection; ClickHouse now enforces the cap itself and reports `TIMEOUT_EXCEEDED`. `POST /v1/ops/schema/refresh` against an unreachable ClickHouse is a `503` with `Retry-After` instead of a `500`. **SDK:** `WaveHouseError.code` and `retryable` now take the server's `code`/`retryable` when the body has them (`HTTP_` and "5xx retries" otherwise), so a rejected query is `clickhouse.rejected` rather than `HTTP_500`, and is not retried. - **An unavailable ClickHouse is retried with backoff instead of dead-lettering every row** (`internal/chconn/errclass.go` (new, + tests), `internal/ingest/{worker,backoff}.go` (`backoff.go` new, + tests), `internal/mq/{mq,embedded}.go`, `internal/testutil/mocks.go`, `tests/integration/ingest_outage_test.go` (new), `AGENTS.md`, `README.md`, `docs/src/content/docs/{ingest-pipeline,architecture,api,deployment,why-wavehouse}.md`, `docs/src/content/docs/{settings-directory,index,access-control}.mdx`): workstream A of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). A failed batch insert used to go through row-by-row isolation whatever the failure, so a ClickHouse that was down, overloaded or read-only failed every row twice and parked the whole batch on the DLQ. `chconn.Classify` now classes the failure first — `Rejected` (any ClickHouse exception code outside the availability and credential lists: the server read the row and refused it), `Unavailable` (connection refused/reset, timeouts, `TOO_MANY_SIMULTANEOUS_QUERIES`, `SERVER_OVERLOADED`, `MEMORY_LIMIT_EXCEEDED`, `TOO_MANY_PARTS`, `READONLY`, `TABLE_IS_READ_ONLY`, `KEEPER_EXCEPTION`, …), `Denied` (`AUTHENTICATION_FAILED`, `ACCESS_DENIED`, …) or `Unknown` (no code, no recognizable transport failure). Only `Rejected` is isolated and dead-lettered as before, and a multi-row batch refused with `TOO_MANY_PARTS` or `MEMORY_LIMIT_EXCEEDED` is split row by row first (`chconn.Splittable`), because a batch spanning too many partitions or too much memory can fail where each of its rows inserts; every other class hands the batch back to the queue with a delayed nak (`mq.Message.NakWithDelay`, new) under a jittered 1 s → 30 s backoff shared by every table on the same ClickHouse pool (a failure of one table — read-only, too many parts or mutations, a grant missing on it, `chconn.TableScoped` — backs off that table alone), which turns rows away without a request while it runs and probes once per window, and ClickHouse going away mid-isolation stops isolation and retries the rows it had not settled. Counted by the new `wavehouse_ingest_retries_total{table, reason}`; logged at `WARN` when an outage starts and at most every 30 s during it. A long outage now shows as a growing ingest stream and, at `mq.max_bytes_gb`, ingest `503`s — not as a full DLQ; a lasting failure of one table holds back its tenant's other tables once its waiting rows reach `maxAckPending`. Retried rows come back out of arrival order, which matters only to a `ReplacingMergeTree` without a version column or a `CollapsingMergeTree`. - **Schema discovery's retry loop jitters its backoff** (`internal/discovery/discovery.go` (+ tests), `internal/app/wire.go`, `internal/api/errors.go`, `AGENTS.md`, `docs/src/content/docs/{architecture,api,deployment}.md`): `RetryRefresh` slept exactly `2s * 2^n` capped at 60s, so instances retrying against one recovering ClickHouse fired in lockstep, every 60s on the same second. Each sleep is now drawn uniformly from below the backoff (full jitter), spreading the retries over the whole window and halving the mean wait — so a failing tenant's retries, their log lines and `wavehouse_schema_refresh_failures_total` come about twice as often ([#141](https://github.com/Wave-RF/WaveHouse/issues/141)). -- **A write that lands while a cached read is running no longer re-homes the pre-write rows under the post-write key** (`internal/cache/{cache,local,version_manager}.go` (+ tests), `internal/testutil/cachetest` (new), `internal/api/{structured_query,pipes}.go` (+ tests), `internal/ingest/worker.go` (+ tests), `internal/query/ident.go` (removed, + tests), `internal/app/wire.go`, `docs/src/content/docs/{api,architecture,deployment}.md`, `AGENTS.md`): fixes [#382](https://github.com/Wave-RF/WaveHouse/issues/382), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `POST /v1/query` rebuilt the version-folded cache key after the query ran, so an insert invalidating the table mid-query filed the rows read before it under the new versions, and they were served as fresh until their TTL (a pipe result's key folded no version, so pipes were unaffected; with the tenant's version in every key they now take the same snapshot). The `Cache` interface now snapshots at lookup: `Lookup(ctx, tenant, sha, deps)` returns the `Entry` and a `Snapshot` of the versions it read, and `Set(ctx, snapshot, value, ttl)` stores under that snapshot, so such a fill is orphaned and the next request reads the post-write rows. The singleflight leader's snapshot is the one used; coalescing is unchanged. The snapshot is taken before any input a bump invalidates is chosen, the tenant's ClickHouse connection included: both handlers now look up before they resolve the tenant's pool, so a reload that moves the tenant to another address or database after a request took the old pool orphans that request's fill instead of filing the old database's rows as fresh under the new tenant version. A tenant on no pool is still a `503` before a cached result is served or a query runs. A `Lookup` whose dependencies name another tenant is refused (`ErrForeignDependency`). `Set` now errors only when the backend failed: a value the cache declines (larger than the pool, a non-positive TTL) is not an error. One behavior change: the tenant's version is folded into every key, a pipe result's included, so `InvalidateTenant` (a tenant back on a pool after an absence, or moved to another ClickHouse address or database) now drops that tenant's cached pipe results as well as its query results; before, a pipe result stayed until its TTL. Inserts still do not invalidate pipe results ([#343](https://github.com/Wave-RF/WaveHouse/pull/343)). A backend-agnostic conformance suite, `cachetest.Run`, pins what a hit, a miss and each kind of bump mean, and `LocalCache` runs it; the Redis-compatible backend will run the same suite. The cache now escapes table and scope names itself: a `Namespace` carries them raw and the version index builds its keys with `internal/keyenc`, so neither the structured-query read nor the ingest worker's invalidation escapes them (`query.SafeEncodeToken` is gone) and no name reaches a key unescaped. The suite pins that a name holding a dot, a space or a `%` is read and bumped under one key, and that names which would run together unescaped (`a.0.b` against `a` with scope `b.0.`) stay two entries. The version index's own entries are unaffected by the escaping; only the rendered key changes: it now carries the tenant's version and escapes the caller's query key whole. All of them live in the process, so nothing stored is orphaned. +- **A write that lands while a cached read is running no longer re-homes the pre-write rows under the post-write key** (`internal/cache/{cache,local,version_manager}.go` (+ tests), `internal/testutil/cachetest` (new), `internal/api/{structured_query,pipes}.go` (+ tests), `internal/ingest/worker.go` (+ tests), `internal/query/ident.go` (removed, + tests), `internal/app/wire.go`, `docs/src/content/docs/{api,architecture,deployment}.md`, `AGENTS.md`): fixes [#382](https://github.com/Wave-RF/WaveHouse/issues/382), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `POST /v1/query` rebuilt the version-folded cache key after the query ran, so an insert invalidating the table mid-query filed the rows read before it under the new versions, and they were served as fresh until their TTL (a pipe result's key folded no version, so pipes were unaffected; with the tenant's version in every key they now take the same snapshot). The `Cache` interface now snapshots at lookup: `Lookup(ctx, tenant, sha, deps)` returns the `Entry` and a `Snapshot` of the versions it read, and `Set(ctx, snapshot, value, ttl)` stores under that snapshot, so such a fill is orphaned and the next request reads the post-write rows. The singleflight leader's snapshot is the one used; coalescing is unchanged. The snapshot is taken before any input a bump invalidates is chosen, the tenant's ClickHouse connection included: both handlers now look up before they resolve the tenant's pool, so a reload that moves the tenant to another address or database after a request took the old pool orphans that request's fill instead of filing the old database's rows as fresh under the new tenant version. A tenant on no pool is still a `503` before a cached result is served or a query runs. A `Lookup` whose dependencies name another tenant is refused (`ErrForeignDependency`). `Set` now errors only when the backend failed: a value the cache declines (larger than the pool, a non-positive TTL) is not an error. One behavior change: the tenant's version is folded into every key, a pipe result's included, so `InvalidateTenant` (a tenant back on a pool after an absence, or moved to another ClickHouse address or database) now drops that tenant's cached pipe results as well as its query results; before, a pipe result stayed until its TTL. Inserts still do not invalidate pipe results ([#343](https://github.com/Wave-RF/WaveHouse/pull/343)). A backend-agnostic conformance suite, `cachetest.Run`, pins what a hit, a miss and each kind of bump mean, and `LocalCache` runs it; the Redis-compatible backend will run the same suite. The cache now escapes table and scope names itself: a `Namespace` carries them raw and the cache renders each result's key with `internal/keyenc`, so neither the structured-query read nor the ingest worker's invalidation escapes them (`query.SafeEncodeToken` is gone) and no name reaches a key unescaped. The suite pins that a name holding a dot, a space or a `%` is read and bumped under one key, and that names which would run together unescaped (`a.0.b` against `a` with scope `b.0.`) stay two entries. The version index's own entries are unaffected by the escaping; only the rendered key changes: it now carries the tenant's version and escapes the caller's query key whole. All of them live in the process, so nothing stored is orphaned. - **The cache's version index no longer grows with every bump, and forgets a tenant no longer served** (`internal/cache/{local,version_manager}.go` (+ tests), `internal/app/wire.go` (+ tests), `docs/src/content/docs/architecture.md`, `AGENTS.md`): part of [#262](https://github.com/Wave-RF/WaveHouse/issues/262) (growth across bumps and a departed tenant's memory; per-table scope cardinality is left open, see the issue) and of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The index nested each table under its tenant's version and each scope under its table's, and never pruned, so every tenant invalidation left the tenant's whole index behind, and it grew with every tenant ever served. It now holds one version per tenant, per (tenant, table) and per (tenant, table, scope), bumped in place. A tenant invalidation drops the tenant's index and hands its next key a generation unique within the process, so nothing cached before it can match again, and a table bump drops the table's scope versions. After each settings reload the index of every tenant no longer served, removed or rejected, is dropped the same way; its cached results are orphaned with it, as they already were when such a tenant came back on a pool. No change to what is cached or served. The Redis-compatible backend (#613) will bound its versions with a TTL instead. - **An explicit `false`, `0` or `""` in `config.yaml` is no longer replaced by the key's default** (`internal/config/config.go`, `internal/config/defaults_test.go` (new), `docs/src/content/docs/configuration.mdx`, `AGENTS.md`): [#631](https://github.com/Wave-RF/WaveHouse/issues/631). Defaults lived in cleanenv `env-default` tags, which cleanenv applies after the YAML decode to any field still at its zero value, so it could not tell a key the file set to its zero value from one the file left out. `otel.traces.enabled: false`, `otel.metrics.enabled: false` and `otel.logs.enabled: false` came back `true`; `otel.traces.sample_rate: 0` and `otel.logs.sample_rate: 0` came back `1.0`; `server.shutdown_timeout: 0` came back `10`; `cache.l1_max_cost: 0`, `prometheus.path: ""` and `data_dir: ""` came back as their defaults; `server.port: 0` came back `8080`. All of it was silent. Defaults now live in one Go function, `defaults()`, which `Load` starts from before decoding the file and then applying `WH_*` variables, so the order is env > YAML > default and a key the file sets always wins. **Behaviour change if your file relied on the bug:** a zero you wrote now takes effect. A file that says `sample_rate: 0` now exports no traces (or no DEBUG/INFO logs), where it silently exported everything; a signal set `enabled: false` is now off; `shutdown_timeout: 0` now skips the drain. `cache.l1_max_cost: 0`, `server.port: 0`, and `data_dir: ""` now refuse boot (`cache init: MaxCost can't be zero`, `server.port 0 out of range`, `data_dir (WH_DATA_DIR) is required`) instead of running on the default; an empty `prometheus.path` refuses boot when `prometheus.enabled` is true. Delete the key to get the default back. Env vars are unchanged: they already honoured an explicit zero. New tests load through `config.Load` for every affected key (a YAML zero is kept, an absent key gets the default, env wins in both directions), refuse an `env-default` tag on any field, and pin each documented default in `configuration.mdx` to `defaults()`. - **An embedded queue store that cannot be created fails boot at once, naming the cause** (`internal/mq/embedded.go` (+ tests)): part of [#617](https://github.com/Wave-RF/WaveHouse/issues/617). A regular file at `/nats`, or a `nats` directory that could not be created there, failed JetStream in the background, so boot waited out the server's 5s readiness check and reported only `nats server not ready`. `NewEmbedded` now creates the directory first (at `0700`, as the server does) and refuses boot with the mkdir error. An existing but unwritable `nats` directory still takes the old path. diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 7aede5bad..678c52a3f 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -114,7 +114,7 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ - **cache.go** — `Cache` interface: `Lookup`, `Set`, `Invalidate`, `InvalidateTenant`, `Close`, plus `QueryTimeToTTL`, which sets a result's TTL from how long its query took (10 s floor, 1 h ceiling). Every entry is one tenant's: `Lookup` takes the tenant, the caller's query key — `:query:`, built by the two cached handlers in `api/` (`queryCacheKey`, with the tenant read off the request's store — `settings.Store.Tenant`), which use it as their [singleflight](https://pkg.go.dev/golang.org/x/sync/singleflight) key too — and the `Namespace`s the result depends on — one for a structured query, none yet for a pipe (a pipe's table dependencies are [#343](https://github.com/Wave-RF/WaveHouse/pull/343)) — each of that tenant (another tenant's is `ErrForeignDependency`) and naming a table and scope by their raw names, which the cache escapes where it builds a key. `Lookup` returns the `Entry` (a nil value is a miss) and a `Snapshot` of the versions it read; on a miss the handler runs the query and passes that snapshot to `Set`, so a result is filed under the versions read *before* its query ran, and a write that lands while it runs orphans the fill rather than re-homing pre-write rows under the post-write versions ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)). The snapshot is taken before any input a bump invalidates is chosen, the tenant's connection included: a reload that moves the tenant to another address or database runs `Pools.Reconcile` and then `InvalidateTenant` (a repoint that keeps both, such as a username or `tls` change, reads the same tables and bumps nothing), so a request that took the old pool files the old database's rows under a version that bump orphans, whether its `Set` lands before the bump or after. The singleflight leader's snapshot is the one used. `Set` returns an error only when the backend failed; a value the cache declines — larger than it keeps, a non-positive TTL, a zero snapshot — is not one. What a hit, a miss and a bump mean is pinned by the conformance suite every backend runs, `internal/testutil/cachetest`. - **local.go** — `LocalCache`, the in-process L1 on [Ristretto](https://github.com/dgraph-io/ristretto): one pool shared by every tenant (a heavier tenant holds more of it), sized by the boot config's `cache.l1_max_cost`. -- **version_manager.go** — `VersionManager`, the invalidation index behind `Invalidate` and `InvalidateTenant`: one version per tenant, per (tenant, table) and per (tenant, table, scope), each keyed by its name alone and bumped in place, so the index holds one entry per live tenant, table and scope however often each is bumped ([#262](https://github.com/Wave-RF/WaveHouse/issues/262)). A query key folds the tenant's version and, for each dependency, its tenant's, table's and scope's, so bumping a table (a scopeless write) orphans every scope of it, and bumping one scope orphans that scope and the whole-table view — scope is reserved and empty today, so every write is the whole-table bump — all without touching the pool. Every field — the caller's query key, the tenant id, and each dependency's table and scope — is escaped and joined by `internal/keyenc` where the key is built, so a dot, a space or a `%` in a name is never read as a separator: each dependency renders as `..
.
..`, and the whole entry key is `|.||…`. The tenant leads every key ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8): the same table under two tenants is two namespaces, so a bump through `Invalidate` under one tenant never touches — and a read under one tenant is never served — the other's results. A tenant's version is a *generation*, unique within the process and handed out by the first key built for the tenant; `BumpTenant` (behind `InvalidateTenant`) drops the tenant's whole index, so the next key gets a fresh generation no cached entry folds, orphaning every cached result of the tenant in one step — a pipe result with no dependencies, and a table no bump ever keyed, included — for a tenant back on a pool after an absence from the fan-out, or moved to another address or database (story 6). `LocalCache.Prune` does the same for every tenant no longer served, which `internal/app` runs after each settings reload, so a tenant removed or rejected stops holding its index. A table bump drops the table's scope versions with it, since every key they were folded into also folds the old table version; and any bump (of a table, a scope or the tenant) under a tenant with no index is a no-op that records nothing, since the next key built for it gets a fresh generation no cached entry folds — so neither the `sharedTables` fan-out nor an insert still in flight for a tenant just pruned brings its index back. The index is per tenant; the cross-tenant invalidation an insert into a shared table needs is not the index's but the wiring's: `internal/app` hands the ingest worker a cache (`sharedTables`) that repeats each bump under every tenant on the same ClickHouse address and database. +- **version_manager.go** — `VersionManager`, the invalidation index behind `Invalidate` and `InvalidateTenant`: one version per tenant, per (tenant, table) and per (tenant, table, scope), each keyed by its name alone and bumped in place, so the index holds one entry per live tenant, table and scope however often each is bumped ([#262](https://github.com/Wave-RF/WaveHouse/issues/262)). An entry's key (`QueryKey`) folds the tenant's version and, for each dependency, its tenant's, table's and scope's, so bumping a table (a scopeless write) orphans every scope of it, and bumping one scope orphans that scope and the whole-table view — scope is reserved and empty today, so every write is the whole-table bump — all without touching the pool. Every field — the caller's query key, the tenant id, and each dependency's table and scope — is escaped and joined by `internal/keyenc` where the key is built, so a dot, a space or a `%` in a name is never read as a separator: each dependency renders as `..
.
..`, and the whole entry key is `|.||…`. The tenant leads every key ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8): the same table under two tenants is two namespaces, so a bump through `Invalidate` under one tenant never touches — and a read under one tenant is never served — the other's results. A tenant's version is a *generation*, unique within the process and handed out by the first key built for the tenant; `BumpTenant` (behind `InvalidateTenant`) drops the tenant's whole index, so the next key gets a fresh generation no cached entry folds, orphaning every cached result of the tenant in one step — a pipe result with no dependencies, and a table no bump ever keyed, included — for a tenant back on a pool after an absence from the fan-out, or moved to another address or database (story 6). `LocalCache.Prune` does the same for every tenant no longer served, which `internal/app` runs after each settings reload, so a tenant removed or rejected stops holding its index. A table bump drops the table's scope versions with it, since every key they were folded into also folds the old table version; and any bump (of a table, a scope or the tenant) under a tenant with no index is a no-op that records nothing, since the next key built for it gets a fresh generation no cached entry folds — so neither the `sharedTables` fan-out nor an insert still in flight for a tenant just pruned brings its index back. The index is per tenant; the cross-tenant invalidation an insert into a shared table needs is not the index's but the wiring's: `internal/app` hands the ingest worker a cache (`sharedTables`) that repeats each bump under every tenant on the same ClickHouse address and database. ### `config/` — Configuration diff --git a/internal/cache/version_manager_test.go b/internal/cache/version_manager_test.go index 847920542..079453060 100644 --- a/internal/cache/version_manager_test.go +++ b/internal/cache/version_manager_test.go @@ -164,10 +164,26 @@ func TestVersionManager_GenerationsNeverRepeat(t *testing.T) { func TestVersionManager_BumpWithoutIndex(t *testing.T) { t.Parallel() vm := NewVersionManager() + vm.BumpTable("acme", "users") + assert.Zero(t, vm.size(), "a table bump for a tenant with no index creates nothing") + vm.BumpNamespace(Namespace{Tenant: "acme", Table: "users", Scope: "org_1"}) + assert.Zero(t, vm.size(), "a namespace bump for a tenant with no index creates nothing") + vm.BumpTenant("acme") - assert.Zero(t, vm.size()) + assert.Zero(t, vm.size(), "bumping a tenant with no index is a no-op") + + // An insert still in flight for a tenant just pruned must not bring its + // index back: a write racing the prune sees the tenant gone and bumps + // blind, same as above. + vm.QueryKey("acme", "h", nil) + vm.Prune(func(tenant.ID) bool { return false }) + assert.Zero(t, vm.size(), "prune released the tenant's index") + + vm.BumpTable("acme", "users") + vm.BumpNamespace(Namespace{Tenant: "acme", Table: "users", Scope: "org_1"}) + assert.Zero(t, vm.size(), "a bump for a tenant just pruned must not recreate its index") } func TestVersionManager_Prune(t *testing.T) { From 5a72724b83afe4b14bcf9f0f43f66ccd105f9048 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 08:40:39 -0400 Subject: [PATCH 35/45] =?UTF-8?q?fix(cache):=20review=20round=202=20?= =?UTF-8?q?=E2=80=94=20stale=20docs=20claims,=20redisConfig=20field=20pin?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - configuration.mdx: isAuthError also matches NOPERM (an ACL user missing a connection command); the boot-error-logging line only named WRONGPASS/NOAUTH. - CHANGELOG.md: the E4 entry claimed the e2e coverage exclude for the cache files was gone; .testcoverage.yml still excludes internal/cache/pending.go, internal/config/cache_redis.go and internal/cache/(local|version_manager).go from e2e, for reasons the entry now states. Also added the docs/.github files this PR touched that the entry's file list was missing. - pipes.mdx, api.md: "L1" claims that are wrong once cache.backend is redis (there is no L1) — reworded to "the query cache"/"cache". - deployment.md: "Persistence is not needed" was incomplete — a restart *with* persistence reloads a stale snapshot and can serve invalidated results as hits until their TTL. Now says to run without persistence, and that restoring a snapshot is a rollback. - development.md: note that shared_cache_test.go starts its own Redis per test and boots extra cache.backend=redis instances over the integration suite's ClickHouse. - app_test.go: TestRedisConfig_FromLoadedDefaults left Username, DB and TLS at their zero value on both sides of its assert.Equal, so deleting any of their three mapping lines in redisConfig wouldn't fail it. Added TestRedisConfig_UsernameDBTLSMapped, which drives all three to a non-zero value through config.Load. Mutation-checked: deleting each of the three lines in redisConfig fails this test (the TLS line fails to compile instead, since dropping it leaves the local `t` unused — still a build failure). Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_017aS7rLrH1RKkUMem7X4ckd --- CHANGELOG.md | 2 +- docs/src/content/docs/api.md | 2 +- docs/src/content/docs/configuration.mdx | 2 +- docs/src/content/docs/deployment.md | 2 +- docs/src/content/docs/development.md | 2 +- docs/src/content/docs/pipes.mdx | 2 +- internal/app/app_test.go | 22 ++++++++++++++++++++++ 7 files changed, 28 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c4e6e2934..472f94af1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **`cache.backend: redis` shares the query cache across instances** (`internal/config/{cache_redis,backends,config}.go` (+ tests), `internal/app/wire.go` (+ tests), `internal/ingest/worker.go`, `tests/integration/shared_cache_test.go`, `scripts/orchestrator/main.go`, `tests/e2e/fixtures/config.yaml`, `.testcoverage.yml`, `deployments/compose/dependencies.yaml`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md,settings-directory.mdx,getting-started.md,pipes.mdx,api.md,development.md}`, `README.md`, `AGENTS.md`): PR E4 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config's `cache.backend` now takes `redis`, configured by a new `cache.redis` block (`WH_CACHE_REDIS_*`): `addrs` (required), `mode` (`standalone` or `cluster`; `sentinel` refuses boot until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `username`, `password` (a secret — set it through the environment), `db`, `tls.{enabled,ca_file,cert_file,key_file,server_name,insecure_skip_verify}`, `key_prefix` (`wh`), `timeout` (`100ms`) and `dial_timeout` (`1s`), each at most `1s` since boot and shutdown each wait out a connection attempt they bound, `max_value_bytes` (1 MiB), `compress_min_bytes` (`1024`; `0` never compresses) and `version_ttl` (`168h`). Every instance pointed at one server shares its cached results, and an insert on any instance invalidates every instance's. It is also the shared cache a split of [`roles`](https://github.com/Wave-RF/WaveHouse/pull/622) needs: the refusal of `api` without `ingest`, or the reverse, over `cache.backend: local` now names `redis`, though every split is still refused while the queue is embedded. A malformed block — no address, an address without a valid port, more than one address in `standalone` mode, a URL-style address (refused without repeating it, since it may carry a password), an unknown mode, `db` other than `0` in cluster mode, an unreadable TLS file, a TLS key set while `tls.enabled` is off — refuses boot; an unreachable server, or one that rejects the password, does not: the process boots with the cache bypassed and keeps reconnecting, logging a rejected password at `ERROR` on every attempt, so a rotated secret cannot crash-loop every instance at once. `insecure_skip_verify`, and a `cache.redis.addrs` set while `cache.backend` is `local`, are logged at `WARN` at boot. The ingest worker's log of an invalidation that did not land drops from `ERROR` to `WARN`, since the shared backend defers and retries it: an outage would otherwise log an `ERROR` for every batch. The e2e suite now runs against a Redis testcontainer with `cache.backend: redis`, so the shared backend is exercised end to end, and the e2e coverage exclude for its files is gone; the unit and integration suites keep covering `local`. An integration test boots two instances over one Redis and one ClickHouse: a result one fills is a hit for the other, and a row ingested through one is served fresh by the other on its next query, well inside the stale entry's TTL; another pauses Redis and checks queries keep succeeding from ClickHouse, then turn back to hits; a third runs the first one's hit, insert and fresh-miss lifecycle on the suite's own `cache.backend: local` app, since e2e no longer exercises that backend. `deployments/compose/dependencies.yaml` gains an optional `redis` profile for local multi-instance work. The deployment guide gains a "Multiple instances and the shared cache" section: what each instance keeps to itself, what a reader on another instance can see and when, `maxmemory-policy`, and the metrics to alert on. +- **`cache.backend: redis` shares the query cache across instances** (`internal/config/{cache_redis,backends,config}.go` (+ tests), `internal/app/wire.go` (+ tests), `internal/ingest/worker.go`, `tests/integration/shared_cache_test.go`, `scripts/orchestrator/main.go`, `tests/e2e/fixtures/config.yaml`, `.testcoverage.yml`, `deployments/compose/dependencies.yaml`, `config.yaml`, `.github/workflows/{ci.yml,README.md}`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md,settings-directory.mdx,getting-started.md,pipes.mdx,api.md,development.md,index.mdx,why-wavehouse.md,sdk/reference.md}`, `README.md`, `AGENTS.md`): PR E4 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config's `cache.backend` now takes `redis`, configured by a new `cache.redis` block (`WH_CACHE_REDIS_*`): `addrs` (required), `mode` (`standalone` or `cluster`; `sentinel` refuses boot until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `username`, `password` (a secret — set it through the environment), `db`, `tls.{enabled,ca_file,cert_file,key_file,server_name,insecure_skip_verify}`, `key_prefix` (`wh`), `timeout` (`100ms`) and `dial_timeout` (`1s`), each at most `1s` since boot and shutdown each wait out a connection attempt they bound, `max_value_bytes` (1 MiB), `compress_min_bytes` (`1024`; `0` never compresses) and `version_ttl` (`168h`). Every instance pointed at one server shares its cached results, and an insert on any instance invalidates every instance's. It is also the shared cache a split of [`roles`](https://github.com/Wave-RF/WaveHouse/pull/622) needs: the refusal of `api` without `ingest`, or the reverse, over `cache.backend: local` now names `redis`, though every split is still refused while the queue is embedded. A malformed block — no address, an address without a valid port, more than one address in `standalone` mode, a URL-style address (refused without repeating it, since it may carry a password), an unknown mode, `db` other than `0` in cluster mode, an unreadable TLS file, a TLS key set while `tls.enabled` is off — refuses boot; an unreachable server, or one that rejects the password, does not: the process boots with the cache bypassed and keeps reconnecting, logging a rejected password at `ERROR` on every attempt, so a rotated secret cannot crash-loop every instance at once. `insecure_skip_verify`, and a `cache.redis.addrs` set while `cache.backend` is `local`, are logged at `WARN` at boot. The ingest worker's log of an invalidation that did not land drops from `ERROR` to `WARN`, since the shared backend defers and retries it: an outage would otherwise log an `ERROR` for every batch. The e2e suite now runs against a Redis testcontainer with `cache.backend: redis`, so the shared backend is exercised end to end; the e2e per-suite exclude now names what its run still can't reach instead — `internal/cache/pending.go` (the retry of an invalidation the server did not take, which needs an outage), `internal/config/cache_redis.go` (the block's own rejection paths) and `internal/cache/(local|version_manager).go` (the `local` backend, which e2e no longer runs) — and the unit and integration suites keep covering them. An integration test boots two instances over one Redis and one ClickHouse: a result one fills is a hit for the other, and a row ingested through one is served fresh by the other on its next query, well inside the stale entry's TTL; another pauses Redis and checks queries keep succeeding from ClickHouse, then turn back to hits; a third runs the first one's hit, insert and fresh-miss lifecycle on the suite's own `cache.backend: local` app, since e2e no longer exercises that backend. `deployments/compose/dependencies.yaml` gains an optional `redis` profile for local multi-instance work. The deployment guide gains a "Multiple instances and the shared cache" section: what each instance keeps to itself, what a reader on another instance can see and when, `maxmemory-policy`, and the metrics to alert on. - **A Redis-compatible shared cache backend** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/cache.go`, `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`, `CONTRIBUTING.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag, with the table and scope escaped into the key (`internal/keyenc`) so no two names share a token; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row, or one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like), open a circuit breaker that skips the server until a probe write succeeds (connections are replaced every minute, so after a failover behind a stable address the process reaches the new primary, and delivers the bumps it owes, within about a minute, provided a fresh connection's dial and handshake fit in the per-operation timeout), and deferred invalidations are retried until they land — the first at once, and at the probe's cadence while the breaker is open — collapsing to one tenant-wide bump per tenant past 100,000 keys; until one lands, the process that owes it bypasses the lookups it would orphan. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. `cache.backend: redis` selects it (the entry above). Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression, server-stops-answering, refused-writes (a demoted primary, a full `noeviction` server), failover-behind-a-stable-address and owed-invalidation cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. - **One conformance suite for every `mq.Broker`, and a transient broker failure is a `503`** (`internal/mq/mqtest/` (new: the suite and the embedded broker's run of it), `internal/mq/{mq,embedded}.go` (+ tests), `internal/api/ingest.go` (+ tests), `.testcoverage.yml`, `docs/src/content/docs/{api,architecture}.md`, `AGENTS.md`): the first piece of the external-NATS workstream of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `mqtest.Run` states the `Broker` contract as behavior — round trips with names that need encoding, per-tenant order, `Nak` and `AckWait` redelivery, the trace context reaching `Subscribe`, dead-lettering that keeps the topic and leaves the original unacked, per-tenant dead-letter counts, replay bounds and isolation, and exactly one `failed` report when delivery ends underneath a consumer — through the interfaces alone, so the external backend runs the same cases, with `mqtest.Caps` for the four places where its semantics legitimately differ. The embedded broker passes it; writing it turned up that a durable deleted on several tenants' queues could report on `failed` more than once, which is fixed and pinned by a test that deletes it on one queue after another. It also turned up a replay that lost its connection mid-pull passing for a caught-up one when the pull ended in a timeout; that is an error now, as the `Replayer` contract says. The interface comments now allow a delivery unit that is a partition holding several tenants, a `CreateConsumer` that finds a durable rather than creating one, a `PurgeAcked` that leaves retention to the operator, and zero dead-letter counts where there is no per-tenant queue. A new sentinel, `mq.ErrUnavailable`, is a broker that cannot be reached or does not answer in time: the ingest handler answers it with `503` + `Retry-After: 5` rather than the `500` "publish failed" it would have been. Nothing returns it yet; the external backend of #613 will. - **Process roles: the API and the background workers can run in separate processes** (`internal/config/config.go` (+ `roles_test.go`, `defaults_test.go`), `internal/config/backends.go`, `internal/app/{app,wire}.go` (+ `roles_test.go`), `internal/api/router.go` (+ tests), `tests/integration/{setup,tenants}_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md}`, `AGENTS.md`), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The boot config gains `roles` (`WH_ROLES`, default `api,ingest,sweeper`, set in `defaults()` like every boot default, so an explicit `roles: []` refuses boot) and `instance_id` (`WH_INSTANCE_ID`, default `-<8 hex>`, fresh at every boot; logged at boot, and recorded as a lease holder once a shared `coord.backend` exists). A process wires only what its roles need: `api` runs the HTTP API with schema discovery, the token verifiers, the dedupe stores and the SSE hub (all per API process); `ingest` runs the ingest worker; `sweeper` runs the sweeper under its lease. A process without `api` serves an ops-only listener on `server.port` (the probes and their aliases, `/version`, the same-port metrics path, and `POST /v1/ops/settings/reload`, which takes the operator key alone); every other route answers 404, under `/v1/ops` once the operator key has passed. Boot refuses any split over the embedded MQ, which no other process can reach, and `api` without `ingest` (or the reverse) over a local cache, which the ingest worker's invalidations would never reach. Until a shared `mq.backend` exists, every process therefore runs every role, which is the default, so nothing changes for an existing deployment. `data_dir` is probed for Pebble only in a process running `api`. A `config.Config` built without `config.Load` must now name its roles (`config.AllRoles()` for all of them): `app.New` refuses an empty set. diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index 0ed66f231..77bbe7921 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -445,7 +445,7 @@ ClickHouse's inline `FORMAT` clause (e.g. `SELECT 1 FORMAT CSV` or `… FORMAT P The proxy buffers the upstream response in memory before forwarding (no row-streaming yet), so a `SELECT *` from a large table can pin RAM on the API server. To avoid an admin OOMing themselves, responses larger than 64 MiB return 502 with a `clickhouse response exceeded N bytes` error. Narrow the query with `LIMIT`, or use a streaming client outside WaveHouse that talks to ClickHouse directly (the standard escape hatch — the same admin credentials work). ::: -This endpoint **does not cache, does not singleflight, and emits `Cache-Control: no-store`** — every request goes straight to ClickHouse, mutation or read, and downstream HTTP caches are explicitly told not to store the response. Raw SQL is an admin escape hatch with infrequent, ad-hoc traffic, so the L1/singleflight machinery would only add complexity without a real hit-rate win. Use [`POST /v1/query?table={table}`](#post-v1querytabletable--structured-query) or [`GET/POST /v1/pipes/{name}`](#getpost-v1pipesname--execute-named-pipe) for the cached read paths (dashboards, high-QPS clients, etc.) — both go through the query cache ([`cache.backend`](/configuration#backends): in-process, or a Redis shared by every instance) with singleflight coalescing. +This endpoint **does not cache, does not singleflight, and emits `Cache-Control: no-store`** — every request goes straight to ClickHouse, mutation or read, and downstream HTTP caches are explicitly told not to store the response. Raw SQL is an admin escape hatch with infrequent, ad-hoc traffic, so the cache/singleflight machinery would only add complexity without a real hit-rate win. Use [`POST /v1/query?table={table}`](#post-v1querytabletable--structured-query) or [`GET/POST /v1/pipes/{name}`](#getpost-v1pipesname--execute-named-pipe) for the cached read paths (dashboards, high-QPS clients, etc.) — both go through the query cache ([`cache.backend`](/configuration#backends): in-process, or a Redis shared by every instance) with singleflight coalescing. :::note[Admin only] The route is mounted under `/v1/ops/*`, behind the `RequireAdmin` gate: only a caller whose JWT role equals the policy `admin_role` (`"admin"` by default) — or who presents the non-JWT [operator key](#authentication) — may use it. A tokenless request (or a valid token without a role claim) resolves to the `default_role` (not the admin role unless `default_role` is deliberately set to it — a loudly-warned dev-only setting) and is rejected with `403`; a present-but-invalid token — expired, malformed, bad signature — keeps its stashed verification error and fails loud with `401` instead. Raw SQL has no per-statement scope check (a full SQL parser would be needed to authorize predicates), so the role gate is the entire authorization story, shared with the rest of `/v1/ops/*` (see [Admin Endpoints](#admin-endpoints)). The normal surfaces for non-admin callers are `POST /v1/ingest?table={table}` for writes, `POST /v1/query?table={table}` for structured reads, and `GET/POST /v1/pipes/{name}` for pre-defined queries — none of which expose raw SQL. diff --git a/docs/src/content/docs/configuration.mdx b/docs/src/content/docs/configuration.mdx index d625b2e46..de970487c 100644 --- a/docs/src/content/docs/configuration.mdx +++ b/docs/src/content/docs/configuration.mdx @@ -170,7 +170,7 @@ The `redis` backend's settings, read only when `cache.backend` is `redis`. It is **When the server is unreachable or misbehaves, the cache is bypassed; queries are not.** A failure or a timeout makes the lookup a miss and the fill a no-op. Five in a row, or one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like), open a circuit breaker that skips the server entirely until a probe write, every 5 s, succeeds. Queries then go straight to ClickHouse, still coalesced per instance by `singleflight`. An invalidation the server did not take is kept and retried until it lands, and until then the instance that owes it bypasses the lookups it would orphan. `/readyz` does not depend on the cache. -**Boot does not wait for the server.** A malformed block (an address without a port, `mode: cluster` with `db` other than `0`, an unreadable or unparsable TLS file) refuses boot. A server that cannot be reached, or that refuses the credentials, does not: the process boots with the cache bypassed and keeps reconnecting, with backoff up to 30 s. A rejected password (`WRONGPASS`, `NOAUTH`) is logged at `ERROR` on every attempt; any other failure at `WARN`. This is deliberate: a rotated Redis password must not crash-loop every instance at once. Watch `wavehouse_cache_breaker_open`, which reads `1` while the cache is bypassed, including before the first connection. +**Boot does not wait for the server.** A malformed block (an address without a port, `mode: cluster` with `db` other than `0`, an unreadable or unparsable TLS file) refuses boot. A server that cannot be reached, or that refuses the credentials, does not: the process boots with the cache bypassed and keeps reconnecting, with backoff up to 30 s. A rejected credential (`WRONGPASS`, `NOAUTH`, or `NOPERM` for an ACL user missing a connection command) is logged at `ERROR` on every attempt; any other failure at `WARN`. This is deliberate: a rotated Redis password must not crash-loop every instance at once. Watch `wavehouse_cache_breaker_open`, which reads `1` while the cache is bypassed, including before the first connection. ### Authentication diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index 1e14399b2..aa29bb6b6 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -437,7 +437,7 @@ The query-result cache is the layer that can be shared today. With the default ` - **A pipe that writes** (an `INSERT` in `pipes.json`) has its result cached like a read, so a repeated identical call is answered from the cache and the write does not run again ([#386](https://github.com/Wave-RF/WaveHouse/issues/386)). With a shared cache that holds on every instance, until the entry's TTL. - **Admin writes through `POST /v1/ops/query`** do not invalidate the cache ([#394](https://github.com/Wave-RF/WaveHouse/issues/394)). With a shared cache, the stale results they leave are served by every instance, not only one. -**Sizing the server.** Every key WaveHouse writes has a TTL, and a version token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses, never bring back an entry it had invalidated. So set `maxmemory` and let the server evict: `maxmemory-policy allkeys-lru` (or `allkeys-lfu`, `volatile-lru`, `volatile-lfu`). Under `noeviction`, a full server refuses the writes: each refusal (a fill's is counted by `wavehouse_cache_set_failures_total{reason="oom"}`) bypasses the cache of the instance that got it, and invalidations are kept and retried, so the pre-insert results above stay served by the others: avoid `noeviction`. A stored result is capped at `cache.redis.max_value_bytes` (1 MiB compressed). A tenant's version tokens share one hash tag, so each lookup reads them in one `MGET` in cluster mode as well. The results themselves carry no hash tag and spread across shards. Persistence is not needed: an empty server after a restart is a cold cache, not a wrong one. +**Sizing the server.** Every key WaveHouse writes has a TTL, and a version token lost to eviction, expiry or `FLUSHALL` can only cause misses, never bring back an entry it had invalidated. So set `maxmemory` and let the server evict: `maxmemory-policy allkeys-lru` (or `allkeys-lfu`, `volatile-lru`, `volatile-lfu`). Under `noeviction`, a full server refuses the writes: each refusal (a fill's is counted by `wavehouse_cache_set_failures_total{reason="oom"}`) bypasses the cache of the instance that got it, and invalidations are kept and retried, so the pre-insert results above stay served by the others: avoid `noeviction`. A stored result is capped at `cache.redis.max_value_bytes` (1 MiB compressed). A tenant's version tokens share one hash tag, so each lookup reads them in one `MGET` in cluster mode as well. The results themselves carry no hash tag and spread across shards. **Run it without persistence** (`save ""` and `appendonly no`): a restart without persistence can only cause misses, the same as any other token loss. With persistence on, a restart is not that — it reloads whatever snapshot or AOF it last wrote, tokens and values it had already invalidated included, so a fresh instance can serve the pre-write rows filed under them as hits until their TTL (up to 1 h) expires. Treat restoring a snapshot as a rollback, not a resume. **The server is inside the trust boundary.** A cached result is served after the access policy has filtered it, so whoever can write to the server can change what any caller reads. Keep it on a private network, require a password or ACL user (`WH_CACHE_REDIS_PASSWORD`), use TLS across links you do not trust, and share it only with deployments you trust as much as this one. diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index 581df72d1..11fcb82d0 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -345,7 +345,7 @@ Each test target writes `covdata` to `tmp/coverage//data/`, renders a tex | E2E tests (SDK) | `tests/e2e/sdk/*.test.ts` | Yes | `make test-e2e` | - **Unit tests** live beside the code they test (e.g., `internal/discovery/discovery_test.go`). They use mocks or embedded NATS (in-process, no Docker needed). -- **Integration tests** use the `//go:build integration` build tag. In `tests/integration`, `TestMain` starts one ClickHouse testcontainer and boots the production wiring against it through `app.New` (embedded NATS, ingest worker, sweeper, hub, the API server on a random loopback port); tests reach it via `env(t)` and create their own tables. DLQ tests use `assert.Eventually` with a 30-second timeout for the 5-second ingest worker batch window. `internal/cache`'s integration tests start their own containers instead — Redis, Valkey, Dragonfly and a one-node Redis Cluster — for the shared backend. +- **Integration tests** use the `//go:build integration` build tag. In `tests/integration`, `TestMain` starts one ClickHouse testcontainer and boots the production wiring against it through `app.New` (embedded NATS, ingest worker, sweeper, hub, the API server on a random loopback port); tests reach it via `env(t)` and create their own tables. DLQ tests use `assert.Eventually` with a 30-second timeout for the 5-second ingest worker batch window. `internal/cache`'s integration tests start their own containers instead — Redis, Valkey, Dragonfly and a one-node Redis Cluster — for the shared backend. `shared_cache_test.go` starts its own Redis testcontainer per test (`startRedis`) and boots extra, independent `cache.backend: redis` instances over that same ClickHouse (`bootRedisApp`), to exercise the cache shared across processes rather than one package in isolation. Shared test utilities live in `internal/testutil/`. The packages log through `slog.Default()`, so tests reach log output through `internal/testutil/logtest`: `logtest.Silence()` in a package's `TestMain` discards it, and `logtest.Capture(t, level)` routes it to a buffer for a test that asserts on log lines — such a test must not call `t.Parallel()`, because the default logger is process-wide. diff --git a/docs/src/content/docs/pipes.mdx b/docs/src/content/docs/pipes.mdx index b64129225..b6ded2005 100644 --- a/docs/src/content/docs/pipes.mdx +++ b/docs/src/content/docs/pipes.mdx @@ -7,7 +7,7 @@ sidebar: A **named pipe** is a saved SQL query, registered under a name, that callers run by name with parameters — without ever sending raw SQL. They turn an ad-hoc query into a stable, cached, access-controlled endpoint: you write the SQL once as an operator in the settings directory's [`pipes.json`](/settings-directory#pipesjson), expose it at `GET/POST /v1/pipes/{name}`, and clients supply only the declared parameters. -Pipes are the right tool when a query is reusable and shouldn't live in client code — dashboards, reports, public APIs over curated slices of data. They sit on the **cached read path** (shared L1 + singleflight, same as structured queries), and authorize through a simple per-pipe allowlist rather than the full [policy engine](/access-control). +Pipes are the right tool when a query is reusable and shouldn't live in client code — dashboards, reports, public APIs over curated slices of data. They sit on the **cached read path** (the query cache + singleflight, same as structured queries), and authorize through a simple per-pipe allowlist rather than the full [policy engine](/access-control). ## Anatomy of a pipe diff --git a/internal/app/app_test.go b/internal/app/app_test.go index 7a23330ed..945f7eb96 100644 --- a/internal/app/app_test.go +++ b/internal/app/app_test.go @@ -832,6 +832,28 @@ func TestRedisConfig_FromLoadedDefaults(t *testing.T) { assert.Equal(t, cache.RedisSentinel, config.RedisSentinel) } +// Username, DB and TLS are zero on both sides of TestRedisConfig_FromLoadedDefaults' +// assert.Equal, so deleting any of their three mapping lines in redisConfig +// would pass it anyway. Drive all three through config.Load to a non-zero +// value and assert on them directly. +func TestRedisConfig_UsernameDBTLSMapped(t *testing.T) { + t.Setenv("WH_SETTINGS_DIR", t.TempDir()) + t.Setenv("WH_CACHE_BACKEND", "redis") + t.Setenv("WH_CACHE_REDIS_ADDRS", "a:6379") + t.Setenv("WH_CACHE_REDIS_USERNAME", "u") + t.Setenv("WH_CACHE_REDIS_DB", "2") + t.Setenv("WH_CACHE_REDIS_TLS_ENABLED", "true") + t.Setenv("WH_CACHE_REDIS_TLS_SERVER_NAME", "r.internal") + loaded, err := config.Load(filepath.Join(t.TempDir(), "none.yaml")) + require.NoError(t, err) + got, err := redisConfig(loaded.Cache.Redis) + require.NoError(t, err) + assert.Equal(t, "u", got.Username) + assert.Equal(t, 2, got.DB) + require.NotNil(t, got.TLS) + assert.Equal(t, "r.internal", got.TLS.ServerName) +} + // keepalive is a config.json patch setting the stream block's keepalive pair. func keepalive(interval, buckets int) map[string]any { return map[string]any{"stream": map[string]any{"keepalive_interval": interval, "keepalive_buckets": buckets, "gap_window_minutes": 15}} From 568f2ddd20a826be68fd7bc8800e2fd2d7524df5 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 08:46:46 -0400 Subject: [PATCH 36/45] docs(cache): version_manager.go's own doc carries the query-key/entry-key mixup too Same overload as architecture.md's version_manager.go bullet: the type doc's "A query key folds all three versions of each dependency" means the rendered entry key QueryKey returns, not the caller's input. Reword to match. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_017aS7rLrH1RKkUMem7X4ckd --- internal/cache/version_manager.go | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/internal/cache/version_manager.go b/internal/cache/version_manager.go index d67a60778..d6fd7496a 100644 --- a/internal/cache/version_manager.go +++ b/internal/cache/version_manager.go @@ -16,9 +16,10 @@ import ( // place, so the index holds one entry per live tenant, table and scope // however often each is bumped, and forgetting a tenant releases all of it. // -// A query key folds all three versions of each dependency, which gives the -// lattice: a table bump orphans every scope, a scope bump that scope and the -// whole-table view, and a tenant bump everything of the tenant's. +// An entry's key (`QueryKey`) folds all three versions of each dependency, +// which gives the lattice: a table bump orphans every scope, a scope bump +// that scope and the whole-table view, and a tenant bump everything of the +// tenant's. type VersionManager struct { mu sync.RWMutex From bf564035e5fca6ba55e6fcb9a4acd2e2d289b325 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 08:53:38 -0400 Subject: [PATCH 37/45] fix(cache): trip on rejected credentials; give the probe a dial budget - A password rotated under a running process refused every new connection's handshake with WRONGPASS, which counted as the server being up: the breaker stayed closed and nothing was logged. WRONGPASS and NOAUTH now open it at once, logged at ERROR. NOPERM still names one key or command and does not. - rueidis redials under the calling operation's context, so a probe bounded by Timeout could never complete a reconnect slower than it. The probe now runs under DialTimeout plus Timeout. It reconnects only the connection it lands on; the client's others still reconnect under the op timeout, and the docs say so (refs #664). - Restoring an RDB/AOF snapshot is a rollback, not a lost token: the docs no longer say a restart can only cause misses, and a test pins the rollback. - The stored-size cap is now tested: a unit count of both size limits, and an incompressible value between them on every server. - Docs: invalidations_total's ok and deferred overlap; the breaker.go and pending.go bullets describe their own files and name the redis.go functions around them; the cluster test covers CROSSSLOT and the topology read, not routing; the timing assertion is stated as written. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_017aS7rLrH1RKkUMem7X4ckd --- CHANGELOG.md | 2 +- docs/src/content/docs/architecture.md | 8 +- internal/cache/breaker.go | 8 +- internal/cache/breaker_test.go | 5 +- internal/cache/metrics.go | 2 +- internal/cache/redis.go | 34 +++- internal/cache/redis_integration_test.go | 199 +++++++++++++++++++---- internal/cache/redis_test.go | 102 ++++++++---- 8 files changed, 279 insertions(+), 81 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 563dd2e76..fde8df608 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **A Redis-compatible shared cache backend, built but not yet selectable** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/cache.go`, `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`, `CONTRIBUTING.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag, with the table and scope escaped into the key (`internal/keyenc`) so no two names share a token; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart can only cause misses — `maxmemory-policy allkeys-lru` is safe. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row, or one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like), open a circuit breaker that skips the server until a probe write succeeds (connections are replaced every minute, so after a failover behind a stable address the process reaches the new primary, and delivers the bumps it owes, within about a minute, provided a fresh connection's dial and handshake fit in the per-operation timeout), and deferred invalidations are retried until they land — the first at once, and at the probe's cadence while the breaker is open — collapsing to one tenant-wide bump per tenant past 100,000 keys; until one lands, the process that owes it bypasses the lookups it would orphan. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. Nothing selects it yet: `cache.backend` accepts only `local` until the wiring (E4) adds `redis` and the `cache.redis.*` settings, so every deployment still runs `LocalCache`. Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, compression, server-stops-answering, refused-writes (a demoted primary, a full `noeviction` server), failover-behind-a-stable-address and owed-invalidation cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. +- **A Redis-compatible shared cache backend, built but not yet selectable** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/cache.go`, `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`, `CONTRIBUTING.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag, with the table and scope escaped into the key (`internal/keyenc`) so no two names share a token; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart without persistence can only cause misses — `maxmemory-policy allkeys-lru` is safe. Restoring an RDB or AOF snapshot, or a backup, is a rollback instead: the old tokens return with their values, so invalidations made since are undone until those entries' TTL. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row, or one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like) or the credentials (`WRONGPASS` or `NOAUTH` after a password rotation, logged at `ERROR`), open a circuit breaker that skips the server until a probe write succeeds (connections are replaced every minute, so after a failover behind a stable address the process reaches the new primary, and delivers the bumps it owes, within about a minute; the probe's budget covers a reconnect slower than the per-operation timeout, but the client's other connections reconnect within that timeout, so the timeout should still exceed a reconnect), and deferred invalidations are retried until they land — the first at once, and at the probe's cadence while the breaker is open — collapsing to one tenant-wide bump per tenant past 100,000 keys; until one lands, the process that owes it bypasses the lookups it would orphan. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. Nothing selects it yet: `cache.backend` accepts only `local` until the wiring (E4) adds `redis` and the `cache.redis.*` settings, so every deployment still runs `LocalCache`. Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, snapshot-rollback, compression, stored-size, server-stops-answering, refused-writes (a demoted primary, a full `noeviction` server), rotated-credentials, slow-reconnect, failover-behind-a-stable-address and owed-invalidation cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. - **One conformance suite for every `mq.Broker`, and a transient broker failure is a `503`** (`internal/mq/mqtest/` (new: the suite and the embedded broker's run of it), `internal/mq/{mq,embedded}.go` (+ tests), `internal/api/ingest.go` (+ tests), `.testcoverage.yml`, `docs/src/content/docs/{api,architecture}.md`, `AGENTS.md`): the first piece of the external-NATS workstream of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `mqtest.Run` states the `Broker` contract as behavior — round trips with names that need encoding, per-tenant order, `Nak` and `AckWait` redelivery, the trace context reaching `Subscribe`, dead-lettering that keeps the topic and leaves the original unacked, per-tenant dead-letter counts, replay bounds and isolation, and exactly one `failed` report when delivery ends underneath a consumer — through the interfaces alone, so the external backend runs the same cases, with `mqtest.Caps` for the four places where its semantics legitimately differ. The embedded broker passes it; writing it turned up that a durable deleted on several tenants' queues could report on `failed` more than once, which is fixed and pinned by a test that deletes it on one queue after another. It also turned up a replay that lost its connection mid-pull passing for a caught-up one when the pull ended in a timeout; that is an error now, as the `Replayer` contract says. The interface comments now allow a delivery unit that is a partition holding several tenants, a `CreateConsumer` that finds a durable rather than creating one, a `PurgeAcked` that leaves retention to the operator, and zero dead-letter counts where there is no per-tenant queue. A new sentinel, `mq.ErrUnavailable`, is a broker that cannot be reached or does not answer in time: the ingest handler answers it with `503` + `Retry-After: 5` rather than the `500` "publish failed" it would have been. Nothing returns it yet; the external backend of #613 will. - **Process roles: the API and the background workers can run in separate processes** (`internal/config/config.go` (+ `roles_test.go`, `defaults_test.go`), `internal/config/backends.go`, `internal/app/{app,wire}.go` (+ `roles_test.go`), `internal/api/router.go` (+ tests), `tests/integration/{setup,tenants}_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md}`, `AGENTS.md`), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The boot config gains `roles` (`WH_ROLES`, default `api,ingest,sweeper`, set in `defaults()` like every boot default, so an explicit `roles: []` refuses boot) and `instance_id` (`WH_INSTANCE_ID`, default `-<8 hex>`, fresh at every boot; logged at boot, and recorded as a lease holder once a shared `coord.backend` exists). A process wires only what its roles need: `api` runs the HTTP API with schema discovery, the token verifiers, the dedupe stores and the SSE hub (all per API process); `ingest` runs the ingest worker; `sweeper` runs the sweeper under its lease. A process without `api` serves an ops-only listener on `server.port` (the probes and their aliases, `/version`, the same-port metrics path, and `POST /v1/ops/settings/reload`, which takes the operator key alone); every other route answers 404, under `/v1/ops` once the operator key has passed. Boot refuses any split over the embedded MQ, which no other process can reach, and `api` without `ingest` (or the reverse) over a local cache, which the ingest worker's invalidations would never reach. Until a shared `mq.backend` exists, every process therefore runs every role, which is the default, so nothing changes for an existing deployment. `data_dir` is probed for Pebble only in a process running `api`. A `config.Config` built without `config.Load` must now name its roles (`config.AllRoles()` for all of them): `app.New` refuses an empty set. - **Leases for work that must run in one process at a time, and the sweeper runs under one** (`internal/coord/` (new: `coord.go`, `local.go`, `elect.go`, `coordtest/`, + tests), `internal/app/{app,wire}.go` (+ tests), `internal/config/backends.go`, `internal/ingest/sweeper.go`, `.github/labeler.yml`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development,ingest-pipeline}.md`, `docs/src/content/docs/configuration.mdx`, `config.yaml`, `AGENTS.md`): part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `coord.Coordinator` hands out named leases (`TryAcquire` → a `Term` with a strictly increasing fencing `Token`, a `Done` channel and `Resign`; `ErrHeld` while another holder's term is live), and `coord.RunElected` runs a loop only while its process holds the lease, resigning when the loop returns and campaigning again every 2s. `coord.Local` is the in-process implementation, and `coordtest.Conformance` is the suite every implementation runs — the NATS KV backend that lets several replicas share one queue comes next. The sweeper now runs through `RunElected` under the `sweeper` lease; with the in-process coordinator the one process always holds it, so nothing changes for a single-process deployment beyond one `coord: elected` log line at startup. `coord.backend` now selects the coordinator (`local`, the only value), so a `config.Config` built without `config.Load` must name it as well as the other three layers' backends. diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 51239a095..519316940 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -115,11 +115,11 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ - **cache.go** — `Cache` interface: `Lookup`, `Set`, `Invalidate`, `InvalidateTenant`, `Close`, plus `QueryTimeToTTL`, which sets a result's TTL from how long its query took (10 s floor, 1 h ceiling). Every entry is one tenant's: `Lookup` takes the tenant, the caller's query key — `:query:`, built by the two cached handlers in `api/` (`queryCacheKey`, with the tenant read off the request's store — `settings.Store.Tenant`), which use it as their [singleflight](https://pkg.go.dev/golang.org/x/sync/singleflight) key too — and the `Namespace`s the result depends on — one for a structured query, none yet for a pipe (a pipe's table dependencies are [#343](https://github.com/Wave-RF/WaveHouse/pull/343)) — each of that tenant (another tenant's is `ErrForeignDependency`) and naming a table and scope by their raw names, which the cache escapes where it builds a key. `Lookup` returns the `Entry` (a nil value is a miss) and a `Snapshot` of the versions it read; on a miss the handler runs the query and passes that snapshot to `Set`, so a result is filed under the versions read *before* its query ran, and a write that lands while it runs orphans the fill rather than re-homing pre-write rows under the post-write versions ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)). The snapshot is taken before any input a bump invalidates is chosen, the tenant's connection included: a reload that moves the tenant to another address or database runs `Pools.Reconcile` and then `InvalidateTenant` (a repoint that keeps both, such as a username or `tls` change, reads the same tables and bumps nothing), so a request that took the old pool files the old database's rows under a version that bump orphans, whether its `Set` lands before the bump or after. The singleflight leader's snapshot is the one used. `Set` returns an error only when the backend failed; a value the cache declines — larger than it keeps, a non-positive TTL, a zero snapshot — is not one. What a hit, a miss and a bump mean is pinned by the conformance suite every backend runs, `internal/testutil/cachetest`. - **local.go** — `LocalCache`, the in-process L1 on [Ristretto](https://github.com/dgraph-io/ristretto): one pool shared by every tenant (a heavier tenant holds more of it), sized by the boot config's `cache.l1_max_cost`. - **version_manager.go** — `VersionManager`, the invalidation index behind `Invalidate` and `InvalidateTenant`: a namespace key is `..
.
.`, its fields joined with `keyenc.Join` so a dot in a table or scope name is escaped rather than read as a separator, and an entry's key, `|.||…` with the caller's query key escaped whole (its `:` become `%3A`), folds in the tenant's version and each dependency's namespace key and namespace version, so bumping a table (a scopeless write) or one scope — scope is reserved and empty today, so every write is the whole-table bump — orphans every dependent entry without touching the pool. The tenant leads every key ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8): the same table under two tenants is two namespaces, so a bump through `Invalidate` under one tenant never touches — and a read under one tenant is never served — the other's results, and the flat directory's single tenant simply carries the `0` prefix. `BumpTenant` (behind `InvalidateTenant`) advances the tenant version that leads every namespace key of one tenant, orphaning its every namespace and every cached query in one step (the tenant version is folded into every query key, so a pipe result with no dependencies is orphaned too) — a table no bump ever keyed included, which an enumeration of the index would miss — for a tenant back on a pool after an absence from the fan-out, or moved to another address or database (story 6). The index is per tenant; the cross-tenant invalidation an insert into a shared table needs is not the index's but the wiring's: `internal/app` hands the ingest worker a cache (`sharedTables`) that repeats each bump under every tenant on the same ClickHouse address and database. -- **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — only `GET`, `SET` and `MGET`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot; the table and scope are escaped with `keyenc` after the fixed prefix, so a `:` in a name is never read as the separator. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, the hash over their escaped forms, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, and a token key holding anything but a token (a string of another length, a list, a hash) is replaced, so a token lost to eviction, expiry or a restart is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. While this process owes a bump on any of a lookup's tokens, that lookup is a bypass that files nothing: the bump would orphan whatever it found (other processes, which cannot know, serve those entries until it lands). `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing, each attempt bounded by `DialTimeout` (1 s), and a cluster client's topology read after the handshake by the larger of it and `Timeout`, which is also how long a connection waits on a silent server before it is redialed. Nothing selects this backend yet: `cache.backend` accepts only `local` until [#613](https://github.com/Wave-RF/WaveHouse/issues/613)'s E4 adds `redis`, the `cache.redis.*` settings and the wiring. +- **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — only `GET`, `SET` and `MGET`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot; the table and scope are escaped with `keyenc` after the fixed prefix, so a `:` in a name is never read as the separator. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, the hash over their escaped forms, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, and a token key holding anything but a token (a string of another length, a list, a hash) is replaced, so a token lost to eviction, expiry, `FLUSHALL` or a restart without persistence is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. Restoring an RDB or AOF snapshot, or a backup, is not a loss but a rollback: the old tokens come back with the values filed under them, so what was invalidated since is served again until its TTL, as after a failover to a replica that missed the bumps. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. While this process owes a bump on any of a lookup's tokens, that lookup is a bypass that files nothing: the bump would orphan whatever it found (other processes, which cannot know, serve those entries until it lands). `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing, each attempt bounded by `DialTimeout` (1 s), and a cluster client's topology read after the handshake by the larger of it and `Timeout`, which is also how long a connection waits on a silent server before it is redialed. Every connection is replaced after a minute (`clientOption`; rueidis retries what was in flight), because one that outlives a failover behind a stable address stays on the demoted node, which answers but refuses writes; the replacement re-resolves the address, so a bypassed process reaches the new primary, and delivers the bumps it owes, within about that long. rueidis dials a replacement lazily, under the context of the operation that lands on it, so a reconnect (dial and handshake, TLS included) slower than `Timeout` fails that operation. The breaker's probe runs under `DialTimeout` plus `Timeout`, so such a reconnect still closes an open breaker; but rueidis spreads commands over several connections (up to four to one server, by `GOMAXPROCS`, and one per cluster node) and the probe reconnects only the one it lands on, so size `Timeout` above a reconnect, or operations that land on the others keep failing. Nothing selects this backend yet: `cache.backend` accepts only `local` until [#613](https://github.com/Wave-RF/WaveHouse/issues/613)'s E4 adds `redis`, the `cache.redis.*` settings and the wiring. - **redis_codec.go** — the key schema (`tokenKeys`, `bumpKeys`, `valueKey`, which take a `Namespace`'s raw names and escape them) and the value frame: format, flags, expiry, the token list, the payload, zstd-compressed from `CompressMinBytes` when that is smaller. A stored value is capped at `MaxValueBytes` and a decoded one at eight times that, which refuses a zip bomb planted in a shared server; an unknown format, as a newer process writes during a rolling upgrade, is a miss. -- **breaker.go** — the circuit breaker: `BreakerThreshold` (5) consecutive failures open it, and while open every operation skips the server at once; after `BreakerOpenFor` (5 s) one background write (`SET :probe`) decides whether it closes, and only that probe's success closes it. Every connection is replaced after a minute (rueidis retries what was in flight), because one that outlives a failover behind a stable address stays on the demoted node, which answers but refuses writes; the replacement re-resolves the address, so a bypassed process reaches the new primary, and delivers the bumps it owes, within about that long. rueidis dials a replacement lazily, under the `Timeout` of the operation that needs it rather than `DialTimeout`, so this holds only where a fresh connection's dial and handshake (TLS included) fit in `Timeout`; size `Timeout` above that, or every reconnect, at the lifetime or after a drop, fails and the next one starts over. A transport failure or a timeout counts against the server; an error reply saying it takes no writes right now — `READONLY` (a demoted primary), `MASTERDOWN`, `OOM` (full under `maxmemory-policy noeviction`, Redis's default, so give the cache an evicting policy), `NOREPLICAS`, `MISCONF`, `LOADING`, `BUSY`, `CLUSTERDOWN` — opens it at once, whatever the threshold, since no bump can land. Any other reply, an error reply about one key (`WRONGTYPE`, `NOPERM`, `TRYAGAIN`) or one the backend cannot use included, counts as a success, and a caller that gave up first counts as nothing. -- **pending.go** — invalidations the server did not take, kept per token key (repeats coalesce) and retried by a background loop until they land: the first one owed at once, then with backoff from 100 ms to 10 s, and while the breaker is open at each probe, which the loop starts when due, so a process that makes no lookups (ingest only) recovers as soon as one that does. `Invalidate` sends its bumps 1,000 to a round trip, as the retry does. Landing late is still correct: a fresh token orphans the pre-write entries and any fill made meanwhile. Past `PendingMax` (100,000) keys the set collapses to one tenant bump per affected tenant. `Close` makes one last attempt at the bumps still owed, past the breaker (an open one is why they are owed); what that attempt cannot deliver is lost, and the entries they would orphan are served until their TTL — the same failure as a worker stopping between an insert and its invalidation. -- **metrics.go** — the shared backend's instruments (meter `wavehouse-cache`, `backend="redis"`, no tenant attribute): `wavehouse_cache_lookups_total` by `result` (`hit`, `miss`, `stale` — filed under since-bumped tokens, `bypass` — server skipped, or held by a bump this process owes, `error`), `wavehouse_cache_op_duration_seconds` by `op` (`lookup`, `set`, `invalidate`), `wavehouse_cache_breaker_open` (1 while bypassed, including before the first connection), `wavehouse_cache_invalidations_total` by `result` (`ok`, `deferred`), `wavehouse_cache_invalidations_pending` (bumps owed: entries they would orphan may be served stale meanwhile), `wavehouse_cache_value_bytes` (stored size), `wavehouse_cache_oversize_total` and `wavehouse_cache_set_failures_total` by `reason` (`oom`, `timeout`, `other`). +- **breaker.go** — the circuit breaker's state machine: `BreakerThreshold` (5) consecutive failures open it, `trip` opens it at once, and while it is open every operation skips the server; once `BreakerOpenFor` (5 s) has passed, `allow` hands one caller the probe, and only the probe's success closes it. What feeds it is `redis.go`'s. `record` counts a transport failure or a timeout against the server, and trips the breaker on an error reply that means no bump can land: one saying the server takes no writes right now, as `refusesWork` lists them — `READONLY` (a demoted primary), `MASTERDOWN`, `OOM` (full under `maxmemory-policy noeviction`, Redis's default, so give the cache an evicting policy), `NOREPLICAS`, `MISCONF`, `LOADING`, `BUSY`, `CLUSTERDOWN` — or one refusing the credentials, as `rejectsCredentials` lists them — `WRONGPASS`, `NOAUTH`, what a new connection's handshake meets after a password rotation — which it logs at `ERROR`. Any other reply, an error reply about one key or command (`WRONGTYPE`, `NOPERM`, `TRYAGAIN`) or one the backend cannot use included, counts as a success, and a caller that gave up first counts as nothing. The probe (`probe`) is a write, `SET :probe`, so a server that answers but refuses writes stays bypassed. +- **pending.go** — the invalidations owed (`pendingBumps`): kept per token key, repeats coalescing, and past `PendingMax` (100,000) keys collapsed to one tenant bump per affected tenant; `owesAny` tells `Lookup` which lookups to hold. `redis.go` delivers them: `Invalidate` sends its bumps 1,000 to a round trip and defers those the server does not take, and `drainLoop` retries them through `drain` until they land — the first one owed at once, then with backoff from 100 ms to 10 s, and while the breaker is open at each probe, which the loop starts when due, so a process that makes no lookups (ingest only) recovers as soon as one that does. Landing late is still correct: a fresh token orphans the pre-write entries and any fill made meanwhile. `Close` makes one last attempt at the bumps still owed, past the breaker (an open one is why they are owed); what that attempt cannot deliver is lost, and the entries they would orphan are served until their TTL — the same failure as a worker stopping between an insert and its invalidation. +- **metrics.go** — the shared backend's instruments (meter `wavehouse-cache`, `backend="redis"`, no tenant attribute): `wavehouse_cache_lookups_total` by `result` (`hit`, `miss`, `stale` — filed under since-bumped tokens, `bypass` — server skipped, or held by a bump this process owes, `error`), `wavehouse_cache_op_duration_seconds` by `op` (`lookup`, `set`, `invalidate`), `wavehouse_cache_breaker_open` (1 while bypassed, including before the first connection), `wavehouse_cache_invalidations_total` by `result` (`ok` counts every bump that lands, retried ones included, and `deferred` each time one is put off, a repeat of one already owed included, so the two overlap), `wavehouse_cache_invalidations_pending` (bumps owed: entries they would orphan may be served stale meanwhile), `wavehouse_cache_value_bytes` (stored size), `wavehouse_cache_oversize_total` and `wavehouse_cache_set_failures_total` by `reason` (`oom`, `timeout`, `other`). ### `config/` — Configuration diff --git a/internal/cache/breaker.go b/internal/cache/breaker.go index a33c94dae..6bcb89d4c 100644 --- a/internal/cache/breaker.go +++ b/internal/cache/breaker.go @@ -65,11 +65,15 @@ func (b *breaker) failure() { } // trip opens the breaker at once, for a reply that says the server cannot -// do the work: one is as conclusive as any number. -func (b *breaker) trip() { +// do the work: one is as conclusive as any number. It reports whether the +// breaker was closed or probing, so a caller logs once per opening and once +// per refused probe rather than once per operation in flight. +func (b *breaker) trip() bool { b.mu.Lock() defer b.mu.Unlock() + opened := !b.open || b.probing b.open, b.openedAt, b.probing = true, b.now(), false + return opened } func (b *breaker) isOpen() bool { diff --git a/internal/cache/breaker_test.go b/internal/cache/breaker_test.go index 3b2109d96..ace2f13f1 100644 --- a/internal/cache/breaker_test.go +++ b/internal/cache/breaker_test.go @@ -68,8 +68,9 @@ func TestBreaker_TripAndProbeSchedule(t *testing.T) { _, open := b.untilProbe() assert.False(t, open) - b.trip() + assert.True(t, b.trip(), "it opened") assert.True(t, b.isOpen(), "no threshold for a refusal") + assert.False(t, b.trip(), "already open") b.success() assert.True(t, b.isOpen(), "a success that is not the probe's leaves it open") d, open := b.untilProbe() @@ -85,7 +86,7 @@ func TestBreaker_TripAndProbeSchedule(t *testing.T) { require.True(t, probe) d, _ = b.untilProbe() assert.Equal(t, 5*time.Second, d, "while the probe runs, wait out a whole period") - b.trip() // the probe was refused too + assert.True(t, b.trip(), "the probe was refused too") d, _ = b.untilProbe() assert.Equal(t, 5*time.Second, d) diff --git a/internal/cache/metrics.go b/internal/cache/metrics.go index 34539f13b..c1dc12dc0 100644 --- a/internal/cache/metrics.go +++ b/internal/cache/metrics.go @@ -45,7 +45,7 @@ func newMetrics(backend string, breakerOpen func() bool, pending func() int) (*m metric.WithDescription("Shared-cache round-trip time by op: lookup, set, invalidate"), metric.WithUnit("s"), metric.WithExplicitBucketBoundaries(.0001, .00025, .0005, .001, .0025, .005, .01, .025, .05, .1, .25)) m.invalidation, errs[2] = meter.Int64Counter("wavehouse_cache_invalidations_total", - metric.WithDescription("Version-token bumps by result: ok, or deferred to the pending retry set")) + metric.WithDescription("Version-token bumps by result: ok counts every bump that lands, retried ones included; deferred counts each time one is put off, a repeat of one already owed included. They overlap: wavehouse_cache_invalidations_pending is what is still owed")) m.valueBytes, errs[3] = meter.Int64Histogram("wavehouse_cache_value_bytes", metric.WithDescription("Size of each value written to the shared cache, after compression"), metric.WithUnit("By"), metric.WithExplicitBucketBoundaries(256, 1<<10, 4<<10, 16<<10, 64<<10, 256<<10, 1<<20, 4<<20)) diff --git a/internal/cache/redis.go b/internal/cache/redis.go index 9d6733d4e..b722e446a 100644 --- a/internal/cache/redis.go +++ b/internal/cache/redis.go @@ -187,9 +187,11 @@ func (c RedisConfig) clientOption() rueidis.ClientOption { // Versions are random tokens, one per tenant, per table and per scope, // under the tenant's hash tag; a bump sets a fresh one. A value carries the // tokens it was computed under and is a hit only while they are all still -// current, so a lost token (eviction, expiry, a restart) can only cause -// misses. A lookup is one round trip. The server failing or timing out is -// a miss, a skipped fill and a deferred invalidation — never a failed query. +// current, so a lost token (eviction, expiry, a restart without persistence) +// can only cause misses. Restoring a snapshot is not a loss but a rollback: +// the old tokens return with their values. A lookup is one round trip. The +// server failing or timing out is a miss, a skipped fill and a deferred +// invalidation — never a failed query. type RedisCache struct { cfg RedisConfig opt rueidis.ClientOption @@ -310,9 +312,12 @@ func (r *RedisCache) conn() rueidis.Client { } // probe decides whether an open breaker closes. It writes: a server that -// answers but refuses writes (refusesWork) would take no bump either. +// answers but refuses writes (refusesWork) would take no bump either. rueidis +// redials under the calling operation's context, so the probe's budget is +// DialTimeout for a reconnect plus Timeout for the write: a reconnect slower +// than Timeout fails the operations waiting on it, but not the probe. func (r *RedisCache) probe(c rueidis.Client) { - ctx, cancel := context.WithTimeout(r.ctx, r.cfg.Timeout) + ctx, cancel := context.WithTimeout(r.ctx, r.cfg.DialTimeout+r.cfg.Timeout) defer cancel() r.record(r.ctx, c.Do(ctx, c.B().Set().Key(r.cfg.KeyPrefix+":probe").Value("1").Ex(time.Minute).Build()).Error()) if r.breaker.isOpen() { @@ -337,9 +342,15 @@ func (r *RedisCache) record(parent context.Context, err error) { return } if re, ok := rueidis.IsRedisErr(err); ok { - if refusesWork(re.Error()) { + switch msg := re.Error(); { + case rejectsCredentials(msg): + if r.breaker.trip() { + slog.ErrorContext(parent, "cache: redis rejected the credentials; bypassing the cache until they work", + "addrs", r.cfg.Addrs, "error", msg) + } + case refusesWork(msg): r.breaker.trip() - } else { + default: r.breaker.success() } return @@ -371,6 +382,15 @@ func refusesWork(msg string) bool { return false } +// rejectsCredentials reports whether an error reply refuses this process's +// credentials, as a connection's handshake after a password rotation does: +// every operation meets it, so it refuses all work. NOPERM is not one: it +// names a key or a command, which the rest of the work may not touch. +func rejectsCredentials(msg string) bool { + code, _, _ := strings.Cut(msg, " ") + return code == "WRONGPASS" || code == "NOAUTH" +} + // Lookup reads the tokens deps fold and the entry for sha in one pipelined // round trip. A token that does not exist yet is created, never read as a // value, so its first use is a miss. diff --git a/internal/cache/redis_integration_test.go b/internal/cache/redis_integration_test.go index 3169e9230..d40a2fae7 100644 --- a/internal/cache/redis_integration_test.go +++ b/internal/cache/redis_integration_test.go @@ -5,6 +5,7 @@ package cache_test import ( "bytes" "context" + "crypto/rand" "crypto/tls" "fmt" "io" @@ -12,6 +13,7 @@ import ( "slices" "strconv" "strings" + "sync" "sync/atomic" "testing" "time" @@ -92,11 +94,12 @@ func startDragonfly(t *testing.T) *server { } // startCluster runs a one-node Redis Cluster owning every slot: enough for -// the server to enforce cluster semantics — CROSSSLOT on a multi-key -// command, MOVED routing through the client — which is what the key schema -// must survive. The node announces 127.0.0.1 on a host port bound to the -// same number, so the address the client learns from CLUSTER SLOTS is -// dialable from the test. +// the server to enforce CROSSSLOT on a multi-key command, which is what the +// key schema must survive, and for the cluster client to read the topology. +// It never routes across nodes: a node owning every slot never answers +// MOVED. The node announces 127.0.0.1 on a host port bound to the same +// number, so the address the client learns from CLUSTER SLOTS is dialable +// from the test. func startCluster(t *testing.T) *server { t.Helper() port := freePort(t) @@ -233,6 +236,10 @@ func TestRedis_Conformance(t *testing.T) { t.Parallel() testCompression(t, s) }) + t.Run("an incompressible value over the stored limit is not stored", func(t *testing.T) { + t.Parallel() + testIncompressible(t, s) + }) }) } } @@ -316,6 +323,24 @@ func testCompression(t *testing.T, s *server) { assert.Less(t, stored, int64(len(rows)/10)) } +// The stored-size limit, past the raw one: random bytes do not compress, so +// a value between the two is refused only once it is encoded. +func testIncompressible(t *testing.T, s *server) { + ctx := context.Background() + c := open(t, s, uniquePrefix()) + rows := make([]byte, maxValue+1<<10) + _, _ = rand.Read(rows) + require.Less(t, len(rows), maxValue*cache.DecodedFactor, "within the raw limit") + deps := []cache.Namespace{{Tenant: "acme", Table: "events"}} + _, snap, err := c.Lookup(ctx, "acme", "big", deps) + require.NoError(t, err) + require.NoError(t, c.Set(ctx, snap, rows, time.Minute), "declined, not failed") + e, _, err := c.Lookup(ctx, "acme", "big", deps) + require.NoError(t, err) + assert.Nil(t, e.Value) + assert.Zero(t, valueCount(raw(t, s))(c), "nothing stored") +} + // A token that is lost — evicted, expired, flushed, a restart without // persistence — is recreated fresh, so a value stored under its predecessor // can only miss. A counter recreated at its initial value would serve the @@ -424,8 +449,9 @@ func dockerClient(t *testing.T) *testcontainers.DockerClient { return d } -// A server that stops answering costs a request at most about the op -// timeout, then nothing: the breaker opens and the cache is bypassed. +// A server that stops answering costs a lookup a bounded wait — asserted +// under ten times the op timeout, as the suite runs in parallel under -race — +// then nothing: the breaker opens and the cache is bypassed. // Invalidations made meanwhile are kept and land once it answers again, and // a process that boots while it is down starts bypassed and connects later. func TestRedis_ServerStopsAnswering(t *testing.T) { @@ -460,7 +486,7 @@ func TestRedis_ServerStopsAnswering(t *testing.T) { e, snap, err := a.Lookup(ctx, "acme", "q", deps) require.Error(t, err, "lookup %d", i) assert.Nil(t, e.Value) - assert.Less(t, time.Since(start), 10*timeout, "a lookup costs at most about the timeout") + assert.Less(t, time.Since(start), 10*timeout, "a lookup returns within ten times the timeout") require.NoError(t, a.Set(ctx, snap, []byte("rows"), time.Minute), "the failed lookup's snapshot files nothing") } require.True(t, cache.Bypassed(a), "three timeouts open the breaker") @@ -560,35 +586,63 @@ func (c unanswered) Write(b []byte) (int, error) { return c.Conn.Write(b) } -// forward proxies each connection it accepts to the address target holds -// at that moment, so a switch moves new connections only, as a stable DNS -// name or a proxy does after a failover. It returns the address to dial. -func forward(t *testing.T, target *atomic.Pointer[string]) string { +// proxy forwards each connection it accepts to the address target holds at +// that moment, so a switch moves new connections only, as a stable DNS name +// or a proxy does after a failover. It holds a new connection for delay +// before forwarding it, as a slow dial and handshake would. +type proxy struct { + addr string // to dial + target atomic.Pointer[string] + delay atomic.Int64 // a time.Duration + + mu sync.Mutex + conns []net.Conn +} + +func newProxy(t *testing.T, target string) *proxy { t.Helper() var lc net.ListenConfig ln, err := lc.Listen(context.Background(), "tcp", "127.0.0.1:0") require.NoError(t, err) t.Cleanup(func() { _ = ln.Close() }) + p := &proxy{addr: ln.Addr().String()} + p.target.Store(&target) go func() { for { c, err := ln.Accept() if err != nil { return } - go func() { - defer func() { _ = c.Close() }() - var d net.Dialer - u, err := d.DialContext(context.Background(), "tcp", *target.Load()) - if err != nil { - return - } - defer func() { _ = u.Close() }() - go func() { _, _ = io.Copy(u, c); _ = u.Close() }() - _, _ = io.Copy(c, u) - }() + p.mu.Lock() + p.conns = append(p.conns, c) + p.mu.Unlock() + go p.serve(c) } }() - return ln.Addr().String() + return p +} + +func (p *proxy) serve(c net.Conn) { + defer func() { _ = c.Close() }() + time.Sleep(time.Duration(p.delay.Load())) + var d net.Dialer + u, err := d.DialContext(context.Background(), "tcp", *p.target.Load()) + if err != nil { + return + } + defer func() { _ = u.Close() }() + go func() { _, _ = io.Copy(u, c); _ = u.Close() }() + _, _ = io.Copy(c, u) +} + +// drop closes every connection p carries, so the client must reconnect. +func (p *proxy) drop() { + p.mu.Lock() + defer p.mu.Unlock() + for _, c := range p.conns { + _ = c.Close() + } + p.conns = nil } // A failover behind a stable address: the connections the process holds @@ -616,9 +670,8 @@ func TestRedis_FailoverBehindAStableAddress(t *testing.T) { return err == nil && strings.Contains(info, "master_link_status:up") }, 30*time.Second, 50*time.Millisecond, "the replica syncs") - var target atomic.Pointer[string] - target.Store(&primary.addr) - stable := &server{addr: forward(t, &target), mode: cache.RedisStandalone} + fwd := newProxy(t, primary.addr) + stable := &server{addr: fwd.addr, mode: cache.RedisStandalone} prefix := uniquePrefix() a := open(t, stable, prefix, func(c *cache.RedisConfig) { c.BreakerThreshold, c.BreakerOpenFor = 1000, 200*time.Millisecond @@ -637,7 +690,7 @@ func TestRedis_FailoverBehindAStableAddress(t *testing.T) { _, err = a.Invalidate(ctx, deps) require.ErrorContains(t, err, "READONLY") require.True(t, cache.Bypassed(a)) - target.Store(&replica.addr) + fwd.target.Store(&replica.addr) deadline := time.Now().Add(15 * time.Second) for cache.Pending(a) > 0 { @@ -814,3 +867,93 @@ func TestRedis_RefusedWrites(t *testing.T) { }) } } + +// Restoring a snapshot — RDB, AOF or a backup — is a rollback, not a lost +// token: the old tokens come back with the values filed under them, so what +// was invalidated since is served again, as after a failover to a replica +// that missed the bumps. The documented exception to "lost tokens miss". +func TestRedis_RestoredSnapshotIsARollback(t *testing.T) { + t.Parallel() + ctx := context.Background() + s := startStandalone(t, redisImage, "redis-server", "--save", "", "--appendonly", "no", "--enable-debug-command", "yes") + r := raw(t, s) + prefix := uniquePrefix() + c := open(t, s, prefix) + deps := []cache.Namespace{{Tenant: "acme", Table: "events"}} + _, snap, err := c.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + require.NoError(t, c.Set(ctx, snap, []byte("pre-write rows"), time.Minute)) + command(t, r, "SAVE") + + _, err = c.Invalidate(ctx, deps) + require.NoError(t, err) + e, _, err := c.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + require.Nil(t, e.Value) + + command(t, r, "DEBUG", "RELOAD", "NOSAVE") + e, _, err = open(t, s, prefix).Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + assert.Equal(t, "pre-write rows", string(e.Value), "the restore brought back the pre-write token") +} + +// A reconnect slower than the op timeout (a dial and TLS handshake across +// zones) fails the operations waiting on it, since rueidis dials under their +// context; they open the breaker, and the probe, whose budget covers a dial, +// completes the reconnect and closes it. Only the connection the probe +// lands on: rueidis spreads commands over several, and the rest still +// reconnect under the op timeout. +func TestRedis_ProbeFitsASlowReconnect(t *testing.T) { + t.Parallel() + ctx := context.Background() + s := startRedis(t) + p := newProxy(t, s.addr) + const timeout = 100 * time.Millisecond + a := open(t, &server{addr: p.addr, mode: cache.RedisStandalone}, uniquePrefix(), func(c *cache.RedisConfig) { + c.Timeout, c.DialTimeout = timeout, 2*time.Second + c.BreakerThreshold, c.BreakerOpenFor = 1, 200*time.Millisecond + }) + deps := []cache.Namespace{{Tenant: "acme", Table: "events"}} + _, _, err := a.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + + p.delay.Store(int64(5 * timeout)) + p.drop() + _, _, err = a.Lookup(ctx, "acme", "q", deps) + require.Error(t, err, "the reconnect does not fit in the lookup's timeout") + require.True(t, cache.Bypassed(a)) + require.Eventually(t, func() bool { return !cache.Bypassed(a) }, 10*time.Second, 10*time.Millisecond, + "the probe reconnects within the dial timeout") +} + +// A password rotated under a running process refuses its next connection's +// handshake, and so every operation: one such reply opens the breaker, +// whatever the threshold, and the probe closes it once the credentials work. +func TestRedis_RejectedCredentialsOpenTheBreaker(t *testing.T) { + t.Parallel() + ctx := context.Background() + s := startRedis(t) + r := raw(t, s) + command(t, r, "ACL", "SETUSER", "rotating", "on", ">old", "+@all", "~*") + a := open(t, s, uniquePrefix(), func(c *cache.RedisConfig) { + c.Username, c.Password = "rotating", "old" + c.BreakerThreshold, c.BreakerOpenFor = 1000, 200*time.Millisecond + }) + deps := []cache.Namespace{{Tenant: "acme", Table: "events"}} + _, _, err := a.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + + command(t, r, "ACL", "SETUSER", "rotating", "resetpass", ">new") + command(t, r, "CLIENT", "KILL", "USER", "rotating") // as the connection lifetime would + require.Eventually(t, func() bool { + _, _, err = a.Lookup(ctx, "acme", "q", deps) + return err != nil && strings.Contains(err.Error(), "WRONGPASS") + }, 5*time.Second, 10*time.Millisecond, "the reconnect is refused") + assert.True(t, cache.Bypassed(a), "one refused handshake opens the breaker") + time.Sleep(time.Second) // several refused probes + assert.True(t, cache.Bypassed(a)) + + command(t, r, "ACL", "SETUSER", "rotating", ">old") + require.Eventually(t, func() bool { return !cache.Bypassed(a) }, 10*time.Second, 10*time.Millisecond, + "the probe closes it once the credentials work") +} diff --git a/internal/cache/redis_test.go b/internal/cache/redis_test.go index a9ccc3711..deb4f2c56 100644 --- a/internal/cache/redis_test.go +++ b/internal/cache/redis_test.go @@ -119,8 +119,11 @@ func TestRedis_UnreachableIsBypassed(t *testing.T) { require.NoError(t, r.Close(), "idempotent") } +// Each size limit is counted: the raw one before compression, the stored one +// after it, which a server-less Set cannot otherwise tell from a bypass. func TestRedis_SetDeclinesWithoutTouchingTheServer(t *testing.T) { - t.Parallel() + // No t.Parallel(): swaps the global meter provider. + reader := meterReader(t) r, err := NewRedis(RedisConfig{Addrs: []string{closedAddr(t)}, DialTimeout: 100 * time.Millisecond, MaxValueBytes: 64}) require.NoError(t, err) t.Cleanup(func() { _ = r.Close() }) @@ -139,6 +142,7 @@ func TestRedis_SetDeclinesWithoutTouchingTheServer(t *testing.T) { } { require.NoError(t, r.Set(ctx, tt.snap, tt.value, tt.ttl), tt.name) } + assert.Equal(t, int64(2), sumOf(t, collect(t, reader), "wavehouse_cache_oversize_total", "", "")) } func TestRedis_Record(t *testing.T) { @@ -156,6 +160,15 @@ func TestRedis_Record(t *testing.T) { assert.True(t, r.breaker.isOpen()) } +func TestRejectsCredentials(t *testing.T) { + t.Parallel() + assert.True(t, rejectsCredentials("WRONGPASS invalid username-password pair or user is disabled.")) + assert.True(t, rejectsCredentials("NOAUTH Authentication required.")) + assert.False(t, rejectsCredentials("NOPERM No permissions to access a key"), "about a key, not the credentials") + assert.False(t, rejectsCredentials("READONLY You can't write against a read only replica.")) + assert.False(t, rejectsCredentials("ERR WRONGPASS")) +} + func TestRefusesWork(t *testing.T) { t.Parallel() for _, msg := range []string{ @@ -224,8 +237,10 @@ func TestReadTokens_NotAnArrayIsMalformed(t *testing.T) { require.ErrorIs(t, err, errMalformedReply) } -func TestRedisMetrics(t *testing.T) { - // No t.Parallel(): swaps the global meter provider. +// meterReader makes a manual reader the global meter provider's for the +// test's duration, which must then not run in parallel. +func meterReader(t *testing.T) *sdkmetric.ManualReader { + t.Helper() saved := otel.GetMeterProvider() reader := sdkmetric.NewManualReader() mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader)) @@ -234,51 +249,66 @@ func TestRedisMetrics(t *testing.T) { _ = mp.Shutdown(context.Background()) otel.SetMeterProvider(saved) }) + return reader +} - r, err := NewRedis(RedisConfig{Addrs: []string{closedAddr(t)}, DialTimeout: 100 * time.Millisecond}) - require.NoError(t, err) - t.Cleanup(func() { _ = r.Close() }) - ctx := context.Background() - _, _, _ = r.Lookup(ctx, "acme", "q", nil) - _, _ = r.Invalidate(ctx, []Namespace{{Tenant: "acme", Table: "events"}}) - r.metrics.op("set", time.Now()) - r.metrics.stored(10) - r.metrics.tooLarge() - r.metrics.setFailed("oom") - +// collect reads every instrument's data from reader, by name. +func collect(t *testing.T, reader *sdkmetric.ManualReader) map[string]metricdata.Aggregation { + t.Helper() var rm metricdata.ResourceMetrics - require.NoError(t, reader.Collect(ctx, &rm)) + require.NoError(t, reader.Collect(context.Background(), &rm)) got := map[string]metricdata.Aggregation{} for _, sm := range rm.ScopeMetrics { for _, m := range sm.Metrics { got[m.Name] = m.Data } } - sumOf := func(name, key, value string) int64 { - t.Helper() - var n int64 - switch d := got[name].(type) { - case metricdata.Sum[int64]: - for _, dp := range d.DataPoints { - if v, ok := dp.Attributes.Value(attribute.Key(key)); key == "" || ok && v.AsString() == value { - n += dp.Value - } - } - case metricdata.Gauge[int64]: - for _, dp := range d.DataPoints { + return got +} + +// sumOf adds up an int64 counter's or gauge's points whose attribute key +// is value; key "" takes every point. +func sumOf(t *testing.T, got map[string]metricdata.Aggregation, name, key, value string) int64 { + t.Helper() + var n int64 + switch d := got[name].(type) { + case metricdata.Sum[int64]: + for _, dp := range d.DataPoints { + if v, ok := dp.Attributes.Value(attribute.Key(key)); key == "" || ok && v.AsString() == value { n += dp.Value } - default: - t.Fatalf("%s: %T", name, got[name]) } - return n + case metricdata.Gauge[int64]: + for _, dp := range d.DataPoints { + n += dp.Value + } + default: + t.Fatalf("%s: %T", name, got[name]) } - assert.Equal(t, int64(1), sumOf("wavehouse_cache_lookups_total", "result", resultBypass)) - assert.Equal(t, int64(1), sumOf("wavehouse_cache_invalidations_total", "result", "deferred")) - assert.Equal(t, int64(1), sumOf("wavehouse_cache_invalidations_pending", "", "")) - assert.Equal(t, int64(1), sumOf("wavehouse_cache_breaker_open", "", "")) - assert.Equal(t, int64(1), sumOf("wavehouse_cache_oversize_total", "", "")) - assert.Equal(t, int64(1), sumOf("wavehouse_cache_set_failures_total", "reason", "oom")) + return n +} + +func TestRedisMetrics(t *testing.T) { + // No t.Parallel(): swaps the global meter provider. + reader := meterReader(t) + r, err := NewRedis(RedisConfig{Addrs: []string{closedAddr(t)}, DialTimeout: 100 * time.Millisecond}) + require.NoError(t, err) + t.Cleanup(func() { _ = r.Close() }) + ctx := context.Background() + _, _, _ = r.Lookup(ctx, "acme", "q", nil) + _, _ = r.Invalidate(ctx, []Namespace{{Tenant: "acme", Table: "events"}}) + r.metrics.op("set", time.Now()) + r.metrics.stored(10) + r.metrics.tooLarge() + r.metrics.setFailed("oom") + + got := collect(t, reader) + assert.Equal(t, int64(1), sumOf(t, got, "wavehouse_cache_lookups_total", "result", resultBypass)) + assert.Equal(t, int64(1), sumOf(t, got, "wavehouse_cache_invalidations_total", "result", "deferred")) + assert.Equal(t, int64(1), sumOf(t, got, "wavehouse_cache_invalidations_pending", "", "")) + assert.Equal(t, int64(1), sumOf(t, got, "wavehouse_cache_breaker_open", "", "")) + assert.Equal(t, int64(1), sumOf(t, got, "wavehouse_cache_oversize_total", "", "")) + assert.Equal(t, int64(1), sumOf(t, got, "wavehouse_cache_set_failures_total", "reason", "oom")) assert.Contains(t, got, "wavehouse_cache_op_duration_seconds") assert.Contains(t, got, "wavehouse_cache_value_bytes") } From c42b6db2603dc8d517441bba981dc6685833c135 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 09:04:44 -0400 Subject: [PATCH 38/45] docs(cache): fix the one query-key/entry-key spot the terminology pass missed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit version_manager.go's tenants field comment still said "the first query key built for it" — the same overload the type doc three lines above and the other doc fixes in this round eliminated (query key = the caller's input, entry key = what QueryKey renders). Both confirmation reviewers caught this independently. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_017aS7rLrH1RKkUMem7X4ckd --- internal/cache/version_manager.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/internal/cache/version_manager.go b/internal/cache/version_manager.go index d6fd7496a..6b1de3b5f 100644 --- a/internal/cache/version_manager.go +++ b/internal/cache/version_manager.go @@ -23,8 +23,8 @@ import ( type VersionManager struct { mu sync.RWMutex - // tenants holds each tenant's index from the first query key built for - // it until the tenant is bumped or pruned. + // tenants holds each tenant's index from the first entry key (`QueryKey`) + // built for it until the tenant is bumped or pruned. tenants map[tenant.ID]*tenantVersions // lastGen is the last generation handed to a tenant; see tenantVersions.gen. From ce0da484740928e07cd721831ed5afd54159c401 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 09:32:52 -0400 Subject: [PATCH 39/45] docs(cache): match configuration.mdx and deployment.md to #626's fixes Verified against the merged internal/cache/{redis,breaker}.go: - rejectsCredentials (WRONGPASS, NOAUTH) now trips the breaker at once at runtime too, logged at ERROR once per opening and once per refused probe (breaker.trip()); NOPERM deliberately does not (it names one key/command, not every operation, so an ACL denial on one tenant's traffic shouldn't bypass the cache for all of them). configuration.mdx's circuit-breaker paragraph didn't mention this at all; added it, distinct from the dial-time isAuthError logging (a separate function, used only in dial()/dialLoop(), where WRONGPASS, NOAUTH and NOPERM are all ERROR since any of them blocks the connection outright before a client exists to send a scoped command on). - deployment.md's failover bullet gets the probe's DialTimeout+Timeout budget and the caveat that every other connection still redials under Timeout alone, matching architecture.md's redis.go bullet. - deployment.md's metrics list presented invalidations_total's `ok`/ `deferred` as if they partition the count; they overlap (a retried landing counts `ok` again), per metrics.go's own doc comment. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_017aS7rLrH1RKkUMem7X4ckd --- docs/src/content/docs/configuration.mdx | 2 +- docs/src/content/docs/deployment.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/src/content/docs/configuration.mdx b/docs/src/content/docs/configuration.mdx index de970487c..c8878afee 100644 --- a/docs/src/content/docs/configuration.mdx +++ b/docs/src/content/docs/configuration.mdx @@ -168,7 +168,7 @@ The `redis` backend's settings, read only when `cache.backend` is `redis`. It is | `cache.redis.compress_min_bytes` | `WH_CACHE_REDIS_COMPRESS_MIN_BYTES` | `1024` | Results at least this large are zstd-compressed when that makes them smaller. `0` never compresses. | | `cache.redis.version_ttl` | `WH_CACHE_REDIS_VERSION_TTL` | `168h` | How long a table's or tenant's version token outlives its last write, so the tokens of dropped tables and removed tenants eventually expire. At least `2s`. An expired token only causes misses. | -**When the server is unreachable or misbehaves, the cache is bypassed; queries are not.** A failure or a timeout makes the lookup a miss and the fill a no-op. Five in a row, or one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like), open a circuit breaker that skips the server entirely until a probe write, every 5 s, succeeds. Queries then go straight to ClickHouse, still coalesced per instance by `singleflight`. An invalidation the server did not take is kept and retried until it lands, and until then the instance that owes it bypasses the lookups it would orphan. `/readyz` does not depend on the cache. +**When the server is unreachable or misbehaves, the cache is bypassed; queries are not.** A failure or a timeout makes the lookup a miss and the fill a no-op. Five in a row, one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like), or one refusing the credentials (`WRONGPASS`, `NOAUTH` — what an already-open connection meets once the password is rotated; logged at `ERROR`, once per opening and once per refused probe) open a circuit breaker that skips the server entirely until a probe write, every 5 s, succeeds. `NOPERM` does not open it: it names one key or command an ACL user cannot use, not every operation, so it should not bypass the cache for every tenant. Queries then go straight to ClickHouse, still coalesced per instance by `singleflight`. An invalidation the server did not take is kept and retried until it lands, and until then the instance that owes it bypasses the lookups it would orphan. `/readyz` does not depend on the cache. **Boot does not wait for the server.** A malformed block (an address without a port, `mode: cluster` with `db` other than `0`, an unreadable or unparsable TLS file) refuses boot. A server that cannot be reached, or that refuses the credentials, does not: the process boots with the cache bypassed and keeps reconnecting, with backoff up to 30 s. A rejected credential (`WRONGPASS`, `NOAUTH`, or `NOPERM` for an ACL user missing a connection command) is logged at `ERROR` on every attempt; any other failure at `WARN`. This is deliberate: a rotated Redis password must not crash-loop every instance at once. Watch `wavehouse_cache_breaker_open`, which reads `1` while the cache is bypassed, including before the first connection. diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index fcbcc2523..b3fdcf801 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -432,7 +432,7 @@ The query-result cache is the layer that can be shared today. With the default ` **What another instance can see.** Ingest is already asynchronous: `/v1/ingest` answers before the batch is inserted. Once the inserting instance's worker has written the batch to ClickHouse, it replaces the table's version token in Redis, and from then on a lookup on any instance misses and reads the new rows. The cache adds no delay of its own beyond that single write. The exceptions: - **The server is unreachable from the inserting instance.** The invalidation is kept and retried until it lands (`wavehouse_cache_invalidations_pending` counts what is owed). Meanwhile other instances that can still reach the server keep serving the older results, for as long as the outage lasts and at most until each entry's TTL. An instance that stops while invalidations are still owed loses them, with the same bound. The same thing happens today when a process stops between an insert and its invalidation. -- **A failover to a replica that had not yet received the latest token writes** can bring back entries filed under the older tokens, bounded by the replication lag at the moment of failover and those entries' TTL. WaveHouse never reads from replicas. Behind a stable address (a managed primary endpoint), an instance still connected to the demoted node has its writes refused, which bypasses its cache; connections are replaced every minute, so it reaches the new primary and delivers the invalidations it owes within about that long. +- **A failover to a replica that had not yet received the latest token writes** can bring back entries filed under the older tokens, bounded by the replication lag at the moment of failover and those entries' TTL. WaveHouse never reads from replicas. Behind a stable address (a managed primary endpoint), an instance still connected to the demoted node has its writes refused, which bypasses its cache; connections are replaced every minute, so it reaches the new primary and delivers the invalidations it owes within about that long. The breaker's own probe gets a longer reconnect budget (`dial_timeout` plus `timeout`) so a slow reconnect still closes it, but every other connection redials under `timeout` alone: size it above how long a reconnect actually takes, or operations that land on one of those keep failing after the probe has already succeeded. - **The server is full and `maxmemory-policy` is `noeviction`.** It refuses the token writes. The inserting instance keeps its invalidations and retries them, bypassing its cache meanwhile, but every other instance serves the results from before the insert until one lands, up to their TTL. - **A pipe that writes** (an `INSERT` in `pipes.json`) has its result cached like a read, so a repeated identical call is answered from the cache and the write does not run again ([#386](https://github.com/Wave-RF/WaveHouse/issues/386)). With a shared cache that holds on every instance, until the entry's TTL. - **Admin writes through `POST /v1/ops/query`** do not invalidate the cache ([#394](https://github.com/Wave-RF/WaveHouse/issues/394)). With a shared cache, the stale results they leave are served by every instance, not only one. @@ -443,7 +443,7 @@ The query-result cache is the layer that can be shared today. With the default ` **Coalescing stays per instance.** `singleflight` collapses identical concurrent queries within each instance, so a cold hot query costs at most one ClickHouse query per instance, not one per request. -**Metrics** (meter `wavehouse-cache`, every series labeled `backend="redis"`, no tenant label): `wavehouse_cache_lookups_total{result}` (`hit`, `miss`, `stale`, `bypass`, `error`), `wavehouse_cache_op_duration_seconds{op}` (`lookup`, `set`, `invalidate`), `wavehouse_cache_breaker_open` (1 while the cache is bypassed), `wavehouse_cache_invalidations_total{result}` (`ok`, `deferred`), `wavehouse_cache_invalidations_pending`, `wavehouse_cache_value_bytes`, `wavehouse_cache_oversize_total` and `wavehouse_cache_set_failures_total{reason}` (`oom`, `timeout`, `other`). Two signals are worth alerting on: `wavehouse_cache_breaker_open` at 1, or `wavehouse_cache_invalidations_pending` above 0, for more than a few minutes. +**Metrics** (meter `wavehouse-cache`, every series labeled `backend="redis"`, no tenant label): `wavehouse_cache_lookups_total{result}` (`hit`, `miss`, `stale`, `bypass`, `error`), `wavehouse_cache_op_duration_seconds{op}` (`lookup`, `set`, `invalidate`), `wavehouse_cache_breaker_open` (1 while the cache is bypassed), `wavehouse_cache_invalidations_total{result}` (`ok` counts every bump that lands, retried ones included, and `deferred` each time one is put off, a repeat of one already owed included — the two overlap, not a split), `wavehouse_cache_invalidations_pending`, `wavehouse_cache_value_bytes`, `wavehouse_cache_oversize_total` and `wavehouse_cache_set_failures_total{reason}` (`oom`, `timeout`, `other`). Two signals are worth alerting on: `wavehouse_cache_breaker_open` at 1, or `wavehouse_cache_invalidations_pending` above 0, for more than a few minutes. For local development, `docker compose -f deployments/compose/dependencies.yaml --profile redis up -d` starts a Redis on `localhost:6379` with no persistence. From 0eea1cbc69558819aeec2ff9c9eed95dd222eb16 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 12:52:53 -0400 Subject: [PATCH 40/45] fix(cache): close the breaker only on a probe write within Timeout The probe's reconnect allowance also covered its write, so a server answering slower than Timeout, but within the allowance, passed every probe: the breaker closed, the next operations timed out on the server, and it reopened, every cycle. A probe write slower than Timeout is now repeated under Timeout alone, and the repeat decides. rueidis bounds a reconnect's dial, TLS included, by DialTimeout and then its handshake by DialTimeout again, so the allowance is now twice DialTimeout. The slow-reconnect test now reconnects over TLS, with the dial and the handshake each taking most of DialTimeout, and a new test keeps a server that answers slower than Timeout bypassed. Docs: a restart after a crash that reloads the last save is a rollback, like restoring a snapshot; what the deferred invalidation count counts; the client's other connections must reconnect within Timeout. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_017aS7rLrH1RKkUMem7X4ckd --- CHANGELOG.md | 2 +- docs/src/content/docs/architecture.md | 4 +- internal/cache/metrics.go | 2 +- internal/cache/redis.go | 38 ++++--- internal/cache/redis_integration_test.go | 122 +++++++++++++++++++---- 5 files changed, 134 insertions(+), 34 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 366b49152..a75b24028 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **A Redis-compatible shared cache backend, built but not yet selectable** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/cache.go`, `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`, `CONTRIBUTING.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag, with the table and scope escaped into the key (`internal/keyenc`) so no two names share a token; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart without persistence can only cause misses — `maxmemory-policy allkeys-lru` is safe. Restoring an RDB or AOF snapshot, or a backup, is a rollback instead: the old tokens return with their values, so invalidations made since are undone until those entries' TTL. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row, or one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like) or the credentials (`WRONGPASS` or `NOAUTH` after a password rotation, logged at `ERROR`), open a circuit breaker that skips the server until a probe write succeeds (connections are replaced every minute, so after a failover behind a stable address the process reaches the new primary, and delivers the bumps it owes, within about a minute; the probe's budget covers a reconnect slower than the per-operation timeout, but the client's other connections reconnect within that timeout, so the timeout should still exceed a reconnect), and deferred invalidations are retried until they land — the first at once, and at the probe's cadence while the breaker is open — collapsing to one tenant-wide bump per tenant past 100,000 keys; until one lands, the process that owes it bypasses the lookups it would orphan. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. Nothing selects it yet: `cache.backend` accepts only `local` until the wiring (E4) adds `redis` and the `cache.redis.*` settings, so every deployment still runs `LocalCache`. Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, snapshot-rollback, compression, stored-size, server-stops-answering, refused-writes (a demoted primary, a full `noeviction` server), rotated-credentials, slow-reconnect, failover-behind-a-stable-address and owed-invalidation cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. +- **A Redis-compatible shared cache backend, built but not yet selectable** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/cache.go`, `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`, `CONTRIBUTING.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag, with the table and scope escaped into the key (`internal/keyenc`) so no two names share a token; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart without persistence can only cause misses — `maxmemory-policy allkeys-lru` is safe. Restoring an RDB or AOF snapshot, or a backup, is a rollback instead (a restart after a crash that reloads the server's last save included, which stock Redis and Valkey make by default): the old tokens return with their values, so invalidations made since are undone until those entries' TTL. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row, or one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like) or the credentials (`WRONGPASS` or `NOAUTH` after a password rotation, logged at `ERROR`), open a circuit breaker that skips the server until a probe write succeeds within the per-operation timeout (connections are replaced every minute, so after a failover behind a stable address the process reaches the new primary, and delivers the bumps it owes, within about a minute; the probe also allows up to twice the dial timeout for a reconnect, but the client's other connections must reconnect within the per-operation timeout, so that timeout should still exceed a reconnect), and deferred invalidations are retried until they land — the first at once, and at the probe's cadence while the breaker is open — collapsing to one tenant-wide bump per tenant past 100,000 keys; until one lands, the process that owes it bypasses the lookups it would orphan. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. Nothing selects it yet: `cache.backend` accepts only `local` until the wiring (E4) adds `redis` and the `cache.redis.*` settings, so every deployment still runs `LocalCache`. Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, snapshot-rollback, compression, stored-size, server-stops-answering, refused-writes (a demoted primary, a full `noeviction` server), rotated-credentials, slow-reconnect (over TLS), slow-server, failover-behind-a-stable-address and owed-invalidation cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. - **One conformance suite for every `mq.Broker`, and a transient broker failure is a `503`** (`internal/mq/mqtest/` (new: the suite and the embedded broker's run of it), `internal/mq/{mq,embedded}.go` (+ tests), `internal/api/ingest.go` (+ tests), `.testcoverage.yml`, `docs/src/content/docs/{api,architecture}.md`, `AGENTS.md`): the first piece of the external-NATS workstream of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `mqtest.Run` states the `Broker` contract as behavior — round trips with names that need encoding, per-tenant order, `Nak` and `AckWait` redelivery, the trace context reaching `Subscribe`, dead-lettering that keeps the topic and leaves the original unacked, per-tenant dead-letter counts, replay bounds and isolation, and exactly one `failed` report when delivery ends underneath a consumer — through the interfaces alone, so the external backend runs the same cases, with `mqtest.Caps` for the four places where its semantics legitimately differ. The embedded broker passes it; writing it turned up that a durable deleted on several tenants' queues could report on `failed` more than once, which is fixed and pinned by a test that deletes it on one queue after another. It also turned up a replay that lost its connection mid-pull passing for a caught-up one when the pull ended in a timeout; that is an error now, as the `Replayer` contract says. The interface comments now allow a delivery unit that is a partition holding several tenants, a `CreateConsumer` that finds a durable rather than creating one, a `PurgeAcked` that leaves retention to the operator, and zero dead-letter counts where there is no per-tenant queue. A new sentinel, `mq.ErrUnavailable`, is a broker that cannot be reached or does not answer in time: the ingest handler answers it with `503` + `Retry-After: 5` rather than the `500` "publish failed" it would have been. Nothing returns it yet; the external backend of #613 will. - **Process roles: the API and the background workers can run in separate processes** (`internal/config/config.go` (+ `roles_test.go`, `defaults_test.go`), `internal/config/backends.go`, `internal/app/{app,wire}.go` (+ `roles_test.go`), `internal/api/router.go` (+ tests), `tests/integration/{setup,tenants}_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md}`, `AGENTS.md`), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The boot config gains `roles` (`WH_ROLES`, default `api,ingest,sweeper`, set in `defaults()` like every boot default, so an explicit `roles: []` refuses boot) and `instance_id` (`WH_INSTANCE_ID`, default `-<8 hex>`, fresh at every boot; logged at boot, and recorded as a lease holder once a shared `coord.backend` exists). A process wires only what its roles need: `api` runs the HTTP API with schema discovery, the token verifiers, the dedupe stores and the SSE hub (all per API process); `ingest` runs the ingest worker; `sweeper` runs the sweeper under its lease. A process without `api` serves an ops-only listener on `server.port` (the probes and their aliases, `/version`, the same-port metrics path, and `POST /v1/ops/settings/reload`, which takes the operator key alone); every other route answers 404, under `/v1/ops` once the operator key has passed. Boot refuses any split over the embedded MQ, which no other process can reach, and `api` without `ingest` (or the reverse) over a local cache, which the ingest worker's invalidations would never reach. Until a shared `mq.backend` exists, every process therefore runs every role, which is the default, so nothing changes for an existing deployment. `data_dir` is probed for Pebble only in a process running `api`. A `config.Config` built without `config.Load` must now name its roles (`config.AllRoles()` for all of them): `app.New` refuses an empty set. - **Leases for work that must run in one process at a time, and the sweeper runs under one** (`internal/coord/` (new: `coord.go`, `local.go`, `elect.go`, `coordtest/`, + tests), `internal/app/{app,wire}.go` (+ tests), `internal/config/backends.go`, `internal/ingest/sweeper.go`, `.github/labeler.yml`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development,ingest-pipeline}.md`, `docs/src/content/docs/configuration.mdx`, `config.yaml`, `AGENTS.md`): part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `coord.Coordinator` hands out named leases (`TryAcquire` → a `Term` with a strictly increasing fencing `Token`, a `Done` channel and `Resign`; `ErrHeld` while another holder's term is live), and `coord.RunElected` runs a loop only while its process holds the lease, resigning when the loop returns and campaigning again every 2s. `coord.Local` is the in-process implementation, and `coordtest.Conformance` is the suite every implementation runs — the NATS KV backend that lets several replicas share one queue comes next. The sweeper now runs through `RunElected` under the `sweeper` lease; with the in-process coordinator the one process always holds it, so nothing changes for a single-process deployment beyond one `coord: elected` log line at startup. `coord.backend` now selects the coordinator (`local`, the only value), so a `config.Config` built without `config.Load` must name it as well as the other three layers' backends. diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 96618a477..2347ec727 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -115,11 +115,11 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ - **cache.go** — `Cache` interface: `Lookup`, `Set`, `Invalidate`, `InvalidateTenant`, `Close`, plus `QueryTimeToTTL`, which sets a result's TTL from how long its query took (10 s floor, 1 h ceiling). Every entry is one tenant's: `Lookup` takes the tenant, the caller's query key — `:query:`, built by the two cached handlers in `api/` (`queryCacheKey`, with the tenant read off the request's store — `settings.Store.Tenant`), which use it as their [singleflight](https://pkg.go.dev/golang.org/x/sync/singleflight) key too — and the `Namespace`s the result depends on — one for a structured query, none yet for a pipe (a pipe's table dependencies are [#343](https://github.com/Wave-RF/WaveHouse/pull/343)) — each of that tenant (another tenant's is `ErrForeignDependency`) and naming a table and scope by their raw names, which the cache escapes where it builds a key. `Lookup` returns the `Entry` (a nil value is a miss) and a `Snapshot` of the versions it read; on a miss the handler runs the query and passes that snapshot to `Set`, so a result is filed under the versions read *before* its query ran, and a write that lands while it runs orphans the fill rather than re-homing pre-write rows under the post-write versions ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)). The snapshot is taken before any input a bump invalidates is chosen, the tenant's connection included: a reload that moves the tenant to another address or database runs `Pools.Reconcile` and then `InvalidateTenant` (a repoint that keeps both, such as a username or `tls` change, reads the same tables and bumps nothing), so a request that took the old pool files the old database's rows under a version that bump orphans, whether its `Set` lands before the bump or after. The singleflight leader's snapshot is the one used. `Set` returns an error only when the backend failed; a value the cache declines — larger than it keeps, a non-positive TTL, a zero snapshot — is not one. What a hit, a miss and a bump mean is pinned by the conformance suite every backend runs, `internal/testutil/cachetest`. - **local.go** — `LocalCache`, the in-process L1 on [Ristretto](https://github.com/dgraph-io/ristretto): one pool shared by every tenant (a heavier tenant holds more of it), sized by the boot config's `cache.l1_max_cost`. - **version_manager.go** — `VersionManager`, the invalidation index behind `Invalidate` and `InvalidateTenant`: a namespace key is `..
.
.`, its fields joined with `keyenc.Join` so a dot in a table or scope name is escaped rather than read as a separator, and an entry's key, `|.||…` with the caller's query key escaped whole (its `:` become `%3A`), folds in the tenant's version and each dependency's namespace key and namespace version, so bumping a table (a scopeless write) or one scope — scope is reserved and empty today, so every write is the whole-table bump — orphans every dependent entry without touching the pool. The tenant leads every key ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8): the same table under two tenants is two namespaces, so a bump through `Invalidate` under one tenant never touches — and a read under one tenant is never served — the other's results, and the flat directory's single tenant simply carries the `0` prefix. `BumpTenant` (behind `InvalidateTenant`) advances the tenant version that leads every namespace key of one tenant, orphaning its every namespace and every cached query in one step (the tenant version is folded into every entry's key, so a pipe result with no dependencies is orphaned too) — a table no bump ever keyed included, which an enumeration of the index would miss — for a tenant back on a pool after an absence from the fan-out, or moved to another address or database (story 6). The index is per tenant; the cross-tenant invalidation an insert into a shared table needs is not the index's but the wiring's: `internal/app` hands the ingest worker a cache (`sharedTables`) that repeats each bump under every tenant on the same ClickHouse address and database. -- **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — only `GET`, `SET` and `MGET`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot; the table and scope are escaped with `keyenc` after the fixed prefix, so a `:` in a name is never read as the separator. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, the hash over their escaped forms, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, and a token key holding anything but a token (a string of another length, a list, a hash) is replaced, so a token lost to eviction, expiry, `FLUSHALL` or a restart without persistence is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. Restoring an RDB or AOF snapshot, or a backup, is not a loss but a rollback: the old tokens come back with the values filed under them, so what was invalidated since is served again until its TTL, as after a failover to a replica that missed the bumps. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. While this process owes a bump on any of a lookup's tokens, that lookup is a bypass that files nothing: the bump would orphan whatever it found (other processes, which cannot know, serve those entries until it lands). `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing, each attempt bounded by `DialTimeout` (1 s), and a cluster client's topology read after the handshake by the larger of it and `Timeout`, which is also how long a connection waits on a silent server before it is redialed. Every connection is replaced after a minute (`clientOption`; rueidis retries what was in flight), because one that outlives a failover behind a stable address stays on the demoted node, which answers but refuses writes; the replacement re-resolves the address, so a bypassed process reaches the new primary, and delivers the bumps it owes, within about that long. rueidis dials a replacement lazily, under the context of the operation that lands on it, so a reconnect (dial and handshake, TLS included) slower than `Timeout` fails that operation. The breaker's probe runs under `DialTimeout` plus `Timeout`, so such a reconnect still closes an open breaker; but rueidis spreads commands over several connections (up to four to one server, by `GOMAXPROCS`, and one per cluster node) and the probe reconnects only the one it lands on, so size `Timeout` above a reconnect, or operations that land on the others keep failing. Nothing selects this backend yet: `cache.backend` accepts only `local` until [#613](https://github.com/Wave-RF/WaveHouse/issues/613)'s E4 adds `redis`, the `cache.redis.*` settings and the wiring. +- **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — only `GET`, `SET` and `MGET`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot; the table and scope are escaped with `keyenc` after the fixed prefix, so a `:` in a name is never read as the separator. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, the hash over their escaped forms, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, and a token key holding anything but a token (a string of another length, a list, a hash) is replaced, so a token lost to eviction, expiry, `FLUSHALL` or a restart without persistence is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. Restoring an RDB or AOF snapshot, or a backup, is not a loss but a rollback — a restart after a crash that reloads the server's last save included, which stock Redis and Valkey make by default: the old tokens come back with the values filed under them, so what was invalidated since is served again until its TTL, as after a failover to a replica that missed the bumps. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. While this process owes a bump on any of a lookup's tokens, that lookup is a bypass that files nothing: the bump would orphan whatever it found (other processes, which cannot know, serve those entries until it lands). `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing, each attempt bounded by `DialTimeout` (1 s), and a cluster client's topology read after the handshake by the larger of it and `Timeout`, which is also how long a connection waits on a silent server before it is redialed. Every connection is replaced after a minute (`clientOption`; rueidis retries what was in flight), because one that outlives a failover behind a stable address stays on the demoted node, which answers but refuses writes; the replacement re-resolves the address, so a bypassed process reaches the new primary, and delivers the bumps it owes, within about that long. rueidis dials a replacement lazily, under the context of the operation that lands on it, bounding the dial (TLS included) by `DialTimeout` and then the handshake by `DialTimeout` again, so a reconnect slower than `Timeout` fails that operation. The breaker's probe gives its write twice `DialTimeout` for a reconnect on top of `Timeout`, so such a reconnect still closes an open breaker. The allowance is for a reconnect only: a probe write slower than `Timeout` is repeated under `Timeout`, and the repeat decides, so a server answering slower than `Timeout` stays bypassed. But rueidis spreads commands over several connections (up to four to one server, by `GOMAXPROCS`, and one per cluster node) and the probe reconnects only the one it lands on, so size `Timeout` above a reconnect, or operations that land on the others keep failing. Nothing selects this backend yet: `cache.backend` accepts only `local` until [#613](https://github.com/Wave-RF/WaveHouse/issues/613)'s E4 adds `redis`, the `cache.redis.*` settings and the wiring. - **redis_codec.go** — the key schema (`tokenKeys`, `bumpKeys`, `valueKey`, which take a `Namespace`'s raw names and escape them) and the value frame: format, flags, expiry, the token list, the payload, zstd-compressed from `CompressMinBytes` when that is smaller. A stored value is capped at `MaxValueBytes` and a decoded one at eight times that, which refuses a zip bomb planted in a shared server; an unknown format, as a newer process writes during a rolling upgrade, is a miss. - **breaker.go** — the circuit breaker's state machine: `BreakerThreshold` (5) consecutive failures open it, `trip` opens it at once, and while it is open every operation skips the server; once `BreakerOpenFor` (5 s) has passed, `allow` hands one caller the probe, and only the probe's success closes it. What feeds it is `redis.go`'s. `record` counts a transport failure or a timeout against the server, and trips the breaker on an error reply that means no bump can land: one saying the server takes no writes right now, as `refusesWork` lists them — `READONLY` (a demoted primary), `MASTERDOWN`, `OOM` (full under `maxmemory-policy noeviction`, Redis's default, so give the cache an evicting policy), `NOREPLICAS`, `MISCONF`, `LOADING`, `BUSY`, `CLUSTERDOWN` — or one refusing the credentials, as `rejectsCredentials` lists them — `WRONGPASS`, `NOAUTH`, what a new connection's handshake meets after a password rotation — which it logs at `ERROR`. Any other reply, an error reply about one key or command (`WRONGTYPE`, `NOPERM`, `TRYAGAIN`) or one the backend cannot use included, counts as a success, and a caller that gave up first counts as nothing. The probe (`probe`) is a write, `SET :probe`, so a server that answers but refuses writes stays bypassed. - **pending.go** — the invalidations owed (`pendingBumps`): kept per token key, repeats coalescing, and past `PendingMax` (100,000) keys collapsed to one tenant bump per affected tenant; `owesAny` tells `Lookup` which lookups to hold. `redis.go` delivers them: `Invalidate` sends its bumps 1,000 to a round trip and defers those the server does not take, and `drainLoop` retries them through `drain` until they land — the first one owed at once, then with backoff from 100 ms to 10 s, and while the breaker is open at each probe, which the loop starts when due, so a process that makes no lookups (ingest only) recovers as soon as one that does. Landing late is still correct: a fresh token orphans the pre-write entries and any fill made meanwhile. `Close` makes one last attempt at the bumps still owed, past the breaker (an open one is why they are owed); what that attempt cannot deliver is lost, and the entries they would orphan are served until their TTL — the same failure as a worker stopping between an insert and its invalidation. -- **metrics.go** — the shared backend's instruments (meter `wavehouse-cache`, `backend="redis"`, no tenant attribute): `wavehouse_cache_lookups_total` by `result` (`hit`, `miss`, `stale` — filed under since-bumped tokens, `bypass` — server skipped, or held by a bump this process owes, `error`), `wavehouse_cache_op_duration_seconds` by `op` (`lookup`, `set`, `invalidate`), `wavehouse_cache_breaker_open` (1 while bypassed, including before the first connection), `wavehouse_cache_invalidations_total` by `result` (`ok` counts every bump that lands, retried ones included, and `deferred` each time one is put off, a repeat of one already owed included, so the two overlap), `wavehouse_cache_invalidations_pending` (bumps owed: entries they would orphan may be served stale meanwhile), `wavehouse_cache_value_bytes` (stored size), `wavehouse_cache_oversize_total` and `wavehouse_cache_set_failures_total` by `reason` (`oom`, `timeout`, `other`). +- **metrics.go** — the shared backend's instruments (meter `wavehouse-cache`, `backend="redis"`, no tenant attribute): `wavehouse_cache_lookups_total` by `result` (`hit`, `miss`, `stale` — filed under since-bumped tokens, `bypass` — server skipped, or held by a bump this process owes, `error`), `wavehouse_cache_op_duration_seconds` by `op` (`lookup`, `set`, `invalidate`), `wavehouse_cache_breaker_open` (1 while bypassed, including before the first connection), `wavehouse_cache_invalidations_total` by `result` (`ok` counts every bump that lands, retried ones included, and `deferred` each bump an invalidation could not deliver when made, a repeat of one already owed included; a failed retry is not counted again, so the two overlap), `wavehouse_cache_invalidations_pending` (bumps owed: entries they would orphan may be served stale meanwhile), `wavehouse_cache_value_bytes` (stored size), `wavehouse_cache_oversize_total` and `wavehouse_cache_set_failures_total` by `reason` (`oom`, `timeout`, `other`). - **cachetest** (`internal/testutil/cachetest`) — `Run(t, factory, Options)`, the conformance suite every `Cache` backend runs, each case on a fresh cache from the factory: what a hit, a miss and each kind of bump mean, independent of where entries and versions live. `Options` describes what a backend can do beyond the `Cache` contract, and leaving one unset skips the cases it enables: `MaxValueBytes` opens the oversize case; `NewPair` returns two instances over one shared store, for the cross-instance cases a shared backend must run; `Entries` counts a cache's entries, and without it the zero-snapshot case — no `Lookup` reads the key a zero `Snapshot` would land under, so only a count shows that one stored nothing — is skipped. ### `config/` — Configuration diff --git a/internal/cache/metrics.go b/internal/cache/metrics.go index c1dc12dc0..855843443 100644 --- a/internal/cache/metrics.go +++ b/internal/cache/metrics.go @@ -45,7 +45,7 @@ func newMetrics(backend string, breakerOpen func() bool, pending func() int) (*m metric.WithDescription("Shared-cache round-trip time by op: lookup, set, invalidate"), metric.WithUnit("s"), metric.WithExplicitBucketBoundaries(.0001, .00025, .0005, .001, .0025, .005, .01, .025, .05, .1, .25)) m.invalidation, errs[2] = meter.Int64Counter("wavehouse_cache_invalidations_total", - metric.WithDescription("Version-token bumps by result: ok counts every bump that lands, retried ones included; deferred counts each time one is put off, a repeat of one already owed included. They overlap: wavehouse_cache_invalidations_pending is what is still owed")) + metric.WithDescription("Version-token bumps by result: ok counts every bump that lands, retried ones included; deferred counts each bump an invalidation could not deliver when made, a repeat of one already owed included; a failed retry is not counted again. They overlap: wavehouse_cache_invalidations_pending is what is still owed")) m.valueBytes, errs[3] = meter.Int64Histogram("wavehouse_cache_value_bytes", metric.WithDescription("Size of each value written to the shared cache, after compression"), metric.WithUnit("By"), metric.WithExplicitBucketBoundaries(256, 1<<10, 4<<10, 16<<10, 64<<10, 256<<10, 1<<20, 4<<20)) diff --git a/internal/cache/redis.go b/internal/cache/redis.go index b722e446a..92719d80e 100644 --- a/internal/cache/redis.go +++ b/internal/cache/redis.go @@ -188,10 +188,12 @@ func (c RedisConfig) clientOption() rueidis.ClientOption { // under the tenant's hash tag; a bump sets a fresh one. A value carries the // tokens it was computed under and is a hit only while they are all still // current, so a lost token (eviction, expiry, a restart without persistence) -// can only cause misses. Restoring a snapshot is not a loss but a rollback: -// the old tokens return with their values. A lookup is one round trip. The -// server failing or timing out is a miss, a skipped fill and a deferred -// invalidation — never a failed query. +// can only cause misses. Restoring a snapshot is not a loss but a rollback — +// a restart after a crash that reloads the server's last save included, +// which stock Redis and Valkey make by default: the old tokens return with +// their values. A lookup is one round trip. The server failing or timing +// out is a miss, a skipped fill and a deferred invalidation — never a +// failed query. type RedisCache struct { cfg RedisConfig opt rueidis.ClientOption @@ -312,18 +314,30 @@ func (r *RedisCache) conn() rueidis.Client { } // probe decides whether an open breaker closes. It writes: a server that -// answers but refuses writes (refusesWork) would take no bump either. rueidis -// redials under the calling operation's context, so the probe's budget is -// DialTimeout for a reconnect plus Timeout for the write: a reconnect slower -// than Timeout fails the operations waiting on it, but not the probe. +// answers but refuses writes (refusesWork) would take no bump either. +// rueidis redials under the calling operation's context, bounding the dial +// (TLS included) by DialTimeout and then the handshake by DialTimeout again, +// so the first write has twice DialTimeout for a reconnect on top of +// Timeout: a reconnect slower than Timeout fails the operations waiting on +// it, but not the probe. The allowance is for a reconnect only: a first +// write slower than Timeout is repeated under Timeout, and the repeat +// decides, so a server answering slower than Timeout stays bypassed. func (r *RedisCache) probe(c rueidis.Client) { - ctx, cancel := context.WithTimeout(r.ctx, r.cfg.DialTimeout+r.cfg.Timeout) - defer cancel() - r.record(r.ctx, c.Do(ctx, c.B().Set().Key(r.cfg.KeyPrefix+":probe").Value("1").Ex(time.Minute).Build()).Error()) + set := func(budget time.Duration) error { + ctx, cancel := context.WithTimeout(r.ctx, budget) + defer cancel() + return c.Do(ctx, c.B().Set().Key(r.cfg.KeyPrefix+":probe").Value("1").Ex(time.Minute).Build()).Error() + } + start := time.Now() + err := set(2*r.cfg.DialTimeout + r.cfg.Timeout) + if time.Since(start) > r.cfg.Timeout { + err = set(r.cfg.Timeout) + } + r.record(r.ctx, err) if r.breaker.isOpen() { return } - slog.InfoContext(ctx, "cache: redis reachable again; cache back in use") + slog.InfoContext(r.ctx, "cache: redis reachable again; cache back in use") r.nudge() } diff --git a/internal/cache/redis_integration_test.go b/internal/cache/redis_integration_test.go index d40a2fae7..13ff66557 100644 --- a/internal/cache/redis_integration_test.go +++ b/internal/cache/redis_integration_test.go @@ -5,10 +5,14 @@ package cache_test import ( "bytes" "context" + "crypto/ecdsa" + "crypto/elliptic" "crypto/rand" "crypto/tls" + "crypto/x509" "fmt" "io" + "math/big" "net" "slices" "strconv" @@ -588,24 +592,28 @@ func (c unanswered) Write(b []byte) (int, error) { // proxy forwards each connection it accepts to the address target holds at // that moment, so a switch moves new connections only, as a stable DNS name -// or a proxy does after a failover. It holds a new connection for delay -// before forwarding it, as a slow dial and handshake would. +// or a proxy does after a failover. Given a TLS config it terminates TLS. +// It holds a new connection for delay before its TLS handshake and again +// before forwarding it, as a slow dial and a slow handshake would, and +// holds everything the server sends for replyDelay, as a slow server would. type proxy struct { - addr string // to dial - target atomic.Pointer[string] - delay atomic.Int64 // a time.Duration + addr string // to dial + serverTLS *tls.Config + target atomic.Pointer[string] + delay atomic.Int64 // a time.Duration + replyDelay atomic.Int64 // a time.Duration mu sync.Mutex conns []net.Conn } -func newProxy(t *testing.T, target string) *proxy { +func newProxy(t *testing.T, target string, serverTLS *tls.Config) *proxy { t.Helper() var lc net.ListenConfig ln, err := lc.Listen(context.Background(), "tcp", "127.0.0.1:0") require.NoError(t, err) t.Cleanup(func() { _ = ln.Close() }) - p := &proxy{addr: ln.Addr().String()} + p := &proxy{addr: ln.Addr().String(), serverTLS: serverTLS} p.target.Store(&target) go func() { for { @@ -624,6 +632,14 @@ func newProxy(t *testing.T, target string) *proxy { func (p *proxy) serve(c net.Conn) { defer func() { _ = c.Close() }() + if p.serverTLS != nil { + time.Sleep(time.Duration(p.delay.Load())) + tc := tls.Server(c, p.serverTLS) + if tc.HandshakeContext(context.Background()) != nil { + return + } + c = tc + } time.Sleep(time.Duration(p.delay.Load())) var d net.Dialer u, err := d.DialContext(context.Background(), "tcp", *p.target.Load()) @@ -632,7 +648,42 @@ func (p *proxy) serve(c net.Conn) { } defer func() { _ = u.Close() }() go func() { _, _ = io.Copy(u, c); _ = u.Close() }() - _, _ = io.Copy(c, u) + buf := make([]byte, 32<<10) + for { + n, err := u.Read(buf) + if n > 0 { + time.Sleep(time.Duration(p.replyDelay.Load())) + if _, err := c.Write(buf[:n]); err != nil { + return + } + } + if err != nil { + return + } + } +} + +// selfSigned returns a server TLS config holding a fresh certificate for +// 127.0.0.1, and a client config that trusts it. +func selfSigned(t *testing.T) (srv, cli *tls.Config) { + t.Helper() + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + require.NoError(t, err) + tmpl := &x509.Certificate{ + SerialNumber: big.NewInt(1), + NotBefore: time.Now().Add(-time.Minute), + NotAfter: time.Now().Add(time.Hour), + IPAddresses: []net.IP{net.IPv4(127, 0, 0, 1)}, + ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, + } + der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key) + require.NoError(t, err) + cert, err := x509.ParseCertificate(der) + require.NoError(t, err) + roots := x509.NewCertPool() + roots.AddCert(cert) + return &tls.Config{Certificates: []tls.Certificate{{Certificate: [][]byte{der}, PrivateKey: key}}, MinVersion: tls.VersionTLS12}, + &tls.Config{RootCAs: roots, MinVersion: tls.VersionTLS12} } // drop closes every connection p carries, so the client must reconnect. @@ -670,7 +721,7 @@ func TestRedis_FailoverBehindAStableAddress(t *testing.T) { return err == nil && strings.Contains(info, "master_link_status:up") }, 30*time.Second, 50*time.Millisecond, "the replica syncs") - fwd := newProxy(t, primary.addr) + fwd := newProxy(t, primary.addr, nil) stable := &server{addr: fwd.addr, mode: cache.RedisStandalone} prefix := uniquePrefix() a := open(t, stable, prefix, func(c *cache.RedisConfig) { @@ -899,31 +950,66 @@ func TestRedis_RestoredSnapshotIsARollback(t *testing.T) { // A reconnect slower than the op timeout (a dial and TLS handshake across // zones) fails the operations waiting on it, since rueidis dials under their -// context; they open the breaker, and the probe, whose budget covers a dial, -// completes the reconnect and closes it. Only the connection the probe -// lands on: rueidis spreads commands over several, and the rest still -// reconnect under the op timeout. +// context; they open the breaker, and the probe, whose budget covers a +// reconnect, completes it and closes the breaker. rueidis bounds the dial, +// TLS included, by the dial timeout and then the handshake by it again, so +// here each takes most of it: together they outlast one dial timeout plus +// the op timeout. Only the connection the probe lands on reconnects: +// rueidis spreads commands over several, and the rest still reconnect under +// the op timeout. func TestRedis_ProbeFitsASlowReconnect(t *testing.T) { t.Parallel() ctx := context.Background() s := startRedis(t) - p := newProxy(t, s.addr) - const timeout = 100 * time.Millisecond + srvTLS, cliTLS := selfSigned(t) + p := newProxy(t, s.addr, srvTLS) + const timeout, dialTimeout = 100 * time.Millisecond, time.Second a := open(t, &server{addr: p.addr, mode: cache.RedisStandalone}, uniquePrefix(), func(c *cache.RedisConfig) { - c.Timeout, c.DialTimeout = timeout, 2*time.Second + c.Timeout, c.DialTimeout, c.TLS = timeout, dialTimeout, cliTLS c.BreakerThreshold, c.BreakerOpenFor = 1, 200*time.Millisecond }) deps := []cache.Namespace{{Tenant: "acme", Table: "events"}} _, _, err := a.Lookup(ctx, "acme", "q", deps) require.NoError(t, err) - p.delay.Store(int64(5 * timeout)) + p.delay.Store(int64(dialTimeout * 7 / 10)) p.drop() _, _, err = a.Lookup(ctx, "acme", "q", deps) require.Error(t, err, "the reconnect does not fit in the lookup's timeout") require.True(t, cache.Bypassed(a)) + require.Eventually(t, func() bool { return !cache.Bypassed(a) }, 20*time.Second, 10*time.Millisecond, + "the probe reconnects within twice the dial timeout") +} + +// A server that answers, but slower than the op timeout, fails every +// operation, so the probe's reconnect allowance must not close the breaker +// on it: a probe write slower than the op timeout is repeated under it, and +// only a repeat that lands closes the breaker. +func TestRedis_ProbeKeepsASlowServerBypassed(t *testing.T) { + t.Parallel() + ctx := context.Background() + s := startRedis(t) + p := newProxy(t, s.addr, nil) + const timeout = 100 * time.Millisecond + a := open(t, &server{addr: p.addr, mode: cache.RedisStandalone}, uniquePrefix(), func(c *cache.RedisConfig) { + c.Timeout, c.DialTimeout = timeout, time.Second + c.BreakerThreshold, c.BreakerOpenFor = 1, 200*time.Millisecond + }) + deps := []cache.Namespace{{Tenant: "acme", Table: "events"}} + _, _, err := a.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + + p.replyDelay.Store(int64(3 * timeout)) + _, _, err = a.Lookup(ctx, "acme", "q", deps) + require.Error(t, err, "a reply slower than the timeout fails the lookup") + require.True(t, cache.Bypassed(a)) + // Several probes, each answered within the reconnect allowance. + for end := time.Now().Add(4 * time.Second); time.Now().Before(end); time.Sleep(5 * time.Millisecond) { + require.True(t, cache.Bypassed(a), "a probe answered slower than the op timeout closed the breaker") + } + p.replyDelay.Store(0) require.Eventually(t, func() bool { return !cache.Bypassed(a) }, 10*time.Second, 10*time.Millisecond, - "the probe reconnects within the dial timeout") + "a probe answered in time closes it") } // A password rotated under a running process refuses its next connection's From ecd21c0890b5e84e5e830547a32e1e7a6c98233f Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 13:10:19 -0400 Subject: [PATCH 41/45] docs(cache): match deployment.md to #626's probe-fix (0eea1cbc) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verified against the merged internal/cache/redis.go: - The failover bullet said the probe gets DialTimeout+Timeout; it's now 2×DialTimeout+Timeout for the first write (rueidis bounds the dial and the handshake by DialTimeout in turn), and a write slower than Timeout is repeated under Timeout alone — only the repeat decides whether the breaker closes, so a server that merely answers slowly stays bypassed instead of flapping open and shut every cycle. - Confirmed the probe is unaffected by, and doesn't affect, the boot/ Close 1s-cap arithmetic in configuration.mdx's dial_timeout row and cache_redis.go's maxRedisTimeout comment: probe() runs in an untracked goroutine (no r.wg.Add before `go r.probe(...)`), so Close's r.wg.Wait() never waits on it. No doc change needed there. - "Run it without persistence" didn't say why: stock Redis and Valkey persist by default (periodic RDB save points), so an ordinary crash- restart reloads the last save on its own — the same rollback as restoring a snapshot by hand. Reworded to match #626's CHANGELOG/ architecture.md wording (grepped for other copies; found none). Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_017aS7rLrH1RKkUMem7X4ckd --- docs/src/content/docs/deployment.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index b3fdcf801..0ba84aa5c 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -432,12 +432,12 @@ The query-result cache is the layer that can be shared today. With the default ` **What another instance can see.** Ingest is already asynchronous: `/v1/ingest` answers before the batch is inserted. Once the inserting instance's worker has written the batch to ClickHouse, it replaces the table's version token in Redis, and from then on a lookup on any instance misses and reads the new rows. The cache adds no delay of its own beyond that single write. The exceptions: - **The server is unreachable from the inserting instance.** The invalidation is kept and retried until it lands (`wavehouse_cache_invalidations_pending` counts what is owed). Meanwhile other instances that can still reach the server keep serving the older results, for as long as the outage lasts and at most until each entry's TTL. An instance that stops while invalidations are still owed loses them, with the same bound. The same thing happens today when a process stops between an insert and its invalidation. -- **A failover to a replica that had not yet received the latest token writes** can bring back entries filed under the older tokens, bounded by the replication lag at the moment of failover and those entries' TTL. WaveHouse never reads from replicas. Behind a stable address (a managed primary endpoint), an instance still connected to the demoted node has its writes refused, which bypasses its cache; connections are replaced every minute, so it reaches the new primary and delivers the invalidations it owes within about that long. The breaker's own probe gets a longer reconnect budget (`dial_timeout` plus `timeout`) so a slow reconnect still closes it, but every other connection redials under `timeout` alone: size it above how long a reconnect actually takes, or operations that land on one of those keep failing after the probe has already succeeded. +- **A failover to a replica that had not yet received the latest token writes** can bring back entries filed under the older tokens, bounded by the replication lag at the moment of failover and those entries' TTL. WaveHouse never reads from replicas. Behind a stable address (a managed primary endpoint), an instance still connected to the demoted node has its writes refused, which bypasses its cache; connections are replaced every minute, so it reaches the new primary and delivers the invalidations it owes within about that long. The breaker's own probe write gets a longer budget for a reconnect — twice `dial_timeout` (the client bounds the dial and the handshake by it in turn) plus `timeout` for the write itself — but only closes the breaker when a write actually lands within `timeout`: one slower than that is repeated under `timeout` alone, and the repeat decides, so a server that merely answers slowly stays bypassed instead of flapping open and shut. Every other connection redials under `timeout` alone: size it above how long a reconnect actually takes, or operations that land on one of those keep failing after the probe has already succeeded. - **The server is full and `maxmemory-policy` is `noeviction`.** It refuses the token writes. The inserting instance keeps its invalidations and retries them, bypassing its cache meanwhile, but every other instance serves the results from before the insert until one lands, up to their TTL. - **A pipe that writes** (an `INSERT` in `pipes.json`) has its result cached like a read, so a repeated identical call is answered from the cache and the write does not run again ([#386](https://github.com/Wave-RF/WaveHouse/issues/386)). With a shared cache that holds on every instance, until the entry's TTL. - **Admin writes through `POST /v1/ops/query`** do not invalidate the cache ([#394](https://github.com/Wave-RF/WaveHouse/issues/394)). With a shared cache, the stale results they leave are served by every instance, not only one. -**Sizing the server.** Every key WaveHouse writes has a TTL, and a version token lost to eviction, expiry or `FLUSHALL` can only cause misses, never bring back an entry it had invalidated. So set `maxmemory` and let the server evict: `maxmemory-policy allkeys-lru` (or `allkeys-lfu`, `volatile-lru`, `volatile-lfu`). Under `noeviction`, a full server refuses the writes: each refusal (a fill's is counted by `wavehouse_cache_set_failures_total{reason="oom"}`) bypasses the cache of the instance that got it, and invalidations are kept and retried, so the pre-insert results above stay served by the others: avoid `noeviction`. A stored result is capped at `cache.redis.max_value_bytes` (1 MiB compressed). A tenant's version tokens share one hash tag, so each lookup reads them in one `MGET` in cluster mode as well. The results themselves carry no hash tag and spread across shards. **Run it without persistence** (`save ""` and `appendonly no`): a restart without persistence can only cause misses, the same as any other token loss. With persistence on, a restart is not that — it reloads whatever snapshot or AOF it last wrote, tokens and values it had already invalidated included, so a fresh instance can serve the pre-write rows filed under them as hits until their TTL (up to 1 h) expires. Treat restoring a snapshot as a rollback, not a resume. +**Sizing the server.** Every key WaveHouse writes has a TTL, and a version token lost to eviction, expiry or `FLUSHALL` can only cause misses, never bring back an entry it had invalidated. So set `maxmemory` and let the server evict: `maxmemory-policy allkeys-lru` (or `allkeys-lfu`, `volatile-lru`, `volatile-lfu`). Under `noeviction`, a full server refuses the writes: each refusal (a fill's is counted by `wavehouse_cache_set_failures_total{reason="oom"}`) bypasses the cache of the instance that got it, and invalidations are kept and retried, so the pre-insert results above stay served by the others: avoid `noeviction`. A stored result is capped at `cache.redis.max_value_bytes` (1 MiB compressed). A tenant's version tokens share one hash tag, so each lookup reads them in one `MGET` in cluster mode as well. The results themselves carry no hash tag and spread across shards. **Run it without persistence** (`save ""` and `appendonly no`): stock Redis and Valkey persist by default (periodic RDB save points), so a crash that is followed by a restart reloads the last save on its own — the same rollback as restoring a snapshot by hand, not a loss. Without persistence, a restart can only cause misses, the same as any other token loss. With it on (the default), a restart reloads whatever snapshot or AOF it last wrote, tokens and values it had already invalidated included, so a fresh instance can serve the pre-write rows filed under them as hits until their TTL (up to 1 h) expires. Treat restoring a snapshot, or a crash-restart on a server that still has its defaults, as a rollback, not a resume. **The server is inside the trust boundary.** A cached result is served after the access policy has filtered it, so whoever can write to the server can change what any caller reads. Keep it on a private network, require a password or ACL user (`WH_CACHE_REDIS_PASSWORD`), use TLS across links you do not trust, and share it only with deployments you trust as much as this one. From b3068ab6563fe952ae10ee9acfed732d90cd2470 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 14:30:20 -0400 Subject: [PATCH 42/45] test(cache): pin the failover test to one connection rueidis keeps up to four connections to a standalone server by GOMAXPROCS and dials each on first use, so an operation landing on one first used after the failover reached the new primary with no connection replaced: without ConnLifetime the test still passed unless GOMAXPROCS was 1. A test-only option gives the client one connection, dialed before the failover, so the test fails without the replacement at any GOMAXPROCS. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01FyrXjhR7iDg33paioLQHFq --- internal/cache/export_test.go | 5 +++++ internal/cache/redis.go | 4 ++++ internal/cache/redis_integration_test.go | 1 + 3 files changed, 10 insertions(+) diff --git a/internal/cache/export_test.go b/internal/cache/export_test.go index 70b8544d2..73a791178 100644 --- a/internal/cache/export_test.go +++ b/internal/cache/export_test.go @@ -24,6 +24,11 @@ func Bypassed(r *RedisCache) bool { return r.bypassed() } // replaced, for a failover test. func SetConnLifetime(c *RedisConfig, d time.Duration) { c.connLifetime = d } +// SetOnePipe gives c's client one connection per node. rueidis otherwise +// keeps up to four by GOMAXPROCS, dialing each on first use, so one first +// used after a failover reaches the new primary without being replaced. +func SetOnePipe(c *RedisConfig) { c.onePipe = true } + // KeyPrefix is the prefix every key r writes leads with. func KeyPrefix(r *RedisCache) string { return r.cfg.KeyPrefix } diff --git a/internal/cache/redis.go b/internal/cache/redis.go index 92719d80e..6e567e899 100644 --- a/internal/cache/redis.go +++ b/internal/cache/redis.go @@ -80,6 +80,7 @@ type RedisConfig struct { BreakerOpenFor time.Duration // how long it stays open before a probe connLifetime time.Duration // defaultConnLifetime; tests shorten it + onePipe bool // one connection per node, whatever GOMAXPROCS; tests only } func (c RedisConfig) withDefaults() (RedisConfig, error) { @@ -174,6 +175,9 @@ func (c RedisConfig) clientOption() rueidis.ClientOption { // bypass reaches the new primary, and delivers its owed bumps, within // about this long. opt.ConnLifetime = c.connLifetime + if c.onePipe { + opt.PipelineMultiplex = -1 + } if c.Mode == RedisSentinel { opt.Sentinel = rueidis.SentinelOption{MasterSet: c.SentinelMaster, TLSConfig: c.TLS, Dialer: opt.Dialer} } diff --git a/internal/cache/redis_integration_test.go b/internal/cache/redis_integration_test.go index 13ff66557..f19ea8059 100644 --- a/internal/cache/redis_integration_test.go +++ b/internal/cache/redis_integration_test.go @@ -727,6 +727,7 @@ func TestRedis_FailoverBehindAStableAddress(t *testing.T) { a := open(t, stable, prefix, func(c *cache.RedisConfig) { c.BreakerThreshold, c.BreakerOpenFor = 1000, 200*time.Millisecond cache.SetConnLifetime(c, time.Second) + cache.SetOnePipe(c) // every operation uses the connection dialed before the failover }) deps := []cache.Namespace{{Tenant: "acme", Table: "events"}} _, snap, err := a.Lookup(ctx, "acme", "q", deps) From 822aecb17bac70d56970532e165c1c5da36790b4 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 14:30:34 -0400 Subject: [PATCH 43/45] fix(cache): log the breaker opening once, at WARN A reply refusing writes and a run of unanswered operations opened the breaker silently. Each opening, a failed probe's included, now logs one WARN with the reply or the error; operations failing while it is open log nothing. Also rewords a comment that named an internal milestone. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01FyrXjhR7iDg33paioLQHFq --- CHANGELOG.md | 2 +- internal/cache/breaker.go | 8 +++++-- internal/cache/breaker_test.go | 7 +++--- internal/cache/redis.go | 38 +++++++++++++++++++++----------- internal/cache/redis_test.go | 40 ++++++++++++++++++++++++++++++++++ 5 files changed, 76 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a75b24028..e14bae9c2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **A Redis-compatible shared cache backend, built but not yet selectable** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/cache.go`, `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`, `CONTRIBUTING.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag, with the table and scope escaped into the key (`internal/keyenc`) so no two names share a token; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart without persistence can only cause misses — `maxmemory-policy allkeys-lru` is safe. Restoring an RDB or AOF snapshot, or a backup, is a rollback instead (a restart after a crash that reloads the server's last save included, which stock Redis and Valkey make by default): the old tokens return with their values, so invalidations made since are undone until those entries' TTL. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row, or one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like) or the credentials (`WRONGPASS` or `NOAUTH` after a password rotation, logged at `ERROR`), open a circuit breaker that skips the server until a probe write succeeds within the per-operation timeout (connections are replaced every minute, so after a failover behind a stable address the process reaches the new primary, and delivers the bumps it owes, within about a minute; the probe also allows up to twice the dial timeout for a reconnect, but the client's other connections must reconnect within the per-operation timeout, so that timeout should still exceed a reconnect), and deferred invalidations are retried until they land — the first at once, and at the probe's cadence while the breaker is open — collapsing to one tenant-wide bump per tenant past 100,000 keys; until one lands, the process that owes it bypasses the lookups it would orphan. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. Nothing selects it yet: `cache.backend` accepts only `local` until the wiring (E4) adds `redis` and the `cache.redis.*` settings, so every deployment still runs `LocalCache`. Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, snapshot-rollback, compression, stored-size, server-stops-answering, refused-writes (a demoted primary, a full `noeviction` server), rotated-credentials, slow-reconnect (over TLS), slow-server, failover-behind-a-stable-address and owed-invalidation cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. +- **A Redis-compatible shared cache backend, built but not yet selectable** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/cache.go`, `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`, `CONTRIBUTING.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag, with the table and scope escaped into the key (`internal/keyenc`) so no two names share a token; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart without persistence can only cause misses — `maxmemory-policy allkeys-lru` is safe. Restoring an RDB or AOF snapshot, or a backup, is a rollback instead (a restart after a crash that reloads the server's last save included, which stock Redis and Valkey make by default): the old tokens return with their values, so invalidations made since are undone until those entries' TTL. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row, or one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like) or the credentials (`WRONGPASS` or `NOAUTH` after a password rotation), open a circuit breaker — logged once per opening, at `ERROR` for the credentials and `WARN` otherwise — that skips the server until a probe write succeeds within the per-operation timeout (connections are replaced every minute, so after a failover behind a stable address the process reaches the new primary, and delivers the bumps it owes, within about a minute; the probe also allows up to twice the dial timeout for a reconnect, but the client's other connections must reconnect within the per-operation timeout, so that timeout should still exceed a reconnect), and deferred invalidations are retried until they land — the first at once, and at the probe's cadence while the breaker is open — collapsing to one tenant-wide bump per tenant past 100,000 keys; until one lands, the process that owes it bypasses the lookups it would orphan. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. Nothing selects it yet: `cache.backend` accepts only `local` until the wiring (E4) adds `redis` and the `cache.redis.*` settings, so every deployment still runs `LocalCache`. Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, snapshot-rollback, compression, stored-size, server-stops-answering, refused-writes (a demoted primary, a full `noeviction` server), rotated-credentials, slow-reconnect (over TLS), slow-server, failover-behind-a-stable-address and owed-invalidation cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. - **One conformance suite for every `mq.Broker`, and a transient broker failure is a `503`** (`internal/mq/mqtest/` (new: the suite and the embedded broker's run of it), `internal/mq/{mq,embedded}.go` (+ tests), `internal/api/ingest.go` (+ tests), `.testcoverage.yml`, `docs/src/content/docs/{api,architecture}.md`, `AGENTS.md`): the first piece of the external-NATS workstream of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `mqtest.Run` states the `Broker` contract as behavior — round trips with names that need encoding, per-tenant order, `Nak` and `AckWait` redelivery, the trace context reaching `Subscribe`, dead-lettering that keeps the topic and leaves the original unacked, per-tenant dead-letter counts, replay bounds and isolation, and exactly one `failed` report when delivery ends underneath a consumer — through the interfaces alone, so the external backend runs the same cases, with `mqtest.Caps` for the four places where its semantics legitimately differ. The embedded broker passes it; writing it turned up that a durable deleted on several tenants' queues could report on `failed` more than once, which is fixed and pinned by a test that deletes it on one queue after another. It also turned up a replay that lost its connection mid-pull passing for a caught-up one when the pull ended in a timeout; that is an error now, as the `Replayer` contract says. The interface comments now allow a delivery unit that is a partition holding several tenants, a `CreateConsumer` that finds a durable rather than creating one, a `PurgeAcked` that leaves retention to the operator, and zero dead-letter counts where there is no per-tenant queue. A new sentinel, `mq.ErrUnavailable`, is a broker that cannot be reached or does not answer in time: the ingest handler answers it with `503` + `Retry-After: 5` rather than the `500` "publish failed" it would have been. Nothing returns it yet; the external backend of #613 will. - **Process roles: the API and the background workers can run in separate processes** (`internal/config/config.go` (+ `roles_test.go`, `defaults_test.go`), `internal/config/backends.go`, `internal/app/{app,wire}.go` (+ `roles_test.go`), `internal/api/router.go` (+ tests), `tests/integration/{setup,tenants}_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md}`, `AGENTS.md`), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The boot config gains `roles` (`WH_ROLES`, default `api,ingest,sweeper`, set in `defaults()` like every boot default, so an explicit `roles: []` refuses boot) and `instance_id` (`WH_INSTANCE_ID`, default `-<8 hex>`, fresh at every boot; logged at boot, and recorded as a lease holder once a shared `coord.backend` exists). A process wires only what its roles need: `api` runs the HTTP API with schema discovery, the token verifiers, the dedupe stores and the SSE hub (all per API process); `ingest` runs the ingest worker; `sweeper` runs the sweeper under its lease. A process without `api` serves an ops-only listener on `server.port` (the probes and their aliases, `/version`, the same-port metrics path, and `POST /v1/ops/settings/reload`, which takes the operator key alone); every other route answers 404, under `/v1/ops` once the operator key has passed. Boot refuses any split over the embedded MQ, which no other process can reach, and `api` without `ingest` (or the reverse) over a local cache, which the ingest worker's invalidations would never reach. Until a shared `mq.backend` exists, every process therefore runs every role, which is the default, so nothing changes for an existing deployment. `data_dir` is probed for Pebble only in a process running `api`. A `config.Config` built without `config.Load` must now name its roles (`config.AllRoles()` for all of them): `app.New` refuses an empty set. - **Leases for work that must run in one process at a time, and the sweeper runs under one** (`internal/coord/` (new: `coord.go`, `local.go`, `elect.go`, `coordtest/`, + tests), `internal/app/{app,wire}.go` (+ tests), `internal/config/backends.go`, `internal/ingest/sweeper.go`, `.github/labeler.yml`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development,ingest-pipeline}.md`, `docs/src/content/docs/configuration.mdx`, `config.yaml`, `AGENTS.md`): part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `coord.Coordinator` hands out named leases (`TryAcquire` → a `Term` with a strictly increasing fencing `Token`, a `Done` channel and `Resign`; `ErrHeld` while another holder's term is live), and `coord.RunElected` runs a loop only while its process holds the lease, resigning when the loop returns and campaigning again every 2s. `coord.Local` is the in-process implementation, and `coordtest.Conformance` is the suite every implementation runs — the NATS KV backend that lets several replicas share one queue comes next. The sweeper now runs through `RunElected` under the `sweeper` lease; with the in-process coordinator the one process always holds it, so nothing changes for a single-process deployment beyond one `coord: elected` log line at startup. `coord.backend` now selects the coordinator (`local`, the only value), so a `config.Config` built without `config.Load` must name it as well as the other three layers' backends. diff --git a/internal/cache/breaker.go b/internal/cache/breaker.go index 6bcb89d4c..f3e065a53 100644 --- a/internal/cache/breaker.go +++ b/internal/cache/breaker.go @@ -54,14 +54,18 @@ func (b *breaker) success() { } // failure records a call the server did not answer in time, and opens the -// breaker at the threshold — or at once, for a failed probe. -func (b *breaker) failure() { +// breaker at the threshold — or at once, for a failed probe. It reports +// whether that opened a closed or probing breaker, as trip does. +func (b *breaker) failure() bool { b.mu.Lock() defer b.mu.Unlock() b.failures++ if b.probing || b.failures >= b.threshold { + opened := !b.open || b.probing b.open, b.openedAt, b.probing = true, b.now(), false + return opened } + return false } // trip opens the breaker at once, for a reply that says the server cannot diff --git a/internal/cache/breaker_test.go b/internal/cache/breaker_test.go index ace2f13f1..273a871c0 100644 --- a/internal/cache/breaker_test.go +++ b/internal/cache/breaker_test.go @@ -26,10 +26,11 @@ func TestBreaker(t *testing.T) { b.failure() b.success() // a success resets the run b.failure() - b.failure() + assert.False(t, b.failure()) assert.False(t, b.isOpen(), "two in a row is below the threshold") - b.failure() + assert.True(t, b.failure(), "it opened") assert.True(t, b.isOpen()) + assert.False(t, b.failure(), "already open") ok, probe = allow() assert.False(t, ok, "open: skip the server") @@ -43,7 +44,7 @@ func TestBreaker(t *testing.T) { assert.False(t, ok) assert.False(t, probe, "one probe at a time") - b.failure() // the probe failed: open for another period + assert.True(t, b.failure(), "the probe failed: open for another period") assert.True(t, b.isOpen()) clock.t = clock.t.Add(4 * time.Second) _, probe = allow() diff --git a/internal/cache/redis.go b/internal/cache/redis.go index 6e567e899..1889ee727 100644 --- a/internal/cache/redis.go +++ b/internal/cache/redis.go @@ -160,7 +160,7 @@ func (c RedisConfig) clientOption() rueidis.ClientOption { TLSConfig: c.TLS, Dialer: net.Dialer{Timeout: c.DialTimeout}, ClientName: "wavehouse", - DisableCache: true, // no client-side caching until the near-cache (E5) + DisableCache: true, // no client-side caching until a local near-cache in front of Redis exists ForceSingleClient: c.Mode == RedisStandalone, } // How long a connection waits on a silent server, 10 s unset. A cluster @@ -360,17 +360,7 @@ func (r *RedisCache) record(parent context.Context, err error) { return } if re, ok := rueidis.IsRedisErr(err); ok { - switch msg := re.Error(); { - case rejectsCredentials(msg): - if r.breaker.trip() { - slog.ErrorContext(parent, "cache: redis rejected the credentials; bypassing the cache until they work", - "addrs", r.cfg.Addrs, "error", msg) - } - case refusesWork(msg): - r.breaker.trip() - default: - r.breaker.success() - } + r.recordReply(parent, re.Error()) return } if errors.Is(err, errMalformedReply) { @@ -380,7 +370,29 @@ func (r *RedisCache) record(parent context.Context, err error) { if parent.Err() != nil { return } - r.breaker.failure() + if r.breaker.failure() { + slog.WarnContext(parent, "cache: redis not answering; bypassing the cache", + "addrs", r.cfg.Addrs, "error", err) + } +} + +// recordReply is record for an error reply. The breaker opening is logged +// once, not once per operation in flight. +func (r *RedisCache) recordReply(parent context.Context, msg string) { + switch { + case rejectsCredentials(msg): + if r.breaker.trip() { + slog.ErrorContext(parent, "cache: redis rejected the credentials; bypassing the cache until they work", + "addrs", r.cfg.Addrs, "error", msg) + } + case refusesWork(msg): + if r.breaker.trip() { + slog.WarnContext(parent, "cache: redis refusing writes; bypassing the cache", + "addrs", r.cfg.Addrs, "reply", msg) + } + default: + r.breaker.success() + } } // refusesWork reports whether an error reply says the server takes no diff --git a/internal/cache/redis_test.go b/internal/cache/redis_test.go index deb4f2c56..5f9d5cec4 100644 --- a/internal/cache/redis_test.go +++ b/internal/cache/redis_test.go @@ -4,7 +4,9 @@ import ( "context" "errors" "fmt" + "log/slog" "net" + "strings" "testing" "time" @@ -17,6 +19,7 @@ import ( "go.opentelemetry.io/otel/sdk/metric/metricdata" "github.com/Wave-RF/WaveHouse/internal/tenant" + "github.com/Wave-RF/WaveHouse/internal/testutil/logtest" ) func TestRedisConfig_Validation(t *testing.T) { @@ -160,6 +163,43 @@ func TestRedis_Record(t *testing.T) { assert.True(t, r.breaker.isOpen()) } +// Each opening of the breaker logs one WARN naming why; the operations that +// fail while it is open log nothing. Not parallel: it captures the default +// logger. +func TestRedis_BreakerOpeningLogsOnce(t *testing.T) { + buf := logtest.Capture(t, slog.LevelWarn) + clock := &fakeClock{t: time.Unix(0, 0)} + r := &RedisCache{breaker: newBreaker(2, time.Second, clock.now)} + live := context.Background() + warns := func() int { return strings.Count(buf.String(), `"level":"WARN"`) } + + r.recordReply(live, "READONLY You can't write against a read only replica.") + r.recordReply(live, "READONLY You can't write against a read only replica.") + r.record(live, context.DeadlineExceeded) + r.record(live, context.DeadlineExceeded) + assert.Equal(t, 1, warns(), buf.String()) + assert.Contains(t, buf.String(), `"reply":"READONLY You can't write against a read only replica."`) + + clock.t = clock.t.Add(time.Second) + _, probe := r.breaker.allow() + require.True(t, probe) + r.record(live, nil) + require.False(t, r.breaker.isOpen()) + + r.record(live, context.DeadlineExceeded) + assert.Equal(t, 1, warns(), "under the threshold") + r.record(live, context.DeadlineExceeded) + r.record(live, context.DeadlineExceeded) + assert.Equal(t, 2, warns(), buf.String()) + assert.Contains(t, buf.String(), "not answering") + + clock.t = clock.t.Add(time.Second) + _, probe = r.breaker.allow() + require.True(t, probe) + r.record(live, context.DeadlineExceeded) + assert.Equal(t, 3, warns(), "a failed probe opens it again") +} + func TestRejectsCredentials(t *testing.T) { t.Parallel() assert.True(t, rejectsCredentials("WRONGPASS invalid username-password pair or user is disabled.")) From aad9b1e5b965bd30f2a05618650f2cec1b184aad Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 14:33:21 -0400 Subject: [PATCH 44/45] docs(cache): say every breaker opening logs, a failed probe's included Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01FyrXjhR7iDg33paioLQHFq --- CHANGELOG.md | 2 +- docs/src/content/docs/architecture.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e14bae9c2..5aa2462e9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **A Redis-compatible shared cache backend, built but not yet selectable** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/cache.go`, `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`, `CONTRIBUTING.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag, with the table and scope escaped into the key (`internal/keyenc`) so no two names share a token; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart without persistence can only cause misses — `maxmemory-policy allkeys-lru` is safe. Restoring an RDB or AOF snapshot, or a backup, is a rollback instead (a restart after a crash that reloads the server's last save included, which stock Redis and Valkey make by default): the old tokens return with their values, so invalidations made since are undone until those entries' TTL. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row, or one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like) or the credentials (`WRONGPASS` or `NOAUTH` after a password rotation), open a circuit breaker — logged once per opening, at `ERROR` for the credentials and `WARN` otherwise — that skips the server until a probe write succeeds within the per-operation timeout (connections are replaced every minute, so after a failover behind a stable address the process reaches the new primary, and delivers the bumps it owes, within about a minute; the probe also allows up to twice the dial timeout for a reconnect, but the client's other connections must reconnect within the per-operation timeout, so that timeout should still exceed a reconnect), and deferred invalidations are retried until they land — the first at once, and at the probe's cadence while the breaker is open — collapsing to one tenant-wide bump per tenant past 100,000 keys; until one lands, the process that owes it bypasses the lookups it would orphan. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. Nothing selects it yet: `cache.backend` accepts only `local` until the wiring (E4) adds `redis` and the `cache.redis.*` settings, so every deployment still runs `LocalCache`. Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, snapshot-rollback, compression, stored-size, server-stops-answering, refused-writes (a demoted primary, a full `noeviction` server), rotated-credentials, slow-reconnect (over TLS), slow-server, failover-behind-a-stable-address and owed-invalidation cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. +- **A Redis-compatible shared cache backend, built but not yet selectable** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/cache.go`, `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`, `CONTRIBUTING.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag, with the table and scope escaped into the key (`internal/keyenc`) so no two names share a token; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart without persistence can only cause misses — `maxmemory-policy allkeys-lru` is safe. Restoring an RDB or AOF snapshot, or a backup, is a rollback instead (a restart after a crash that reloads the server's last save included, which stock Redis and Valkey make by default): the old tokens return with their values, so invalidations made since are undone until those entries' TTL. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row, or one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like) or the credentials (`WRONGPASS` or `NOAUTH` after a password rotation), open a circuit breaker — logged once per opening, at `ERROR` for the credentials and `WARN` otherwise, which a failed probe repeats every 5 s while the server stays down — that skips the server until a probe write succeeds within the per-operation timeout (connections are replaced every minute, so after a failover behind a stable address the process reaches the new primary, and delivers the bumps it owes, within about a minute; the probe also allows up to twice the dial timeout for a reconnect, but the client's other connections must reconnect within the per-operation timeout, so that timeout should still exceed a reconnect), and deferred invalidations are retried until they land — the first at once, and at the probe's cadence while the breaker is open — collapsing to one tenant-wide bump per tenant past 100,000 keys; until one lands, the process that owes it bypasses the lookups it would orphan. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. Nothing selects it yet: `cache.backend` accepts only `local` until the wiring (E4) adds `redis` and the `cache.redis.*` settings, so every deployment still runs `LocalCache`. Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, snapshot-rollback, compression, stored-size, server-stops-answering, refused-writes (a demoted primary, a full `noeviction` server), rotated-credentials, slow-reconnect (over TLS), slow-server, failover-behind-a-stable-address and owed-invalidation cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. - **One conformance suite for every `mq.Broker`, and a transient broker failure is a `503`** (`internal/mq/mqtest/` (new: the suite and the embedded broker's run of it), `internal/mq/{mq,embedded}.go` (+ tests), `internal/api/ingest.go` (+ tests), `.testcoverage.yml`, `docs/src/content/docs/{api,architecture}.md`, `AGENTS.md`): the first piece of the external-NATS workstream of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `mqtest.Run` states the `Broker` contract as behavior — round trips with names that need encoding, per-tenant order, `Nak` and `AckWait` redelivery, the trace context reaching `Subscribe`, dead-lettering that keeps the topic and leaves the original unacked, per-tenant dead-letter counts, replay bounds and isolation, and exactly one `failed` report when delivery ends underneath a consumer — through the interfaces alone, so the external backend runs the same cases, with `mqtest.Caps` for the four places where its semantics legitimately differ. The embedded broker passes it; writing it turned up that a durable deleted on several tenants' queues could report on `failed` more than once, which is fixed and pinned by a test that deletes it on one queue after another. It also turned up a replay that lost its connection mid-pull passing for a caught-up one when the pull ended in a timeout; that is an error now, as the `Replayer` contract says. The interface comments now allow a delivery unit that is a partition holding several tenants, a `CreateConsumer` that finds a durable rather than creating one, a `PurgeAcked` that leaves retention to the operator, and zero dead-letter counts where there is no per-tenant queue. A new sentinel, `mq.ErrUnavailable`, is a broker that cannot be reached or does not answer in time: the ingest handler answers it with `503` + `Retry-After: 5` rather than the `500` "publish failed" it would have been. Nothing returns it yet; the external backend of #613 will. - **Process roles: the API and the background workers can run in separate processes** (`internal/config/config.go` (+ `roles_test.go`, `defaults_test.go`), `internal/config/backends.go`, `internal/app/{app,wire}.go` (+ `roles_test.go`), `internal/api/router.go` (+ tests), `tests/integration/{setup,tenants}_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md}`, `AGENTS.md`), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The boot config gains `roles` (`WH_ROLES`, default `api,ingest,sweeper`, set in `defaults()` like every boot default, so an explicit `roles: []` refuses boot) and `instance_id` (`WH_INSTANCE_ID`, default `-<8 hex>`, fresh at every boot; logged at boot, and recorded as a lease holder once a shared `coord.backend` exists). A process wires only what its roles need: `api` runs the HTTP API with schema discovery, the token verifiers, the dedupe stores and the SSE hub (all per API process); `ingest` runs the ingest worker; `sweeper` runs the sweeper under its lease. A process without `api` serves an ops-only listener on `server.port` (the probes and their aliases, `/version`, the same-port metrics path, and `POST /v1/ops/settings/reload`, which takes the operator key alone); every other route answers 404, under `/v1/ops` once the operator key has passed. Boot refuses any split over the embedded MQ, which no other process can reach, and `api` without `ingest` (or the reverse) over a local cache, which the ingest worker's invalidations would never reach. Until a shared `mq.backend` exists, every process therefore runs every role, which is the default, so nothing changes for an existing deployment. `data_dir` is probed for Pebble only in a process running `api`. A `config.Config` built without `config.Load` must now name its roles (`config.AllRoles()` for all of them): `app.New` refuses an empty set. - **Leases for work that must run in one process at a time, and the sweeper runs under one** (`internal/coord/` (new: `coord.go`, `local.go`, `elect.go`, `coordtest/`, + tests), `internal/app/{app,wire}.go` (+ tests), `internal/config/backends.go`, `internal/ingest/sweeper.go`, `.github/labeler.yml`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development,ingest-pipeline}.md`, `docs/src/content/docs/configuration.mdx`, `config.yaml`, `AGENTS.md`): part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `coord.Coordinator` hands out named leases (`TryAcquire` → a `Term` with a strictly increasing fencing `Token`, a `Done` channel and `Resign`; `ErrHeld` while another holder's term is live), and `coord.RunElected` runs a loop only while its process holds the lease, resigning when the loop returns and campaigning again every 2s. `coord.Local` is the in-process implementation, and `coordtest.Conformance` is the suite every implementation runs — the NATS KV backend that lets several replicas share one queue comes next. The sweeper now runs through `RunElected` under the `sweeper` lease; with the in-process coordinator the one process always holds it, so nothing changes for a single-process deployment beyond one `coord: elected` log line at startup. `coord.backend` now selects the coordinator (`local`, the only value), so a `config.Config` built without `config.Load` must name it as well as the other three layers' backends. diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 2347ec727..9a18f8307 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -117,7 +117,7 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ - **version_manager.go** — `VersionManager`, the invalidation index behind `Invalidate` and `InvalidateTenant`: a namespace key is `..
.
.`, its fields joined with `keyenc.Join` so a dot in a table or scope name is escaped rather than read as a separator, and an entry's key, `|.||…` with the caller's query key escaped whole (its `:` become `%3A`), folds in the tenant's version and each dependency's namespace key and namespace version, so bumping a table (a scopeless write) or one scope — scope is reserved and empty today, so every write is the whole-table bump — orphans every dependent entry without touching the pool. The tenant leads every key ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8): the same table under two tenants is two namespaces, so a bump through `Invalidate` under one tenant never touches — and a read under one tenant is never served — the other's results, and the flat directory's single tenant simply carries the `0` prefix. `BumpTenant` (behind `InvalidateTenant`) advances the tenant version that leads every namespace key of one tenant, orphaning its every namespace and every cached query in one step (the tenant version is folded into every entry's key, so a pipe result with no dependencies is orphaned too) — a table no bump ever keyed included, which an enumeration of the index would miss — for a tenant back on a pool after an absence from the fan-out, or moved to another address or database (story 6). The index is per tenant; the cross-tenant invalidation an insert into a shared table needs is not the index's but the wiring's: `internal/app` hands the ingest worker a cache (`sharedTables`) that repeats each bump under every tenant on the same ClickHouse address and database. - **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — only `GET`, `SET` and `MGET`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot; the table and scope are escaped with `keyenc` after the fixed prefix, so a `:` in a name is never read as the separator. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, the hash over their escaped forms, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, and a token key holding anything but a token (a string of another length, a list, a hash) is replaced, so a token lost to eviction, expiry, `FLUSHALL` or a restart without persistence is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. Restoring an RDB or AOF snapshot, or a backup, is not a loss but a rollback — a restart after a crash that reloads the server's last save included, which stock Redis and Valkey make by default: the old tokens come back with the values filed under them, so what was invalidated since is served again until its TTL, as after a failover to a replica that missed the bumps. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. While this process owes a bump on any of a lookup's tokens, that lookup is a bypass that files nothing: the bump would orphan whatever it found (other processes, which cannot know, serve those entries until it lands). `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing, each attempt bounded by `DialTimeout` (1 s), and a cluster client's topology read after the handshake by the larger of it and `Timeout`, which is also how long a connection waits on a silent server before it is redialed. Every connection is replaced after a minute (`clientOption`; rueidis retries what was in flight), because one that outlives a failover behind a stable address stays on the demoted node, which answers but refuses writes; the replacement re-resolves the address, so a bypassed process reaches the new primary, and delivers the bumps it owes, within about that long. rueidis dials a replacement lazily, under the context of the operation that lands on it, bounding the dial (TLS included) by `DialTimeout` and then the handshake by `DialTimeout` again, so a reconnect slower than `Timeout` fails that operation. The breaker's probe gives its write twice `DialTimeout` for a reconnect on top of `Timeout`, so such a reconnect still closes an open breaker. The allowance is for a reconnect only: a probe write slower than `Timeout` is repeated under `Timeout`, and the repeat decides, so a server answering slower than `Timeout` stays bypassed. But rueidis spreads commands over several connections (up to four to one server, by `GOMAXPROCS`, and one per cluster node) and the probe reconnects only the one it lands on, so size `Timeout` above a reconnect, or operations that land on the others keep failing. Nothing selects this backend yet: `cache.backend` accepts only `local` until [#613](https://github.com/Wave-RF/WaveHouse/issues/613)'s E4 adds `redis`, the `cache.redis.*` settings and the wiring. - **redis_codec.go** — the key schema (`tokenKeys`, `bumpKeys`, `valueKey`, which take a `Namespace`'s raw names and escape them) and the value frame: format, flags, expiry, the token list, the payload, zstd-compressed from `CompressMinBytes` when that is smaller. A stored value is capped at `MaxValueBytes` and a decoded one at eight times that, which refuses a zip bomb planted in a shared server; an unknown format, as a newer process writes during a rolling upgrade, is a miss. -- **breaker.go** — the circuit breaker's state machine: `BreakerThreshold` (5) consecutive failures open it, `trip` opens it at once, and while it is open every operation skips the server; once `BreakerOpenFor` (5 s) has passed, `allow` hands one caller the probe, and only the probe's success closes it. What feeds it is `redis.go`'s. `record` counts a transport failure or a timeout against the server, and trips the breaker on an error reply that means no bump can land: one saying the server takes no writes right now, as `refusesWork` lists them — `READONLY` (a demoted primary), `MASTERDOWN`, `OOM` (full under `maxmemory-policy noeviction`, Redis's default, so give the cache an evicting policy), `NOREPLICAS`, `MISCONF`, `LOADING`, `BUSY`, `CLUSTERDOWN` — or one refusing the credentials, as `rejectsCredentials` lists them — `WRONGPASS`, `NOAUTH`, what a new connection's handshake meets after a password rotation — which it logs at `ERROR`. Any other reply, an error reply about one key or command (`WRONGTYPE`, `NOPERM`, `TRYAGAIN`) or one the backend cannot use included, counts as a success, and a caller that gave up first counts as nothing. The probe (`probe`) is a write, `SET :probe`, so a server that answers but refuses writes stays bypassed. +- **breaker.go** — the circuit breaker's state machine: `BreakerThreshold` (5) consecutive failures open it, `trip` opens it at once, and while it is open every operation skips the server; once `BreakerOpenFor` (5 s) has passed, `allow` hands one caller the probe, and only the probe's success closes it. What feeds it is `redis.go`'s. `record` counts a transport failure or a timeout against the server, and trips the breaker on an error reply that means no bump can land: one saying the server takes no writes right now, as `refusesWork` lists them — `READONLY` (a demoted primary), `MASTERDOWN`, `OOM` (full under `maxmemory-policy noeviction`, Redis's default, so give the cache an evicting policy), `NOREPLICAS`, `MISCONF`, `LOADING`, `BUSY`, `CLUSTERDOWN` — or one refusing the credentials, as `rejectsCredentials` lists them — `WRONGPASS`, `NOAUTH`, what a new connection's handshake meets after a password rotation. Each opening is logged once, at `ERROR` for the credentials and `WARN` otherwise, not once per operation in flight; a failed probe opens it afresh, so a server that stays down logs once every `BreakerOpenFor`. Any other reply, an error reply about one key or command (`WRONGTYPE`, `NOPERM`, `TRYAGAIN`) or one the backend cannot use included, counts as a success, and a caller that gave up first counts as nothing. The probe (`probe`) is a write, `SET :probe`, so a server that answers but refuses writes stays bypassed. - **pending.go** — the invalidations owed (`pendingBumps`): kept per token key, repeats coalescing, and past `PendingMax` (100,000) keys collapsed to one tenant bump per affected tenant; `owesAny` tells `Lookup` which lookups to hold. `redis.go` delivers them: `Invalidate` sends its bumps 1,000 to a round trip and defers those the server does not take, and `drainLoop` retries them through `drain` until they land — the first one owed at once, then with backoff from 100 ms to 10 s, and while the breaker is open at each probe, which the loop starts when due, so a process that makes no lookups (ingest only) recovers as soon as one that does. Landing late is still correct: a fresh token orphans the pre-write entries and any fill made meanwhile. `Close` makes one last attempt at the bumps still owed, past the breaker (an open one is why they are owed); what that attempt cannot deliver is lost, and the entries they would orphan are served until their TTL — the same failure as a worker stopping between an insert and its invalidation. - **metrics.go** — the shared backend's instruments (meter `wavehouse-cache`, `backend="redis"`, no tenant attribute): `wavehouse_cache_lookups_total` by `result` (`hit`, `miss`, `stale` — filed under since-bumped tokens, `bypass` — server skipped, or held by a bump this process owes, `error`), `wavehouse_cache_op_duration_seconds` by `op` (`lookup`, `set`, `invalidate`), `wavehouse_cache_breaker_open` (1 while bypassed, including before the first connection), `wavehouse_cache_invalidations_total` by `result` (`ok` counts every bump that lands, retried ones included, and `deferred` each bump an invalidation could not deliver when made, a repeat of one already owed included; a failed retry is not counted again, so the two overlap), `wavehouse_cache_invalidations_pending` (bumps owed: entries they would orphan may be served stale meanwhile), `wavehouse_cache_value_bytes` (stored size), `wavehouse_cache_oversize_total` and `wavehouse_cache_set_failures_total` by `reason` (`oom`, `timeout`, `other`). - **cachetest** (`internal/testutil/cachetest`) — `Run(t, factory, Options)`, the conformance suite every `Cache` backend runs, each case on a fresh cache from the factory: what a hit, a miss and each kind of bump mean, independent of where entries and versions live. `Options` describes what a backend can do beyond the `Cache` contract, and leaving one unset skips the cases it enables: `MaxValueBytes` opens the oversize case; `NewPair` returns two instances over one shared store, for the cross-instance cases a shared backend must run; `Entries` counts a cache's entries, and without it the zero-snapshot case — no `Lookup` reads the key a zero `Snapshot` would land under, so only a count shows that one stored nothing — is skipped. From 0281ca31efe0327b9428b8087f68a4abef2df103 Mon Sep 17 00:00:00 2001 From: Eric Andrechek Date: Sat, 26 Sep 2026 14:36:05 -0400 Subject: [PATCH 45/45] docs(cache): correct the Close bound, deferred count and probe wording Close can wait up to 4s, not 3s: a dial in flight, then the 1s final drain. deferred counts each deferral once, not failed retries. The probe write closes the breaker only within timeout. The breaker's WARN on opening is documented alongside the credentials ERROR. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01FyrXjhR7iDg33paioLQHFq --- docs/src/content/docs/configuration.mdx | 2 +- docs/src/content/docs/deployment.md | 2 +- internal/config/cache_redis.go | 5 +++-- 3 files changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/src/content/docs/configuration.mdx b/docs/src/content/docs/configuration.mdx index c8878afee..dab012d38 100644 --- a/docs/src/content/docs/configuration.mdx +++ b/docs/src/content/docs/configuration.mdx @@ -168,7 +168,7 @@ The `redis` backend's settings, read only when `cache.backend` is `redis`. It is | `cache.redis.compress_min_bytes` | `WH_CACHE_REDIS_COMPRESS_MIN_BYTES` | `1024` | Results at least this large are zstd-compressed when that makes them smaller. `0` never compresses. | | `cache.redis.version_ttl` | `WH_CACHE_REDIS_VERSION_TTL` | `168h` | How long a table's or tenant's version token outlives its last write, so the tokens of dropped tables and removed tenants eventually expire. At least `2s`. An expired token only causes misses. | -**When the server is unreachable or misbehaves, the cache is bypassed; queries are not.** A failure or a timeout makes the lookup a miss and the fill a no-op. Five in a row, one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like), or one refusing the credentials (`WRONGPASS`, `NOAUTH` — what an already-open connection meets once the password is rotated; logged at `ERROR`, once per opening and once per refused probe) open a circuit breaker that skips the server entirely until a probe write, every 5 s, succeeds. `NOPERM` does not open it: it names one key or command an ACL user cannot use, not every operation, so it should not bypass the cache for every tenant. Queries then go straight to ClickHouse, still coalesced per instance by `singleflight`. An invalidation the server did not take is kept and retried until it lands, and until then the instance that owes it bypasses the lookups it would orphan. `/readyz` does not depend on the cache. +**When the server is unreachable or misbehaves, the cache is bypassed; queries are not.** A failure or a timeout makes the lookup a miss and the fill a no-op. Five in a row, one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like), or one refusing the credentials (`WRONGPASS`, `NOAUTH` — what an already-open connection meets once the password is rotated) open a circuit breaker that skips the server entirely until a probe write, every 5 s, succeeds within `timeout`. Each opening is logged once, at `ERROR` for the credentials and `WARN` otherwise; a failed probe opens it again, so a server that stays down logs every 5 s. `NOPERM` does not open it: it names one key or command an ACL user cannot use, not every operation, so it should not bypass the cache for every tenant. Queries then go straight to ClickHouse, still coalesced per instance by `singleflight`. An invalidation the server did not take is kept and retried until it lands, and until then the instance that owes it bypasses the lookups it would orphan. `/readyz` does not depend on the cache. **Boot does not wait for the server.** A malformed block (an address without a port, `mode: cluster` with `db` other than `0`, an unreadable or unparsable TLS file) refuses boot. A server that cannot be reached, or that refuses the credentials, does not: the process boots with the cache bypassed and keeps reconnecting, with backoff up to 30 s. A rejected credential (`WRONGPASS`, `NOAUTH`, or `NOPERM` for an ACL user missing a connection command) is logged at `ERROR` on every attempt; any other failure at `WARN`. This is deliberate: a rotated Redis password must not crash-loop every instance at once. Watch `wavehouse_cache_breaker_open`, which reads `1` while the cache is bypassed, including before the first connection. diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index 0ba84aa5c..1d0c5ff16 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -443,7 +443,7 @@ The query-result cache is the layer that can be shared today. With the default ` **Coalescing stays per instance.** `singleflight` collapses identical concurrent queries within each instance, so a cold hot query costs at most one ClickHouse query per instance, not one per request. -**Metrics** (meter `wavehouse-cache`, every series labeled `backend="redis"`, no tenant label): `wavehouse_cache_lookups_total{result}` (`hit`, `miss`, `stale`, `bypass`, `error`), `wavehouse_cache_op_duration_seconds{op}` (`lookup`, `set`, `invalidate`), `wavehouse_cache_breaker_open` (1 while the cache is bypassed), `wavehouse_cache_invalidations_total{result}` (`ok` counts every bump that lands, retried ones included, and `deferred` each time one is put off, a repeat of one already owed included — the two overlap, not a split), `wavehouse_cache_invalidations_pending`, `wavehouse_cache_value_bytes`, `wavehouse_cache_oversize_total` and `wavehouse_cache_set_failures_total{reason}` (`oom`, `timeout`, `other`). Two signals are worth alerting on: `wavehouse_cache_breaker_open` at 1, or `wavehouse_cache_invalidations_pending` above 0, for more than a few minutes. +**Metrics** (meter `wavehouse-cache`, every series labeled `backend="redis"`, no tenant label): `wavehouse_cache_lookups_total{result}` (`hit`, `miss`, `stale`, `bypass`, `error`), `wavehouse_cache_op_duration_seconds{op}` (`lookup`, `set`, `invalidate`), `wavehouse_cache_breaker_open` (1 while the cache is bypassed), `wavehouse_cache_invalidations_total{result}` (`ok` counts every bump that lands, retried ones included, and `deferred` each bump an invalidation could not deliver when made, a repeat of one already owed included; a failed retry is not counted again — the two overlap, not a split), `wavehouse_cache_invalidations_pending`, `wavehouse_cache_value_bytes`, `wavehouse_cache_oversize_total` and `wavehouse_cache_set_failures_total{reason}` (`oom`, `timeout`, `other`). Two signals are worth alerting on: `wavehouse_cache_breaker_open` at 1, or `wavehouse_cache_invalidations_pending` above 0, for more than a few minutes. For local development, `docker compose -f deployments/compose/dependencies.yaml --profile redis up -d` starts a Redis on `localhost:6379` with no persistence. diff --git a/internal/config/cache_redis.go b/internal/config/cache_redis.go index be065266f..456006670 100644 --- a/internal/config/cache_redis.go +++ b/internal/config/cache_redis.go @@ -23,8 +23,9 @@ const ( // maxRedisTimeout caps cache.redis.timeout and cache.redis.dial_timeout. // Boot and the cache's Close each wait out a dial in flight: a connect and // a handshake, each bounded by dial_timeout, then for a cluster a topology -// read bounded by the larger of the two. At the caps that is at most 3s, -// inside the 5s budget Close shares with the stores released after it. +// read bounded by the larger of the two. At the caps that is at most 3s; +// Close then spends up to 1s delivering owed invalidations, so at most 4s, +// inside the 5s budget it shares with the stores released after it. const maxRedisTimeout = time.Second // CacheRedisConfig configures cache.backend=redis: one Redis-compatible server