diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 575a83893..470f82f8b 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -44,7 +44,7 @@ Break one of these knowingly or not at all. 2. **A dedicated `coverage` job applies the consolidated gate, polling — not `needs`-ing — the suites.** Each suite (`unit`, `integration`, `e2e`) runs with `COV_DEFER=1` and uploads a `coverage-` fragment; the `coverage` job runs `make cov` (merge + every threshold gate) over all three — exactly like local `make ci`'s final step. Keeping it a separate job (not folded into e2e's tail) decouples the gate result from the e2e suite's pass/fail. Crucially it is `needs: changes` **only, not the suites**: a `needs` edge is a *scheduling* barrier — GitHub won't pick up a runner, check out, restore caches, or `pnpm install` until the needed jobs finish — so needing the suites would serialize this job's ~50s of setup onto the critical path after the last suite, for nothing (the setup doesn't depend on their results). Instead it starts at run creation, runs its setup in parallel with the suites, and blocks only at the merge by polling for the three fragments with [`scripts/ci/wait-artifact.sh`](../../scripts/ci/wait-artifact.sh) (fails fast if a producer concluded without producing). Tail on the critical path: ~10s, not ~50s. **The aggregator and `docs-deploy` must keep `coverage` *and* every suite in their `needs`** — the suites directly (a suite failure must red the gate even though `coverage` no longer needs them), and `coverage` (else a coverage-gate failure wouldn't block merge or a prod deploy). -3. **e2e builds its own inputs and mirrors local `make test-e2e`.** It compiles the SDK dist + cover binary itself (`make -j test-e2e`, warm per-suffix cache) rather than waiting on a builder job, and runs the suite exactly as a developer does — one orchestrator, one ClickHouse testcontainer, sequential files. The ClickHouse image pulls in the background while caches restore (also in the integration job). +3. **e2e builds its own inputs and mirrors local `make test-e2e`.** It compiles the SDK dist + cover binary itself (`make -j test-e2e`, warm per-suffix cache) rather than waiting on a builder job, and runs the suite exactly as a developer does — one orchestrator, one ClickHouse and one Redis testcontainer, sequential files. The ClickHouse image pulls in the background while caches restore (also in the integration job). 4. **One change classifier, split into a pure core + a CI wrapper.** The pure allowlist — file list on stdin ⇒ `code`/`docs` — lives in [`scripts/classify-paths.sh`](../../scripts/classify-paths.sh), dependency-free and unit-tested by [`scripts/classify-paths.test.sh`](../../scripts/classify-paths.test.sh) (`make test-classify-paths`, a `verify` leaf) so the allowlists can't silently regress. The `changes` job runs the thin wrapper [`scripts/ci/classify-changes.sh`](../../scripts/ci/classify-changes.sh), which adds the CI-only policy (API file-list fetch + fail-closed: pushes, dispatches, API hiccups ⇒ `code=true`) on top. Keeping the core pure means the local git hooks can share it (`git diff --name-only | scripts/classify-paths.sh`). The `code`/`docs` outputs gate the suites and docs jobs — gate on these, never on workflow-level `paths:` filters, which would orphan the required check (invariant 1). diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f47619274..fa572e414 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -280,8 +280,8 @@ jobs: with: go-cache-suffix: "-e2e-cov" # `-j` builds the prereqs (build-ts ∥ build-cover) concurrently, - # then runs the orchestrator: ClickHouse testcontainer + the cover - # binary + the SDK vitest suite. + # then runs the orchestrator: ClickHouse and Redis testcontainers + + # the cover binary + the SDK vitest suite. - name: Build SDK dist + cover binary, run E2E suite run: make -j "$(nproc)" test-e2e COV_DEFER=1 - name: Upload coverage fragment diff --git a/.testcoverage.yml b/.testcoverage.yml index 99d3b1cd7..451014a32 100644 --- a/.testcoverage.yml +++ b/.testcoverage.yml @@ -79,3 +79,14 @@ exclude: - ^internal/settings/ - ^cmd/wavehouse/validate\.go$ - ^cmd/wavehouse/bootstrap\.go$ + # The in-process cache backend: the e2e stack runs cache.backend=redis + # (#613), so the binary carries LocalCache and its version index but e2e + # never reaches them. The unit suite and the integration suite's main + # app (cache.backend=local) cover them; the merged total still counts them. + - ^internal/cache/(local|version_manager)\.go$ + # What e2e's Redis never makes it run: the cache.redis block's rejection + # paths (boot adopts a valid fixture, as with internal/settings above) + # and the retry of invalidations the server did not take, which needs an + # outage. The unit and integration suites cover both. + - ^internal/config/cache_redis\.go$ + - ^internal/cache/pending\.go$ diff --git a/AGENTS.md b/AGENTS.md index 5a55f6e1b..d60f9705f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -29,17 +29,17 @@ One binary: Twenty internal packages under `internal/` (plus `internal/testutil/` for shared test helpers): - **`api/`** — Chi HTTP router, JWT/JWKS middleware (from `auth/`), ingest/query/structured-query/SSE/schema/DLQ/pipes handlers; `ch_errors.go` (`writeCHError`) is the one mapping from a failed ClickHouse query to status, `code` and `retryable` -- **`app/`** — the process wiring: `New` builds every component from the boot config and the settings directory (each one wired in one place — what it opens, what it loops, what it releases — with the settings registry handed to its wiring function whole, the injection point of the per-tenant registry of #583: store-keyed getters for the handlers, `perTenant` for the async paths (with the tenant each message's `mq.Topic` names for the stream hub and the ingest worker), the `chconn.Pools` and the per-tenant `discoveries` reconciled from `AfterAdopt`, `shortestKeepalive` for the one setting folded over every tenant served, `gapWindows` handing the sweeper each tenant's own gap window (a rejected tenant's as its folder last had it, unbounded for one rejected since boot) and the `mq.max_bytes_gb` reconcile each served tenant's byte budget, and `defaultPolicy` for the one setting that still follows tenant `0`, a flat directory's ops-gate admin role; the auth verifiers are per tenant, reconfigured (rebuilt only on changed wiring) and pruned from `AfterAdopt`, and the same hook's `Hub.Prune` ends the open streams of a tenant no longer served), `Run` drives the long-lived ones under one `errgroup` until the context is cancelled or one fails, `Close` releases them in reverse order. `New` wires only what the process's `roles` need (discovery, dedupe, auth verifiers, the hub bridge and keepalive per API process; the ingest worker per ingest process; the sweeper under its lease through `elected`); a process without `api` serves `api.NewOpsRouter` — probes, `/version`, metrics, and the settings reload behind the operator key alone. `cmd/wavehouse` and `tests/integration` both boot through it +- **`app/`** — the process wiring: `New` builds every component from the boot config and the settings directory (each one wired in one place — what it opens, what it loops, what it releases — with the settings registry handed to its wiring function whole, the injection point of the per-tenant registry of #583: store-keyed getters for the handlers, `perTenant` for the async paths (with the tenant each message's `mq.Topic` names for the stream hub and the ingest worker), the `chconn.Pools` and the per-tenant `discoveries` reconciled from `AfterAdopt`, `shortestKeepalive` for the one setting folded over every tenant served, `gapWindows` handing the sweeper each tenant's own gap window (a rejected tenant's as its folder last had it, unbounded for one rejected since boot) and the `mq.max_bytes_gb` reconcile each served tenant's byte budget, and `defaultPolicy` for the one setting that still follows tenant `0`, a flat directory's ops-gate admin role; the auth verifiers are per tenant, reconfigured (rebuilt only on changed wiring) and pruned from `AfterAdopt`, the same hook's `Hub.Prune` ends the open streams of a tenant no longer served, and `wireCache`'s hook drops, through `LocalCache.Prune`, the cache version index of a tenant no longer served ([#262](https://github.com/Wave-RF/WaveHouse/issues/262))), `Run` drives the long-lived ones under one `errgroup` until the context is cancelled or one fails, `Close` releases them in reverse order. `New` wires only what the process's `roles` need (discovery, dedupe, auth verifiers, the hub bridge and keepalive per API process; the ingest worker per ingest process; the sweeper under its lease through `elected`); a process without `api` serves `api.NewOpsRouter` — probes, `/version`, metrics, and the settings reload behind the operator key alone. `cmd/wavehouse` and `tests/integration` both boot through it - **`auth/`** — JWT auth middleware: HMAC **or** JWKS verification with `alg` pinned to the active verifier, role extraction from a configurable claim path; always runs, never rejects (bad token → empty role + stashed reason). One verifier per tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 9): `Authenticator` keys them by `tenant.ID` — the request store's `settings.Store.Tenant()`, through an injected `TenantSource`; `tenant.Default` on the tenant-exempt routes — built from each tenant's `auth` block by `Reconfigure`, dropped by `Prune` once the tenant stops being served (rejected or removed), released by `Close`; the secrets (`Config`) are boot-level and shared. A JWKS key set is fetched off the boot and reload paths: until one has been stored the verifier is pending and a token-bearing request gets `503` + `Retry-After` from `api.refuseUnverifiable` (`auth.ErrVerifierPending`), never a `default_role` evaluation; refresh is library-managed (Eric, 2026-09-22), response capped at 1 MiB; the operator key's admin role is the request tenant's -- **`cache/`** — `Cache` interface → `LocalCache` (Ristretto: one pool for every tenant) + `VersionManager` (the invalidation index). Every key leads with the tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8) — `:query:` for the caller's query key and its singleflight (escaped whole as the lead field of the stored key, `|.||…`), `...
.` for a namespace, each field escaped by `keyenc` where the key is built (a `Namespace` carries the raw table and scope, so no caller escapes) — so no cached read or coalesced flight crosses tenants, a bump through `Invalidate` names one tenant's namespaces and no other's, and `InvalidateTenant` advances the tenant version folded into every namespace key and entry key of one tenant, orphaning its every cached result in one step, pipe results included (no insert reaches a pipe result until [#343](https://github.com/Wave-RF/WaveHouse/pull/343)); `Lookup` returns a `Snapshot` of the versions it read, taken before the handler chooses any input a bump invalidates — the tenant's connection included — and `Set` files the fill under it, so a write landing mid-query, or a reload moving the tenant to another address or database after the request took its connection, orphans the fill ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)), and every backend runs the conformance suite `internal/testutil/cachetest`; the one crossing is the wiring's, above the package: `internal/app` hands the ingest worker the cache through `sharedTables`, which repeats each of the worker's bumps under every tenant on the same ClickHouse address and database (`chconn.Pools.SharingTables`, whatever their user or tls block — they read the same tables), and orphans the whole cache of a tenant back on a pool after an absence, since it was out of that fan-out while away, or moved to another address or database, since it now reads other tables (story 6) +- **`cache/`** — `Cache` interface → `LocalCache` (Ristretto: one pool for every tenant) + `VersionManager` (the invalidation index), and `RedisCache`, the Redis-compatible shared backend (random version tokens under the tenant's hash tag, one-round-trip lookups, bypass on failure behind a circuit breaker, deferred invalidations retried; selected by `cache.backend: redis`, configured by the boot config's `cache.redis` block — [#613](https://github.com/Wave-RF/WaveHouse/issues/613)). Every key carries the tenant (in `RedisCache`, after the key prefix: `:{}:…` for a version token, `:q::…` for a value); in `LocalCache` and the version index it leads ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8) — `:query:` for the caller's query key and its singleflight, escaped whole as the lead field of the stored key `|.||…`, where each raw table and scope name is escaped by `keyenc` (a `Namespace` carries them raw, so no caller escapes); the index holds a version per tenant, per (tenant, table) and per (tenant, table, scope), keyed by raw name and bumped in place (one entry per live namespace however often it is bumped, [#262](https://github.com/Wave-RF/WaveHouse/issues/262)) — so no cached read or coalesced flight crosses tenants, a bump through `Invalidate` names one tenant's namespaces and no other's, and `InvalidateTenant` drops the tenant's index so its next key gets a process-unique generation, orphaning its every cached result in one step, pipe results included (no insert reaches a pipe result until [#343](https://github.com/Wave-RF/WaveHouse/pull/343)); `Lookup` returns a `Snapshot` of the versions it read, taken before the handler chooses any input a bump invalidates — the tenant's connection included — and `Set` files the fill under it, so a write landing mid-query, or a reload moving the tenant to another address or database after the request took its connection, orphans the fill ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)), and every backend runs the conformance suite `internal/testutil/cachetest`; the one crossing is the wiring's, above the package: `internal/app` hands the ingest worker the cache through `sharedTables`, which repeats each of the worker's bumps under every tenant on the same ClickHouse address and database (`chconn.Pools.SharingTables`, whatever their user or tls block — they read the same tables), and orphans the whole cache of a tenant back on a pool after an absence, since it was out of that fan-out while away, or moved to another address or database, since it now reads other tables (story 6) - **`chconn/`** — `Pools`, one `Manager` (a `driver.Conn`) per distinct `Identity{Addr, Database, Username, Password, TLS}` tuple among the served tenants, reconciled from the settings registry's `AfterAdopt` after every reload ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 6): tenants naming one tuple share its pool, sized to their largest `max_open_conns`/`max_idle_conns`; a tenant whose tuple changed is repointed; a tuple no tenant names is released after the longest `query_timeout` among the tenants it had (never dials; a resize swaps the connection with the same grace). The boot config's `clickhouse.max_total_conns` bounds the open pools' `max_open_conns` together: boot refuses naming sum and ceiling; at a reload a resize above it keeps the pool's size, and a tuple that cannot be opened (the ceiling, an unreadable certificate, or options the driver refuses) leaves its tenants on the pool they had or on none — logged, retried by the next reload. Every consumer resolves its tenant's pool per call: `For` (nil for a tenant on no pool, a `503`), `Target` (the tenant's own HTTP wiring over its pool's TLS config), `SharingTables`, `Ping` (every pool at once, ready at the first answer). `HTTPClients` keeps one `http.Client` per TLS config. `Classify` (`errclass.go`) says what a failed ClickHouse request means for the request — `Unavailable`, `Denied`, `Rejected` (any unlisted exception code: the server read it and refused it), or `Unknown` (no code, no recognizable transport failure) — over the driver's error types and the HTTP interface's `HTTPError`; the ingest worker and the query handlers (`api/ch_errors.go` `writeCHError`, [#403](https://github.com/Wave-RF/WaveHouse/issues/403), [#271](https://github.com/Wave-RF/WaveHouse/issues/271)) both use it - **`chsql/`** — dependency-free ClickHouse SQL helpers shared by `query`/`policy` (avoids an import cycle): `QuoteIdent` (backtick-quote every identifier) + `BindUnsafe` (reject names with a literal `?`) -- **`config/`** — YAML + env var config loading (cleanenv); strict on both sides (undeclared YAML key, unbound `WH_*` variable) and probes `data_dir` writability when a selected backend keeps state there (`NeedsDataDir`); `backends.go` holds each layer's `.backend` (only the in-process value today); `config.go` holds `roles` (`Has(Role)`) and `instance_id`, and `Validate` refuses a role split the backends cannot serve (any split over the embedded MQ; `api` without `ingest`, or the reverse, over a local cache) — boot is the validator, there is no dry run +- **`config/`** — YAML + env var config loading (cleanenv); strict on both sides (undeclared YAML key, unbound `WH_*` variable) and probes `data_dir` writability when a selected backend keeps state there (`NeedsDataDir`); `backends.go` holds each layer's `.backend` (the in-process value by default; `cache.backend` also takes `redis`, whose sub-block is `cache_redis.go`); `config.go` holds `roles` (`Has(Role)`) and `instance_id`, and `Validate` refuses a role split the backends cannot serve (any split over the embedded MQ; `api` without `ingest`, or the reverse, over a local cache) — boot is the validator, there is no dry run - **`coord/`** — leases for work that must run in one process at a time: `Coordinator.TryAcquire(ctx, name)` → a `Term` (fencing `Token`, strictly increasing per name; `Done`/`Err`, `ErrLost` on loss; `Resign`), `ErrHeld` while another holder's — or this coordinator's own — term is live; `RunElected` runs a loop only while holding its lease, resigning when the loop returns and campaigning again every `RetryPeriod`. `Local` is the in-process implementation (first taker wins, never expires; `Peer` is a second handle over the same table for tests); every implementation runs `coordtest.Conformance`. Imports only the standard library, so a distributed backend lives beside its connection (NATS KV in `internal/mq`). `internal/app`'s `wireCoord` opens the one `coord.backend` selects and the sweeper runs through `RunElected` under the `sweeper` lease - **`dedupe/`** — `Deduplicator` interface → `Embedded` (Pebble: every tenant's seen ids in one instance at `data_dir/pebble`, each key led by its tenant, open while any tenant's store is — the layout is the implementation's call, and the wiring hands it `data_dir` once; its `Stats` feed the system gauges), wrapped by `Managed` whose open/closed state follows the hot-reloadable `dedupe.enabled` in the settings directory's `config.json`; `Stores` holds one `Managed` per tenant, built through a `Factory` (`func(tenant.ID) *Managed`, `Embedded.Tenant` in production; `Managed` opens its store through a function, so every backend gets the same switch), and reconciled from the registry's `AfterAdopt` hook — open exactly when the tenant is served with its switch on, closed with its seen ids kept otherwise ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) stories 7 and 3) - **`discovery/`** — `SchemaRegistry`, one per served tenant over a `Source` read once per refresh — the tenant's pool's connection and the database that pool was opened for, one snapshot, so a refused move keeps discovering the database the tenant's queries still use (`internal/app`'s `discoveries` builds, runs and stops them from `AfterAdopt` and `App.Close`: `RetryRefresh` until the first success, then `StartAutoRefresh` with a random first tick; `Lookup` answers `ErrNotLoaded` before the first success — the handlers' `503` with `Retry-After` — and `ErrUnknownTable` after; a failed loop attempt counts in `wavehouse_schema_refresh_failures_total{tenant}`), that introspects ClickHouse `system.columns` (name/type/nullability plus `default_expression` and 1-based `position`) and `system.tables` (each table's `create_table_query`, kept in-process and never serialized — an external-engine table renders its wiring there unconditionally — endpoint, bucket/host, database, username, S3 access key id; ClickHouse masks the password as `[HIDDEN]` from ~23.9, so the exposure is the topology, not the secret), records the server version, + `Validate()` for ingest payloads + `CanonicalizeTimestamps()` rewriting top-level `DateTime`/`DateTime64` column values to the canonical RFC 3339 UTC wire form pre-publish (Key Design Decision #19) - **`ingest/`** — Ingest worker pipeline (`worker.go`: JetStream input → per-table batch INSERT with DLQ output). The pipeline is **insert-only**. The wire format `EventMessage` (`types.go`) carries `{table_name, scope, received_timestamp, format, columns, row}` and nothing else — `row` is one positional `JSONCompactEachRow` line and `columns` names its slots, the table's **insertable** columns (a `MATERIALIZED`/`ALIAS` column cannot be named in an `INSERT`); the worker batches per (tenant, table, column list), the tenant read off each message's `mq.Topic`, and inserts each batch into its tenant's own ClickHouse (`chconn.Pools.Target`); the worker accepts whatever table name the envelope carries (table existence was already checked by the HTTP ingest handler, which `404`s an unknown table before publish; the worker doesn't re-validate), then bulk-INSERTs. In the embedded-NATS deployment (the default), the server runs with `DontListen: true` (`internal/mq/embedded.go`), so the only Publishers reachable on the `ingest.>` subjects are in-process Go code — today, only the HTTP `/v1/ingest?table={table}` handler. Non-insert mutations (`DELETE`/`UPDATE`/`TRUNCATE`/…) must go through `POST /v1/ops/query` under the admin role (the same `RequireAdmin` gate as the rest of `/v1/ops/*`), so non-admin callers never reach the proxy. A request with no token (or an invalid one) resolves to the `default_role`, which in a production config is not the admin role (setting them equal is a loudly-warned dev-only setting), so it can't reach this endpoint. Plus `Sweeper` (Active Sweeper for NATS message lifecycle) + `EventMessage`/`BufferConsumerName` types (`types.go`) -- **`keyenc/`** — the one escaping composite keys are built from: `Escape` keeps `[A-Za-z0-9_-]` (exactly the tenant-id grammar, so a tenant id is its own escaped form) and writes every other byte as `%XX`, `Unescape` is `url.PathUnescape` (lenient: either hex case, and a byte left unescaped reads as itself, so a `%2D` an earlier build wrote still reads), `Join`/`AppendJoin` escape each field and put a separator between them (they panic on no fields, and on a separator the escaping could write or one outside ASCII) and `Split` reverses them. The package that builds a key takes raw names and escapes them itself, so no caller has to and no field reaches a key unescaped: NATS subjects (`Join`/`Split` after the verbatim tenant) and the cache's keys (`cache.VersionManager`) use it; changing what it keeps orphans every stored key +- **`keyenc/`** — the one escaping composite keys are built from: `Escape` keeps `[A-Za-z0-9_-]` (exactly the tenant-id grammar, so a tenant id is its own escaped form) and writes every other byte as `%XX`, `Unescape` is `url.PathUnescape` (lenient: either hex case, and a byte left unescaped reads as itself, so a `%2D` an earlier build wrote still reads), `Join`/`AppendJoin` escape each field and put a separator between them (they panic on no fields, and on a separator the escaping could write or one outside ASCII) and `Split` reverses them. The package that builds a key takes raw names and escapes them itself, so no caller has to and no field reaches a key unescaped: NATS subjects (`Join`/`Split` after the verbatim tenant) and the cache's keys — the version index and the shared backend's Redis keys (`internal/cache`) — use it; changing what it keeps orphans every stored key, and on the shared backend, whose keys every process builds for itself, splits them between builds for the length of a rolling upgrade (a bump one build makes misses the entries the other filed, served until their TTL) - **`mq/`** — the message-queue boundary: the **only** package that imports NATS/JetStream (Key Design Decision #20). and the only one that knows how the broker works. Everything else addresses events by `Topic{Tenant, Table, Scope}` (a validated tenant id and raw names — the tenant leads every subject, `ingest..
`, so one wildcard selects a tenant's traffic, and a topic without one is refused) and states intent through the interfaces — `Publisher` (`ErrQueueFull` is the backpressure signal, `ErrUnavailable` a broker that cannot be reached — both a `503`, with `Retry-After` `30` and `5`), `Subscriber`, `ConsumerManager`/`Consumer`/`ConsumerConfig` (the ingest worker's durable consumer), `DeadLetterer` and `DeadLetterStats` (park a message, count what is parked), `Purger` (drop what is both acked and older than a cutoff — the sweeper), `Replayer` (SSE gap-fill) — composed into `Broker`, which adds each tenant's byte budget (`SetMaxBytes`/`MaxBytes`: the `mq.max_bytes_gb` reload, which opens a tenant's queue the first time) and `Stats` (the system gauges' source). Every interface speaks per tenant, never per stream: the embedded implementation gives each tenant a queue of its own (a stream pair, `INGEST_`/`DLQ_`), and nothing outside the package may assume that layout — an external implementation may keep one shared stream. Subjects, prefixes, wildcards, stream names, sequences, and ack floors are private to the one implementation, `EmbeddedNATS` (`embedded.go`, `subject.go`, `purge.go`, `deadletter.go`), whose subject tokens are escaped by the shared `internal/keyenc`; `internal/app` constructs it and hands everything else a `mq.Broker`. Every implementation passes the conformance suite in `internal/mq/mqtest` (`mqtest.Run`), which states the `Broker` contract as behavior; a new backend runs it from its own test, with `mqtest.Caps` only where its semantics legitimately differ - **`observability/`** — OpenTelemetry pipeline: `InitProvider` wires trace/metric/log providers via OTLP gRPC (each signal independently gated). A top-level `Prometheus` config block drives an optional `/metrics` scrape endpoint that runs independently of OTLP push — standalone (Alloy/Mimir scrape, no collector), alongside OTLP, or off. `NewLogger` produces a slog handler that fans out to stdout AND OTLP (stdout always 100%, OTLP sample-rate-aware). `TraceHandler` injects trace_id/span_id from active spans. `tracer.go` provides W3C trace context propagation over message headers (`InjectHeaders`/`ExtractHeaders` on a plain header map; `internal/mq` injects on every publish and extracts on the `Subscribe` path, so this package never sees a NATS type). - **`pipes/`** — Named query pipes: `NamedQuery` type + `BindParams` + `Source` (read per request; `settings.Store` in production, `Static(q...)` in tests) @@ -129,7 +129,7 @@ Tooling notes (the non-obvious bits `make help` won't tell you): - **Shared mocks in `internal/testutil/`**: Use `MockPublisher` (records `Publish` and `DeadLetter`), `MockCache`, `MockDeduplicator`, `MockSubscriber`, `MockMessage`, `MockPurger`, `MockDeadLetterStats` instead of creating ad-hoc mocks. See `testutil/mocks.go`. - **JWT helpers**: Use `testutil.MakeJWT(t, claims)` and `testutil.MakeExpiredJWT(t, claims)` for auth tests. See `testutil/jwt.go`. - **Schema helpers**: Use `testutil.NewTestSchemaRegistry(t, tables)` for schema-aware tests — it builds the registry through the real discovery path (`Refresh` against a mock ClickHouse connection), so timestamp specs are precomputed like production. -- **Cache backends**: every `cache.Cache` backend runs `cachetest.Run` (`internal/testutil/cachetest`), the backend-agnostic conformance suite; a behavior the contract promises goes there, not in one backend's tests. +- **Cache backends**: every `cache.Cache` backend runs `cachetest.Run` (`internal/testutil/cachetest`), the backend-agnostic conformance suite; a behavior the contract promises goes there, not in one backend's tests. `RedisCache` runs it from `internal/cache/redis_integration_test.go` (`//go:build integration`, pinned Redis, Valkey, Dragonfly and Redis Cluster containers), which `make test-integration` includes. - **Policy helpers**: Use `policy.Static(p)` for a fixed `policy.Source` in tests. - **Pipes helpers**: Use `pipes.Static(queries...)` for a fixed `pipes.Source` in tests. - **Response assertions**: Use `testutil.AssertJSONResponse(t, rec, status, expected)` and `testutil.AssertJSONContains(t, rec, status, substring)`. @@ -152,7 +152,7 @@ If `make ci` passes locally, your commit has crossed the same gates CI will run ### Running `make ci` (for agents) -`make ci` is **self-contained**: the integration suite (`tests/integration/`) and the E2E orchestrator (`scripts/orchestrator/`) each boot ClickHouse via **testcontainers on random host ports**. The only prerequisite is a running **Docker daemon** — do **not** `make deps-up` or start ClickHouse first (`deps-up` is for `make dev` only). +`make ci` is **self-contained**: the integration suite (`tests/integration/`) and the E2E orchestrator (`scripts/orchestrator/`) each boot ClickHouse and a Redis via **testcontainers on random host ports**, and the shared cache backend's integration tests (`internal/cache/`) start their own Redis, Valkey, Dragonfly and one-node Redis Cluster containers the same way. The only prerequisite is a running **Docker daemon** — do **not** `make deps-up` or start ClickHouse first (`deps-up` is for `make dev` only). Run it via the **background Bash tool** (`run_in_background: true`) and wait for the completion notification; the harness re-invokes you on exit, so polling the log with `tail` only burns context: @@ -430,7 +430,7 @@ cmd/ → Binary entry points (thin — argv, logger, config, internal/api/ → HTTP layer (handlers, router, middleware, schema/DLQ/pipes endpoints) internal/app/ → Process wiring (build every component, run them under one errgroup, release in reverse) internal/auth/ → JWT/JWKS authentication middleware (HMAC or JWKS, role extraction from claims) -internal/cache/ → Query cache (interface, Ristretto L1, tenant-led version index) +internal/cache/ → Query cache (interface, Ristretto L1, tenant-led version index, Redis-compatible shared backend) internal/chconn/ → ClickHouse pools, one per connection tuple among the served tenants (reconciled on settings reload) internal/chsql/ → Shared ClickHouse SQL helpers (identifier quoting + bind-safety) internal/config/ → Configuration structs + loader @@ -449,8 +449,8 @@ internal/stream/ → SSE fan-out (event Hub: project once per role, Subsc internal/tenant/ → Tenant id (type, grammar, reserved default, request header name) internal/testutil/ → Shared test helpers (mocks, JWT + schema helpers; logtest/ captures or silences the default logger; cachetest/ is the conformance suite every cache.Cache backend runs) tests/ → Integration & E2E tests -tests/integration/ → Go integration tests (//go:build integration; ClickHouse testcontainer) -tests/e2e/ → E2E test stack (scripts/orchestrator boots a ClickHouse testcontainer + the wavehouse-cov binary) +tests/integration/ → Go integration tests (//go:build integration; ClickHouse testcontainer, and Redis for shared_cache_test.go). A package tested against its own external server keeps them beside it: internal/cache/redis_integration_test.go (Redis, Valkey, Dragonfly, Redis Cluster testcontainers) +tests/e2e/ → E2E test stack (scripts/orchestrator boots ClickHouse and Redis testcontainers + the wavehouse-cov binary) tests/e2e/fixtures/ → Idempotent ClickHouse DDL scripts for test tables tests/e2e/sdk/ → E2E integration tests via TypeScript SDK (Vitest) deployments/compose/ → Docker Compose files (standalone.yaml, dependencies.yaml) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8492d1576..b978fe90d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added +- **`cache.backend: redis` shares the query cache across instances** (`internal/config/{cache_redis,backends,config}.go` (+ tests), `internal/app/wire.go` (+ tests), `internal/ingest/worker.go`, `tests/integration/shared_cache_test.go`, `scripts/orchestrator/main.go`, `tests/e2e/fixtures/config.yaml`, `.testcoverage.yml`, `deployments/compose/dependencies.yaml`, `config.yaml`, `.github/workflows/{ci.yml,README.md}`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md,settings-directory.mdx,getting-started.md,pipes.mdx,api.md,development.md,index.mdx,why-wavehouse.md,sdk/reference.md}`, `README.md`, `AGENTS.md`): PR E4 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). The boot config's `cache.backend` now takes `redis`, configured by a new `cache.redis` block (`WH_CACHE_REDIS_*`): `addrs` (required), `mode` (`standalone` or `cluster`; `sentinel` refuses boot until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `username`, `password` (a secret — set it through the environment), `db`, `tls.{enabled,ca_file,cert_file,key_file,server_name,insecure_skip_verify}`, `key_prefix` (`wh`), `timeout` (`100ms`) and `dial_timeout` (`1s`), each at most `1s` since boot and shutdown each wait out a connection attempt they bound, `max_value_bytes` (1 MiB), `compress_min_bytes` (`1024`; `0` never compresses) and `version_ttl` (`168h`). Every instance pointed at one server shares its cached results, and an insert on any instance invalidates every instance's. It is also the shared cache a split of [`roles`](https://github.com/Wave-RF/WaveHouse/pull/622) needs: the refusal of `api` without `ingest`, or the reverse, over `cache.backend: local` now names `redis`, though every split is still refused while the queue is embedded. A malformed block — no address, an address without a valid port, more than one address in `standalone` mode, a URL-style address (refused without repeating it, since it may carry a password), an unknown mode, `db` other than `0` in cluster mode, an unreadable TLS file, a TLS key set while `tls.enabled` is off — refuses boot; an unreachable server, or one that rejects the password, does not: the process boots with the cache bypassed and keeps reconnecting, logging a rejected password at `ERROR` on every attempt, so a rotated secret cannot crash-loop every instance at once. `insecure_skip_verify`, and a `cache.redis.addrs` set while `cache.backend` is `local`, are logged at `WARN` at boot. The ingest worker's log of an invalidation that did not land drops from `ERROR` to `WARN`, since the shared backend defers and retries it: an outage would otherwise log an `ERROR` for every batch. The e2e suite now runs against a Redis testcontainer with `cache.backend: redis`, so the shared backend is exercised end to end; the e2e per-suite exclude now names what its run still can't reach instead — `internal/cache/pending.go` (the retry of an invalidation the server did not take, which needs an outage), `internal/config/cache_redis.go` (the block's own rejection paths) and `internal/cache/(local|version_manager).go` (the `local` backend, which e2e no longer runs) — and the unit and integration suites keep covering them. An integration test boots two instances over one Redis and one ClickHouse: a result one fills is a hit for the other, and a row ingested through one is served fresh by the other on its next query, well inside the stale entry's TTL; another pauses Redis and checks queries keep succeeding from ClickHouse, then turn back to hits; a third runs the first one's hit, insert and fresh-miss lifecycle on the suite's own `cache.backend: local` app, since e2e no longer exercises that backend. `deployments/compose/dependencies.yaml` gains an optional `redis` profile for local multi-instance work. The deployment guide gains a "Multiple instances and the shared cache" section: what each instance keeps to itself, what a reader on another instance can see and when, `maxmemory-policy`, and the metrics to alert on. +- **A Redis-compatible shared cache backend** (`internal/cache/{redis,redis_codec,breaker,pending,metrics}.go` (+ tests), `internal/cache/cache.go`, `internal/cache/redis_integration_test.go`, `Makefile`, `go.mod`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development}.md`, `AGENTS.md`, `CONTRIBUTING.md`): PR E3 of the distributed-deployment epic ([#613](https://github.com/Wave-RF/WaveHouse/issues/613)). `cache.RedisCache` keeps query results and their versions in one Redis, Valkey, Dragonfly, ElastiCache or MemoryDB server shared by every process, so an insert one process makes invalidates what every other process has cached. Versions are random tokens, one per tenant, table and scope, under the tenant's hash tag, with the table and scope escaped into the key (`internal/keyenc`) so no two names share a token; a bump sets a fresh one, and a value carries the tokens it was computed under, so a lookup is one pipelined round trip (`MGET` of the tokens plus `GET` of the value, no scripts) and a token lost to eviction, expiry, `FLUSHALL` or a restart without persistence can only cause misses — `maxmemory-policy allkeys-lru` is safe. Restoring an RDB or AOF snapshot, or a backup, is a rollback instead (a restart after a crash that reloads the server's last save included, which stock Redis and Valkey make by default): the old tokens return with their values, so invalidations made since are undone until those entries' TTL. Values of 1 KiB or more are zstd-compressed when that makes them smaller, and a value over 1 MiB stored is not cached. A server that fails or takes longer than the per-operation timeout (100 ms) is a miss, a skipped fill and a deferred invalidation, never a failed query; five failures in a row, or one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like) or the credentials (`WRONGPASS` or `NOAUTH` after a password rotation), open a circuit breaker — logged once per opening, at `ERROR` for the credentials and `WARN` otherwise, which a failed probe repeats every 5 s while the server stays down — that skips the server until a probe write succeeds within the per-operation timeout (connections are replaced every minute, so after a failover behind a stable address the process reaches the new primary, and delivers the bumps it owes, within about a minute; the probe also allows up to twice the dial timeout for a reconnect, but the client's other connections must reconnect within the per-operation timeout, so that timeout should still exceed a reconnect), and deferred invalidations are retried until they land — the first at once, and at the probe's cadence while the breaker is open — collapsing to one tenant-wide bump per tenant past 100,000 keys; until one lands, the process that owes it bypasses the lookups it would orphan. New metrics: `wavehouse_cache_lookups_total{backend,result}`, `wavehouse_cache_op_duration_seconds{backend,op}`, `wavehouse_cache_breaker_open{backend}`, `wavehouse_cache_invalidations_total{backend,result}`, `wavehouse_cache_invalidations_pending{backend}`, `wavehouse_cache_value_bytes{backend}`, `wavehouse_cache_oversize_total{backend}`, `wavehouse_cache_set_failures_total{backend,reason}`. `cache.backend: redis` selects it (the entry above). Tested against Redis 8.10, Valkey 8.1, Dragonfly 2.0 and a Redis Cluster node by the conformance suite, plus lost-token, snapshot-rollback, compression, stored-size, server-stops-answering, refused-writes (a demoted primary, a full `noeviction` server), rotated-credentials, slow-reconnect (over TLS), slow-server, failover-behind-a-stable-address and owed-invalidation cases; `make test-integration` now also runs `internal/cache`'s integration-tagged tests. Adds `github.com/redis/rueidis` (Redis org, Apache-2.0; its only runtime dependency is `golang.org/x/sys`) and makes `github.com/klauspost/compress` a direct dependency. - **One conformance suite for every `mq.Broker`, and a transient broker failure is a `503`** (`internal/mq/mqtest/` (new: the suite and the embedded broker's run of it), `internal/mq/{mq,embedded}.go` (+ tests), `internal/api/ingest.go` (+ tests), `.testcoverage.yml`, `docs/src/content/docs/{api,architecture}.md`, `AGENTS.md`): the first piece of the external-NATS workstream of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `mqtest.Run` states the `Broker` contract as behavior — round trips with names that need encoding, per-tenant order, `Nak` and `AckWait` redelivery, the trace context reaching `Subscribe`, dead-lettering that keeps the topic and leaves the original unacked, per-tenant dead-letter counts, replay bounds and isolation, and exactly one `failed` report when delivery ends underneath a consumer — through the interfaces alone, so the external backend runs the same cases, with `mqtest.Caps` for the four places where its semantics legitimately differ. The embedded broker passes it; writing it turned up that a durable deleted on several tenants' queues could report on `failed` more than once, which is fixed and pinned by a test that deletes it on one queue after another. It also turned up a replay that lost its connection mid-pull passing for a caught-up one when the pull ended in a timeout; that is an error now, as the `Replayer` contract says. The interface comments now allow a delivery unit that is a partition holding several tenants, a `CreateConsumer` that finds a durable rather than creating one, a `PurgeAcked` that leaves retention to the operator, and zero dead-letter counts where there is no per-tenant queue. A new sentinel, `mq.ErrUnavailable`, is a broker that cannot be reached or does not answer in time: the ingest handler answers it with `503` + `Retry-After: 5` rather than the `500` "publish failed" it would have been. Nothing returns it yet; the external backend of #613 will. - **Process roles: the API and the background workers can run in separate processes** (`internal/config/config.go` (+ `roles_test.go`, `defaults_test.go`), `internal/config/backends.go`, `internal/app/{app,wire}.go` (+ `roles_test.go`), `internal/api/router.go` (+ tests), `tests/integration/{setup,tenants}_test.go`, `config.yaml`, `docs/src/content/docs/{configuration.mdx,deployment.md,architecture.md}`, `AGENTS.md`), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The boot config gains `roles` (`WH_ROLES`, default `api,ingest,sweeper`, set in `defaults()` like every boot default, so an explicit `roles: []` refuses boot) and `instance_id` (`WH_INSTANCE_ID`, default `-<8 hex>`, fresh at every boot; logged at boot, and recorded as a lease holder once a shared `coord.backend` exists). A process wires only what its roles need: `api` runs the HTTP API with schema discovery, the token verifiers, the dedupe stores and the SSE hub (all per API process); `ingest` runs the ingest worker; `sweeper` runs the sweeper under its lease. A process without `api` serves an ops-only listener on `server.port` (the probes and their aliases, `/version`, the same-port metrics path, and `POST /v1/ops/settings/reload`, which takes the operator key alone); every other route answers 404, under `/v1/ops` once the operator key has passed. Boot refuses any split over the embedded MQ, which no other process can reach, and `api` without `ingest` (or the reverse) over a local cache, which the ingest worker's invalidations would never reach. Until a shared `mq.backend` exists, every process therefore runs every role, which is the default, so nothing changes for an existing deployment. `data_dir` is probed for Pebble only in a process running `api`. A `config.Config` built without `config.Load` must now name its roles (`config.AllRoles()` for all of them): `app.New` refuses an empty set. - **Leases for work that must run in one process at a time, and the sweeper runs under one** (`internal/coord/` (new: `coord.go`, `local.go`, `elect.go`, `coordtest/`, + tests), `internal/app/{app,wire}.go` (+ tests), `internal/config/backends.go`, `internal/ingest/sweeper.go`, `.github/labeler.yml`, `.testcoverage.yml`, `docs/src/content/docs/{architecture,development,ingest-pipeline}.md`, `docs/src/content/docs/configuration.mdx`, `config.yaml`, `AGENTS.md`): part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `coord.Coordinator` hands out named leases (`TryAcquire` → a `Term` with a strictly increasing fencing `Token`, a `Done` channel and `Resign`; `ErrHeld` while another holder's term is live), and `coord.RunElected` runs a loop only while its process holds the lease, resigning when the loop returns and campaigning again every 2s. `coord.Local` is the in-process implementation, and `coordtest.Conformance` is the suite every implementation runs — the NATS KV backend that lets several replicas share one queue comes next. The sweeper now runs through `RunElected` under the `sweeper` lease; with the in-process coordinator the one process always holds it, so nothing changes for a single-process deployment beyond one `coord: elected` log line at startup. `coord.backend` now selects the coordinator (`local`, the only value), so a `config.Config` built without `config.Load` must name it as well as the other three layers' backends. @@ -88,7 +90,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), - **A failed ClickHouse query answers by what went wrong, not a flat `500`/`502`** (`internal/api/ch_errors.go` (new, + tests), `internal/api/{errors,query,structured_query,pipes,schema,ch_settings}.go`, `internal/chconn/errclass.go` (`HTTPStatus` exported), `clients/ts/src/errors.ts` (+ tests), `tests/integration/query_errors_test.go` (new), `tests/integration/query_limits_test.go`, `internal/app/app_test.go`, `tests/e2e/sdk/{admin,query}.test.ts`, `AGENTS.md`, `docs/src/content/docs/{api,architecture}.md`, `docs/src/content/docs/{access-control,configuration}.mdx`, `docs/src/content/docs/sdk/{reference.md,index.mdx}`): fixes [#403](https://github.com/Wave-RF/WaveHouse/issues/403) and [#271](https://github.com/Wave-RF/WaveHouse/issues/271), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). ClickHouse answers a syntax error, a missing grant and an overloaded server alike with HTTP `500`, so `/v1/ops/query` turned a bad statement into a `502` and `/v1/query` and pipes into a `500` the SDK retried. All three now class the failure with `chconn.Classify` through one helper, `writeCHError`: a statement ClickHouse refused is `400 clickhouse.rejected`; a query over a rows/bytes limit, the role's own memory cap, or its time cap where that is no longer than `query_timeout` is `400 clickhouse.limit_exceeded`; `ACCESS_DENIED` is `403 clickhouse.access_denied`; credentials, user or database refused, or a redirect or `4xx` with no exception code from whatever fronts ClickHouse, is `502 clickhouse.misconfigured`; ClickHouse down, unreachable or overloaded is `503 clickhouse.unavailable` with `Retry-After: 5`; a failure with no verdict stays `500` (`502` on the proxy) as `clickhouse.unknown`. The error envelope gains `code` and `retryable` next to `error` on these responses — additive. A role with `max_execution_time` now queries with no context deadline and a cancel two seconds past the cap instead: clickhouse-go overwrote the cap's `max_execution_time` with deadline+5s for any deadline over 1s, so an overrun came back as a bare deadline, indistinguishable from waiting for a pooled connection; ClickHouse now enforces the cap itself and reports `TIMEOUT_EXCEEDED`. `POST /v1/ops/schema/refresh` against an unreachable ClickHouse is a `503` with `Retry-After` instead of a `500`. **SDK:** `WaveHouseError.code` and `retryable` now take the server's `code`/`retryable` when the body has them (`HTTP_` and "5xx retries" otherwise), so a rejected query is `clickhouse.rejected` rather than `HTTP_500`, and is not retried. - **An unavailable ClickHouse is retried with backoff instead of dead-lettering every row** (`internal/chconn/errclass.go` (new, + tests), `internal/ingest/{worker,backoff}.go` (`backoff.go` new, + tests), `internal/mq/{mq,embedded}.go`, `internal/testutil/mocks.go`, `tests/integration/ingest_outage_test.go` (new), `AGENTS.md`, `README.md`, `docs/src/content/docs/{ingest-pipeline,architecture,api,deployment,why-wavehouse}.md`, `docs/src/content/docs/{settings-directory,index,access-control}.mdx`): workstream A of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). A failed batch insert used to go through row-by-row isolation whatever the failure, so a ClickHouse that was down, overloaded or read-only failed every row twice and parked the whole batch on the DLQ. `chconn.Classify` now classes the failure first — `Rejected` (any ClickHouse exception code outside the availability and credential lists: the server read the row and refused it), `Unavailable` (connection refused/reset, timeouts, `TOO_MANY_SIMULTANEOUS_QUERIES`, `SERVER_OVERLOADED`, `MEMORY_LIMIT_EXCEEDED`, `TOO_MANY_PARTS`, `READONLY`, `TABLE_IS_READ_ONLY`, `KEEPER_EXCEPTION`, …), `Denied` (`AUTHENTICATION_FAILED`, `ACCESS_DENIED`, …) or `Unknown` (no code, no recognizable transport failure). Only `Rejected` is isolated and dead-lettered as before, and a multi-row batch refused with `TOO_MANY_PARTS` or `MEMORY_LIMIT_EXCEEDED` is split row by row first (`chconn.Splittable`), because a batch spanning too many partitions or too much memory can fail where each of its rows inserts; every other class hands the batch back to the queue with a delayed nak (`mq.Message.NakWithDelay`, new) under a jittered 1 s → 30 s backoff shared by every table on the same ClickHouse pool (a failure of one table — read-only, too many parts or mutations, a grant missing on it, `chconn.TableScoped` — backs off that table alone), which turns rows away without a request while it runs and probes once per window, and ClickHouse going away mid-isolation stops isolation and retries the rows it had not settled. Counted by the new `wavehouse_ingest_retries_total{table, reason}`; logged at `WARN` when an outage starts and at most every 30 s during it. A long outage now shows as a growing ingest stream and, at `mq.max_bytes_gb`, ingest `503`s — not as a full DLQ; a lasting failure of one table holds back its tenant's other tables once its waiting rows reach `maxAckPending`. Retried rows come back out of arrival order, which matters only to a `ReplacingMergeTree` without a version column or a `CollapsingMergeTree`. - **Schema discovery's retry loop jitters its backoff** (`internal/discovery/discovery.go` (+ tests), `internal/app/wire.go`, `internal/api/errors.go`, `AGENTS.md`, `docs/src/content/docs/{architecture,api,deployment}.md`): `RetryRefresh` slept exactly `2s * 2^n` capped at 60s, so instances retrying against one recovering ClickHouse fired in lockstep, every 60s on the same second. Each sleep is now drawn uniformly from below the backoff (full jitter), spreading the retries over the whole window and halving the mean wait — so a failing tenant's retries, their log lines and `wavehouse_schema_refresh_failures_total` come about twice as often ([#141](https://github.com/Wave-RF/WaveHouse/issues/141)). -- **A write that lands while a cached read is running no longer re-homes the pre-write rows under the post-write key** (`internal/cache/{cache,local,version_manager}.go` (+ tests), `internal/testutil/cachetest` (new), `internal/api/{structured_query,pipes}.go` (+ tests), `internal/ingest/worker.go` (+ tests), `internal/query/ident.go` (removed, + tests), `internal/app/wire.go`, `docs/src/content/docs/{api,architecture,deployment}.md`, `AGENTS.md`): fixes [#382](https://github.com/Wave-RF/WaveHouse/issues/382), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `POST /v1/query` rebuilt the version-folded cache key after the query ran, so an insert invalidating the table mid-query filed the rows read before it under the new versions, and they were served as fresh until their TTL (a pipe result's key folded no version, so pipes were unaffected; with the tenant's version in every key they now take the same snapshot). The `Cache` interface now snapshots at lookup: `Lookup(ctx, tenant, sha, deps)` returns the `Entry` and a `Snapshot` of the versions it read, and `Set(ctx, snapshot, value, ttl)` stores under that snapshot, so such a fill is orphaned and the next request reads the post-write rows. The singleflight leader's snapshot is the one used; coalescing is unchanged. The snapshot is taken before any input a bump invalidates is chosen, the tenant's ClickHouse connection included: both handlers now look up before they resolve the tenant's pool, so a reload that moves the tenant to another address or database after a request took the old pool orphans that request's fill instead of filing the old database's rows as fresh under the new tenant version. A tenant on no pool is still a `503` before a cached result is served or a query runs. A `Lookup` whose dependencies name another tenant is refused (`ErrForeignDependency`). `Set` now errors only when the backend failed: a value the cache declines (larger than the cache holds (`cache.l1_max_cost`), a non-positive TTL) is not an error. One behavior change: the tenant's version is folded into every key, a pipe result's included, so `InvalidateTenant` (a tenant back on a pool after an absence, or moved to another ClickHouse address or database) now drops that tenant's cached pipe results as well as its query results; before, a pipe result stayed until its TTL. Inserts still do not invalidate pipe results ([#343](https://github.com/Wave-RF/WaveHouse/pull/343)). A backend-agnostic conformance suite, `cachetest.Run`, pins what a hit, a miss and each kind of bump mean, and `LocalCache` runs it; the Redis-compatible backend will run the same suite. The cache now escapes table and scope names itself: a `Namespace` carries them raw and the version index builds its keys with `internal/keyenc`, so neither the structured-query read nor the ingest worker's invalidation escapes them (`query.SafeEncodeToken` is gone) and no name reaches a key unescaped. The suite pins that a name holding a dot, a space or a `%` is read and bumped under one key, and that names which would run together unescaped (`a.0.b` against `a` with scope `b.0.`) stay two entries. Namespace keys are unchanged; an entry's key now carries the tenant's version and escapes the caller's query key whole. All of them live in the process, so nothing stored is orphaned. +- **A write that lands while a cached read is running no longer re-homes the pre-write rows under the post-write key** (`internal/cache/{cache,local,version_manager}.go` (+ tests), `internal/testutil/cachetest` (new), `internal/api/{structured_query,pipes}.go` (+ tests), `internal/ingest/worker.go` (+ tests), `internal/query/ident.go` (removed, + tests), `internal/app/wire.go`, `docs/src/content/docs/{api,architecture,deployment}.md`, `AGENTS.md`): fixes [#382](https://github.com/Wave-RF/WaveHouse/issues/382), part of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). `POST /v1/query` rebuilt the version-folded cache key after the query ran, so an insert invalidating the table mid-query filed the rows read before it under the new versions, and they were served as fresh until their TTL (a pipe result's key folded no version, so pipes were unaffected; with the tenant's version in every key they now take the same snapshot). The `Cache` interface now snapshots at lookup: `Lookup(ctx, tenant, sha, deps)` returns the `Entry` and a `Snapshot` of the versions it read, and `Set(ctx, snapshot, value, ttl)` stores under that snapshot, so such a fill is orphaned and the next request reads the post-write rows. The singleflight leader's snapshot is the one used; coalescing is unchanged. The snapshot is taken before any input a bump invalidates is chosen, the tenant's ClickHouse connection included: both handlers now look up before they resolve the tenant's pool, so a reload that moves the tenant to another address or database after a request took the old pool orphans that request's fill instead of filing the old database's rows as fresh under the new tenant version. A tenant on no pool is still a `503` before a cached result is served or a query runs. A `Lookup` whose dependencies name another tenant is refused (`ErrForeignDependency`). `Set` now errors only when the backend failed: a value the cache declines (larger than the cache holds (`cache.l1_max_cost`), a non-positive TTL) is not an error. One behavior change: the tenant's version is folded into every key, a pipe result's included, so `InvalidateTenant` (a tenant back on a pool after an absence, or moved to another ClickHouse address or database) now drops that tenant's cached pipe results as well as its query results; before, a pipe result stayed until its TTL. Inserts still do not invalidate pipe results ([#343](https://github.com/Wave-RF/WaveHouse/pull/343)). A backend-agnostic conformance suite, `cachetest.Run`, pins what a hit, a miss and each kind of bump mean, and `LocalCache` runs it; the Redis-compatible backend will run the same suite. The cache now escapes table and scope names itself: a `Namespace` carries them raw and the cache renders each result's key with `internal/keyenc`, so neither the structured-query read nor the ingest worker's invalidation escapes them (`query.SafeEncodeToken` is gone) and no name reaches a key unescaped. The suite pins that a name holding a dot, a space or a `%` is read and bumped under one key, and that names which would run together unescaped (`a.0.b` against `a` with scope `b.0.`) stay two entries. The version index's own entries are unaffected by the escaping; only the rendered key changes: it now carries the tenant's version and escapes the caller's query key whole. All of them live in the process, so nothing stored is orphaned. +- **The cache's version index no longer grows with every bump, and forgets a tenant no longer served** (`internal/cache/{local,version_manager}.go` (+ tests), `internal/app/wire.go` (+ tests), `docs/src/content/docs/architecture.md`, `AGENTS.md`): part of [#262](https://github.com/Wave-RF/WaveHouse/issues/262) (growth across bumps and a departed tenant's memory; per-table scope cardinality is left open, see the issue) and of [#613](https://github.com/Wave-RF/WaveHouse/issues/613). The index nested each table under its tenant's version and each scope under its table's, and never pruned, so every tenant invalidation left the tenant's whole index behind, and it grew with every tenant ever served. It now holds one version per tenant, per (tenant, table) and per (tenant, table, scope), bumped in place. A tenant invalidation drops the tenant's index and hands its next key a generation unique within the process, so nothing cached before it can match again, and a table bump drops the table's scope versions. After each settings reload the index of every tenant no longer served, removed or rejected, is dropped the same way; its cached results are orphaned with it, as they already were when such a tenant came back on a pool. No change to what is cached or served. The Redis-compatible backend (#613) will bound its versions with a TTL instead. - **An explicit `false`, `0` or `""` in `config.yaml` is no longer replaced by the key's default** (`internal/config/config.go`, `internal/config/defaults_test.go` (new), `docs/src/content/docs/configuration.mdx`, `AGENTS.md`): [#631](https://github.com/Wave-RF/WaveHouse/issues/631). Defaults lived in cleanenv `env-default` tags, which cleanenv applies after the YAML decode to any field still at its zero value, so it could not tell a key the file set to its zero value from one the file left out. `otel.traces.enabled: false`, `otel.metrics.enabled: false` and `otel.logs.enabled: false` came back `true`; `otel.traces.sample_rate: 0` and `otel.logs.sample_rate: 0` came back `1.0`; `server.shutdown_timeout: 0` came back `10`; `cache.l1_max_cost: 0`, `prometheus.path: ""` and `data_dir: ""` came back as their defaults; `server.port: 0` came back `8080`. All of it was silent. Defaults now live in one Go function, `defaults()`, which `Load` starts from before decoding the file and then applying `WH_*` variables, so the order is env > YAML > default and a key the file sets always wins. **Behaviour change if your file relied on the bug:** a zero you wrote now takes effect. A file that says `sample_rate: 0` now exports no traces (or no DEBUG/INFO logs), where it silently exported everything; a signal set `enabled: false` is now off; `shutdown_timeout: 0` now skips the drain. `cache.l1_max_cost: 0`, `server.port: 0`, and `data_dir: ""` now refuse boot (`cache init: MaxCost can't be zero`, `server.port 0 out of range`, `data_dir (WH_DATA_DIR) is required`) instead of running on the default; an empty `prometheus.path` refuses boot when `prometheus.enabled` is true. Delete the key to get the default back. Env vars are unchanged: they already honoured an explicit zero. New tests load through `config.Load` for every affected key (a YAML zero is kept, an absent key gets the default, env wins in both directions), refuse an `env-default` tag on any field, and pin each documented default in `configuration.mdx` to `defaults()`. - **An embedded queue store that cannot be created fails boot at once, naming the cause** (`internal/mq/embedded.go` (+ tests)): part of [#617](https://github.com/Wave-RF/WaveHouse/issues/617). A regular file at `/nats`, or a `nats` directory that could not be created there, failed JetStream in the background, so boot waited out the server's 5s readiness check and reported only `nats server not ready`. `NewEmbedded` now creates the directory first (at `0700`, as the server does) and refuses boot with the mkdir error. An existing but unwritable `nats` directory still takes the old path. - **An insert invalidates a table's cached results under every tenant the directory holds** (`internal/app/wire.go` (+ tests), `internal/settings/registry.go` (+ tests), `AGENTS.md`, `docs/src/content/docs/{deployment,architecture,ingest-pipeline}.md`): until [#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 6 gives each tenant its own ClickHouse, every tenant reads the same tables, but the ingest worker — which writes every event as tenant `0`'s until story 5 — bumped only tenant `0`'s cache namespaces after an insert, so another tenant's cached query could answer stale rows for up to its TTL (an hour at most). The cache the worker invalidates through now fans each bumped namespace out to every tenant the registry knows (the new `Registry.Known`), the named one and a rejected one included — a rejected tenant comes back into service with the entries it has, so leaving it out would let a folder repaired inside a TTL serve pre-insert rows; reads are untouched, so a tenant is still never served another's cached rows. The residual, a folder removed and restored inside a TTL, is closed since #610: a tenant back on a pool after an absence has its cached results orphaned at once (`Cache.InvalidateTenant`). Raised by CodeRabbit on #602. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c02dfa1d9..70bbb00ba 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -39,7 +39,7 @@ Open a [feature request issue](https://github.com/Wave-RF/WaveHouse/issues/new?t The pre-push hook (installed by `make tools`) blocks a push until the tree has been validated locally: a code change needs `make ci`, a docs/prose-only change needs only `make verify` (the same split CI makes). `make lint` / `make test` / `make build` are fast inner-loop subsets. -2. Write tests for new functionality. Unit tests go alongside the code in `internal/`. Integration tests go in `tests/` with the `//go:build integration` tag. +2. Write tests for new functionality. Unit tests go alongside the code in `internal/`. Integration tests carry the `//go:build integration` tag and go in `tests/integration/`, or beside the package when they test one package against its own external server (e.g. `internal/cache/redis_integration_test.go`), with the package added to the `test-integration` target. 3. Update documentation if your change affects: - API endpoints → update `docs/src/content/docs/api.md` diff --git a/Makefile b/Makefile index 8dcbe35d0..5fff09f2b 100644 --- a/Makefile +++ b/Makefile @@ -764,7 +764,7 @@ test-integration: go-mod-download ## Run Go integration tests + render coverage @rm -rf $(COV_INT)/data && mkdir -p $(COV_INT)/data @GOCOVERDIR="$(CURDIR)/$(COV_INT)/data" go tool gotestsum --format $(GOTESTSUM_FMT) -- \ -tags="integration $(TAGS)" -timeout 240s -coverpkg=./... -race -count=1 \ - ./tests/integration/... $(ARGS) \ + ./tests/integration/... ./internal/cache/... $(ARGS) \ -args -test.gocoverdir="$(CURDIR)/$(COV_INT)/data" @if [ -z "$(COV_DEFER)" ]; then go run ./scripts/cov render integration; fi diff --git a/README.md b/README.md index 7d14981c2..025822839 100644 --- a/README.md +++ b/README.md @@ -72,7 +72,7 @@ ClickHouse is a phenomenal OLAP database, but pointing a frontend right at it le If you're building user-facing analytics, WaveHouse is like **Supabase for ClickHouse**. Or an **open-source Tinybird** that pushes data to the frontend in real time over SSE, not just pull-based REST. - **Ingest** — async durable WAL (embedded NATS JetStream), `200 OK` instantly, background batch-flush; schema-validated against `system.columns`; optional ID-based dedup (idempotent ingest); dead-letter queue for rows ClickHouse rejects (an unavailable ClickHouse is retried with backoff, not dead-lettered). -- **Query** — in-process Ristretto cache + `singleflight` coalescing; type-safe structured query AST; Tinybird-style named pipes (parameterized SQL endpoints). +- **Query** — result cache (in-process Ristretto, or a Redis shared by every instance) + `singleflight` coalescing; type-safe structured query AST; Tinybird-style named pipes (parameterized SQL endpoints). - **Real-time** — native SSE push, broadcast *before* the ClickHouse flush, with JetStream gap-fill for late/reconnecting clients. - **Security** — Hasura-style per-table, per-role column + row policies with JWT claim templating, defined in the hot-reloadable settings directory. - **Client** — `@wavehouse/sdk`: TypeScript client with query builder, live queries, streaming, and schema codegen; one runtime dependency (an SSE frame parser, ~1.4 KB gzipped). diff --git a/config.yaml b/config.yaml index ee8d0e652..0f0a6ad59 100644 --- a/config.yaml +++ b/config.yaml @@ -51,8 +51,8 @@ clickhouse: password: "" max_total_conns: 0 # ceiling on open native connections across pools; 0 = none -# Each layer's implementation, chosen at boot. Only the in-process backend -# exists for each today, and it is the default. +# Each layer's implementation, chosen at boot. The in-process backend is the +# default for each, and the only one for these three. mq: backend: embedded # NATS JetStream under /nats dedupe: @@ -60,11 +60,18 @@ dedupe: coord: backend: local # leases (the sweeper's) held in this process -# In-process L1 cache size. The query time-bucket +# The query-result cache: local (in-process, sized by l1_max_cost) or redis +# (one Redis-compatible server shared by every instance; see the redis block +# below and the Configuration page for every key). The query time-bucket # (query.timestamp_bucket_seconds) is a settings key. cache: backend: local l1_max_cost: 67108864 + # redis: # read only with backend: redis + # addrs: ["localhost:6379"] # `docker compose -f deployments/compose/dependencies.yaml --profile redis up -d` + # key_prefix: wh + # timeout: 100ms # per operation; slower is a miss, never a failed query + # The password is a secret: WH_CACHE_REDIS_PASSWORD, not this file. # Auth has no on/off switch — the JWT middleware always runs. A request with no # token, or an invalid/expired one, falls back to the policy default_role; diff --git a/deployments/compose/dependencies.yaml b/deployments/compose/dependencies.yaml index de147c726..4cc4f2fd6 100644 --- a/deployments/compose/dependencies.yaml +++ b/deployments/compose/dependencies.yaml @@ -33,5 +33,18 @@ services: timeout: 2s retries: 15 + # Optional shared cache for trying cache.backend=redis locally, e.g. two + # host-side instances on different ports: `--profile redis`. No + # persistence, and /data on tmpfs, so it leaves no volume behind. + redis: + profiles: [redis] + # Pinned to match internal/cache's integration suite. + image: redis:8.10.2-alpine + command: ["redis-server", "--save", "", "--appendonly", "no", "--maxmemory", "256mb", "--maxmemory-policy", "allkeys-lru"] + ports: + - "6379:6379" + tmpfs: + - /data + volumes: clickhouse-data: diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index b80e73b12..77bbe7921 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -445,7 +445,7 @@ ClickHouse's inline `FORMAT` clause (e.g. `SELECT 1 FORMAT CSV` or `… FORMAT P The proxy buffers the upstream response in memory before forwarding (no row-streaming yet), so a `SELECT *` from a large table can pin RAM on the API server. To avoid an admin OOMing themselves, responses larger than 64 MiB return 502 with a `clickhouse response exceeded N bytes` error. Narrow the query with `LIMIT`, or use a streaming client outside WaveHouse that talks to ClickHouse directly (the standard escape hatch — the same admin credentials work). ::: -This endpoint **does not cache, does not singleflight, and emits `Cache-Control: no-store`** — every request goes straight to ClickHouse, mutation or read, and downstream HTTP caches are explicitly told not to store the response. Raw SQL is an admin escape hatch with infrequent, ad-hoc traffic, so the L1/singleflight machinery would only add complexity without a real hit-rate win. Use [`POST /v1/query?table={table}`](#post-v1querytabletable--structured-query) or [`GET/POST /v1/pipes/{name}`](#getpost-v1pipesname--execute-named-pipe) for the cached read paths (dashboards, high-QPS clients, etc.) — both share an in-process L1 (Ristretto) with singleflight coalescing. +This endpoint **does not cache, does not singleflight, and emits `Cache-Control: no-store`** — every request goes straight to ClickHouse, mutation or read, and downstream HTTP caches are explicitly told not to store the response. Raw SQL is an admin escape hatch with infrequent, ad-hoc traffic, so the cache/singleflight machinery would only add complexity without a real hit-rate win. Use [`POST /v1/query?table={table}`](#post-v1querytabletable--structured-query) or [`GET/POST /v1/pipes/{name}`](#getpost-v1pipesname--execute-named-pipe) for the cached read paths (dashboards, high-QPS clients, etc.) — both go through the query cache ([`cache.backend`](/configuration#backends): in-process, or a Redis shared by every instance) with singleflight coalescing. :::note[Admin only] The route is mounted under `/v1/ops/*`, behind the `RequireAdmin` gate: only a caller whose JWT role equals the policy `admin_role` (`"admin"` by default) — or who presents the non-JWT [operator key](#authentication) — may use it. A tokenless request (or a valid token without a role claim) resolves to the `default_role` (not the admin role unless `default_role` is deliberately set to it — a loudly-warned dev-only setting) and is rejected with `403`; a present-but-invalid token — expired, malformed, bad signature — keeps its stashed verification error and fails loud with `401` instead. Raw SQL has no per-statement scope check (a full SQL parser would be needed to authorize predicates), so the role gate is the entire authorization story, shared with the rest of `/v1/ops/*` (see [Admin Endpoints](#admin-endpoints)). The normal surfaces for non-admin callers are `POST /v1/ingest?table={table}` for writes, `POST /v1/query?table={table}` for structured reads, and `GET/POST /v1/pipes/{name}` for pre-defined queries — none of which expose raw SQL. @@ -561,7 +561,7 @@ Table, column, and alias names may contain any characters ClickHouse accepts — **Response:** -JSON array of result rows. Top-level `DateTime`/`DateTime64` values are returned in canonical RFC 3339 UTC (`2026-06-21T04:00:00.123Z`) — `Nullable` timestamp columns included (a SQL `NULL` renders as JSON `null`), while timestamps nested inside `Array`/`Map`/`Tuple` columns are rendered in the column's declared zone, else the ClickHouse server's, as the driver returns them — byte-identical to the [SSE stream](#get-v1stream--server-sent-events-stream) for values [canonicalized at ingest](#timestamp-canonicalization) (a fail-open pass-through that ClickHouse accepted still comes back canonical here, though it streamed in the producer's spelling). The response carries an `X-Cache: HIT` or `X-Cache: MISS` header — this endpoint shares the in-process L1 (Ristretto) + singleflight machinery (unlike `/v1/ops/query`, which always hits ClickHouse), keyed by [tenant](/deployment#multi-tenant-deployments): a request is never served from, or coalesced with, another tenant's. +JSON array of result rows. Top-level `DateTime`/`DateTime64` values are returned in canonical RFC 3339 UTC (`2026-06-21T04:00:00.123Z`) — `Nullable` timestamp columns included (a SQL `NULL` renders as JSON `null`), while timestamps nested inside `Array`/`Map`/`Tuple` columns are rendered in the column's declared zone, else the ClickHouse server's, as the driver returns them — byte-identical to the [SSE stream](#get-v1stream--server-sent-events-stream) for values [canonicalized at ingest](#timestamp-canonicalization) (a fail-open pass-through that ClickHouse accepted still comes back canonical here, though it streamed in the producer's spelling). The response carries an `X-Cache: HIT` or `X-Cache: MISS` header — this endpoint shares the query cache + singleflight machinery (unlike `/v1/ops/query`, which always hits ClickHouse), keyed by [tenant](/deployment#multi-tenant-deployments): a request is never served from, or coalesced with, another tenant's. The inbound request body is capped at 1 MiB; a body over the cap is rejected with `413`. A query AST is bounded by nature (far under 1 MiB even with a large `in`-list), and the cap blocks a single-request memory-exhaustion vector on this public endpoint. Set a tighter or higher outer limit at your [reverse proxy](/reverse-proxy#request-body-size-limits) — but it can only narrow the effective limit, not raise it past this cap. @@ -585,7 +585,7 @@ The inbound request body is capped at 1 MiB; a body over the cap is rejected wit ### `GET/POST /v1/pipes/{name}` — Execute Named Pipe -Executes a pre-defined named query (pipe) with parameter binding. Parameters can be supplied via query string and/or JSON body. Results are cached in the shared L1 (Ristretto) with singleflight coalescing — same machinery as the structured query endpoint, keyed by [tenant](/deployment#multi-tenant-deployments) like it, and again, unlike `/v1/ops/query`. +Executes a pre-defined named query (pipe) with parameter binding. Parameters can be supplied via query string and/or JSON body. Results are cached in the query cache ([`cache.backend`](/configuration#backends): in-process, or a Redis shared by every instance) with singleflight coalescing — same machinery as the structured query endpoint, keyed by [tenant](/deployment#multi-tenant-deployments) like it, and again, unlike `/v1/ops/query`. **Query Parameters:** Any key matching a pipe parameter name. @@ -600,7 +600,7 @@ Executes a pre-defined named query (pipe) with parameter binding. Parameters can **Response:** -JSON array of result rows, with `X-Cache: HIT` or `X-Cache: MISS` indicating whether the row came from the in-process L1. +JSON array of result rows, with `X-Cache: HIT` or `X-Cache: MISS` indicating whether the rows came from the query cache. The POST parameter body is capped at 1 MiB; a body over the cap is rejected with `413` (the same 1 MiB parameter/AST-body cap as [`POST /v1/query`](#post-v1querytabletable--structured-query) — see [reverse proxy → body limits](/reverse-proxy#request-body-size-limits)). A malformed-but-within-cap body is ignored rather than rejected, since parameters may legitimately come from the query string alone. diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 5672ddd16..b1577349e 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -28,7 +28,7 @@ flowchart TD MQ --> BC["Buffer Consumer
(batch flush)"] BC -.->|rejected rows| DLQ["DLQ"]:::fail - QH["Query Handler"] --> Cache["Cache
(Ristretto + singleflight)"] + QH["Query Handler"] --> Cache["Cache
(local or Redis + singleflight)"] SSH["SSE Handler"] --> Hub["Stream Hub
(project once per role)"] @@ -54,7 +54,7 @@ internal/ ├── api/ HTTP layer (Chi router, handlers, middleware, the cached read paths' singleflight) ├── app/ Process wiring: build every component, run them under one errgroup, release in reverse ├── auth/ JWT/JWKS authentication middleware (HMAC or JWKS, role extraction) -├── cache/ Query cache: Ristretto L1 + the tenant-led version index +├── cache/ Query cache: Ristretto L1 + the tenant-led version index; the Redis-compatible shared backend ├── chconn/ One ClickHouse pool per connection tuple among the served tenants, reconciled on reload under the ceiling ├── chsql/ Shared ClickHouse SQL helpers (identifier quoting, bind-safety) ├── config/ YAML + env var configuration loading @@ -93,7 +93,7 @@ The API layer uses [Chi](https://github.com/go-chi/chi) for routing with Request ### `app/` — Process wiring - **app.go** — `New(ctx, Options)` builds every component from the boot config (`Options.Config`) and the settings directory it names, in dependency order: settings registry, observability (after which each `Config.Warnings` line is logged at `WARN`), ClickHouse pools, schema discovery, the dedupe stores, embedded NATS (ingest + DLQ streams), cache, the lease coordinator, sweeper, streaming (hub, MQ→hub bridge, keepalive wheel), ingest worker, auth, reload triggers, HTTP. The boot config's `roles` decide which of them a process wires: every process gets the settings registry, observability, the MQ, the coordinator, the reload triggers and a listener; `api` adds schema discovery, the dedupe stores, streaming, auth and the full router; `ingest` adds the ingest worker; `sweeper` adds the sweeper; the ClickHouse pools and the cache come with `api` or `ingest`. A process without `api` serves `api.NewOpsRouter` (probes, `/version`, the metrics path, and the settings reload behind the operator key alone, `wireOpsAuth`) on `server.port`. `config.Validate` refuses a role set the backends cannot serve (a split over the embedded MQ, or `api` without `ingest` and the reverse over a local cache), and `New` refuses a `Config` with no roles, which only one built without `config.Load` can have. Each is one `component` value — what it opens, what it loops, what it releases — so a failure part-way releases what was already opened and returns the error. `Run(ctx)` drives every loop under one `errgroup` until `ctx` is canceled (a clean stop: every loop drains, the API server and the ingest worker within `server.shutdown_timeout`; open SSE streams are ended as the drain begins rather than waited on) or a component fails, which stops the rest and returns that error. `Close(ctx)` releases what `New` opened, newest first, under the caller's release budget (`ReleaseTimeout`, 5s), a real bound: a remote implementation's close gives up at the deadline itself, and a close that ignores the context (the local stores) is abandoned at it, with the components below it left unreleased rather than overlapping it, both named in the error — and then flushes telemetry under its own 3s budget, so the flush that reports on the stop is never handed a deadline a slow close already spent. The SIGHUP registration is released last of all. `Handler`, `Registry`, and `MQ` expose the pieces a harness needs; `Options.Listener` lets one serve the API on its own listener instead of `server.port`. -- **wire.go** — one `wire*` function per component, each handed the settings registry whole and deriving the per-call getters the internal packages take (`DLQFor`, `DedupeFor`, `GapWindow`, …) and registering its `AfterAdopt` hook there where it has one. A layer with a choice of implementation — `wireMQ`, `wireCache`, `wireDedupe`, `wireCoord` — picks it there and nowhere else, in a `switch` on the boot config's `.backend` with one case per backend; the default case refuses boot, which only a `config.Config` built without `config.Load` reaches, since `Validate` refuses a value no case handles. Those wiring functions are where the per-tenant registry of [#583](https://github.com/Wave-RF/WaveHouse/issues/583) is injected, not `main`: `wireSettings` opens the `settings.Registry`, the HTTP handlers get store-keyed getters (method expressions such as `(*settings.Store).Policy`), and `perTenant` adapts a store accessor into the `func(tenant.ID) T` getter the async packages take, with the tenant each message's `mq.Topic` names for the stream hub and the ingest worker — a tenant the registry is not serving is logged and read as the zero value, except in `dlqFor`, the ingest worker's DLQ switch, where it reads as on so a message the worker cannot read is parked rather than dropped, and a removed or rejected tenant's queued rows are parked rather than left unacked, where each would be redelivered every ack wait for as long as the tenant is away and would hold that tenant's ack floor, so the sweeper could purge none of its queue past it. The ClickHouse pools (`chconn.Pools`) and the per-tenant schema registries (`discoveries`, in `discoveries.go`) are reconciled from `AfterAdopt` after every reload ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 6): `wireClickHouse` builds each served tenant's `chconn.Member` from its store and logs what the reconcile refused; `wireDiscovery` builds a registry over `pools.For` for each newly served tenant — a flat directory's tenant `0` refreshed synchronously first, as before — runs its loop under the App's stop context, stops the loop of a tenant no longer served, and drives the `BootState` from the first tenant's first discovery, sticky from there; before that, a diagnostic naming a tenant a reload stopped serving goes back to the no-tenant one. The handlers resolve both per request through store-keyed getters (`chConnFor`, `registryFor`, `chTargetFor`, `queryTimeout`), the hub and the ingest worker through tenant-keyed ones (`discoveries.For`, `pools.Target`) called with the tenant the message's topic names; a tenant on no pool is an untyped nil connection, the handlers' `503`. The ingest worker is handed the cache through `sharedTables`, which bumps each namespace the worker invalidates under every tenant on the same ClickHouse address and database (`pools.SharingTables`), and the pools hook orphans the whole cache — structured-query and pipe results — of a tenant back on a pool after an absence (`Cache.InvalidateTenant`), since it was out of that fan-out while away, and of a tenant moved to another address or database, since it now reads other tables (both returned by `Pools.Reconcile`). The one setting that still follows the default tenant is read per request, the admin role of a flat directory's ops gate: `defaultPolicy` reads it through the registry, where a flat directory's tenant `0` is always served. The auth verifiers are per tenant: `wireAuth` builds one for each tenant being served, its `AfterAdopt` hook reconfigures the adopted tenants' (rebuilt only when their wiring changed) and prunes the ones no longer served, and the operator key's admin role is read from the request tenant's policy. `wireStreaming`'s hook prunes the stream hub the same way (`Hub.Prune`, with the one `served` predicate the auth and dedupe hooks use too), ending the open streams of a tenant no longer served. One setting is shared by folding over the tenants being served rather than by following tenant `0`: the keepalive wheel runs at the shortest `stream.keepalive_interval` among them (`shortestKeepalive`), re-derived after every reload the registry applies — an adoption, a rejection, or a removal — so a dropped tenant's interval leaves the wheel at once ([#597](https://github.com/Wave-RF/WaveHouse/issues/597)). The sweeper is handed each tenant's own `stream.gap_window_minutes` (`gapWindows`, read every sweep over `Registry.Known`, so a rejected tenant keeps the window its folder last had, and all of its history if the folder has been rejected since boot), since each tenant's events have a queue of their own, and runs only while its process holds the `sweeper` lease (`elected`, which wraps `coord.RunElected` over the coordinator `wireCoord` opens; `local`, the only `coord.backend`, keeps leases in the process, so the one process always holds it). The dedupe stores are per tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 7): `wireDedupe`'s `pebble` case builds a `dedupe.Stores` over the `Tenant` factory of the embedded Pebble implementation (`dedupe.NewEmbedded`), handing it `data_dir` once; the implementation decides where every tenant's store lives — one instance, each key led by its tenant (story 3) — and one reconcile closure, the boot apply and the `AfterAdopt` hook alike, sets every store to what the registry says: open exactly when its tenant is served with `dedupe.enabled` on, closed with its seen ids kept when the tenant is switched off, rejected, or removed. An instance that cannot open follows the registry's rule for the shape: fatal at boot over a flat directory, fail-closed for every tenant with dedupe on over a nested one. The system gauges report that one instance's figures (`Embedded.Stats`), not a sum over tenants. The ingest handler picks the tenant's store off the request's `settings.Store` (`Store.Tenant()`). The reload triggers only start in `Run`, after `New` has registered every hook, so the watcher's first reload already drives all of them: SIGHUP in both shapes, the directory watcher for a flat directory only. `wireMQ`'s `embedded` case hands each served tenant's `mq.max_bytes_gb` to `mq.Broker.SetMaxBytes` at boot, under `New`'s context (so a stop signaled mid-boot is not held up by opening many queues), and again after every reload, under the App's stop context; the first apply opens that tenant's queue. A queue that cannot be opened or resized follows the registry's rule for the shape — fatal at boot over a flat directory, logged over a nested one — and is retried by the next reload, a queue that did not open by the next publish too. How the budget is split across the tenant's streams, the time bounds, the rollback, and the dead-letter shrink guard are `internal/mq`'s. +- **wire.go** — one `wire*` function per component, each handed the settings registry whole and deriving the per-call getters the internal packages take (`DLQFor`, `DedupeFor`, `GapWindow`, …) and registering its `AfterAdopt` hook there where it has one. A layer with a choice of implementation — `wireMQ`, `wireCache`, `wireDedupe`, `wireCoord` — picks it there and nowhere else, in a `switch` on the boot config's `.backend` with one case per backend; the default case refuses boot, which only a `config.Config` built without `config.Load` reaches, since `Validate` refuses a value no case handles. `wireCache` has two: `local`, the in-process `LocalCache`, and `redis`, the shared `RedisCache` built from the `cache.redis` block, which boots bypassed rather than failing when its server is unreachable. Those wiring functions are where the per-tenant registry of [#583](https://github.com/Wave-RF/WaveHouse/issues/583) is injected, not `main`: `wireSettings` opens the `settings.Registry`, the HTTP handlers get store-keyed getters (method expressions such as `(*settings.Store).Policy`), and `perTenant` adapts a store accessor into the `func(tenant.ID) T` getter the async packages take, with the tenant each message's `mq.Topic` names for the stream hub and the ingest worker — a tenant the registry is not serving is logged and read as the zero value, except in `dlqFor`, the ingest worker's DLQ switch, where it reads as on so a message the worker cannot read is parked rather than dropped, and a removed or rejected tenant's queued rows are parked rather than left unacked, where each would be redelivered every ack wait for as long as the tenant is away and would hold that tenant's ack floor, so the sweeper could purge none of its queue past it. The ClickHouse pools (`chconn.Pools`) and the per-tenant schema registries (`discoveries`, in `discoveries.go`) are reconciled from `AfterAdopt` after every reload ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 6): `wireClickHouse` builds each served tenant's `chconn.Member` from its store and logs what the reconcile refused; `wireDiscovery` builds a registry over `pools.For` for each newly served tenant — a flat directory's tenant `0` refreshed synchronously first, as before — runs its loop under the App's stop context, stops the loop of a tenant no longer served, and drives the `BootState` from the first tenant's first discovery, sticky from there; before that, a diagnostic naming a tenant a reload stopped serving goes back to the no-tenant one. The handlers resolve both per request through store-keyed getters (`chConnFor`, `registryFor`, `chTargetFor`, `queryTimeout`), the hub and the ingest worker through tenant-keyed ones (`discoveries.For`, `pools.Target`) called with the tenant the message's topic names; a tenant on no pool is an untyped nil connection, the handlers' `503`. The ingest worker is handed the cache through `sharedTables`, which bumps each namespace the worker invalidates under every tenant on the same ClickHouse address and database (`pools.SharingTables`), and the pools hook orphans the whole cache — structured-query and pipe results — of a tenant back on a pool after an absence (`Cache.InvalidateTenant`), since it was out of that fan-out while away, and of a tenant moved to another address or database, since it now reads other tables (both returned by `Pools.Reconcile`). The one setting that still follows the default tenant is read per request, the admin role of a flat directory's ops gate: `defaultPolicy` reads it through the registry, where a flat directory's tenant `0` is always served. The auth verifiers are per tenant: `wireAuth` builds one for each tenant being served, its `AfterAdopt` hook reconfigures the adopted tenants' (rebuilt only when their wiring changed) and prunes the ones no longer served, and the operator key's admin role is read from the request tenant's policy. `wireStreaming`'s hook prunes the stream hub the same way (`Hub.Prune`, with the one `served` predicate the auth, dedupe and cache hooks use too), ending the open streams of a tenant no longer served, and `wireCache`'s hook prunes the cache's version index the same way (`LocalCache.Prune`, [#262](https://github.com/Wave-RF/WaveHouse/issues/262)), so a tenant no longer served stops holding it. One setting is shared by folding over the tenants being served rather than by following tenant `0`: the keepalive wheel runs at the shortest `stream.keepalive_interval` among them (`shortestKeepalive`), re-derived after every reload the registry applies — an adoption, a rejection, or a removal — so a dropped tenant's interval leaves the wheel at once ([#597](https://github.com/Wave-RF/WaveHouse/issues/597)). The sweeper is handed each tenant's own `stream.gap_window_minutes` (`gapWindows`, read every sweep over `Registry.Known`, so a rejected tenant keeps the window its folder last had, and all of its history if the folder has been rejected since boot), since each tenant's events have a queue of their own, and runs only while its process holds the `sweeper` lease (`elected`, which wraps `coord.RunElected` over the coordinator `wireCoord` opens; `local`, the only `coord.backend`, keeps leases in the process, so the one process always holds it). The dedupe stores are per tenant ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 7): `wireDedupe`'s `pebble` case builds a `dedupe.Stores` over the `Tenant` factory of the embedded Pebble implementation (`dedupe.NewEmbedded`), handing it `data_dir` once; the implementation decides where every tenant's store lives — one instance, each key led by its tenant (story 3) — and one reconcile closure, the boot apply and the `AfterAdopt` hook alike, sets every store to what the registry says: open exactly when its tenant is served with `dedupe.enabled` on, closed with its seen ids kept when the tenant is switched off, rejected, or removed. An instance that cannot open follows the registry's rule for the shape: fatal at boot over a flat directory, fail-closed for every tenant with dedupe on over a nested one. The system gauges report that one instance's figures (`Embedded.Stats`), not a sum over tenants. The ingest handler picks the tenant's store off the request's `settings.Store` (`Store.Tenant()`). The reload triggers only start in `Run`, after `New` has registered every hook, so the watcher's first reload already drives all of them: SIGHUP in both shapes, the directory watcher for a flat directory only. `wireMQ`'s `embedded` case hands each served tenant's `mq.max_bytes_gb` to `mq.Broker.SetMaxBytes` at boot, under `New`'s context (so a stop signaled mid-boot is not held up by opening many queues), and again after every reload, under the App's stop context; the first apply opens that tenant's queue. A queue that cannot be opened or resized follows the registry's rule for the shape — fatal at boot over a flat directory, logged over a nested one — and is retried by the next reload, a queue that did not open by the next publish too. How the budget is split across the tenant's streams, the time bounds, the rollback, and the dead-letter shrink guard are `internal/mq`'s. ### `stream/` — SSE keepalive & fan-out @@ -114,14 +114,20 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ - **cache.go** — `Cache` interface: `Lookup`, `Set`, `Invalidate`, `InvalidateTenant`, `Close`, plus `QueryTimeToTTL`, which sets a result's TTL from how long its query took (10 s floor, 1 h ceiling). Every entry is one tenant's: `Lookup` takes the tenant, the caller's query key — `:query:`, built by the two cached handlers in `api/` (`queryCacheKey`, with the tenant read off the request's store — `settings.Store.Tenant`), which use it as their [singleflight](https://pkg.go.dev/golang.org/x/sync/singleflight) key too — and the `Namespace`s the result depends on — one for a structured query, none yet for a pipe (a pipe's table dependencies are [#343](https://github.com/Wave-RF/WaveHouse/pull/343)) — each of that tenant (another tenant's is `ErrForeignDependency`) and naming a table and scope by their raw names, which the cache escapes where it builds a key. `Lookup` returns the `Entry` (a nil value is a miss) and a `Snapshot` of the versions it read; on a miss the handler runs the query and passes that snapshot to `Set`, so a result is filed under the versions read *before* its query ran, and a write that lands while it runs orphans the fill rather than re-homing pre-write rows under the post-write versions ([#382](https://github.com/Wave-RF/WaveHouse/issues/382)). The snapshot is taken before any input a bump invalidates is chosen, the tenant's connection included: a reload that moves the tenant to another address or database runs `Pools.Reconcile` and then `InvalidateTenant` (a repoint that keeps both, such as a username or `tls` change, reads the same tables and bumps nothing), so a request that took the old pool files the old database's rows under a version that bump orphans, whether its `Set` lands before the bump or after. The singleflight leader's snapshot is the one used. `Set` returns an error only when the backend failed; a value the cache declines — larger than it keeps, a non-positive TTL, a zero snapshot — is not one. What a hit, a miss and a bump mean is pinned by the conformance suite every backend runs, `internal/testutil/cachetest`. - **local.go** — `LocalCache`, the in-process L1 on [Ristretto](https://github.com/dgraph-io/ristretto): one pool shared by every tenant (a heavier tenant holds more of it), sized by the boot config's `cache.l1_max_cost`. -- **version_manager.go** — `VersionManager`, the invalidation index behind `Invalidate` and `InvalidateTenant`: a namespace key is `..
.
.`, its fields joined with `keyenc.Join` so a dot in a table or scope name is escaped rather than read as a separator, and an entry's key, `|.||…` with the caller's query key escaped whole (its `:` become `%3A`), folds in the tenant's version and each dependency's namespace key and namespace version, so bumping a table (a scopeless write) or one scope — scope is reserved and empty today, so every write is the whole-table bump — orphans every dependent entry without touching the pool. The tenant leads every key ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8): the same table under two tenants is two namespaces, so a bump through `Invalidate` under one tenant never touches — and a read under one tenant is never served — the other's results, and the flat directory's single tenant simply carries the `0` prefix. `BumpTenant` (behind `InvalidateTenant`) advances the tenant version that leads every namespace key of one tenant, orphaning its every namespace and every cached query in one step (the tenant version is folded into every entry's key, so a pipe result with no dependencies is orphaned too) — a table no bump ever keyed included, which an enumeration of the index would miss — for a tenant back on a pool after an absence from the fan-out, or moved to another address or database (story 6). The index is per tenant; the cross-tenant invalidation an insert into a shared table needs is not the index's but the wiring's: `internal/app` hands the ingest worker a cache (`sharedTables`) that repeats each bump under every tenant on the same ClickHouse address and database. +- **version_manager.go** — `VersionManager`, the invalidation index behind `Invalidate` and `InvalidateTenant`: one version per tenant, per (tenant, table) and per (tenant, table, scope), each keyed by its name alone and bumped in place, so the index holds one entry per live tenant, table and scope however often each is bumped ([#262](https://github.com/Wave-RF/WaveHouse/issues/262)). A query key folds the tenant's version and, for each dependency, its tenant's, table's and scope's, so bumping a table (a scopeless write) orphans every scope of it, and bumping one scope orphans that scope and the whole-table view — scope is reserved and empty today, so every write is the whole-table bump — all without touching the pool. Every field — the caller's query key, the tenant id, and each dependency's table and scope — is escaped and joined by `internal/keyenc` where the key is built, so a dot, a space or a `%` in a name is never read as a separator: each dependency renders as `..
.
..`, and the whole entry key is `|.||…`. The tenant leads every key ([#583](https://github.com/Wave-RF/WaveHouse/issues/583) story 8): the same table under two tenants is two namespaces, so a bump through `Invalidate` under one tenant never touches — and a read under one tenant is never served — the other's results. A tenant's version is a *generation*, unique within the process and handed out by the first key built for the tenant; `BumpTenant` (behind `InvalidateTenant`) drops the tenant's whole index, so the next key gets a fresh generation no cached entry folds, orphaning every cached result of the tenant in one step — a pipe result with no dependencies, and a table no bump ever keyed, included — for a tenant back on a pool after an absence from the fan-out, or moved to another address or database (story 6). `LocalCache.Prune` does the same for every tenant no longer served, which `internal/app` runs after each settings reload, so a tenant removed or rejected stops holding its index. A table bump drops the table's scope versions with it, since every key they were folded into also folds the old table version; and any bump (of a table, a scope or the tenant) under a tenant with no index is a no-op that records nothing, since the next key built for it gets a fresh generation no cached entry folds — so neither the `sharedTables` fan-out nor an insert still in flight for a tenant just pruned brings its index back. The index is per tenant; the cross-tenant invalidation an insert into a shared table needs is not the index's but the wiring's: `internal/app` hands the ingest worker a cache (`sharedTables`) that repeats each bump under every tenant on the same ClickHouse address and database. +- **redis.go** — `RedisCache`, the shared backend: one Redis-compatible server (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB — only `GET`, `SET` and `MGET`, no scripts, no client tracking) holds every process's results and versions, so a bump one process makes orphans what every process cached. Versions are random 8-byte tokens in a flat key space, one per tenant (`:{t}:T`), table (`:B:
`) and scope (`:S:
:`, the empty scope being the whole-table view), all under the tenant's hash tag so they share one cluster slot; the table and scope are escaped with `keyenc` after the fixed prefix, so a `:` in a name is never read as the separator. A dependency folds three: the tenant's, its table's and its scope's; a scopeless write bumps `B:
`, a scoped one `S:
:` and `S:
:`, `InvalidateTenant` the tenant's — the lattice `VersionManager` encodes. `Lookup` pipelines an `MGET` of the tokens with a `GET` of the value in one round trip; the value (`:q::`, the hash over their escaped forms, no hash tag, so a tenant's values spread across shards) carries the tokens it was filed under and is a hit only while they are all current. A missing token is created (`SET NX`) and read back, never read as a value, and a token key holding anything but a token (a string of another length, a list, a hash) is replaced, so a token lost to eviction, expiry, `FLUSHALL` or a restart without persistence is a miss for everything under it, never a revival — any `maxmemory-policy` that evicts is safe. Restoring an RDB or AOF snapshot, or a backup, is not a loss but a rollback — a restart after a crash that reloads the server's last save included, which stock Redis and Valkey make by default: the old tokens come back with the values filed under them, so what was invalidated since is served again until its TTL, as after a failover to a replica that missed the bumps. A failure or a timeout past `Timeout` (100 ms) is a miss, a skipped fill and a deferred invalidation; queries never fail on the cache. While this process owes a bump on any of a lookup's tokens, that lookup is a bypass that files nothing: the bump would orphan whatever it found (other processes, which cannot know, serve those entries until it lands). `NewRedis` never fails on an unreachable server: the cache starts bypassed and keeps dialing, each attempt bounded by `DialTimeout` (1 s), and a cluster client's topology read after the handshake by the larger of it and `Timeout`, which is also how long a connection waits on a silent server before it is redialed. Every connection is replaced after a minute (`clientOption`; rueidis retries what was in flight), because one that outlives a failover behind a stable address stays on the demoted node, which answers but refuses writes; the replacement re-resolves the address, so a bypassed process reaches the new primary, and delivers the bumps it owes, within about that long. rueidis dials a replacement lazily, under the context of the operation that lands on it, bounding the dial (TLS included) by `DialTimeout` and then the handshake by `DialTimeout` again, so a reconnect slower than `Timeout` fails that operation. The breaker's probe gives its write twice `DialTimeout` for a reconnect on top of `Timeout`, so such a reconnect still closes an open breaker. The allowance is for a reconnect only: a probe write slower than `Timeout` is repeated under `Timeout`, and the repeat decides, so a server answering slower than `Timeout` stays bypassed. But rueidis spreads commands over several connections (up to four to one server, by `GOMAXPROCS`, and one per cluster node) and the probe reconnects only the one it lands on, so size `Timeout` above a reconnect, or operations that land on the others keep failing. `cache.backend: redis` selects it: `internal/app`'s `wireCache` maps the boot config's `cache.redis` block onto `RedisConfig`, reading the TLS files, and releases it with the other components. +- **redis_codec.go** — the key schema (`tokenKeys`, `bumpKeys`, `valueKey`, which take a `Namespace`'s raw names and escape them) and the value frame: format, flags, expiry, the token list, the payload, zstd-compressed from `CompressMinBytes` when that is smaller. A stored value is capped at `MaxValueBytes` and a decoded one at eight times that, which refuses a zip bomb planted in a shared server; an unknown format, as a newer process writes during a rolling upgrade, is a miss. +- **breaker.go** — the circuit breaker's state machine: `BreakerThreshold` (5) consecutive failures open it, `trip` opens it at once, and while it is open every operation skips the server; once `BreakerOpenFor` (5 s) has passed, `allow` hands one caller the probe, and only the probe's success closes it. What feeds it is `redis.go`'s. `record` counts a transport failure or a timeout against the server, and trips the breaker on an error reply that means no bump can land: one saying the server takes no writes right now, as `refusesWork` lists them — `READONLY` (a demoted primary), `MASTERDOWN`, `OOM` (full under `maxmemory-policy noeviction`, Redis's default, so give the cache an evicting policy), `NOREPLICAS`, `MISCONF`, `LOADING`, `BUSY`, `CLUSTERDOWN` — or one refusing the credentials, as `rejectsCredentials` lists them — `WRONGPASS`, `NOAUTH`, what a new connection's handshake meets after a password rotation. Each opening is logged once, at `ERROR` for the credentials and `WARN` otherwise, not once per operation in flight; a failed probe opens it afresh, so a server that stays down logs once every `BreakerOpenFor`. Any other reply, an error reply about one key or command (`WRONGTYPE`, `NOPERM`, `TRYAGAIN`) or one the backend cannot use included, counts as a success, and a caller that gave up first counts as nothing. The probe (`probe`) is a write, `SET :probe`, so a server that answers but refuses writes stays bypassed. +- **pending.go** — the invalidations owed (`pendingBumps`): kept per token key, repeats coalescing, and past `PendingMax` (100,000) keys collapsed to one tenant bump per affected tenant; `owesAny` tells `Lookup` which lookups to hold. `redis.go` delivers them: `Invalidate` sends its bumps 1,000 to a round trip and defers those the server does not take, and `drainLoop` retries them through `drain` until they land — the first one owed at once, then with backoff from 100 ms to 10 s, and while the breaker is open at each probe, which the loop starts when due, so a process that makes no lookups (ingest only) recovers as soon as one that does. Landing late is still correct: a fresh token orphans the pre-write entries and any fill made meanwhile. `Close` makes one last attempt at the bumps still owed, past the breaker (an open one is why they are owed); what that attempt cannot deliver is lost, and the entries they would orphan are served until their TTL — the same failure as a worker stopping between an insert and its invalidation. +- **metrics.go** — the shared backend's instruments (meter `wavehouse-cache`, `backend="redis"`, no tenant attribute): `wavehouse_cache_lookups_total` by `result` (`hit`, `miss`, `stale` — filed under since-bumped tokens, `bypass` — server skipped, or held by a bump this process owes, `error`), `wavehouse_cache_op_duration_seconds` by `op` (`lookup`, `set`, `invalidate`), `wavehouse_cache_breaker_open` (1 while bypassed, including before the first connection), `wavehouse_cache_invalidations_total` by `result` (`ok` counts every bump that lands, retried ones included, and `deferred` each bump an invalidation could not deliver when made, a repeat of one already owed included; a failed retry is not counted again, so the two overlap), `wavehouse_cache_invalidations_pending` (bumps owed: entries they would orphan may be served stale meanwhile), `wavehouse_cache_value_bytes` (stored size), `wavehouse_cache_oversize_total` and `wavehouse_cache_set_failures_total` by `reason` (`oom`, `timeout`, `other`). - **cachetest** (`internal/testutil/cachetest`) — `Run(t, factory, Options)`, the conformance suite every `Cache` backend runs, each case on a fresh cache from the factory: what a hit, a miss and each kind of bump mean, independent of where entries and versions live. `Options` describes what a backend can do beyond the `Cache` contract, and leaving one unset skips the cases it enables: `MaxValueBytes` opens the oversize case; `NewPair` returns two instances over one shared store, for the cross-instance cases a shared backend must run; `Entries` counts a cache's entries, and without it the zero-snapshot case — no `Lookup` reads the key a zero `Snapshot` would land under, so only a count shows that one stored nothing — is skipped. ### `config/` — Configuration -- **config.go** — Loads *boot* configuration from a YAML file with environment variable overrides (using [cleanenv](https://github.com/ilyakaznacheev/cleanenv)); every key has a `WH_`-prefixed env var. Boot config is only what can't change under a running process — the implementation each layer runs on, the process's `roles`, resource sizing, listeners, observability exporters, the settings-directory path, and the secrets (`clickhouse.password`, `auth.jwt_secret`, `auth.operator_key`). Everything tenant-tunable lives in the settings directory (`settings/`). Both sources are strict: `Load` refuses to boot naming every YAML key the struct doesn't declare (`strict.go`) and every `WH_*` environment variable no field binds (`check.go`), so a tunable that moved to the settings directory can't be read, ignored, and believed. Boot is the validator for this half — there is no dry-run command. See [Configuration Reference](/configuration). +- **config.go** — Loads *boot* configuration from a YAML file with environment variable overrides (using [cleanenv](https://github.com/ilyakaznacheev/cleanenv)); every key has a `WH_`-prefixed env var. Boot config is only what can't change under a running process — the implementation each layer runs on, the process's `roles`, resource sizing, listeners, observability exporters, the settings-directory path, and the secrets (`clickhouse.password`, `cache.redis.password`, `auth.jwt_secret`, `auth.operator_key`). Everything tenant-tunable lives in the settings directory (`settings/`). Both sources are strict: `Load` refuses to boot naming every YAML key the struct doesn't declare (`strict.go`) and every `WH_*` environment variable no field binds (`check.go`), so a tunable that moved to the settings directory can't be read, ignored, and believed. Boot is the validator for this half — there is no dry-run command. See [Configuration Reference](/configuration). - **check.go** — `rejectUnboundEnv` is the environment half of the strict loader: `unboundEnv` walks the struct's `env` tags (plus the two process-level names, `WH_CONFIG` and `WH_LOG_LEVEL`) against the environment; `CheckDataDir` probes `data_dir` — run by `main` right after `Load` when `Config.NeedsDataDir` says a selected backend keeps state there, so an unusable `data_dir` refuses boot before anything dials out. It refuses an empty value (reachable through `WH_DATA_DIR=`) outright rather than probing the working directory; a path that exists and is not a directory; a dangling symlink at `data_dir` or any component above it (the walk to the nearest existing ancestor uses `Lstat`, so a failed mount is not skipped over as "does not exist"); and a directory the process cannot write to — or, when it does not exist, an unwritable nearest ancestor — probed by creating and removing one temp file. A permission denial, on the probe or on reaching the path through a parent without search permission, carries the UID-65532 hint, since a bind mount owned by root is the typical cause. -- **backends.go** — the `.backend` keys: one string type per layer (`MQBackend`, `CacheBackend`, `DedupeBackend`, `CoordBackend`), each with its list of the backends this build has, and a `validate` per layer block (`checkBackend`, run by `validateBackends` in `Validate`, between `validateRoles` and `validateTopology`), which refuses a value not on the list and names the ones that are. A backend's own settings go in a `.` sub-block that is its `validate`'s case to check. `Distributed` reports whether the MQ is shared with other processes, `NeedsDataDir` whether a selected backend keeps state under `data_dir`, and `Warnings` returns the valid combinations that are correct for one replica only (a shared MQ over a local cache or Pebble dedupe), which `app.New` logs at `WARN`. +- **backends.go** — the `.backend` keys: one string type per layer (`MQBackend`, `CacheBackend`, `DedupeBackend`, `CoordBackend`), each with its list of the backends this build has, and a `validate` per layer block (`checkBackend`, run by `validateBackends` in `Validate`, between `validateRoles` and `validateTopology`), which refuses a value not on the list and names the ones that are. A backend's own settings go in a `.` sub-block that is its `validate`'s case to check. `Distributed` reports whether the MQ is shared with other processes, `NeedsDataDir` whether a selected backend keeps state under `data_dir`, and `Warnings` returns what a valid configuration is still likely to get wrong — the combinations that are correct for one replica only (a shared MQ over a local cache or Pebble dedupe), a `cache.redis` block that is not read, certificate verification turned off — which `app.New` logs at `WARN`. +- **cache_redis.go** — `CacheRedisConfig`, the `cache.redis` sub-block, and its checks: an address (exactly one in `standalone` mode, which dials only the first), each `host:port` with a port from 1 to 65535 (a URL or `user:password@` form refused without repeating it, since it may hold a password), a known mode (`sentinel` is refused until [#656](https://github.com/Wave-RF/WaveHouse/issues/656)), `db` 0 in cluster mode, positive timeouts and sizes, a `timeout` and `dial_timeout` of at most 1 s each (boot and `Close` each wait out a dial: a connect and a handshake bounded by `dial_timeout`, and a cluster's topology read bounded by the larger of the two), a `version_ttl` of at least 2 s, and a `compress_min_bytes` that is not negative (`0` never compresses); its defaults are in `defaults()` with the rest. `CacheRedisTLS.Config` builds the `tls.Config`, reading the files; `Validate` calls it so an unreadable file refuses boot, and `internal/app` calls it again to build the connection. A TLS key set while `tls.enabled` is off is an error rather than a plaintext connection. - **config.go**, roles — `roles` (`[]Role`: `api`, `ingest`, `sweeper`; `AllRoles` by default; `Has(Role)`) picks which components `internal/app` wires, and `instance_id` names the process (`-<8 hex>` when empty, resolved in `Load`; today only logged at boot, and a distributed coordinator will record it as a lease's holder). `validateRoles` refuses an empty list, an empty entry, an unknown or a repeated role; `validateTopology` refuses a role set the backends cannot serve: any split over the embedded MQ, and a process with exactly one of `api` and `ingest` over a local cache. `NeedsDataDir` counts Pebble only for a process running `api`, and `Warnings` is empty without `api`, since only that role opens a cache it reads or a dedupe store. - **strict.go** — `rejectUnknownKeys`, the YAML half: re-reads the file as a generic tree and walks it against the struct's `yaml` tags, listing every key the struct doesn't declare. cleanenv itself is lenient by design, which is exactly wrong for boot config once keys have moved to the settings directory. - **persistence.go** — `WarnIfFreshDataDir` logs the startup `WARN` when `data_dir` is missing or empty (on a redeploy, the sign that the volume didn't persist); `LogStorageInitError` attaches the UID-65532 `permissionHint` to a NATS or Pebble open failure that looks like a permission denial — the same hint string `CheckDataDir` uses. @@ -219,7 +225,7 @@ The hot-reloadable half of configuration: a directory of four JSON files (`confi ### `keyenc/` — Key Escaping -- **keyenc.go** — The one escaping composite keys are built from, so a name can never be mistaken for a separator: `Escape` keeps ASCII letters, digits, `_` and `-` — exactly the tenant-id grammar, so a tenant id is its own escaped form — and writes every other byte as `%XX` (uppercase hex); `Unescape` is `url.PathUnescape`, which decodes `%XX` in either case and takes any other byte as itself, so a `%2D` for `-` that an earlier build wrote still reads. `Join`/`AppendJoin` escape each field and put a separator between them, panicking on no fields and on a separator the escaping could write or one outside ASCII, and `Split` reverses them. The package that builds a key takes raw names and escapes them itself, so no caller has to: NATS subject tokens (`internal/mq`) and the cache's keys (`internal/cache`) both use it. Keys built from it are stored, so changing what it keeps orphans them. +- **keyenc.go** — The one escaping composite keys are built from, so a name can never be mistaken for a separator: `Escape` keeps ASCII letters, digits, `_` and `-` — exactly the tenant-id grammar, so a tenant id is its own escaped form — and writes every other byte as `%XX` (uppercase hex); `Unescape` is `url.PathUnescape`, which decodes `%XX` in either case and takes any other byte as itself, so a `%2D` for `-` that an earlier build wrote still reads. `Join`/`AppendJoin` escape each field and put a separator between them, panicking on no fields and on a separator the escaping could write or one outside ASCII, and `Split` reverses them. The package that builds a key takes raw names and escapes them itself, so no caller has to: NATS subject tokens (`internal/mq`) and the cache's keys — the version index and the shared backend's Redis keys (`internal/cache`) — use it. Keys built from it are stored, so changing what it keeps orphans them — and on the shared backend, whose keys every process builds for itself, it splits them for the length of a rolling upgrade: a bump one build makes does not reach the entries the other build filed, which are served until their TTL. ## Data Flows @@ -388,6 +394,7 @@ Client GET /v1/stream | Analytics DB | ClickHouse | Primary data store + schema source of truth | | Message Queue | NATS + JetStream | Durable event streaming | | L1 Cache | Ristretto v2 | In-process memory cache | +| Shared Cache | [rueidis](https://github.com/redis/rueidis) | Redis-compatible client for the shared backend (`cache.backend: redis`) | | Embedded KV | Pebble | Optional deduplication | | Config | cleanenv | YAML + env var config loading | | Release | GoReleaser | Cross-platform binary builds | diff --git a/docs/src/content/docs/configuration.mdx b/docs/src/content/docs/configuration.mdx index 5fac488b7..dab012d38 100644 --- a/docs/src/content/docs/configuration.mdx +++ b/docs/src/content/docs/configuration.mdx @@ -39,16 +39,16 @@ This page is boot config only — what the platform operator owns (wiring, lifec ### Backends -Each layer's implementation is chosen once, at boot. Today every layer has one backend, the in-process one, and it is the default, so a config that sets none of these keys runs as it always has. A value this build has no backend for refuses boot and names the valid ones. +Each layer's implementation is chosen once, at boot. Every layer defaults to its in-process backend, so a config that sets none of these keys runs as it always has; the cache also has a shared one, `redis`. A value this build has no backend for refuses boot and names the valid ones. | YAML Key | Env Var | Default | Description | | --- | --- | ------- | ----------- | | `mq.backend` | `WH_MQ_BACKEND` | `embedded` | The message queue. `embedded`: NATS JetStream inside this process, under `/nats`. It listens on no port, so no other process can reach its queue. | -| `cache.backend` | `WH_CACHE_BACKEND` | `local` | The query-result cache. `local`: in this process, sized by `cache.l1_max_cost`. | +| `cache.backend` | `WH_CACHE_BACKEND` | `local` | The query-result cache. `local`: in this process, sized by `cache.l1_max_cost`. `redis`: one Redis-compatible server shared by every process, configured by [`cache.redis`](#cache), so an insert one process makes invalidates what every process has cached. | | `dedupe.backend` | `WH_DEDUPE_BACKEND` | `pebble` | Where ingest dedupe keeps the event ids it has seen. `pebble`: in this process, under `/pebble`, open while any tenant has dedupe on. | | `coord.backend` | `WH_COORD_BACKEND` | `local` | Where the leases for work only one process may do at a time, such as the sweeper, are held. `local`: in this process, so the one process always holds them. It shares nothing with another process, so every process runs its own sweeper. | -Settings for one backend will go in a sub-block named after it, `.`, read only when that backend is selected. No backend has settings yet, so today any such sub-block, `mq.embedded` included, is an unknown key and refuses boot. `mq` and `dedupe` also appear in the settings directory's `config.json`, with different keys (`mq.max_bytes_gb`, `dedupe.enabled`, …): those are per-tenant tunables and stay there, and one written in `config.yaml` refuses boot as an unknown key. +Settings for one backend go in a sub-block named after it, `.`, read only when that backend is selected. `cache.redis` is the only one so far; any other, `mq.embedded` included, is an unknown key and refuses boot. A `cache.redis.addrs` set while `cache.backend` is `local` is logged at `WARN` at boot, since the block is not read. `mq` and `dedupe` also appear in the settings directory's `config.json`, with different keys (`mq.max_bytes_gb`, `dedupe.enabled`, …): those are per-tenant tunables and stay there, and one written in `config.yaml` refuses boot as an unknown key. ### Process roles @@ -70,7 +70,7 @@ Every process, whatever its roles, reads the settings directory and reloads it ( Boot refuses a role set the selected backends cannot serve: - **Any split with `mq.backend=embedded`.** The embedded queue lives inside its process and listens on no port, so a process without every role could not reach it. Until a shared `mq.backend` exists, every process runs every role. -- **`api` without `ingest`, or `ingest` without `api`, with `cache.backend=local`.** The ingest worker invalidates the cache the API reads, and a local cache in another process never sees that invalidation. Run `api` and `ingest` together, or choose a shared `cache.backend`. A `sweeper`-only process holds no cache, so this rule does not apply to it. +- **`api` without `ingest`, or `ingest` without `api`, with `cache.backend=local`.** The ingest worker invalidates the cache the API reads, and a local cache in another process never sees that invalidation. Run `api` and `ingest` together, or set [`cache.backend: redis`](#backends), one cache every process shares. A `sweeper`-only process holds no cache, so this rule does not apply to it. ### Server @@ -144,7 +144,33 @@ Each tenant's queue has its own disk budget, `mq.max_bytes_gb`, a hot-reloadable | YAML Key | Env Var | Default | Description | | --- | --- | ------- | ----------- | -| `cache.l1_max_cost` | `WH_CACHE_L1_MAX_COST` | `67108864` | Maximum L1 cache size in bytes (~64 MB). The time-range bucket structured queries normalize to is `query.timestamp_bucket_seconds` in the [Settings Directory](/settings-directory#configjson-keys). | +| `cache.l1_max_cost` | `WH_CACHE_L1_MAX_COST` | `67108864` | Maximum size in bytes (~64 MB) of the `local` backend's in-process cache. The time-range bucket structured queries normalize to is `query.timestamp_bucket_seconds` in the [Settings Directory](/settings-directory#configjson-keys). | + +The `redis` backend's settings, read only when `cache.backend` is `redis`. It is tested on Redis, Valkey, Dragonfly and a Redis Cluster node, and its data commands are only `GET`, `SET` and `MGET` (no scripts, no client tracking), which ElastiCache and MemoryDB also serve. An ACL user also needs the connection commands the client sends when it dials: `HELLO`, `CLIENT`, `SELECT`, and `CLUSTER` in cluster mode; without them it is refused (`NOPERM`) and the cache stays bypassed. Whoever can write to the server can replace cached query results, which are served after the access policy has already been applied, so treat the server as part of WaveHouse's trust boundary (see [Deployment](/deployment#multiple-instances-and-the-shared-cache)). [Deployment](/deployment#multiple-instances-and-the-shared-cache) covers sizing, `maxmemory-policy` and what a reader on another instance can see. + +| YAML Key | Env Var | Default | Description | +| --- | --- | ------- | ----------- | +| `cache.redis.addrs` | `WH_CACHE_REDIS_ADDRS` | *(empty)* | **Required** with `backend: redis`. `host:port` of the server: exactly one in `standalone` mode, where a second would be ignored and so refuses boot; several are a cluster's seed nodes. Comma-separated in the env var. Not a URL: `redis://user:password@host:port` refuses boot, without repeating the value, so set the credentials through `username` and `password`, and `rediss://` through `tls.enabled`. | +| `cache.redis.mode` | `WH_CACHE_REDIS_MODE` | `standalone` | `standalone` or `cluster`. `sentinel` refuses boot until [#656](https://github.com/Wave-RF/WaveHouse/issues/656): the cache does not yet authenticate to the sentinels or refresh their topology, so it could not be trusted to follow a failover. | +| `cache.redis.username` | `WH_CACHE_REDIS_USERNAME` | *(empty)* | ACL user. Empty uses the server's `default` user. | +| `cache.redis.password` | `WH_CACHE_REDIS_PASSWORD` | *(empty)* | A secret: set it through the environment (or your secret store's env injection), not in a tracked `config.yaml`. | +| `cache.redis.db` | `WH_CACHE_REDIS_DB` | `0` | Database number (`SELECT`). Standalone only: a cluster has only database `0`, and any other value refuses boot. | +| `cache.redis.tls.enabled` | `WH_CACHE_REDIS_TLS_ENABLED` | `false` | Connect over TLS, verifying the server against the system roots or `ca_file`. Any other `tls` key set while this is off refuses boot, rather than connecting in plaintext. | +| `cache.redis.tls.ca_file` | `WH_CACHE_REDIS_TLS_CA_FILE` | *(empty)* | PEM file of the authorities to trust instead of the system roots. | +| `cache.redis.tls.cert_file` | `WH_CACHE_REDIS_TLS_CERT_FILE` | *(empty)* | Client certificate (PEM) for mutual TLS. Set together with `key_file`. | +| `cache.redis.tls.key_file` | `WH_CACHE_REDIS_TLS_KEY_FILE` | *(empty)* | The client certificate's private key (PEM). | +| `cache.redis.tls.server_name` | `WH_CACHE_REDIS_TLS_SERVER_NAME` | *(empty)* | Name to verify the server's certificate against, when it differs from the address. | +| `cache.redis.tls.insecure_skip_verify` | `WH_CACHE_REDIS_TLS_INSECURE_SKIP_VERIFY` | `false` | Accept any server certificate. Logged at `WARN` at boot: whoever can intercept the connection can read and replace cached results. | +| `cache.redis.key_prefix` | `WH_CACHE_REDIS_KEY_PREFIX` | `wh` | Leads every key, so several deployments can share one server, provided you trust each as much as the others: any of them can overwrite what the rest serve. No `{` or `}`. | +| `cache.redis.timeout` | `WH_CACHE_REDIS_TIMEOUT` | `100ms` | Per operation, at most `1s`. A lookup or fill that takes longer is a miss or a skipped fill, never a failed query. | +| `cache.redis.dial_timeout` | `WH_CACHE_REDIS_DIAL_TIMEOUT` | `1s` | Per connection attempt, at most `1s`. An attempt is a connect and a handshake, each bounded by this, and in `cluster` mode a topology read bounded by the larger of this and `timeout`. Boot and shutdown each wait out one in flight, so the two caps keep that under 3s, inside a stop's fixed 5s release budget (see [Stopping](/deployment#stopping)). | +| `cache.redis.max_value_bytes` | `WH_CACHE_REDIS_MAX_VALUE_BYTES` | `1048576` | Largest result stored, after compression (1 MiB). A larger one is returned to the caller but not cached. | +| `cache.redis.compress_min_bytes` | `WH_CACHE_REDIS_COMPRESS_MIN_BYTES` | `1024` | Results at least this large are zstd-compressed when that makes them smaller. `0` never compresses. | +| `cache.redis.version_ttl` | `WH_CACHE_REDIS_VERSION_TTL` | `168h` | How long a table's or tenant's version token outlives its last write, so the tokens of dropped tables and removed tenants eventually expire. At least `2s`. An expired token only causes misses. | + +**When the server is unreachable or misbehaves, the cache is bypassed; queries are not.** A failure or a timeout makes the lookup a miss and the fill a no-op. Five in a row, one reply refusing writes (`READONLY` from a demoted primary, `OOM` when full under `noeviction`, and the like), or one refusing the credentials (`WRONGPASS`, `NOAUTH` — what an already-open connection meets once the password is rotated) open a circuit breaker that skips the server entirely until a probe write, every 5 s, succeeds within `timeout`. Each opening is logged once, at `ERROR` for the credentials and `WARN` otherwise; a failed probe opens it again, so a server that stays down logs every 5 s. `NOPERM` does not open it: it names one key or command an ACL user cannot use, not every operation, so it should not bypass the cache for every tenant. Queries then go straight to ClickHouse, still coalesced per instance by `singleflight`. An invalidation the server did not take is kept and retried until it lands, and until then the instance that owes it bypasses the lookups it would orphan. `/readyz` does not depend on the cache. + +**Boot does not wait for the server.** A malformed block (an address without a port, `mode: cluster` with `db` other than `0`, an unreadable or unparsable TLS file) refuses boot. A server that cannot be reached, or that refuses the credentials, does not: the process boots with the cache bypassed and keeps reconnecting, with backoff up to 30 s. A rejected credential (`WRONGPASS`, `NOAUTH`, or `NOPERM` for an ACL user missing a connection command) is logged at `ERROR` on every attempt; any other failure at `WARN`. This is deliberate: a rotated Redis password must not crash-loop every instance at once. Watch `wavehouse_cache_breaker_open`, which reads `1` while the cache is bypassed, including before the first connection. ### Authentication @@ -237,8 +263,27 @@ mq: backend: embedded # in-process NATS JetStream under /nats cache: - backend: local - l1_max_cost: 67108864 + backend: local # local | redis + l1_max_cost: 67108864 # the local backend's size + redis: # read only with backend: redis + addrs: [] # required with backend: redis, e.g. ["redis:6379"] + mode: standalone # standalone | cluster + username: "" + password: "" # a secret: set WH_CACHE_REDIS_PASSWORD instead + db: 0 + tls: + enabled: false + ca_file: "" + cert_file: "" + key_file: "" + server_name: "" + insecure_skip_verify: false + key_prefix: wh + timeout: 100ms + dial_timeout: 1s + max_value_bytes: 1048576 + compress_min_bytes: 1024 # 0 = never + version_ttl: 168h dedupe: backend: pebble # in-process Pebble under /pebble @@ -298,6 +343,24 @@ WH_CH_MAX_TOTAL_CONNS=0 WH_MQ_BACKEND=embedded WH_CACHE_BACKEND=local WH_CACHE_L1_MAX_COST=67108864 +# Read only with WH_CACHE_BACKEND=redis; WH_CACHE_REDIS_ADDRS is then required. +WH_CACHE_REDIS_ADDRS= +WH_CACHE_REDIS_MODE=standalone +WH_CACHE_REDIS_USERNAME= +WH_CACHE_REDIS_PASSWORD= +WH_CACHE_REDIS_DB=0 +WH_CACHE_REDIS_TLS_ENABLED=false +WH_CACHE_REDIS_TLS_CA_FILE= +WH_CACHE_REDIS_TLS_CERT_FILE= +WH_CACHE_REDIS_TLS_KEY_FILE= +WH_CACHE_REDIS_TLS_SERVER_NAME= +WH_CACHE_REDIS_TLS_INSECURE_SKIP_VERIFY=false +WH_CACHE_REDIS_KEY_PREFIX=wh +WH_CACHE_REDIS_TIMEOUT=100ms +WH_CACHE_REDIS_DIAL_TIMEOUT=1s +WH_CACHE_REDIS_MAX_VALUE_BYTES=1048576 +WH_CACHE_REDIS_COMPRESS_MIN_BYTES=1024 +WH_CACHE_REDIS_VERSION_TTL=168h WH_DEDUPE_BACKEND=pebble WH_COORD_BACKEND=local diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index 08f538c65..1d0c5ff16 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -151,6 +151,12 @@ WH_AUTH_JWT_SECRET= # as an admin secret — inject from your secret store, serve only over TLS. WH_AUTH_OPERATOR_KEY= +# Optional shared query cache for several instances (see Multiple instances +# and the shared cache below); the password is a secret like the ones above. +# WH_CACHE_BACKEND=redis +# WH_CACHE_REDIS_ADDRS=redis:6379 +# WH_CACHE_REDIS_PASSWORD= + # Settings directory (required): roles.json, policies.json, pipes.json, # config.json — the hot-reloadable configuration: the access-control policy # and its roles, the named pipes, and the tunables including the ClickHouse @@ -341,7 +347,7 @@ By default one process runs all of WaveHouse. [`roles`](/configuration#process-r - **Ingest.** Every ingest pod consumes the same shared durable consumer and competes for its messages, so throughput scales with the pod count. The rows of one table are then split across pods: each pod writes smaller batches, and rows written by different pods do not reach ClickHouse in publish order. - **Sweeper.** The sweeper runs under a lease held in the shared `coord.backend`, so only one pod sweeps at a time. A second replica waits and takes over when the first stops. -A split needs backends that every process can reach: a shared `mq.backend`, so that every process reaches the same queue; a shared `cache.backend`, so that the ingest pods' invalidations reach the API pods' cache; and a shared `coord.backend`, so that the sweeper lease spans pods. **This build has only the in-process backends, so boot refuses any split** and names the backend to change. Until shared backends ship, run every role in one process, the default. +A split needs backends that every process can reach: a shared `mq.backend`, so that every process reaches the same queue; a shared `cache.backend` ([`redis`](#multiple-instances-and-the-shared-cache)), so that the ingest pods' invalidations reach the API pods' cache; and a shared `coord.backend`, so that the sweeper lease spans pods. **This build has a shared cache but only the in-process queue and leases, so boot refuses any split** and names the backend to change. Until a shared `mq.backend` and `coord.backend` ship, run every role in one process, the default. A pod without the `api` role serves an ops listener on `:8080`: `/livez`, `/readyz` and their aliases, `/version`, the metrics path when `prometheus.port` is `0`, and `POST /v1/ops/settings/reload`. Every other route answers 404 (under `/v1/ops`, 403 without the operator key, and 401 for a bearer token). Point the same probes at it as at an API pod. `/livez` does not wait for schema discovery there, because only the API runs it. `/readyz` checks ClickHouse in an ingest pod, and is ready once a sweeper pod has booted. Every pod reads the settings directory, so mount it in every Deployment. The reload route on the ops listener accepts only the operator key, so whatever reloads your API pods over HTTP must send the operator key to the worker pods too, or rely on `SIGHUP` (or, over a flat directory, the directory watcher) instead. @@ -417,6 +423,30 @@ The folder name is the tenant id, and each folder is a complete settings directo `X-Tenant-ID` is a generic name, and some gateways and service meshes stamp one on every request. WaveHouse used to ignore it; now, over a settings directory that holds the four files, any value other than `0` names an unknown tenant, so **every `/v1` route outside `/v1/ops/*` answers `404 unknown tenant: `** (a `400` when the value is not a tenant id at all, a dotted hostname, say) — the SDK's `/v1/health` reachability ping included, while the bare probes and the admin surface stay green. Strip the inbound header at the edge ([header forwarding](/reverse-proxy#header-and-auth-forwarding)) unless you are using it deliberately. +## Multiple instances and the shared cache + +Several WaveHouse instances can serve one ClickHouse behind a load balancer, but most of what each one holds is its own. The message queue is embedded, so an event is inserted by the instance that took its `POST /v1/ingest`, and reaches only that instance's SSE subscribers. The dedupe store is per instance too, so an id one instance has seen is new to another. + +The query-result cache is the layer that can be shared today. With the default `cache.backend: local`, each instance caches in its own memory, and an insert invalidates only the cache of the instance that made it. Every other instance keeps serving its cached results for the rows before the insert until each entry's TTL runs out, between 10 s and 1 h depending on how long the query took. With [`cache.backend: redis`](/configuration#cache), every instance reads and fills one Redis-compatible server, and an insert on any instance invalidates the cached results of every instance. The server is a standalone one or a Redis Cluster; Sentinel (`mode: sentinel`) refuses boot until [#656](https://github.com/Wave-RF/WaveHouse/issues/656), since the cache does not yet authenticate to the sentinels or refresh their topology. + +**What another instance can see.** Ingest is already asynchronous: `/v1/ingest` answers before the batch is inserted. Once the inserting instance's worker has written the batch to ClickHouse, it replaces the table's version token in Redis, and from then on a lookup on any instance misses and reads the new rows. The cache adds no delay of its own beyond that single write. The exceptions: + +- **The server is unreachable from the inserting instance.** The invalidation is kept and retried until it lands (`wavehouse_cache_invalidations_pending` counts what is owed). Meanwhile other instances that can still reach the server keep serving the older results, for as long as the outage lasts and at most until each entry's TTL. An instance that stops while invalidations are still owed loses them, with the same bound. The same thing happens today when a process stops between an insert and its invalidation. +- **A failover to a replica that had not yet received the latest token writes** can bring back entries filed under the older tokens, bounded by the replication lag at the moment of failover and those entries' TTL. WaveHouse never reads from replicas. Behind a stable address (a managed primary endpoint), an instance still connected to the demoted node has its writes refused, which bypasses its cache; connections are replaced every minute, so it reaches the new primary and delivers the invalidations it owes within about that long. The breaker's own probe write gets a longer budget for a reconnect — twice `dial_timeout` (the client bounds the dial and the handshake by it in turn) plus `timeout` for the write itself — but only closes the breaker when a write actually lands within `timeout`: one slower than that is repeated under `timeout` alone, and the repeat decides, so a server that merely answers slowly stays bypassed instead of flapping open and shut. Every other connection redials under `timeout` alone: size it above how long a reconnect actually takes, or operations that land on one of those keep failing after the probe has already succeeded. +- **The server is full and `maxmemory-policy` is `noeviction`.** It refuses the token writes. The inserting instance keeps its invalidations and retries them, bypassing its cache meanwhile, but every other instance serves the results from before the insert until one lands, up to their TTL. +- **A pipe that writes** (an `INSERT` in `pipes.json`) has its result cached like a read, so a repeated identical call is answered from the cache and the write does not run again ([#386](https://github.com/Wave-RF/WaveHouse/issues/386)). With a shared cache that holds on every instance, until the entry's TTL. +- **Admin writes through `POST /v1/ops/query`** do not invalidate the cache ([#394](https://github.com/Wave-RF/WaveHouse/issues/394)). With a shared cache, the stale results they leave are served by every instance, not only one. + +**Sizing the server.** Every key WaveHouse writes has a TTL, and a version token lost to eviction, expiry or `FLUSHALL` can only cause misses, never bring back an entry it had invalidated. So set `maxmemory` and let the server evict: `maxmemory-policy allkeys-lru` (or `allkeys-lfu`, `volatile-lru`, `volatile-lfu`). Under `noeviction`, a full server refuses the writes: each refusal (a fill's is counted by `wavehouse_cache_set_failures_total{reason="oom"}`) bypasses the cache of the instance that got it, and invalidations are kept and retried, so the pre-insert results above stay served by the others: avoid `noeviction`. A stored result is capped at `cache.redis.max_value_bytes` (1 MiB compressed). A tenant's version tokens share one hash tag, so each lookup reads them in one `MGET` in cluster mode as well. The results themselves carry no hash tag and spread across shards. **Run it without persistence** (`save ""` and `appendonly no`): stock Redis and Valkey persist by default (periodic RDB save points), so a crash that is followed by a restart reloads the last save on its own — the same rollback as restoring a snapshot by hand, not a loss. Without persistence, a restart can only cause misses, the same as any other token loss. With it on (the default), a restart reloads whatever snapshot or AOF it last wrote, tokens and values it had already invalidated included, so a fresh instance can serve the pre-write rows filed under them as hits until their TTL (up to 1 h) expires. Treat restoring a snapshot, or a crash-restart on a server that still has its defaults, as a rollback, not a resume. + +**The server is inside the trust boundary.** A cached result is served after the access policy has filtered it, so whoever can write to the server can change what any caller reads. Keep it on a private network, require a password or ACL user (`WH_CACHE_REDIS_PASSWORD`), use TLS across links you do not trust, and share it only with deployments you trust as much as this one. + +**Coalescing stays per instance.** `singleflight` collapses identical concurrent queries within each instance, so a cold hot query costs at most one ClickHouse query per instance, not one per request. + +**Metrics** (meter `wavehouse-cache`, every series labeled `backend="redis"`, no tenant label): `wavehouse_cache_lookups_total{result}` (`hit`, `miss`, `stale`, `bypass`, `error`), `wavehouse_cache_op_duration_seconds{op}` (`lookup`, `set`, `invalidate`), `wavehouse_cache_breaker_open` (1 while the cache is bypassed), `wavehouse_cache_invalidations_total{result}` (`ok` counts every bump that lands, retried ones included, and `deferred` each bump an invalidation could not deliver when made, a repeat of one already owed included; a failed retry is not counted again — the two overlap, not a split), `wavehouse_cache_invalidations_pending`, `wavehouse_cache_value_bytes`, `wavehouse_cache_oversize_total` and `wavehouse_cache_set_failures_total{reason}` (`oom`, `timeout`, `other`). Two signals are worth alerting on: `wavehouse_cache_breaker_open` at 1, or `wavehouse_cache_invalidations_pending` above 0, for more than a few minutes. + +For local development, `docker compose -f deployments/compose/dependencies.yaml --profile redis up -d` starts a Redis on `localhost:6379` with no persistence. + ## ClickHouse Schema WaveHouse uses a **Bring Your Own Schema** model. You create your tables in ClickHouse with whatever columns and engines you need. WaveHouse discovers the schemas automatically via `system.columns` and validates ingest data against them — see [Schema Validation](/api#post-v1ingesttabletable--ingest-data) for the rules a record must satisfy. diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index 4d91f6710..22c8e6beb 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -16,7 +16,7 @@ You need these on your `PATH` before any `make` recipe will work end-to-end: | **Go** | 1.26+ (matches `go.mod`) | Compiles `cmd/wavehouse`; also runs the pinned `tool` deps (`gotestsum`, `gofumpt`, `goimports`, `govulncheck`, `deadcode`, `gsa`, `goda`) via `go tool` | [go.dev/dl](https://go.dev/dl/) | | **GNU Make** | **4.0+** | The Makefile uses `--output-sync=target` (Make 4 only) and bash-pinned recipes. macOS ships with BSD Make 3.81, which **will not work** | macOS: `brew install make` then use `gmake` or put `$(brew --prefix make)/libexec/gnubin` on your PATH. Linux: usually already installed | | **bash** | 4+ recommended | Recipes are pinned to `bash`; the helper scripts under `scripts/` use `set -euo pipefail` and bash arrays | macOS default is bash 3.2 (works for current recipes, but `brew install bash` is safer); Linux distros ship 4+ | -| **Docker** *(or Podman)* | Engine 20.10+ with the Compose **v2** plugin (`docker compose`, no hyphen) | Compose stacks under `deployments/compose/`; the E2E and integration suites boot ClickHouse via testcontainers (no compose file) | [Docker Desktop](https://docs.docker.com/get-docker/), [colima](https://github.com/abiosoft/colima), or [Podman](https://podman.io) with `podman-compose` / the `podman compose` plugin. The testcontainers Go library also honors `DOCKER_HOST` for rootless Podman setups | +| **Docker** *(or Podman)* | Engine 20.10+ with the Compose **v2** plugin (`docker compose`, no hyphen) | Compose stacks under `deployments/compose/`; the E2E and integration suites boot ClickHouse and a Redis via testcontainers (no compose file), and the integration suite also runs the shared cache backend against Redis, Valkey, Dragonfly (pulled from `docker.dragonflydb.io`) and a one-node Redis Cluster | [Docker Desktop](https://docs.docker.com/get-docker/), [colima](https://github.com/abiosoft/colima), or [Podman](https://podman.io) with `podman-compose` / the `podman compose` plugin. The testcontainers Go library also honors `DOCKER_HOST` for rootless Podman setups | | **Node.js** | 22 LTS — pinned via `.nvmrc` at the repo root | Runtime for pnpm and the Vitest suites. Pinned to match CI (`setup-node` uses 22) and to avoid Node-major surprises; older Vitest versions in this repo were known to crash on Node 26 with a V8 heap-allocation abort | [nodejs.org](https://nodejs.org/) or `nvm use` / `fnm use` / `volta` (all read `.nvmrc`) | | **pnpm** | 11.21+ (pinned via `packageManager` in the root `package.json`) | Package manager for the TypeScript SDK, E2E test harness, and docs site (managed as a single pnpm workspace from the repo root); `make build-ts`, `make test-ts`, `make test-e2e`, `make build-docs`, `make dev-docs`, `make preview-docs` all shell out to `pnpm` | `corepack enable && corepack prepare pnpm@11.21.0 --activate` (recommended), or `npm i -g pnpm` | | **git** + **curl** | any recent | `git` for source + version metadata in builds; `curl` is used by the Makefile to fetch the pinned `golangci-lint` binary into `.bin/` | usually preinstalled | @@ -82,7 +82,7 @@ make dev WaveHouse is now running at `http://localhost:8080` in standalone mode with: - **Embedded NATS** (JetStream) — no external MQ needed -- **L1 cache only** (Ristretto) — no external cache needed +- **In-process cache** (Ristretto, `cache.backend: local`) — no external cache needed; to try the shared one, start Redis with `docker compose -f deployments/compose/dependencies.yaml --profile redis up -d` and set `WH_CACHE_BACKEND=redis WH_CACHE_REDIS_ADDRS=localhost:6379` - **Trial policy** — the dev settings directory `./settings` is seeded on first run with the compose stack's permissive `public` policy, so tokenless requests to the demo tables work (see [Test the API](#test-the-api)) - **Dedup disabled** by default — no Pebble needed - **Schema discovery** — automatically finds your ClickHouse tables @@ -341,18 +341,18 @@ Each test target writes `covdata` to `tmp/coverage//data/`, renders a tex | -------- | -------- | ------- | ------- | | Unit tests | `internal/*/_test.go` | No | `make test` | | SDK unit tests | `clients/ts/src/**/*.test.ts` | No | `make test-ts` (always includes coverage + gate) | -| Integration tests (Go) | `tests/integration/*_test.go` | Yes | `make test-integration` | +| Integration tests (Go) | `tests/integration/*_test.go`, `internal/cache/*_integration_test.go` | Yes | `make test-integration` | | E2E tests (SDK) | `tests/e2e/sdk/*.test.ts` | Yes | `make test-e2e` | - **Unit tests** live beside the code they test (e.g., `internal/discovery/discovery_test.go`). They use mocks or embedded NATS (in-process, no Docker needed). -- **Integration tests** use the `//go:build integration` build tag. `TestMain` starts one ClickHouse testcontainer and boots the production wiring against it through `app.New` (embedded NATS, ingest worker, sweeper, hub, the API server on a random loopback port); tests reach it via `env(t)` and create their own tables. DLQ tests use `assert.Eventually` with a 30-second timeout for the 5-second ingest worker batch window. +- **Integration tests** use the `//go:build integration` build tag. In `tests/integration`, `TestMain` starts one ClickHouse testcontainer and boots the production wiring against it through `app.New` (embedded NATS, ingest worker, sweeper, hub, the API server on a random loopback port); tests reach it via `env(t)` and create their own tables. DLQ tests use `assert.Eventually` with a 30-second timeout for the 5-second ingest worker batch window. `internal/cache`'s integration tests start their own containers instead — Redis, Valkey, Dragonfly and a one-node Redis Cluster — for the shared backend. `shared_cache_test.go` starts its own Redis testcontainer per test (`startRedis`) and boots extra, independent `cache.backend: redis` instances over that same ClickHouse (`bootRedisApp`), to exercise the cache shared across processes rather than one package in isolation. Shared test utilities live in `internal/testutil/`. The packages log through `slog.Default()`, so tests reach log output through `internal/testutil/logtest`: `logtest.Silence()` in a package's `TestMain` discards it, and `logtest.Capture(t, level)` routes it to a buffer for a test that asserts on log lines — such a test must not call `t.Parallel()`, because the default logger is process-wide. ### Adding New Tests - **Unit test for `internal/foo/`** → create `internal/foo/foo_test.go` (same package). -- **Integration test needing Docker** → add a subtest under `tests/integration/` (e.g. a new file with `//go:build integration`). +- **Integration test needing Docker** → add a subtest under `tests/integration/` (e.g. a new file with `//go:build integration`). A test of one package against its own external server — the shared cache backend against Redis, Valkey and Dragonfly containers — lives beside the package instead (`internal/cache/redis_integration_test.go`, same build tag), and the package is listed in the `test-integration` target. - **E2E test via SDK** → add a `tests/e2e/sdk/*.test.ts` file. These tests exercise the full pipeline (ingest → ClickHouse → query) through the TypeScript SDK. Run with `make test-e2e`. - **Test helpers** → add to `internal/testutil/` (Go) or `tests/e2e/sdk/helpers.ts` (E2E). @@ -362,7 +362,7 @@ The primary E2E integration test suite lives in `tests/e2e/sdk/`. It uses the Ty **Architecture**: -- `scripts/orchestrator` — the E2E entrypoint behind `make test-e2e`: it starts a clean ClickHouse **testcontainer** per run, launches the `wavehouse-cov` binary on a random free port, runs the SDK suite against it, then SIGINTs the binary to flush coverage. No Compose file is involved. CI runs the exact same path. +- `scripts/orchestrator` — the E2E entrypoint behind `make test-e2e`: it starts a clean ClickHouse **testcontainer** and a Redis one (the fixture's shared cache, `cache.backend: redis`) per run, launches the `wavehouse-cov` binary on a random free port, runs the SDK suite against it, then SIGINTs the binary to flush coverage. No Compose file is involved. CI runs the exact same path. - `tests/e2e/sdk/setup.ts` — `globalSetup`. Probes the `CLICKHOUSE_URL` / `WAVEHOUSE_URL` the orchestrator injects, creates the per-suite tables, refreshes the schema, and writes the baseline policy into the run's settings directory (adopted via `POST /v1/ops/settings/reload` — files are the only write path). It starts nothing itself and fails fast if either URL isn't up. It also prints the active Node/undici version, warning when the local Node major differs from `.nvmrc` — a runtime-specific transport bug is otherwise indistinguishable from a code failure (see [#440](https://github.com/Wave-RF/WaveHouse/issues/440)). - `tests/e2e/sdk/helpers.ts` — JWT factories, typed client constructors, async wait helpers, direct ClickHouse query helper. @@ -375,10 +375,11 @@ make test-e2e `make test-e2e` builds `bin/wavehouse-cov` (coverage-instrumented) and runs the orchestrator under `scripts/orchestrator/` to wire ClickHouse + the cover binary into the suite. covdata flushes on SIGINT into `tmp/coverage/e2e/data/`. -The orchestrator always provisions its own stack — a fresh ClickHouse testcontainer plus `wavehouse-cov` on a random free port — so a running `make dev` on `:8080` is neither detected nor reused, and the two don't collide. To run vitest against a stack you manage yourself, start the server from the **repo root** with the E2E fixture config: +The orchestrator always provisions its own stack — fresh ClickHouse and Redis testcontainers plus `wavehouse-cov` on a random free port — so a running `make dev` on `:8080` is neither detected nor reused, and the two don't collide. To run vitest against a stack you manage yourself, start a Redis for the fixture's `cache.backend: redis`, then the server from the **repo root** with the E2E fixture config: ```bash -WH_CONFIG=tests/e2e/fixtures/config.yaml go run ./cmd/wavehouse +docker compose -f deployments/compose/dependencies.yaml --profile redis up -d +WH_CONFIG=tests/e2e/fixtures/config.yaml WH_CACHE_REDIS_ADDRS=localhost:6379 go run ./cmd/wavehouse ``` The fixture matters: the suite signs its tokens with its `sdk-dev-secret` and depends on its dedupe, DLQ, and 5s schema-refresh settings. Point the suite at a default `make dev` server (`jwt_secret: change-me-in-production`) and setup's schema calls are rejected, then global setup dies 30s later on a misleading `schema not refreshed within 30s`. The repo root matters too — the fixture's `settings.dir` is relative to the working directory. The fixture's settings directory (policy, pipes, and tunables) points at ClickHouse on `localhost:9000`; if yours isn't there, edit `clickhouse.addr` / `http_port` in `tests/e2e/fixtures/settings/config.json` (the orchestrator patches them itself for its testcontainer). @@ -454,7 +455,7 @@ WaveHouse/ │ ├── api/ # HTTP handlers, router, middleware │ ├── app/ # Process wiring (build every component, run under one errgroup, release in reverse) │ ├── auth/ # JWT/JWKS authentication middleware -│ ├── cache/ # Query cache: Ristretto L1 + the tenant-led version index +│ ├── cache/ # Query cache: Ristretto L1 + the tenant-led version index; the Redis-compatible shared backend │ ├── chconn/ # ClickHouse pools, one per connection tuple (reconciled on settings reload) │ ├── chsql/ # Shared ClickHouse SQL helpers (quoting + bind-safety) │ ├── config/ # YAML + env var configuration @@ -474,7 +475,7 @@ WaveHouse/ │ └── testutil/ # Shared test helpers and mocks (cachetest suite) ├── tests/ # Integration & E2E tests │ ├── integration/ # Go integration tests (//go:build integration) -│ └── e2e/ # E2E suite (orchestrator + ClickHouse testcontainer) +│ └── e2e/ # E2E suite (orchestrator + ClickHouse and Redis testcontainers) │ ├── fixtures/ # ClickHouse DDL + config and settings-directory fixtures │ └── sdk/ # E2E specs driven through the TypeScript SDK (Vitest) ├── clients/ # Client SDKs diff --git a/docs/src/content/docs/getting-started.md b/docs/src/content/docs/getting-started.md index f24c3ee6f..667137d1a 100644 --- a/docs/src/content/docs/getting-started.md +++ b/docs/src/content/docs/getting-started.md @@ -75,7 +75,7 @@ curl -s -X POST "http://localhost:8080/v1/query?table=clicks" \ -d '{"columns": ["page", "button", "score"], "limit": 10}' ``` -`POST /v1/query?table={table}` and `GET/POST /v1/pipes/{name}` are cached in-process (L1 Ristretto) with singleflight coalescing — duplicate concurrent queries hit ClickHouse once. For raw SQL there's `POST /v1/ops/query` (an admin escape hatch that never caches, emitting `Cache-Control: no-store`), but it's **admin-only** — the trial `public` role can't reach it. To use it, swap the public default for real auth: configure a JWT secret and present a token whose role is the policy [`admin_role`](/access-control#admin_role--the-privileged-role). +`POST /v1/query?table={table}` and `GET/POST /v1/pipes/{name}` are cached — in-process by default, or in a Redis shared by every instance with [`cache.backend: redis`](/configuration#cache) — with singleflight coalescing, so duplicate concurrent queries hit ClickHouse once. For raw SQL there's `POST /v1/ops/query` (an admin escape hatch that never caches, emitting `Cache-Control: no-store`), but it's **admin-only** — the trial `public` role can't reach it. To use it, swap the public default for real auth: configure a JWT secret and present a token whose role is the policy [`admin_role`](/access-control#admin_role--the-privileged-role). :::tip[Prefer a type-safe client?] The [TypeScript SDK](/sdk) wraps this endpoint in a chainable query builder with autocomplete on your table names and row types — plus live queries and streaming. The raw shapes are in the [structured query reference](/api#post-v1querytabletable--structured-query). diff --git a/docs/src/content/docs/index.mdx b/docs/src/content/docs/index.mdx index 3a6143ca5..205a6f845 100644 --- a/docs/src/content/docs/index.mdx +++ b/docs/src/content/docs/index.mdx @@ -97,7 +97,7 @@ If you're building user-facing analytics, **WaveHouse is like Supabase for Click Every event is broadcast to SSE subscribers **before** it's flushed to ClickHouse. Gap-fill from JetStream history for late-connecting clients. - Ristretto cache plus Go `singleflight` coalesces identical concurrent queries — dashboards survive thundering herds without an extra cache tier to operate. + Ristretto cache plus Go `singleflight` coalesces identical concurrent queries — dashboards survive thundering herds without an extra cache tier to operate. Running several instances? Share one Redis-compatible cache with `cache.backend: redis`. Per-table, per-role column and row-level policies with JWT claim templating, defined in the hot-reloadable settings directory. diff --git a/docs/src/content/docs/pipes.mdx b/docs/src/content/docs/pipes.mdx index b8c7efa04..b6ded2005 100644 --- a/docs/src/content/docs/pipes.mdx +++ b/docs/src/content/docs/pipes.mdx @@ -7,7 +7,7 @@ sidebar: A **named pipe** is a saved SQL query, registered under a name, that callers run by name with parameters — without ever sending raw SQL. They turn an ad-hoc query into a stable, cached, access-controlled endpoint: you write the SQL once as an operator in the settings directory's [`pipes.json`](/settings-directory#pipesjson), expose it at `GET/POST /v1/pipes/{name}`, and clients supply only the declared parameters. -Pipes are the right tool when a query is reusable and shouldn't live in client code — dashboards, reports, public APIs over curated slices of data. They sit on the **cached read path** (shared L1 + singleflight, same as structured queries), and authorize through a simple per-pipe allowlist rather than the full [policy engine](/access-control). +Pipes are the right tool when a query is reusable and shouldn't live in client code — dashboards, reports, public APIs over curated slices of data. They sit on the **cached read path** (the query cache + singleflight, same as structured queries), and authorize through a simple per-pipe allowlist rather than the full [policy engine](/access-control). ## Anatomy of a pipe @@ -169,7 +169,7 @@ curl -X POST http://localhost:8080/v1/pipes/top_pages \ -d '{"start_date": "2024-01-01", "limit": 20}' ``` -The response is a JSON array of rows. Results flow through the shared in-process L1 cache (Ristretto) with singleflight coalescing, so concurrent identical calls hit ClickHouse once; an `X-Cache: HIT` or `X-Cache: MISS` header tells you which path served the response. +The response is a JSON array of rows. Results flow through the query cache (in-process by default, or a Redis shared by every instance with [`cache.backend: redis`](/configuration#cache)) with singleflight coalescing, so concurrent identical calls hit ClickHouse once; an `X-Cache: HIT` or `X-Cache: MISS` header tells you which path served the response. | Status | Body | Cause | | ------ | ---- | ----- | diff --git a/docs/src/content/docs/sdk/reference.md b/docs/src/content/docs/sdk/reference.md index fa6983da4..91df43af6 100644 --- a/docs/src/content/docs/sdk/reference.md +++ b/docs/src/content/docs/sdk/reference.md @@ -184,8 +184,8 @@ export interface ClicksRow { The SDK doubles as the E2E integration test harness. Tests in `tests/e2e/sdk/` exercise the full pipeline (ingest → ClickHouse → query) through the SDK, validating both the backend and the client library in one pass. ```bash -# Run all E2E tests: the orchestrator boots a ClickHouse testcontainer + -# the wavehouse-cov binary, then runs the SDK suite +# Run all E2E tests: the orchestrator boots ClickHouse and Redis +# testcontainers + the wavehouse-cov binary, then runs the SDK suite make test-e2e ``` diff --git a/docs/src/content/docs/settings-directory.mdx b/docs/src/content/docs/settings-directory.mdx index 900e6a2c2..67b97faaa 100644 --- a/docs/src/content/docs/settings-directory.mdx +++ b/docs/src/content/docs/settings-directory.mdx @@ -179,7 +179,7 @@ The tenant tunables. Every key is required (a missing one is a validation error) } ``` -What stays in boot config is only what cannot change under a running process — the implementation each layer runs on (`mq.backend`, `cache.backend`, `dedupe.backend`, `coord.backend`), the process's `roles`, resource sizing (`data_dir`, `cache.l1_max_cost`, `clickhouse.max_total_conns`), the listeners, the observability exporters — and the **secrets**: `clickhouse.password`, `auth.jwt_secret`, `auth.operator_key`. Secrets never belong in a tracked JSON file, so they stay in the environment and are combined with the wiring here on every (re)connect; rotating one is a restart. See [Configuration](/configuration). Everything else lives here and reloads. +What stays in boot config is only what cannot change under a running process — the implementation each layer runs on (`mq.backend`, `cache.backend`, `dedupe.backend`, `coord.backend`) and a shared backend's connection (`cache.redis`), the process's `roles`, resource sizing (`data_dir`, `cache.l1_max_cost`, `clickhouse.max_total_conns`), the listeners, the observability exporters — and the **secrets**: `clickhouse.password`, `cache.redis.password`, `auth.jwt_secret`, `auth.operator_key`. Secrets never belong in a tracked JSON file, so they stay in the environment and are combined with the wiring here on every (re)connect; rotating one is a restart. See [Configuration](/configuration). Everything else lives here and reloads. ## Deduplication diff --git a/docs/src/content/docs/why-wavehouse.md b/docs/src/content/docs/why-wavehouse.md index 766d62144..883d19284 100644 --- a/docs/src/content/docs/why-wavehouse.md +++ b/docs/src/content/docs/why-wavehouse.md @@ -152,7 +152,7 @@ flowchart TB | ---------- | --------- | --------- | | Durable ingest buffer | Kafka / Redpanda cluster (3+ brokers, Zookeeper/KRaft) | Embedded NATS JetStream | | Batch consumer | Custom Go/Rust/Java service you write and operate | Built in | -| Query cache | Redis + singleflight middleware you write | Built in (Ristretto + singleflight) | +| Query cache | Redis + singleflight middleware you write | Built in (Ristretto + singleflight; or one Redis shared by every instance, `cache.backend: redis`) | | Real-time push | WebSocket service + bridge from Kafka | Built in (`/v1/stream`) | | Schema validation | Custom code in ingest API | Built in (discovers `system.columns`) | | Row/column access control | Custom middleware or a dedicated service | Built in (Hasura-style, JWT-driven) | diff --git a/go.mod b/go.mod index ca418e893..d75355341 100644 --- a/go.mod +++ b/go.mod @@ -26,9 +26,13 @@ require ( github.com/golang-jwt/jwt/v5 v5.3.1 github.com/google/uuid v1.6.0 github.com/ilyakaznacheev/cleanenv v1.5.0 + github.com/klauspost/compress v1.19.2 + github.com/moby/moby/api v1.55.0 + github.com/moby/moby/client v0.5.1 github.com/nats-io/nats-server/v2 v2.14.6 github.com/nats-io/nats.go v1.53.1 github.com/prometheus/client_golang v1.24.1 + github.com/redis/rueidis v1.0.78 github.com/samber/slog-multi v1.8.0 github.com/samber/slog-sampling v1.7.0 github.com/stretchr/testify v1.12.1 @@ -132,7 +136,6 @@ require ( github.com/jedib0t/go-pretty/v6 v6.7.10 // indirect github.com/jmespath/go-jmespath v0.4.0 // indirect github.com/joho/godotenv v1.5.1 // indirect - github.com/klauspost/compress v1.19.2 // indirect github.com/knadh/profiler v0.2.0 // indirect github.com/kr/pretty v0.3.1 // indirect github.com/kr/text v0.2.0 // indirect @@ -147,8 +150,6 @@ require ( github.com/minio/highwayhash v1.0.4 // indirect github.com/moby/docker-image-spec v1.3.1 // indirect github.com/moby/go-archive v0.3.0 // indirect - github.com/moby/moby/api v1.55.0 // indirect - github.com/moby/moby/client v0.5.1 // indirect github.com/moby/patternmatcher v0.6.1 // indirect github.com/moby/sys/sequential v0.7.0 // indirect github.com/moby/sys/user v0.4.1 // indirect diff --git a/go.sum b/go.sum index 71dc27272..3b203e221 100644 --- a/go.sum +++ b/go.sum @@ -286,6 +286,8 @@ github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw= github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c= github.com/nikolaydubina/treemap v1.2.5 h1:oSC5z/qnsGLbkU2IihSrh2pS7uDjUq7ipGj8aw8bfII= github.com/nikolaydubina/treemap v1.2.5/go.mod h1:8+wLGh917AyeJqBN1D5KM26tv6W/XfvsY+nfJd04/u8= +github.com/onsi/gomega v1.42.1 h1:iN1rCUX+44NZ1Dc97MPoeFYbFR0vh8zxoxMFwKdyZ6I= +github.com/onsi/gomega v1.42.1/go.mod h1:REff/hsDsodHoKlWsP2mAPhu1+5/6hVYNf9rIEBpeSg= github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U= github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM= github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJwooC2xJA040= @@ -320,6 +322,8 @@ github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+ github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY= github.com/puzpuzpuz/xsync/v4 v4.5.0 h1:vOSWu6b57/emh+L/Cw0BeQfvxa/cogFywXHeGUxQxAg= github.com/puzpuzpuz/xsync/v4 v4.5.0/go.mod h1:VJDmTCJMBt8igNxnkQd86r+8KUeN1quSfNKu5bLYFQo= +github.com/redis/rueidis v1.0.78 h1:hJXpEgC9IYfdwY4hCdaGYsfK+oUaAqvhI/GMy5akVJI= +github.com/redis/rueidis v1.0.78/go.mod h1:L8mnCQJJaSNL6I4pIR6Rz732HTGS9vmuXm0yT9dRvjo= github.com/rivo/uniseg v0.1.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc= github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc= github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ= diff --git a/internal/app/app_test.go b/internal/app/app_test.go index 3d4f20c59..945f7eb96 100644 --- a/internal/app/app_test.go +++ b/internal/app/app_test.go @@ -16,6 +16,7 @@ import ( "os" "path/filepath" "strings" + "sync" "sync/atomic" "syscall" "testing" @@ -698,7 +699,7 @@ func TestReload_ReadmittedTenantCacheIsOrphaned(t *testing.T) { rewriteSettings(t, filepath.Join(root, "globex"), invalidQuery) a.tenants.Reload("test") - assert.Empty(t, mock.GetTenants(), "a rejection releases; it orphans nothing yet") + assert.Empty(t, mock.GetTenants(), "a rejection calls no InvalidateTenant (Prune drops its index)") rewriteSettings(t, filepath.Join(root, "globex"), nil) _, adopted = a.tenants.Reload("test") require.True(t, adopted) @@ -711,6 +712,148 @@ func TestReload_ReadmittedTenantCacheIsOrphaned(t *testing.T) { assert.Equal(t, []tenant.ID{"globex", "acme"}, mock.GetTenants(), "restored: the same") } +// pruneRecorder is a cache that records, at each Prune, which of the tenants +// it is asked about are still served. +type pruneRecorder struct { + testutil.MockCache + mu sync.Mutex + served []map[tenant.ID]bool +} + +func (p *pruneRecorder) Prune(served func(tenant.ID) bool) { + p.mu.Lock() + defer p.mu.Unlock() + p.served = append(p.served, map[tenant.ID]bool{"acme": served("acme"), "globex": served("globex")}) +} + +func (p *pruneRecorder) last() map[tenant.ID]bool { + p.mu.Lock() + defer p.mu.Unlock() + if len(p.served) == 0 { + return nil + } + return p.served[len(p.served)-1] +} + +// Every reload prunes the cache's version index down to the tenants served, +// so a tenant rejected or removed stops holding it (#262). +func TestReload_PrunesCacheIndexToServedTenants(t *testing.T) { + root := writeNestedSettings(t, map[string]map[string]any{"acme": nil, "globex": nil}) + a := newApp(t, testConfig(t, root), Options{}) + _, ok := a.cache.(pruner) + require.True(t, ok, "the wired cache prunes") + + rec := &pruneRecorder{} + a.cache = rec + + rewriteSettings(t, filepath.Join(root, "globex"), invalidQuery) + a.tenants.Reload("test") + assert.Equal(t, map[tenant.ID]bool{"acme": true, "globex": false}, rec.last(), "rejected") + + rewriteSettings(t, filepath.Join(root, "globex"), nil) + require.NoError(t, os.RemoveAll(filepath.Join(root, "acme"))) + a.tenants.Reload("test") + assert.Equal(t, map[tenant.ID]bool{"acme": false, "globex": true}, rec.last(), "removed; the repaired one served again") +} + +// redisTestConfig is testConfig with cache.backend=redis at addr, carrying +// the defaults Load would apply. +func redisTestConfig(t *testing.T, settingsDir, addr string) *config.Config { + t.Helper() + cfg := testConfig(t, settingsDir) + cfg.Cache = config.Cache{Backend: config.CacheRedis, Redis: config.CacheRedisConfig{ + Addrs: []string{addr}, Mode: config.RedisStandalone, KeyPrefix: "wh", + Timeout: 100 * time.Millisecond, DialTimeout: 200 * time.Millisecond, + MaxValueBytes: 1 << 20, CompressMinBytes: 1 << 10, VersionTTL: time.Hour, + }} + require.NoError(t, cfg.Validate()) + return cfg +} + +// cache.backend=redis wires the shared backend. A server that cannot be +// reached does not refuse boot: the cache starts bypassed, and the reload +// hook that prunes an in-process index leaves it alone. +func TestNew_RedisCacheBootsBypassedWhenUnreachable(t *testing.T) { + root := writeNestedSettings(t, map[string]map[string]any{"acme": nil, "globex": nil}) + a := newApp(t, redisTestConfig(t, root, closedAddr(t)), Options{}) + _, ok := a.cache.(*cache.RedisCache) + require.True(t, ok, "cache is %T", a.cache) + assert.Contains(t, componentNames(a), "cache") + + require.NoError(t, os.RemoveAll(filepath.Join(root, "acme"))) + a.tenants.Reload("test") // the prune hook must not trip on a non-pruner + + entry, snap, err := a.cache.Lookup(t.Context(), "globex", "sha", nil) + require.NoError(t, err) + assert.Nil(t, entry.Value, "bypassed: a miss") + assert.NoError(t, a.cache.Set(t.Context(), snap, []byte("v"), time.Minute), "and the fill a no-op") +} + +// A TLS file that went missing between validation and wiring refuses boot, +// naming the key. +func TestNew_RedisCacheRefusesAnUnreadableTLSFile(t *testing.T) { + guardGlobals(t) + cfg := redisTestConfig(t, writeSettings(t, nil), closedAddr(t)) + cfg.Cache.Redis.TLS = config.CacheRedisTLS{Enabled: true, CAFile: filepath.Join(t.TempDir(), "gone.pem")} + _, err := New(t.Context(), Options{Config: cfg}) + require.ErrorContains(t, err, "cache init: cache.redis.tls.ca_file") +} + +// The boot config's defaults are the backend's, and a compress_min_bytes of +// 0 reaches the backend as its "never compress" rather than its default. +// Driven from Load, not a literal, so a default changed on one side only +// fails here. +func TestRedisConfig_FromLoadedDefaults(t *testing.T) { + t.Setenv("WH_SETTINGS_DIR", t.TempDir()) + t.Setenv("WH_CACHE_BACKEND", "redis") + t.Setenv("WH_CACHE_REDIS_ADDRS", "a:6379") + t.Setenv("WH_CACHE_REDIS_PASSWORD", "pw") + loaded, err := config.Load(filepath.Join(t.TempDir(), "none.yaml")) + require.NoError(t, err) + got, err := redisConfig(loaded.Cache.Redis) + require.NoError(t, err) + assert.Equal(t, cache.RedisConfig{ + Addrs: []string{"a:6379"}, Mode: cache.RedisStandalone, Password: "pw", + KeyPrefix: cache.DefaultRedisKeyPrefix, Timeout: cache.DefaultRedisTimeout, + DialTimeout: cache.DefaultRedisDialTimeout, MaxValueBytes: cache.DefaultRedisMaxValueBytes, + CompressMinBytes: cache.DefaultRedisCompressMinBytes, VersionTTL: cache.DefaultRedisVersionTTL, + }, got) + + t.Setenv("WH_CACHE_REDIS_COMPRESS_MIN_BYTES", "0") + t.Setenv("WH_CACHE_REDIS_MODE", "cluster") + t.Setenv("WH_CACHE_REDIS_ADDRS", "a:6379,b:6379") + loaded, err = config.Load(filepath.Join(t.TempDir(), "none.yaml")) + require.NoError(t, err) + got, err = redisConfig(loaded.Cache.Redis) + require.NoError(t, err) + assert.Zero(t, got.CompressMinBytes, "the backend's never, not its default") + assert.Equal(t, cache.RedisCluster, got.Mode) + assert.Equal(t, []string{"a:6379", "b:6379"}, got.Addrs, "a cluster's seeds") + assert.Equal(t, cache.RedisSentinel, config.RedisSentinel) +} + +// Username, DB and TLS are zero on both sides of TestRedisConfig_FromLoadedDefaults' +// assert.Equal, so deleting any of their three mapping lines in redisConfig +// would pass it anyway. Drive all three through config.Load to a non-zero +// value and assert on them directly. +func TestRedisConfig_UsernameDBTLSMapped(t *testing.T) { + t.Setenv("WH_SETTINGS_DIR", t.TempDir()) + t.Setenv("WH_CACHE_BACKEND", "redis") + t.Setenv("WH_CACHE_REDIS_ADDRS", "a:6379") + t.Setenv("WH_CACHE_REDIS_USERNAME", "u") + t.Setenv("WH_CACHE_REDIS_DB", "2") + t.Setenv("WH_CACHE_REDIS_TLS_ENABLED", "true") + t.Setenv("WH_CACHE_REDIS_TLS_SERVER_NAME", "r.internal") + loaded, err := config.Load(filepath.Join(t.TempDir(), "none.yaml")) + require.NoError(t, err) + got, err := redisConfig(loaded.Cache.Redis) + require.NoError(t, err) + assert.Equal(t, "u", got.Username) + assert.Equal(t, 2, got.DB) + require.NotNil(t, got.TLS) + assert.Equal(t, "r.internal", got.TLS.ServerName) +} + // keepalive is a config.json patch setting the stream block's keepalive pair. func keepalive(interval, buckets int) map[string]any { return map[string]any{"stream": map[string]any{"keepalive_interval": interval, "keepalive_buckets": buckets, "gap_window_minutes": 15}} diff --git a/internal/app/wire.go b/internal/app/wire.go index efcb44fc9..fd1eb2e12 100644 --- a/internal/app/wire.go +++ b/internal/app/wire.go @@ -132,8 +132,9 @@ func gapWindows(tenants *settings.Registry) map[tenant.ID]time.Duration { const keepEverything = time.Duration(math.MaxInt64) // served reports whether the registry is serving tenant id: what the -// per-tenant resources — verifiers, dedupe stores, open streams — are pruned -// by once a reload removes or rejects their tenant. +// per-tenant resources — verifiers, dedupe stores, open streams, the cache +// version index — are pruned by once a reload removes or rejects their +// tenant. func (a *App) served(id tenant.ID) bool { _, ok := a.tenants.For(id) return ok @@ -321,7 +322,7 @@ func (a *App) wireClickHouse() error { // cached is stale, so all of it is orphaned at once. for _, id := range stale { if err := a.cache.InvalidateTenant(a.stopCtx, id); err != nil { - slog.Error("cache invalidation of a stale tenant failed; it may serve stale rows until they expire", "tenant", id, "error", err) + slog.Warn("cache invalidation of a stale tenant did not land; it may serve stale rows until it does", "tenant", id, "error", err) } } }) @@ -609,21 +610,75 @@ func (a *App) wireEmbeddedMQ(ctx context.Context) error { return nil } +// pruner is a cache whose version index lives in the process and would +// otherwise keep a tenant that stopped being served (cache.LocalCache). +type pruner interface { + Prune(served func(tenant.ID) bool) +} + +// The hook below asserts pruner at run time; this keeps LocalCache from +// silently dropping out of it. +var _ pruner = (*cache.LocalCache)(nil) + // wireCache opens the query-result cache — the one place the implementation -// is chosen. +// is chosen. After every reload a tenant no longer served, removed or +// rejected alike, has its in-process version index dropped (#262); its cache +// is orphaned with it, as it would be anyway when it came back +// (wireClickHouse). A shared backend keeps no such index and is skipped. func (a *App) wireCache() error { + var c cache.Cache switch b := a.cfg.Cache.Backend; b { case config.CacheLocal: l1, err := cache.NewLocal(a.cfg.Cache.L1MaxCost) if err != nil { return fmt.Errorf("cache init: %w", err) } - a.cache = l1 - a.add(component{name: "cache", close: withoutContext(l1.Close)}) - return nil + c = l1 + case config.CacheRedis: + rc, err := redisConfig(a.cfg.Cache.Redis) + if err != nil { + return fmt.Errorf("cache init: %w", err) + } + r, err := cache.NewRedis(rc) + if err != nil { + return fmt.Errorf("cache init: %w", err) + } + c = r default: return unreachableBackend("cache.backend", b) } + a.cache = c + a.add(component{name: "cache", close: withoutContext(c.Close)}) + a.tenants.AfterAdopt(func([]tenant.ID) { + if p, ok := a.cache.(pruner); ok { + p.Prune(a.served) + } + }) + return nil +} + +// redisConfig maps the boot config's cache.redis block onto the backend's +// config. Load has applied every default and validated the block; the TLS +// files are read again here, so the connection uses what is on disk now. +func redisConfig(r config.CacheRedisConfig) (cache.RedisConfig, error) { + t, err := r.TLS.Config() + if err != nil { + return cache.RedisConfig{}, err + } + return cache.RedisConfig{ + Addrs: r.Addrs, + Mode: r.Mode, + Username: r.Username, + Password: r.Password, + DB: r.DB, + TLS: t, + KeyPrefix: r.KeyPrefix, + Timeout: r.Timeout, + DialTimeout: r.DialTimeout, + MaxValueBytes: r.MaxValueBytes, + CompressMinBytes: r.CompressMinBytes, + VersionTTL: r.VersionTTL, + }, nil } // unreachableBackend is each layer switch's default case. config.Validate diff --git a/internal/cache/breaker.go b/internal/cache/breaker.go new file mode 100644 index 000000000..f3e065a53 --- /dev/null +++ b/internal/cache/breaker.go @@ -0,0 +1,102 @@ +package cache + +import ( + "sync" + "time" +) + +// breaker stops a failing cache server from costing every request its full +// timeout: after threshold consecutive failures it opens — at once, for a +// reply refusing the work — and while open callers skip the server +// entirely. Once openFor has passed, one caller is told to probe; the +// probe's outcome closes the breaker or reopens it. +type breaker struct { + threshold int + openFor time.Duration + now func() time.Time + + mu sync.Mutex + failures int + open bool + openedAt time.Time + probing bool +} + +func newBreaker(threshold int, openFor time.Duration, now func() time.Time) *breaker { + return &breaker{threshold: threshold, openFor: openFor, now: now} +} + +// allow reports whether a call may go to the server, and whether the caller +// should start the one probe that decides whether an open breaker closes. +func (b *breaker) allow() (ok, probe bool) { + b.mu.Lock() + defer b.mu.Unlock() + if !b.open { + return true, false + } + if b.probing || b.now().Sub(b.openedAt) < b.openFor { + return false, false + } + b.probing = true + return false, true +} + +// success records a call the server answered. Only a probe's closes an +// open breaker: any other set out before it opened, and a server that +// answers reads may still be refusing writes. +func (b *breaker) success() { + b.mu.Lock() + defer b.mu.Unlock() + if b.open && !b.probing { + return + } + b.failures, b.open, b.probing = 0, false, false +} + +// failure records a call the server did not answer in time, and opens the +// breaker at the threshold — or at once, for a failed probe. It reports +// whether that opened a closed or probing breaker, as trip does. +func (b *breaker) failure() bool { + b.mu.Lock() + defer b.mu.Unlock() + b.failures++ + if b.probing || b.failures >= b.threshold { + opened := !b.open || b.probing + b.open, b.openedAt, b.probing = true, b.now(), false + return opened + } + return false +} + +// trip opens the breaker at once, for a reply that says the server cannot +// do the work: one is as conclusive as any number. It reports whether the +// breaker was closed or probing, so a caller logs once per opening and once +// per refused probe rather than once per operation in flight. +func (b *breaker) trip() bool { + b.mu.Lock() + defer b.mu.Unlock() + opened := !b.open || b.probing + b.open, b.openedAt, b.probing = true, b.now(), false + return opened +} + +func (b *breaker) isOpen() bool { + b.mu.Lock() + defer b.mu.Unlock() + return b.open +} + +// untilProbe reports how long until an open breaker is due its probe, and +// whether it is open. While a probe runs it reports openFor: the probe ends +// by closing the breaker or by opening it afresh. +func (b *breaker) untilProbe() (time.Duration, bool) { + b.mu.Lock() + defer b.mu.Unlock() + switch { + case !b.open: + return 0, false + case b.probing: + return b.openFor, true + } + return b.openedAt.Add(b.openFor).Sub(b.now()), true +} diff --git a/internal/cache/breaker_test.go b/internal/cache/breaker_test.go new file mode 100644 index 000000000..273a871c0 --- /dev/null +++ b/internal/cache/breaker_test.go @@ -0,0 +1,99 @@ +package cache + +import ( + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +type fakeClock struct{ t time.Time } + +func (c *fakeClock) now() time.Time { return c.t } + +func TestBreaker(t *testing.T) { + t.Parallel() + clock := &fakeClock{t: time.Unix(0, 0)} + b := newBreaker(3, 5*time.Second, clock.now) + allow := func() (bool, bool) { return b.allow() } + + ok, probe := allow() + assert.True(t, ok) + assert.False(t, probe) + + b.failure() + b.failure() + b.success() // a success resets the run + b.failure() + assert.False(t, b.failure()) + assert.False(t, b.isOpen(), "two in a row is below the threshold") + assert.True(t, b.failure(), "it opened") + assert.True(t, b.isOpen()) + assert.False(t, b.failure(), "already open") + + ok, probe = allow() + assert.False(t, ok, "open: skip the server") + assert.False(t, probe, "not due for a probe yet") + + clock.t = clock.t.Add(5 * time.Second) + ok, probe = allow() + assert.False(t, ok) + assert.True(t, probe, "due: this caller probes") + ok, probe = allow() + assert.False(t, ok) + assert.False(t, probe, "one probe at a time") + + assert.True(t, b.failure(), "the probe failed: open for another period") + assert.True(t, b.isOpen()) + clock.t = clock.t.Add(4 * time.Second) + _, probe = allow() + assert.False(t, probe) + clock.t = clock.t.Add(time.Second) + _, probe = allow() + assert.True(t, probe) + + b.success() + assert.False(t, b.isOpen()) + ok, probe = allow() + assert.True(t, ok) + assert.False(t, probe) +} + +// A reply refusing the work opens the breaker at once, and only the probe's +// success closes it: a call that set out before it opened proves nothing. +func TestBreaker_TripAndProbeSchedule(t *testing.T) { + t.Parallel() + clock := &fakeClock{t: time.Unix(0, 0)} + b := newBreaker(100, 5*time.Second, clock.now) + _, open := b.untilProbe() + assert.False(t, open) + + assert.True(t, b.trip(), "it opened") + assert.True(t, b.isOpen(), "no threshold for a refusal") + assert.False(t, b.trip(), "already open") + b.success() + assert.True(t, b.isOpen(), "a success that is not the probe's leaves it open") + d, open := b.untilProbe() + assert.True(t, open) + assert.Equal(t, 5*time.Second, d) + + clock.t = clock.t.Add(2 * time.Second) + d, _ = b.untilProbe() + assert.Equal(t, 3*time.Second, d) + + clock.t = clock.t.Add(3 * time.Second) + _, probe := b.allow() + require.True(t, probe) + d, _ = b.untilProbe() + assert.Equal(t, 5*time.Second, d, "while the probe runs, wait out a whole period") + assert.True(t, b.trip(), "the probe was refused too") + d, _ = b.untilProbe() + assert.Equal(t, 5*time.Second, d) + + clock.t = clock.t.Add(5 * time.Second) + _, probe = b.allow() + require.True(t, probe) + b.success() + assert.False(t, b.isOpen()) +} diff --git a/internal/cache/cache.go b/internal/cache/cache.go index 745adb58a..ec9e51513 100644 --- a/internal/cache/cache.go +++ b/internal/cache/cache.go @@ -21,7 +21,8 @@ type Entry struct { // what was read before the bump under the post-bump versions (#382). The // zero Snapshot makes Set a no-op. type Snapshot struct { - key string // the backend's key for the entry at the observed versions + key string // the backend's key for the entry at the observed versions + tokens []byte // RedisCache: the version tokens read, in tokenKeys order } // ErrForeignDependency is a Lookup whose dependencies name a tenant other diff --git a/internal/cache/export_test.go b/internal/cache/export_test.go index 5e45e3ef6..73a791178 100644 --- a/internal/cache/export_test.go +++ b/internal/cache/export_test.go @@ -1,5 +1,43 @@ package cache +import ( + "time" + + "github.com/redis/rueidis" +) + +// Hooks for the integration tests in package cache_test. + +// ClientOption is the rueidis option a RedisCache built from c dials with. +func ClientOption(c RedisConfig) (rueidis.ClientOption, error) { + c, err := c.withDefaults() + return c.clientOption(), err +} + +// Pending reports how many token bumps r still owes the server. +func Pending(r *RedisCache) int { return r.pending.len() } + +// Bypassed reports whether r is skipping the server. +func Bypassed(r *RedisCache) bool { return r.bypassed() } + +// SetConnLifetime shortens how long c's connections live before they are +// replaced, for a failover test. +func SetConnLifetime(c *RedisConfig, d time.Duration) { c.connLifetime = d } + +// SetOnePipe gives c's client one connection per node. rueidis otherwise +// keeps up to four by GOMAXPROCS, dialing each on first use, so one first +// used after a failover reaches the new primary without being replaced. +func SetOnePipe(c *RedisConfig) { c.onePipe = true } + +// KeyPrefix is the prefix every key r writes leads with. +func KeyPrefix(r *RedisCache) string { return r.cfg.KeyPrefix } + +// ZeroSnapshot reports whether s files nothing. +func ZeroSnapshot(s Snapshot) bool { return s.key == "" && s.tokens == nil } + +// DecodedFactor is how many times MaxValueBytes a value may decompress to. +const DecodedFactor = decodedFactor + // Len counts the unexpired entries l holds, for the conformance suite's // Options.Entries: no Lookup reads the key a zero snapshot would land under. func (l *LocalCache) Len() int { diff --git a/internal/cache/local.go b/internal/cache/local.go index 1a755e058..3e0e14b5e 100644 --- a/internal/cache/local.go +++ b/internal/cache/local.go @@ -69,10 +69,11 @@ func (l *LocalCache) Set(_ context.Context, snap Snapshot, value []byte, ttl tim // view. Returns the number of namespaces processed. // // This bumps exactly what it's given. A whole-table bump already subsumes every -// per-scope bump for the same table (the table version is embedded in every -// namespace key), so a caller that knows a whole-table bump is coming should drop -// the now-redundant scope entries itself — the ingest worker does this as it -// builds the batch, where it already loops once and knows it's a single table. +// per-scope bump for the same table (every key that folds a scope version +// folds the table version too), so a caller that knows a whole-table bump is +// coming should drop the now-redundant scope entries itself — the ingest +// worker does this as it builds the batch, where it already loops once and +// knows it's a single table. func (l *LocalCache) Invalidate(_ context.Context, namespaces []Namespace) (uint64, error) { for _, ns := range namespaces { if ns.Scope == "" { @@ -85,13 +86,21 @@ func (l *LocalCache) Invalidate(_ context.Context, namespaces []Namespace) (uint } // InvalidateTenant orphans every cached result of tenant id, pipe results -// included: one version bump, nothing enumerated (see +// included: its version index is dropped, nothing enumerated (see // VersionManager.BumpTenant). func (l *LocalCache) InvalidateTenant(_ context.Context, id tenant.ID) error { l.versionManager.BumpTenant(id) return nil } +// Prune drops the version index of every tenant served rejects, orphaning +// its entries as InvalidateTenant would, so a tenant removed or rejected at +// a reload stops holding memory (#262). The entries themselves go with +// their TTL or Ristretto's eviction. +func (l *LocalCache) Prune(served func(tenant.ID) bool) { + l.versionManager.Prune(served) +} + // Wait blocks until all buffered writes have been applied. // Exposed for testing; production callers rarely need this. func (l *LocalCache) Wait() { diff --git a/internal/cache/local_test.go b/internal/cache/local_test.go index 5a62ebda6..d60915fe4 100644 --- a/internal/cache/local_test.go +++ b/internal/cache/local_test.go @@ -2,10 +2,13 @@ package cache_test import ( "testing" + "time" + "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" "github.com/Wave-RF/WaveHouse/internal/cache" + "github.com/Wave-RF/WaveHouse/internal/tenant" "github.com/Wave-RF/WaveHouse/internal/testutil/cachetest" ) @@ -26,3 +29,33 @@ func TestLocalCache_Conformance(t *testing.T) { Entries: func(c cache.Cache) int { return c.(*cache.LocalCache).Len() }, }) } + +// A tenant that stops being served has its index dropped: what it cached is +// orphaned — it misses when served again — and a tenant still served keeps +// its entries. +func TestLocalCache_Prune(t *testing.T) { + t.Parallel() + c, err := cache.NewLocal(localMaxCost) + require.NoError(t, err) + t.Cleanup(func() { _ = c.Close() }) + ctx := t.Context() + fill := func(id tenant.ID) { + _, snap, err := c.Lookup(ctx, id, "q", []cache.Namespace{{Tenant: id, Table: "events"}}) + require.NoError(t, err) + require.NoError(t, c.Set(ctx, snap, []byte("rows"), time.Minute)) + } + get := func(id tenant.ID) []byte { + e, _, err := c.Lookup(ctx, id, "q", []cache.Namespace{{Tenant: id, Table: "events"}}) + require.NoError(t, err) + return e.Value + } + fill("acme") + fill("globex") + c.Wait() + require.NotNil(t, get("acme")) + require.NotNil(t, get("globex")) + + c.Prune(func(id tenant.ID) bool { return id == "acme" }) + assert.NotNil(t, get("acme"), "still served") + assert.Nil(t, get("globex"), "pruned: orphaned, never revived") +} diff --git a/internal/cache/metrics.go b/internal/cache/metrics.go new file mode 100644 index 000000000..855843443 --- /dev/null +++ b/internal/cache/metrics.go @@ -0,0 +1,106 @@ +package cache + +import ( + "context" + "errors" + "time" + + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" +) + +// Lookup outcomes for the "result" attribute of wavehouse_cache_lookups_total. +const ( + resultHit = "hit" + resultMiss = "miss" // nothing stored + resultStale = "stale" // stored under versions since bumped + resultBypass = "bypass" // server skipped: breaker open, not yet connected, or a bump this process owes would orphan the entry + resultError = "error" // server failed or timed out +) + +// metrics are a shared-cache backend's instruments. No tenant attribute: +// lookups are the hot path, and tenants are unbounded. +type metrics struct { + backend attribute.KeyValue + lookups metric.Int64Counter + duration metric.Float64Histogram + invalidation metric.Int64Counter + valueBytes metric.Int64Histogram + oversize metric.Int64Counter + setFailures metric.Int64Counter + registration metric.Registration +} + +// newMetrics builds the instruments on the global meter provider, with +// gauges read from breakerOpen and pending on every collection. Call it +// after observability.InitProvider, like every other instrument. +func newMetrics(backend string, breakerOpen func() bool, pending func() int) (*metrics, error) { + meter := otel.Meter("wavehouse-cache") + m := &metrics{backend: attribute.String("backend", backend)} + var errs [9]error + m.lookups, errs[0] = meter.Int64Counter("wavehouse_cache_lookups_total", + metric.WithDescription("Shared-cache lookups by result: hit, miss, stale (stored under since-bumped versions), bypass (server skipped, or held by an invalidation this process has yet to deliver), error")) + m.duration, errs[1] = meter.Float64Histogram("wavehouse_cache_op_duration_seconds", + metric.WithDescription("Shared-cache round-trip time by op: lookup, set, invalidate"), metric.WithUnit("s"), + metric.WithExplicitBucketBoundaries(.0001, .00025, .0005, .001, .0025, .005, .01, .025, .05, .1, .25)) + m.invalidation, errs[2] = meter.Int64Counter("wavehouse_cache_invalidations_total", + metric.WithDescription("Version-token bumps by result: ok counts every bump that lands, retried ones included; deferred counts each bump an invalidation could not deliver when made, a repeat of one already owed included; a failed retry is not counted again. They overlap: wavehouse_cache_invalidations_pending is what is still owed")) + m.valueBytes, errs[3] = meter.Int64Histogram("wavehouse_cache_value_bytes", + metric.WithDescription("Size of each value written to the shared cache, after compression"), metric.WithUnit("By"), + metric.WithExplicitBucketBoundaries(256, 1<<10, 4<<10, 16<<10, 64<<10, 256<<10, 1<<20, 4<<20)) + m.oversize, errs[4] = meter.Int64Counter("wavehouse_cache_oversize_total", + metric.WithDescription("Results not cached because they exceed the value size limit")) + m.setFailures, errs[5] = meter.Int64Counter("wavehouse_cache_set_failures_total", + metric.WithDescription("Shared-cache writes that failed, by reason: oom, timeout, other")) + breakerGauge, err := meter.Int64ObservableGauge("wavehouse_cache_breaker_open", + metric.WithDescription("1 while the shared cache is being bypassed (circuit breaker open, or never connected), else 0")) + errs[6] = err + pendingGauge, err := meter.Int64ObservableGauge("wavehouse_cache_invalidations_pending", + metric.WithDescription("Version-token bumps not yet delivered to the shared cache; entries they would orphan may be served stale meanwhile")) + errs[7] = err + m.registration, errs[8] = meter.RegisterCallback(func(_ context.Context, o metric.Observer) error { + var open int64 + if breakerOpen() { + open = 1 + } + o.ObserveInt64(breakerGauge, open, metric.WithAttributes(m.backend)) + o.ObserveInt64(pendingGauge, int64(pending()), metric.WithAttributes(m.backend)) + return nil + }, breakerGauge, pendingGauge) + if err := errors.Join(errs[:]...); err != nil { + return nil, err + } + return m, nil +} + +func (m *metrics) lookup(result string) { + m.lookups.Add(context.Background(), 1, metric.WithAttributes(m.backend, attribute.String("result", result))) +} + +func (m *metrics) op(op string, start time.Time) { + m.duration.Record(context.Background(), time.Since(start).Seconds(), + metric.WithAttributes(m.backend, attribute.String("op", op))) +} + +func (m *metrics) invalidated(result string, n int) { + if n > 0 { + m.invalidation.Add(context.Background(), int64(n), metric.WithAttributes(m.backend, attribute.String("result", result))) + } +} + +func (m *metrics) stored(n int) { + m.valueBytes.Record(context.Background(), int64(n), metric.WithAttributes(m.backend)) +} + +func (m *metrics) tooLarge() { + m.oversize.Add(context.Background(), 1, metric.WithAttributes(m.backend)) +} + +func (m *metrics) setFailed(reason string) { + m.setFailures.Add(context.Background(), 1, metric.WithAttributes(m.backend, attribute.String("reason", reason))) +} + +func (m *metrics) close() { + _ = m.registration.Unregister() +} diff --git a/internal/cache/pending.go b/internal/cache/pending.go new file mode 100644 index 000000000..481ab9ac7 --- /dev/null +++ b/internal/cache/pending.go @@ -0,0 +1,92 @@ +package cache + +import ( + "sync" + "sync/atomic" + + "github.com/Wave-RF/WaveHouse/internal/tenant" +) + +// pendingBumps holds the token bumps an invalidation could not deliver, until +// a retry lands them. Repeats of one key coalesce; past max keys the set +// collapses to one tenant bump per affected tenant — coarser, never less. +// Landing a bump late is still correct: a fresh token orphans the pre-write +// entries and any fill in between. +type pendingBumps struct { + prefix string + max int + + mu sync.Mutex + keys map[string]pendingKey + gen uint64 + n atomic.Int64 // len(keys), read without mu on every lookup +} + +type pendingKey struct { + tenant tenant.ID + gen uint64 // when last added; take drops a key only if it is unchanged +} + +func newPendingBumps(prefix string, maxKeys int) *pendingBumps { + return &pendingBumps{prefix: prefix, max: maxKeys, keys: map[string]pendingKey{}} +} + +// add records keys of tenant id as owed a bump. +func (p *pendingBumps) add(id tenant.ID, keys ...string) { + p.mu.Lock() + defer p.mu.Unlock() + p.gen++ + for _, k := range keys { + p.keys[k] = pendingKey{tenant: id, gen: p.gen} + } + if len(p.keys) > p.max { + collapsed := make(map[string]pendingKey, len(p.keys)) + for _, pk := range p.keys { + collapsed[tenantTokenKey(p.prefix, pk.tenant)] = pendingKey{tenant: pk.tenant, gen: p.gen} + } + p.keys = collapsed + } + p.n.Store(int64(len(p.keys))) +} + +// snapshot returns the keys owed a bump with the generation each was added +// at, for a later done. +func (p *pendingBumps) snapshot() map[string]uint64 { + p.mu.Lock() + defer p.mu.Unlock() + out := make(map[string]uint64, len(p.keys)) + for k, pk := range p.keys { + out[k] = pk.gen + } + return out +} + +// done drops keys whose bump landed — unless one was added again since the +// snapshot, whose bump the landed one may have preceded. +func (p *pendingBumps) done(landed map[string]uint64) { + p.mu.Lock() + defer p.mu.Unlock() + for k, gen := range landed { + if pk, ok := p.keys[k]; ok && pk.gen == gen { + delete(p.keys, k) + } + } + p.n.Store(int64(len(p.keys))) +} + +// owesAny reports whether any of keys is owed a bump. +func (p *pendingBumps) owesAny(keys []string) bool { + if p.n.Load() == 0 { + return false + } + p.mu.Lock() + defer p.mu.Unlock() + for _, k := range keys { + if _, ok := p.keys[k]; ok { + return true + } + } + return false +} + +func (p *pendingBumps) len() int { return int(p.n.Load()) } diff --git a/internal/cache/pending_test.go b/internal/cache/pending_test.go new file mode 100644 index 000000000..1c8390896 --- /dev/null +++ b/internal/cache/pending_test.go @@ -0,0 +1,71 @@ +package cache + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestPendingBumps_Coalesce(t *testing.T) { + t.Parallel() + p := newPendingBumps("wh", 10) + p.add("acme", "k1", "k2") + p.add("acme", "k1") + assert.Equal(t, 2, p.len()) + + snap := p.snapshot() + p.done(snap) + assert.Zero(t, p.len()) +} + +// A key added again after the snapshot a drain worked from stays owed: the +// drain's bump may have landed before the write the new add is for. +func TestPendingBumps_ReAddedKeyStays(t *testing.T) { + t.Parallel() + p := newPendingBumps("wh", 10) + p.add("acme", "k1", "k2") + snap := p.snapshot() + p.add("acme", "k1") + p.done(snap) + assert.Equal(t, map[string]uint64{"k1": 2}, p.snapshot()) +} + +func TestPendingBumps_OverflowCollapsesToTenants(t *testing.T) { + t.Parallel() + p := newPendingBumps("wh", 3) + p.add("acme", "wh:{acme}:B:a", "wh:{acme}:B:b") + p.add("globex", "wh:{globex}:B:a") + assert.Equal(t, 3, p.len()) + + p.add("acme", "wh:{acme}:B:c") + got := p.snapshot() + assert.Len(t, got, 2) + assert.Contains(t, got, "wh:{acme}:T") + assert.Contains(t, got, "wh:{globex}:T") +} + +// A lookup is held by a bump owed on any of its token keys, including the +// tenant token a set past its maximum collapses to. +func TestPendingBumps_OwesAny(t *testing.T) { + t.Parallel() + events := tokenKeys("wh", "acme", []Namespace{{Tenant: "acme", Table: "events", Scope: "org_1"}}) + orders := tokenKeys("wh", "acme", []Namespace{{Tenant: "acme", Table: "orders"}}) + globex := tokenKeys("wh", "globex", []Namespace{{Tenant: "globex", Table: "events"}}) + p := newPendingBumps("wh", 3) + assert.False(t, p.owesAny(events)) + + p.add("acme", bumpKeys("wh", Namespace{Tenant: "acme", Table: "events", Scope: "org_1"})...) + assert.True(t, p.owesAny(events)) + assert.False(t, p.owesAny(orders), "another table's lookups are not held") + assert.False(t, p.owesAny(globex)) + + p.add("acme", "wh:{acme}:B:a", "wh:{acme}:B:b") // past the maximum + assert.Equal(t, map[string]uint64{"wh:{acme}:T": 2}, p.snapshot()) + assert.True(t, p.owesAny(orders), "collapsed to the tenant token, which every lookup of the tenant reads") + assert.True(t, p.owesAny(tokenKeys("wh", "acme", nil))) + assert.False(t, p.owesAny(globex)) + + p.done(p.snapshot()) + assert.False(t, p.owesAny(events)) + assert.Zero(t, p.len()) +} diff --git a/internal/cache/redis.go b/internal/cache/redis.go new file mode 100644 index 000000000..1889ee727 --- /dev/null +++ b/internal/cache/redis.go @@ -0,0 +1,811 @@ +package cache + +import ( + "bytes" + "context" + "crypto/tls" + "errors" + "fmt" + "log/slog" + "maps" + "net" + "slices" + "strings" + "sync" + "sync/atomic" + "time" + + "github.com/redis/rueidis" + + "github.com/Wave-RF/WaveHouse/internal/tenant" +) + +// Redis deployment modes for RedisConfig.Mode. +const ( + RedisStandalone = "standalone" + RedisCluster = "cluster" + RedisSentinel = "sentinel" +) + +// Defaults for RedisConfig's zero values. +const ( + DefaultRedisKeyPrefix = "wh" + DefaultRedisTimeout = 100 * time.Millisecond + DefaultRedisDialTimeout = time.Second + DefaultRedisMaxValueBytes = 1 << 20 + DefaultRedisCompressMinBytes = 1 << 10 + DefaultRedisVersionTTL = 7 * 24 * time.Hour + DefaultRedisPendingMax = 100_000 + defaultBreakerThreshold = 5 + defaultBreakerOpenFor = 5 * time.Second + defaultConnLifetime = time.Minute +) + +const ( + drainMinBackoff = 100 * time.Millisecond + drainMaxBackoff = 10 * time.Second + drainIdle = time.Second + drainBatch = 1000 + dialMaxBackoff = 30 * time.Second + closeDrainBudget = time.Second +) + +var ( + errBypassed = errors.New("cache: redis unavailable") + // errMalformedReply is a reply the server sent that this code cannot + // use: the server is up, so it never counts against the breaker. + errMalformedReply = errors.New("cache: malformed reply") +) + +// RedisConfig configures a RedisCache. A zero field takes its Default* +// value, except CompressMinBytes, where 0 means never compress. +type RedisConfig struct { + Addrs []string // host:port; several are seeds (cluster) or sentinels + Mode string // RedisStandalone (""), RedisCluster or RedisSentinel + SentinelMaster string // the master set name, Mode RedisSentinel + Username string + Password string + DB int // standalone and sentinel only + TLS *tls.Config // nil: plaintext + + KeyPrefix string // leads every key; separates deployments sharing a server + Timeout time.Duration // per operation + DialTimeout time.Duration + MaxValueBytes int // largest value stored, after compression + CompressMinBytes int // zstd-compress values at least this large + VersionTTL time.Duration // a token's lifetime from its last bump, jittered ±10%; reads do not extend it + PendingMax int // undelivered bumps kept before collapsing to tenant bumps + + BreakerThreshold int // consecutive failures that open the breaker + BreakerOpenFor time.Duration // how long it stays open before a probe + + connLifetime time.Duration // defaultConnLifetime; tests shorten it + onePipe bool // one connection per node, whatever GOMAXPROCS; tests only +} + +func (c RedisConfig) withDefaults() (RedisConfig, error) { + if len(c.Addrs) == 0 { + return c, errors.New("cache: redis needs at least one address") + } + for _, a := range c.Addrs { + if _, _, err := net.SplitHostPort(a); err != nil { + return c, fmt.Errorf("cache: redis address %q: %w", a, err) + } + } + switch c.Mode { + case "": + c.Mode = RedisStandalone + case RedisStandalone, RedisCluster, RedisSentinel: + default: + return c, fmt.Errorf("cache: redis mode %q: want %s, %s or %s", c.Mode, RedisStandalone, RedisCluster, RedisSentinel) + } + if c.Mode == RedisCluster && c.DB != 0 { + return c, errors.New("cache: redis cluster has only database 0") + } + if c.Mode == RedisSentinel && c.SentinelMaster == "" { + return c, errors.New("cache: redis sentinel mode needs the master set name") + } + if c.DB < 0 { + return c, fmt.Errorf("cache: redis db %d is negative", c.DB) + } + if strings.ContainsAny(c.KeyPrefix, "{}") { + return c, fmt.Errorf("cache: redis key prefix %q may not contain a hash tag brace", c.KeyPrefix) + } + for _, v := range []struct { + name string + n int64 + }{ + {"timeout", int64(c.Timeout)}, + {"dial timeout", int64(c.DialTimeout)}, + {"max value bytes", int64(c.MaxValueBytes)}, + {"compress min bytes", int64(c.CompressMinBytes)}, + {"version ttl", int64(c.VersionTTL)}, + {"pending max", int64(c.PendingMax)}, + {"breaker threshold", int64(c.BreakerThreshold)}, + {"breaker open for", int64(c.BreakerOpenFor)}, + } { + if v.n < 0 { + return c, fmt.Errorf("cache: redis %s is negative", v.name) + } + } + if c.VersionTTL != 0 && c.VersionTTL < 2*time.Second { + return c, fmt.Errorf("cache: redis version ttl %s is under 2s: EX has one-second resolution, and jittered below ~1.1s it could be EX 0", c.VersionTTL) + } + c.KeyPrefix = cmpOr(c.KeyPrefix, DefaultRedisKeyPrefix) + c.Timeout = cmpOr(c.Timeout, DefaultRedisTimeout) + c.DialTimeout = cmpOr(c.DialTimeout, DefaultRedisDialTimeout) + c.MaxValueBytes = cmpOr(c.MaxValueBytes, DefaultRedisMaxValueBytes) + c.VersionTTL = cmpOr(c.VersionTTL, DefaultRedisVersionTTL) + c.PendingMax = cmpOr(c.PendingMax, DefaultRedisPendingMax) + c.BreakerThreshold = cmpOr(c.BreakerThreshold, defaultBreakerThreshold) + c.BreakerOpenFor = cmpOr(c.BreakerOpenFor, defaultBreakerOpenFor) + c.connLifetime = cmpOr(c.connLifetime, defaultConnLifetime) + return c, nil +} + +func cmpOr[T comparable](v, def T) T { + var zero T + if v == zero { + return def + } + return v +} + +func (c RedisConfig) clientOption() rueidis.ClientOption { + opt := rueidis.ClientOption{ + InitAddress: c.Addrs, + Username: c.Username, + Password: c.Password, + SelectDB: c.DB, + TLSConfig: c.TLS, + Dialer: net.Dialer{Timeout: c.DialTimeout}, + ClientName: "wavehouse", + DisableCache: true, // no client-side caching until a local near-cache in front of Redis exists + ForceSingleClient: c.Mode == RedisStandalone, + } + // How long a connection waits on a silent server, 10 s unset. A cluster + // client reads the topology under it after the handshake, which boot and + // Close would wait out; never under Timeout, so no connection is cut + // while an operation may still wait on it. + opt.ConnWriteTimeout = max(c.DialTimeout, c.Timeout) + // A connection outlives a failover behind a stable address: the demoted + // node still answers, refusing writes, and rueidis does not redial on + // READONLY. Replacing each connection this often (rueidis retries what + // was in flight) re-resolves the address, so a process the refusals + // bypass reaches the new primary, and delivers its owed bumps, within + // about this long. + opt.ConnLifetime = c.connLifetime + if c.onePipe { + opt.PipelineMultiplex = -1 + } + if c.Mode == RedisSentinel { + opt.Sentinel = rueidis.SentinelOption{MasterSet: c.SentinelMaster, TLSConfig: c.TLS, Dialer: opt.Dialer} + } + return opt +} + +// RedisCache is a Cache shared by every process pointed at one Redis — +// or Valkey, Dragonfly, ElastiCache, MemoryDB: it uses only GET, SET and +// MGET, no scripts and no client tracking. +// +// Versions are random tokens, one per tenant, per table and per scope, +// under the tenant's hash tag; a bump sets a fresh one. A value carries the +// tokens it was computed under and is a hit only while they are all still +// current, so a lost token (eviction, expiry, a restart without persistence) +// can only cause misses. Restoring a snapshot is not a loss but a rollback — +// a restart after a crash that reloads the server's last save included, +// which stock Redis and Valkey make by default: the old tokens return with +// their values. A lookup is one round trip. The server failing or timing +// out is a miss, a skipped fill and a deferred invalidation — never a +// failed query. +type RedisCache struct { + cfg RedisConfig + opt rueidis.ClientOption + maxDecoded int + + client atomic.Pointer[rueidis.Client] // nil until the first connection + codec *codec + breaker *breaker + pending *pendingBumps + metrics *metrics + + ctx context.Context // cancelled by Close + cancel context.CancelFunc + wg sync.WaitGroup + wake chan struct{} + closeOnce sync.Once +} + +var _ Cache = (*RedisCache)(nil) + +// NewRedis builds a RedisCache. A malformed cfg is an error; a server that +// cannot be reached within the dial timeout is not — the cache starts +// bypassed and keeps dialing in the background. +func NewRedis(cfg RedisConfig) (*RedisCache, error) { + cfg, err := cfg.withDefaults() + if err != nil { + return nil, err + } + maxDecoded := cfg.MaxValueBytes * decodedFactor + cd, err := newCodec(cfg.CompressMinBytes, maxDecoded) + if err != nil { + return nil, fmt.Errorf("cache: zstd: %w", err) + } + ctx, cancel := context.WithCancel(context.Background()) + r := &RedisCache{ + cfg: cfg, + opt: cfg.clientOption(), + maxDecoded: maxDecoded, + codec: cd, + breaker: newBreaker(cfg.BreakerThreshold, cfg.BreakerOpenFor, time.Now), + pending: newPendingBumps(cfg.KeyPrefix, cfg.PendingMax), + ctx: ctx, + cancel: cancel, + wake: make(chan struct{}, 1), + } + if r.metrics, err = newMetrics("redis", r.bypassed, r.pending.len); err != nil { + cancel() + cd.close() + return nil, fmt.Errorf("cache: metrics: %w", err) + } + r.wg.Add(1) + go r.drainLoop() + if err := r.dial(); err != nil { + r.wg.Add(1) + go r.dialLoop() + } + return r, nil +} + +func (r *RedisCache) dial() error { + c, err := rueidis.NewClient(r.opt) + if err != nil { + level := slog.LevelWarn + if isAuthError(err) { + level = slog.LevelError + } + slog.Log(r.ctx, level, "cache: redis unreachable; bypassing the cache and retrying", + "addrs", r.cfg.Addrs, "error", err) + return err + } + if r.ctx.Err() != nil { + c.Close() + return r.ctx.Err() + } + r.client.Store(&c) + r.nudge() + return nil +} + +func (r *RedisCache) dialLoop() { + defer r.wg.Done() + backoff := time.Second + for { + select { + case <-r.ctx.Done(): + return + case <-time.After(backoff): + } + if r.dial() == nil { + slog.InfoContext(r.ctx, "cache: redis connected", "addrs", r.cfg.Addrs) + return + } + backoff = min(backoff*2, dialMaxBackoff) + } +} + +func isAuthError(err error) bool { + msg := err.Error() + return strings.Contains(msg, "WRONGPASS") || strings.Contains(msg, "NOAUTH") || strings.Contains(msg, "NOPERM") +} + +// conn returns the client to send to, or nil when the server is to be +// skipped: not connected yet, or the breaker open. The caller that finds an +// open breaker due for a probe starts it. +func (r *RedisCache) conn() rueidis.Client { + cp := r.client.Load() + if cp == nil { + return nil + } + ok, probe := r.breaker.allow() + if probe && r.ctx.Err() == nil { + go r.probe(*cp) + } + if !ok { + return nil + } + return *cp +} + +// probe decides whether an open breaker closes. It writes: a server that +// answers but refuses writes (refusesWork) would take no bump either. +// rueidis redials under the calling operation's context, bounding the dial +// (TLS included) by DialTimeout and then the handshake by DialTimeout again, +// so the first write has twice DialTimeout for a reconnect on top of +// Timeout: a reconnect slower than Timeout fails the operations waiting on +// it, but not the probe. The allowance is for a reconnect only: a first +// write slower than Timeout is repeated under Timeout, and the repeat +// decides, so a server answering slower than Timeout stays bypassed. +func (r *RedisCache) probe(c rueidis.Client) { + set := func(budget time.Duration) error { + ctx, cancel := context.WithTimeout(r.ctx, budget) + defer cancel() + return c.Do(ctx, c.B().Set().Key(r.cfg.KeyPrefix+":probe").Value("1").Ex(time.Minute).Build()).Error() + } + start := time.Now() + err := set(2*r.cfg.DialTimeout + r.cfg.Timeout) + if time.Since(start) > r.cfg.Timeout { + err = set(r.cfg.Timeout) + } + r.record(r.ctx, err) + if r.breaker.isOpen() { + return + } + slog.InfoContext(r.ctx, "cache: redis reachable again; cache back in use") + r.nudge() +} + +// bypassed reports whether operations are skipping the server. +func (r *RedisCache) bypassed() bool { + return r.client.Load() == nil || r.breaker.isOpen() +} + +// record feeds an operation's outcome to the breaker. A reply from the +// server, even an error reply or one this code cannot use, shows it is up — +// unless it refuses the work outright, which opens the breaker at once. A +// caller that gave up first shows nothing about it. +func (r *RedisCache) record(parent context.Context, err error) { + if err == nil || rueidis.IsRedisNil(err) { + r.breaker.success() + return + } + if re, ok := rueidis.IsRedisErr(err); ok { + r.recordReply(parent, re.Error()) + return + } + if errors.Is(err, errMalformedReply) { + r.breaker.success() + return + } + if parent.Err() != nil { + return + } + if r.breaker.failure() { + slog.WarnContext(parent, "cache: redis not answering; bypassing the cache", + "addrs", r.cfg.Addrs, "error", err) + } +} + +// recordReply is record for an error reply. The breaker opening is logged +// once, not once per operation in flight. +func (r *RedisCache) recordReply(parent context.Context, msg string) { + switch { + case rejectsCredentials(msg): + if r.breaker.trip() { + slog.ErrorContext(parent, "cache: redis rejected the credentials; bypassing the cache until they work", + "addrs", r.cfg.Addrs, "error", msg) + } + case refusesWork(msg): + if r.breaker.trip() { + slog.WarnContext(parent, "cache: redis refusing writes; bypassing the cache", + "addrs", r.cfg.Addrs, "reply", msg) + } + default: + r.breaker.success() + } +} + +// refusesWork reports whether an error reply says the server takes no +// writes from anyone right now, so no bump can land: a replica (READONLY, +// or MASTERDOWN, which refuses reads too), memory full under noeviction +// (OOM), writes stopped by min-replicas-to-write (NOREPLICAS) or a failed +// snapshot (MISCONF), a dataset still loading (LOADING), a script holding +// the server (BUSY), a cluster not serving the slot (CLUSTERDOWN). Replies +// about one key or one moment — WRONGTYPE, NOPERM, TRYAGAIN during a slot +// migration — are not. +func refusesWork(msg string) bool { + code, _, _ := strings.Cut(msg, " ") + switch code { + case "READONLY", "MASTERDOWN", "OOM", "NOREPLICAS", "MISCONF", "LOADING", "BUSY", "CLUSTERDOWN": + return true + } + return false +} + +// rejectsCredentials reports whether an error reply refuses this process's +// credentials, as a connection's handshake after a password rotation does: +// every operation meets it, so it refuses all work. NOPERM is not one: it +// names a key or a command, which the rest of the work may not touch. +func rejectsCredentials(msg string) bool { + code, _, _ := strings.Cut(msg, " ") + return code == "WRONGPASS" || code == "NOAUTH" +} + +// Lookup reads the tokens deps fold and the entry for sha in one pipelined +// round trip. A token that does not exist yet is created, never read as a +// value, so its first use is a miss. +func (r *RedisCache) Lookup(ctx context.Context, id tenant.ID, sha string, deps []Namespace) (Entry, Snapshot, error) { + for _, d := range deps { + if d.Tenant != id { + return Entry{}, Snapshot{}, fmt.Errorf("%w: %q under %q", ErrForeignDependency, d.Tenant, id) + } + } + keys := tokenKeys(r.cfg.KeyPrefix, id, deps) + if len(keys) > maxTokenKeys { + return Entry{}, Snapshot{}, fmt.Errorf("cache: %d dependencies is more than a value can record", len(deps)) + } + // A bump this process owes would orphan what a lookup reading its key + // finds, so that lookup is a bypass: no hit, and a zero snapshot, so no + // fill either. A lookup's token keys are exactly those whose bumps orphan + // its entry, and include the tenant token a set past PendingMax collapses + // to. Other processes cannot know what this one owes, and serve those + // entries until the bump lands. + if r.pending.owesAny(keys) { + r.metrics.lookup(resultBypass) + return Entry{}, Snapshot{}, nil + } + c := r.conn() + if c == nil { + r.metrics.lookup(resultBypass) + return Entry{}, Snapshot{}, nil + } + defer r.metrics.op("lookup", time.Now()) + opCtx, cancel := context.WithTimeout(ctx, r.cfg.Timeout) + defer cancel() + + vkey := valueKey(r.cfg.KeyPrefix, id, sha, deps) + res := c.DoMulti(opCtx, c.B().Mget().Key(keys...).Build(), c.B().Get().Key(vkey).Build()) + if err := res[0].Error(); err != nil { + return r.lookupFailed(ctx, err) + } + // An error reply such as WRONGTYPE: the value key holds something else, + // which the fill's plain SET replaces, so it is a miss to fill. + valErr := res[1].Error() + _, valReplied := rueidis.IsRedisErr(valErr) + if valErr != nil && !rueidis.IsRedisNil(valErr) && !valReplied { + return r.lookupFailed(ctx, valErr) + } + r.record(ctx, nil) + tokens, missing, foreign, err := readTokens(res[0]) + if err != nil { + return r.lookupFailed(ctx, err) + } + var val []byte + if valErr == nil { + if val, err = res[1].AsBytes(); err != nil { + return r.lookupFailed(ctx, fmt.Errorf("%w: %w", errMalformedReply, err)) + } + } + + if len(missing) > 0 || len(foreign) > 0 { + if tokens, err = r.createTokens(opCtx, c, keys, missing, foreign); err != nil { + return r.lookupFailed(ctx, err) + } + r.metrics.lookup(resultMiss) + return Entry{}, Snapshot{key: vkey, tokens: tokens}, nil + } + snap := Snapshot{key: vkey, tokens: tokens} + if val == nil { + r.metrics.lookup(resultMiss) + return Entry{}, snap, nil + } + stored, expiresAt, payload, err := r.codec.decode(val) + if err != nil { + slog.DebugContext(ctx, "cache: unreadable value; treating as a miss", "key", vkey, "error", err) + r.metrics.lookup(resultMiss) + return Entry{}, snap, nil + } + remaining := time.Until(expiresAt) + if !bytes.Equal(stored, tokens) || remaining <= 0 { + r.metrics.lookup(resultStale) + return Entry{}, snap, nil + } + r.metrics.lookup(resultHit) + return Entry{Value: payload, TTL: remaining}, snap, nil +} + +func (r *RedisCache) lookupFailed(ctx context.Context, err error) (Entry, Snapshot, error) { + r.record(ctx, err) + r.metrics.lookup(resultError) + return Entry{}, Snapshot{}, fmt.Errorf("cache: redis lookup: %w", err) +} + +// readTokens concatenates an MGET reply's tokens, listing the indexes of the +// keys that do not exist and of those holding something that is not a token. +func readTokens(res rueidis.RedisResult) (tokens []byte, missing, foreign []int, err error) { + msgs, err := res.ToArray() + if err != nil { + return nil, nil, nil, fmt.Errorf("%w: %w", errMalformedReply, err) + } + tokens = make([]byte, 0, len(msgs)*tokenLen) + for i := range msgs { + if msgs[i].IsNil() { + missing = append(missing, i) + tokens = append(tokens, make([]byte, tokenLen)...) + continue + } + s, err := msgs[i].ToString() + if err != nil || len(s) != tokenLen { + foreign = append(foreign, i) + tokens = append(tokens, make([]byte, tokenLen)...) + continue + } + tokens = append(tokens, s...) + } + return tokens, missing, foreign, nil +} + +// createTokens sets each missing token — only if still missing, as another +// process may create it first — replaces each foreign one (a string that is +// not a token, or, on a second round trip, a key of another type), and reads +// them all back, in one round trip: the tokens share a slot, so the pipeline runs +// in order on one node. A fresh token can only cause misses, so replacing +// whatever held a token key is safe. +func (r *RedisCache) createTokens(ctx context.Context, c rueidis.Client, keys []string, missing, foreign []int) ([]byte, error) { + if len(foreign) > 0 { + slog.WarnContext(ctx, "cache: replacing values that are not version tokens; is another program writing under this key prefix?", + "keys", len(foreign), "prefix", r.cfg.KeyPrefix) + } + cmds := make(rueidis.Commands, 0, len(missing)+len(foreign)+1) + for _, i := range missing { + tok := newToken() + cmds = append(cmds, c.B().Set().Key(keys[i]).Value(rueidis.BinaryString(tok)).Nx().Ex(jitter(r.cfg.VersionTTL, tok)).Build()) + } + for _, i := range foreign { + cmds = append(cmds, r.bumpCmd(c, keys[i])) + } + cmds = append(cmds, c.B().Mget().Key(keys...).Build()) + res := c.DoMulti(ctx, cmds...) + for _, rr := range res { + if err := rr.Error(); err != nil && !rueidis.IsRedisNil(err) { + return nil, err + } + } + tokens, still, bad, err := readTokens(res[len(res)-1]) + if err != nil { + return nil, err + } + if len(still) == 0 && len(bad) == 0 { + return tokens, nil + } + // Still nil after SET NX: the key holds a list, hash or other non-string, + // which MGET reads as nil and NX will not overwrite. Replace it too. + if len(foreign) == 0 && len(still) > 0 { + return r.createTokens(ctx, c, keys, nil, still) + } + return nil, fmt.Errorf("%w: version token gone or replaced as it was written", errMalformedReply) +} + +// Set stores value with the tokens snap read, for ttl. A value over the size +// limit is not stored; neither is anything while the server is bypassed. +func (r *RedisCache) Set(ctx context.Context, snap Snapshot, value []byte, ttl time.Duration) error { + if snap.key == "" || ttl <= 0 { + return nil + } + if len(value) > r.maxDecoded { + r.metrics.tooLarge() + return nil + } + b := r.codec.encode(snap.tokens, time.Now().Add(ttl), value) + if len(b) > r.cfg.MaxValueBytes { + r.metrics.tooLarge() + return nil + } + c := r.conn() + if c == nil { + return nil + } + defer r.metrics.op("set", time.Now()) + opCtx, cancel := context.WithTimeout(ctx, r.cfg.Timeout) + defer cancel() + err := c.Do(opCtx, c.B().Set().Key(snap.key).Value(rueidis.BinaryString(b)).Px(max(ttl, time.Millisecond)).Build()).Error() + r.record(ctx, err) + if err != nil { + r.metrics.setFailed(setFailureReason(err)) + return fmt.Errorf("cache: redis set: %w", err) + } + r.metrics.stored(len(b)) + return nil +} + +func setFailureReason(err error) string { + if re, ok := rueidis.IsRedisErr(err); ok && strings.HasPrefix(re.Error(), "OOM") { + return "oom" + } + if errors.Is(err, context.DeadlineExceeded) { + return "timeout" + } + return "other" +} + +// Invalidate sets a fresh token for every token the namespaces' writes +// reach, in pipelined batches of drainBatch, one round trip each. Bumps the server does not take are +// kept and retried until it does, and reported as an error meanwhile. +func (r *RedisCache) Invalidate(ctx context.Context, namespaces []Namespace) (uint64, error) { + owner := map[string]tenant.ID{} + for _, ns := range namespaces { + for _, k := range bumpKeys(r.cfg.KeyPrefix, ns) { + owner[k] = ns.Tenant + } + } + return uint64(len(namespaces)), r.bump(ctx, owner) +} + +// InvalidateTenant sets a fresh tenant token, orphaning every entry of id. +func (r *RedisCache) InvalidateTenant(ctx context.Context, id tenant.ID) error { + return r.bump(ctx, map[string]tenant.ID{tenantTokenKey(r.cfg.KeyPrefix, id): id}) +} + +func (r *RedisCache) bump(ctx context.Context, owner map[string]tenant.ID) error { + if len(owner) == 0 { + return nil + } + c := r.conn() + if c == nil { + r.deferBumps(owner) + return fmt.Errorf("%w: %d invalidations deferred", errBypassed, len(owner)) + } + defer r.metrics.op("invalidate", time.Now()) + failed := map[string]tenant.ID{} + var firstErr error + // In batches, so a wide fan-out is several round trips each within the + // timeout; past a failure the rest are deferred unsent. + for batch := range slices.Chunk(slices.Collect(maps.Keys(owner)), drainBatch) { + var landed []bool + if firstErr == nil { + landed, firstErr = r.sendBumps(ctx, c, batch) + } + for i, k := range batch { + if landed == nil || !landed[i] { + failed[k] = owner[k] + } + } + } + r.record(ctx, firstErr) + r.metrics.invalidated("ok", len(owner)-len(failed)) + if len(failed) > 0 { + r.deferBumps(failed) + return fmt.Errorf("cache: redis invalidate (%d deferred): %w", len(failed), firstErr) + } + return nil +} + +func (r *RedisCache) bumpCmd(c rueidis.Client, key string) rueidis.Completed { + tok := newToken() + return c.B().Set().Key(key).Value(rueidis.BinaryString(tok)).Ex(jitter(r.cfg.VersionTTL, tok)).Build() +} + +// sendBumps sets a fresh token under each key in one pipeline, within the +// op timeout, reporting which landed and the first error. +func (r *RedisCache) sendBumps(ctx context.Context, c rueidis.Client, keys []string) (landed []bool, firstErr error) { + cmds := make(rueidis.Commands, 0, len(keys)) + for _, k := range keys { + cmds = append(cmds, r.bumpCmd(c, k)) + } + opCtx, cancel := context.WithTimeout(ctx, r.cfg.Timeout) + defer cancel() + landed = make([]bool, len(keys)) + for i, rr := range c.DoMulti(opCtx, cmds...) { + err := rr.Error() + landed[i] = err == nil + firstErr = cmpOr(firstErr, err) + } + return landed, firstErr +} + +func (r *RedisCache) deferBumps(owner map[string]tenant.ID) { + first := r.pending.len() == 0 + for k, id := range owner { + r.pending.add(id, k) + } + r.metrics.invalidated("deferred", len(owner)) + // The first bump owed wakes the drain now, not at its next tick: it is + // retried at once, or, past an open breaker, when the probe is due. + // Later ones join the retry already backing off. + if first { + r.nudge() + } +} + +// nudge wakes the drain loop now, rather than at its next tick. +func (r *RedisCache) nudge() { + select { + case r.wake <- struct{}{}: + default: + } +} + +func (r *RedisCache) drainLoop() { + defer r.wg.Done() + backoff := drainMinBackoff + timer := time.NewTimer(drainIdle) + defer timer.Stop() + for { + select { + case <-r.ctx.Done(): + return + case <-r.wake: + backoff = drainMinBackoff + case <-timer.C: + } + wait := drainIdle + if r.breaker.isOpen() { + r.conn() // starts the probe when due + } + if r.pending.len() > 0 { + if r.drain(r.ctx, r.conn) { + backoff = drainMinBackoff + } else { + wait, backoff = backoff, min(backoff*2, drainMaxBackoff) + } + } + // Lookups start the probe that closes an open breaker; a process + // with none (ingest only) has this loop, which wakes when the probe + // is due rather than at the drain's backoff, so it recovers as soon. + if d, open := r.breaker.untilProbe(); open { + wait = min(wait, d) + } + timer.Reset(wait) + } +} + +// drain delivers the pending bumps through the client conn returns, +// reporting whether none remain. +func (r *RedisCache) drain(ctx context.Context, conn func() rueidis.Client) bool { + owed := r.pending.snapshot() + keys := make([]string, 0, len(owed)) + for k := range owed { + keys = append(keys, k) + } + for batch := range slices.Chunk(keys, drainBatch) { + c := conn() + if c == nil { + return false + } + sent, err := r.sendBumps(ctx, c, batch) + landed := map[string]uint64{} + for i, k := range batch { + if sent[i] { + landed[k] = owed[k] + } + } + r.record(ctx, err) + r.pending.done(landed) + r.metrics.invalidated("ok", len(landed)) + if err != nil { + return false + } + } + return r.pending.len() == 0 +} + +// Close stops the background loops, makes one last attempt at the pending +// bumps, and closes the connection. Bumps still undelivered are lost: the +// entries they would orphan are served until their TTL. +func (r *RedisCache) Close() error { + r.closeOnce.Do(func() { + r.cancel() + r.wg.Wait() + if r.pending.len() > 0 { + ctx, cancel := context.WithTimeout(context.Background(), closeDrainBudget) + // Past the breaker: an open one is why bumps are pending, and this + // is the last chance to deliver them. + r.drain(ctx, func() rueidis.Client { + if cp := r.client.Load(); cp != nil { + return *cp + } + return nil + }) + cancel() + if n := r.pending.len(); n > 0 { + slog.Warn("cache: closing with undelivered invalidations; entries they orphan stay cached until their TTL", "pending", n) + } + } + if cp := r.client.Load(); cp != nil { + (*cp).Close() + } + r.codec.close() + r.metrics.close() + }) + return nil +} diff --git a/internal/cache/redis_codec.go b/internal/cache/redis_codec.go new file mode 100644 index 000000000..bb9a853ef --- /dev/null +++ b/internal/cache/redis_codec.go @@ -0,0 +1,217 @@ +package cache + +import ( + "cmp" + "crypto/rand" + "crypto/sha256" + "encoding/binary" + "encoding/hex" + "errors" + "fmt" + "slices" + "time" + + "github.com/klauspost/compress/zstd" + + "github.com/Wave-RF/WaveHouse/internal/keyenc" + "github.com/Wave-RF/WaveHouse/internal/tenant" +) + +// tokenLen is the size of a version token: random, so a token key that is +// lost (evicted, expired, flushed) and recreated can never match a value +// stored under its predecessor, as a counter restarting at 0 would. +const tokenLen = 8 + +// decodedFactor bounds a value's decompressed size at this multiple of the +// stored-size limit, refusing a zip bomb planted in a shared server. +const decodedFactor = 8 + +const encoderWindow = 1 << 20 + +// Value layout: format, flags, expires-at (unix ms), token count, tokens, +// payload. Big-endian. +const ( + valueFormat = 1 + flagZstd = 1 << 0 + headerLen = 1 + 1 + 8 + 2 + maxTokenKeys = 1<<16 - 1 +) + +var errCorruptValue = errors.New("cache: corrupt value") + +func newToken() []byte { + b := make([]byte, tokenLen) + _, _ = rand.Read(b) // never fails (crypto/rand, Go ≥ 1.24) + return b +} + +// tenantTokenKey is the key of tenant id's token. Every token key carries +// the tenant as a hash tag, so all of a tenant's tokens share one cluster +// slot and a lookup reads them with one MGET. The tenant goes in verbatim: +// its grammar is keyenc's kept bytes, so it is its own escaped form. +func tenantTokenKey(prefix string, id tenant.ID) string { + return prefix + ":{" + string(id) + "}:T" +} + +// tableTokenKey and scopeTokenKey take raw names and escape them after the +// fixed prefix (keyenc), so no ':' in a name reads as the separator. +// +// Their layout is a protocol between builds: every process on the server +// reads and bumps these keys for itself, so two builds that lay them out +// differently — a change here, or to what keyenc keeps — split them, and one +// build's bumps miss the entries the other filed, which are served until +// their TTL for the whole rolling deploy. Such a change needs the new build +// to read and bump both layouts (fold the old tokens into what it files) +// until no old build is left, a later build dropping the old; or an upgrade +// that never runs two builds against the server at once. The tenant token, +// placed verbatim, stays shared; a valueKey change only orphans values, +// which is safe to roll. +func tableTokenKey(prefix string, id tenant.ID, table string) string { + return string(keyenc.AppendJoin([]byte(prefix+":{"+string(id)+"}:B:"), ':', table)) +} + +func scopeTokenKey(prefix string, id tenant.ID, table, scope string) string { + return string(keyenc.AppendJoin([]byte(prefix+":{"+string(id)+"}:S:"), ':', table, scope)) +} + +// sortedDeps returns deps in canonical order without duplicates. +func sortedDeps(deps []Namespace) []Namespace { + out := slices.Clone(deps) + slices.SortFunc(out, func(a, b Namespace) int { + return cmp.Or(cmp.Compare(a.Table, b.Table), cmp.Compare(a.Scope, b.Scope)) + }) + return slices.Compact(out) +} + +// tokenKeys lists the tokens a result for deps of tenant id is filed under, +// in canonical order: the tenant's, then each dep's table and scope tokens. +// A dep with scope s folds B:table (bumped by a whole-table write) and +// S:table:s (bumped by a write to s, and — for s == "" — by any scoped write +// to the table), the same lattice LocalCache's version index encodes. +func tokenKeys(prefix string, id tenant.ID, deps []Namespace) []string { + keys := []string{tenantTokenKey(prefix, id)} + for _, d := range sortedDeps(deps) { + keys = append(keys, tableTokenKey(prefix, id, d.Table), scopeTokenKey(prefix, id, d.Table, d.Scope)) + } + slices.Sort(keys[1:]) + return append(keys[:1], slices.Compact(keys[1:])...) +} + +// bumpKeys lists the tokens an invalidation of ns replaces: a whole-table +// write the table's, a scoped write its scope's and the whole-table view's. +func bumpKeys(prefix string, ns Namespace) []string { + if ns.Scope == "" { + return []string{tableTokenKey(prefix, ns.Tenant, ns.Table)} + } + return []string{scopeTokenKey(prefix, ns.Tenant, ns.Table, ns.Scope), scopeTokenKey(prefix, ns.Tenant, ns.Table, "")} +} + +// valueKey names the entry for sha over deps. It carries no versions, so a +// refill overwrites in place, and no hash tag, so one tenant's values spread +// across a cluster's shards. It hashes the escaped sha and each dep's +// escaped, joined table and scope, each ended by a NUL, which escaping never +// writes, so no two sets of names hash the same input. +func valueKey(prefix string, id tenant.ID, sha string, deps []Namespace) string { + h := sha256.New() + b := keyenc.AppendEscape(nil, sha) + h.Write(append(b, 0)) + for _, d := range sortedDeps(deps) { + b = keyenc.AppendJoin(b[:0], ':', d.Table, d.Scope) + h.Write(append(b, 0)) + } + return prefix + ":q:" + string(id) + ":" + hex.EncodeToString(h.Sum(nil)) +} + +// codec compresses and frames values. Its zstd encoder and decoder are safe +// for concurrent EncodeAll/DecodeAll. +// +// The encoder keeps one window-sized history per concurrent caller for the +// life of the process; 1 MiB, not SpeedFastest's 4, cuts that about +// threefold at no measurable cost in speed or ratio on row payloads. The +// decoder takes any window up to maxDecoded, so a value written with a +// larger one still reads. +type codec struct { + enc *zstd.Encoder + dec *zstd.Decoder + compressMin int + maxDecoded int +} + +func newCodec(compressMin, maxDecoded int) (*codec, error) { + enc, err := zstd.NewWriter(nil, zstd.WithEncoderLevel(zstd.SpeedFastest), zstd.WithWindowSize(encoderWindow)) + if err != nil { + return nil, err + } + dec, err := zstd.NewReader(nil, zstd.WithDecoderMaxMemory(uint64(maxDecoded)), zstd.WithDecoderConcurrency(0)) //nolint:gosec // maxDecoded is a positive config-derived int + if err != nil { + return nil, err + } + return &codec{enc: enc, dec: dec, compressMin: compressMin, maxDecoded: maxDecoded}, nil +} + +func (c *codec) close() { + _ = c.enc.Close() + c.dec.Close() +} + +// encode frames payload with the tokens it was computed under, compressing +// it when that is enabled, the payload is large enough, and it helps. +func (c *codec) encode(tokens []byte, expiresAt time.Time, payload []byte) []byte { + var flags byte + body := payload + if c.compressMin > 0 && len(payload) >= c.compressMin { + if z := c.enc.EncodeAll(payload, nil); len(z) < len(payload) { + body, flags = z, flagZstd + } + } + out := make([]byte, headerLen, headerLen+len(tokens)+len(body)) + out[0] = valueFormat + out[1] = flags + binary.BigEndian.PutUint64(out[2:10], uint64(expiresAt.UnixMilli())) + binary.BigEndian.PutUint16(out[10:12], uint16(len(tokens)/tokenLen)) //nolint:gosec // bounded by maxTokenKeys at Lookup + out = append(out, tokens...) + return append(out, body...) +} + +// decode is encode's inverse. An unknown format — a newer process's value +// during a rolling upgrade — is an error, which the caller reads as a miss. +func (c *codec) decode(b []byte) (tokens []byte, expiresAt time.Time, payload []byte, err error) { + if len(b) < headerLen { + return nil, time.Time{}, nil, errCorruptValue + } + if b[0] != valueFormat { + return nil, time.Time{}, nil, fmt.Errorf("%w: format %d", errCorruptValue, b[0]) + } + flags := b[1] + expiresAt = time.UnixMilli(int64(binary.BigEndian.Uint64(b[2:10]))) //nolint:gosec // written by encode + n := int(binary.BigEndian.Uint16(b[10:12])) * tokenLen + if len(b) < headerLen+n { + return nil, time.Time{}, nil, errCorruptValue + } + tokens = b[headerLen : headerLen+n] + payload = b[headerLen+n:] + switch flags { + case 0: + case flagZstd: + if payload, err = c.dec.DecodeAll(payload, nil); err != nil { + return nil, time.Time{}, nil, fmt.Errorf("%w: %w", errCorruptValue, err) + } + if len(payload) > c.maxDecoded { + return nil, time.Time{}, nil, fmt.Errorf("%w: decodes to %d bytes", errCorruptValue, len(payload)) + } + default: + return nil, time.Time{}, nil, fmt.Errorf("%w: flags %#x", errCorruptValue, flags) + } + return tokens, expiresAt, payload, nil +} + +// jitter spreads d by ±10%, drawing on the token being written so a batch of +// bumps doesn't expire together. +func jitter(d time.Duration, token []byte) time.Duration { + span := int64(d) / 5 + if span <= 0 { + return d + } + r := int64(binary.BigEndian.Uint64(token) % uint64(span)) //nolint:gosec // r < span, an int64 + return d - time.Duration(span/2) + time.Duration(r) +} diff --git a/internal/cache/redis_codec_test.go b/internal/cache/redis_codec_test.go new file mode 100644 index 000000000..a61d84dd4 --- /dev/null +++ b/internal/cache/redis_codec_test.go @@ -0,0 +1,258 @@ +package cache + +import ( + "bytes" + "strings" + "testing" + "time" + + "github.com/klauspost/compress/zstd" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestTokenKeys(t *testing.T) { + t.Parallel() + tests := []struct { + name string + deps []Namespace + want []string + }{ + {"no deps is the tenant token alone", nil, []string{"wh:{acme}:T"}}, + { + "a dep folds its table and scope tokens", + []Namespace{{Tenant: "acme", Table: "events", Scope: "org_1"}}, + []string{"wh:{acme}:T", "wh:{acme}:B:events", "wh:{acme}:S:events:org_1"}, + }, + { + "a scopeless dep reads the whole-table view", + []Namespace{{Tenant: "acme", Table: "events"}}, + []string{"wh:{acme}:T", "wh:{acme}:B:events", "wh:{acme}:S:events:"}, + }, + { + "names arrive raw and are escaped after the hash tag", + []Namespace{{Tenant: "acme", Table: "default.clicks", Scope: "org:1"}}, + []string{"wh:{acme}:T", "wh:{acme}:B:default%2Eclicks", "wh:{acme}:S:default%2Eclicks:org%3A1"}, + }, + { + "shared table tokens and duplicate deps appear once, sorted", + []Namespace{ + {Tenant: "acme", Table: "orders"}, + {Tenant: "acme", Table: "events", Scope: "b"}, + {Tenant: "acme", Table: "events", Scope: "a"}, + {Tenant: "acme", Table: "orders"}, + }, + []string{ + "wh:{acme}:T", "wh:{acme}:B:events", "wh:{acme}:B:orders", + "wh:{acme}:S:events:a", "wh:{acme}:S:events:b", "wh:{acme}:S:orders:", + }, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + assert.Equal(t, tt.want, tokenKeys("wh", "acme", tt.deps)) + }) + } +} + +func TestBumpKeys(t *testing.T) { + t.Parallel() + assert.Equal(t, []string{"p:{acme}:B:events"}, bumpKeys("p", Namespace{Tenant: "acme", Table: "events"})) + assert.Equal(t, []string{"p:{acme}:S:events:org_1", "p:{acme}:S:events:"}, + bumpKeys("p", Namespace{Tenant: "acme", Table: "events", Scope: "org_1"})) + assert.Equal(t, []string{"p:{acme}:S:my%20table:a%3Ab", "p:{acme}:S:my%20table:"}, + bumpKeys("p", Namespace{Tenant: "acme", Table: "my table", Scope: "a:b"})) + + // A ':' in a name never reads as the separator: table "a:b" with scope + // "c" and table "a" with scope "b:c" bump two tokens. + assert.NotEqual(t, + bumpKeys("p", Namespace{Tenant: "acme", Table: "a:b", Scope: "c"}), + bumpKeys("p", Namespace{Tenant: "acme", Table: "a", Scope: "b:c"})) +} + +// Every key a bump writes is one some lookup reads: otherwise the bump +// orphans nothing. +func TestBumpKeysAreReadByLookups(t *testing.T) { + t.Parallel() + for _, ns := range []Namespace{ + {Tenant: "acme", Table: "events"}, + {Tenant: "acme", Table: "events", Scope: "org_1"}, + {Tenant: "acme", Table: "default.clicks"}, + {Tenant: "acme", Table: "my table", Scope: "org:1"}, + } { + read := map[string]bool{} + for _, scope := range []string{"", ns.Scope} { + for _, k := range tokenKeys("wh", "acme", []Namespace{{Tenant: "acme", Table: ns.Table, Scope: scope}}) { + read[k] = true + } + } + for _, k := range bumpKeys("wh", ns) { + assert.True(t, read[k], k) + } + } +} + +func TestValueKey(t *testing.T) { + t.Parallel() + a, b := Namespace{Tenant: "acme", Table: "events"}, Namespace{Tenant: "acme", Table: "orders", Scope: "x"} + k := valueKey("wh", "acme", "acme:query:abc", []Namespace{a, b}) + assert.True(t, strings.HasPrefix(k, "wh:q:acme:"), k) + assert.NotContains(t, k, "{", "values carry no hash tag, so they spread across shards") + assert.Equal(t, k, valueKey("wh", "acme", "acme:query:abc", []Namespace{b, a, b}), "order and duplicates do not matter") + assert.NotEqual(t, k, valueKey("wh", "acme", "acme:query:abc", []Namespace{a})) + assert.NotEqual(t, k, valueKey("wh", "acme", "acme:query:abd", []Namespace{a, b})) + // Names that would run together unescaped hash apart: at the table and + // scope boundary, at a NUL, and across dependencies. + for _, pair := range [][2][]Namespace{ + {{{Tenant: "acme", Table: "ab", Scope: "c"}}, {{Tenant: "acme", Table: "a", Scope: "bc"}}}, + {{{Tenant: "acme", Table: "a\x00b"}}, {{Tenant: "acme", Table: "a", Scope: "b\x00"}}}, + {{{Tenant: "acme", Table: "a:b"}}, {{Tenant: "acme", Table: "a", Scope: "b"}}}, + {{{Tenant: "acme", Table: "a"}, {Tenant: "acme", Table: "b"}}, {{Tenant: "acme", Table: "a", Scope: "\x00b\x00"}}}, + } { + assert.NotEqual(t, valueKey("wh", "acme", "q", pair[0]), valueKey("wh", "acme", "q", pair[1]), "%q", pair) + } +} + +func newTestCodec(t *testing.T, compressMin, maxDecoded int) *codec { + t.Helper() + c, err := newCodec(compressMin, maxDecoded) + require.NoError(t, err) + t.Cleanup(c.close) + return c +} + +func TestCodec_RoundTrip(t *testing.T) { + t.Parallel() + c := newTestCodec(t, 64, 1<<20) + tokens := append(newToken(), newToken()...) + exp := time.UnixMilli(time.Now().Add(time.Minute).UnixMilli()) + tests := []struct { + name string + payload []byte + compressed bool + }{ + {"empty", []byte{}, false}, + {"below the threshold", []byte(`[{"a":1}]`), false}, + {"compressible", bytes.Repeat([]byte(`{"user":"u1","n":42},`), 200), true}, + {"incompressible stays raw", randomBytes(4096), false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + b := c.encode(tokens, exp, tt.payload) + assert.Equal(t, tt.compressed, b[1]&flagZstd != 0) + if tt.compressed { + assert.Less(t, len(b), len(tt.payload)) + } + gotTokens, gotExp, gotPayload, err := c.decode(b) + require.NoError(t, err) + assert.Equal(t, tokens, gotTokens) + assert.True(t, exp.Equal(gotExp)) + assert.Equal(t, tt.payload, gotPayload) + }) + } +} + +func TestCodec_CompressionOff(t *testing.T) { + t.Parallel() + c := newTestCodec(t, 0, 1<<20) + b := c.encode(newToken(), time.Now(), bytes.Repeat([]byte("a"), 4096)) + assert.Zero(t, b[1]) +} + +func TestCodec_RefusesBadValues(t *testing.T) { + t.Parallel() + c := newTestCodec(t, 1, 1<<10) + good := c.encode(newToken(), time.Now(), []byte("rows")) + bomb := newTestCodec(t, 1, 1<<30).encode(newToken(), time.Now(), make([]byte, 1<<20)) + require.Less(t, len(bomb), 1<<10, "the bomb is small when stored") + + withByte := func(i int, v byte) []byte { + b := bytes.Clone(good) + b[i] = v + return b + } + tests := []struct { + name string + b []byte + }{ + {"shorter than the header", good[:headerLen-1]}, + {"unknown format", withByte(0, 2)}, + {"unknown flags", withByte(1, 0x80)}, + {"fewer tokens than it declares", withByte(11, 200)}, + {"not zstd", append(withByte(1, flagZstd)[:headerLen+tokenLen], "not zstd"...)}, + {"decodes past the limit", bomb}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + _, _, _, err := c.decode(tt.b) + require.ErrorIs(t, err, errCorruptValue) + }) + } +} + +// The encoder's window bounds the history it keeps per concurrent caller, +// and a value an encoder with a wider one wrote (an earlier build's) still +// decodes. +func TestCodec_EncoderWindow(t *testing.T) { + t.Parallel() + window := func(t *testing.T, b []byte) uint64 { + t.Helper() + var h zstd.Header + require.NoError(t, h.Decode(b[headerLen:])) + if h.SingleSegment { + return h.FrameContentSize + } + return h.WindowSize + } + c := newTestCodec(t, 1, 8<<20) + wide, err := zstd.NewWriter(nil, zstd.WithEncoderLevel(zstd.SpeedFastest)) + require.NoError(t, err) + t.Cleanup(func() { _ = wide.Close() }) + earlier := &codec{enc: wide, compressMin: 1} + + for _, n := range []int{2 << 20, 6 << 20} { + payload := bytes.Repeat([]byte(`{"user_id":"u-1","event":"click","value":42.5},`), n/48) + b := c.encode(nil, time.Now(), payload) + require.Equal(t, byte(flagZstd), b[1]) + assert.LessOrEqual(t, window(t, b), uint64(encoderWindow), n) + + b = earlier.encode(nil, time.Now(), payload) + require.Greater(t, window(t, b), uint64(encoderWindow), n) + _, _, got, err := c.decode(b) + require.NoError(t, err, n) + assert.Equal(t, payload, got, n) + } +} + +func TestJitter(t *testing.T) { + t.Parallel() + d := 100 * time.Second + for range 1000 { + j := jitter(d, newToken()) + assert.GreaterOrEqual(t, j, 90*time.Second) + assert.Less(t, j, 110*time.Second) + } + assert.Equal(t, time.Nanosecond, jitter(time.Nanosecond, newToken())) +} + +func TestNewToken(t *testing.T) { + t.Parallel() + seen := map[string]bool{} + for range 1000 { + tok := newToken() + require.Len(t, tok, tokenLen) + require.False(t, seen[string(tok)]) + seen[string(tok)] = true + } +} + +func randomBytes(n int) []byte { + b := make([]byte, 0, n) + for len(b) < n { + b = append(b, newToken()...) + } + return b[:n] +} diff --git a/internal/cache/redis_integration_test.go b/internal/cache/redis_integration_test.go new file mode 100644 index 000000000..f19ea8059 --- /dev/null +++ b/internal/cache/redis_integration_test.go @@ -0,0 +1,1046 @@ +//go:build integration + +package cache_test + +import ( + "bytes" + "context" + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/tls" + "crypto/x509" + "fmt" + "io" + "math/big" + "net" + "slices" + "strconv" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/moby/moby/api/types/container" + "github.com/moby/moby/api/types/network" + "github.com/moby/moby/client" + "github.com/redis/rueidis" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "github.com/testcontainers/testcontainers-go" + "github.com/testcontainers/testcontainers-go/wait" + + "github.com/Wave-RF/WaveHouse/internal/cache" + "github.com/Wave-RF/WaveHouse/internal/tenant" + "github.com/Wave-RF/WaveHouse/internal/testutil/cachetest" +) + +// Pinned: the servers the shared cache is documented to run on. +const ( + redisImage = "redis:8.10.2-alpine" + valkeyImage = "valkey/valkey:8.1.10-alpine" + dragonflyImage = "docker.dragonflydb.io/dragonflydb/dragonfly:v2.0.0" +) + +// maxValue is the stored-size limit the tests run with; small, so the +// oversize case stays cheap. +const maxValue = 64 << 10 + +type server struct { + ctr testcontainers.Container + addr string + mode string +} + +// noPersistence keeps the image's VOLUME /data off an anonymous volume. +func noPersistence(hc *container.HostConfig) { + hc.Tmpfs = map[string]string{"/data": ""} +} + +func startContainer(t *testing.T, req testcontainers.ContainerRequest, port string) (testcontainers.Container, string) { + t.Helper() + ctx := context.Background() + if req.HostConfigModifier == nil { + req.HostConfigModifier = noPersistence + } + req.WaitingFor = wait.ForListeningPort(port).WithStartupTimeout(90 * time.Second) + ctr, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{ContainerRequest: req, Started: true}) + testcontainers.CleanupContainer(t, ctr) + require.NoError(t, err) + host, err := ctr.Host(ctx) + require.NoError(t, err) + mapped, err := ctr.MappedPort(ctx, port) + require.NoError(t, err) + return ctr, net.JoinHostPort(host, mapped.Port()) +} + +func startStandalone(t *testing.T, image string, cmd ...string) *server { + t.Helper() + ctr, addr := startContainer(t, testcontainers.ContainerRequest{ + Image: image, Cmd: cmd, ExposedPorts: []string{"6379/tcp"}, + }, "6379/tcp") + s := &server{ctr: ctr, addr: addr, mode: cache.RedisStandalone} + waitReady(t, s, nil) + return s +} + +func startRedis(t *testing.T) *server { + return startStandalone(t, redisImage, "redis-server", "--save", "", "--appendonly", "no") +} + +func startValkey(t *testing.T) *server { + return startStandalone(t, valkeyImage, "valkey-server", "--save", "", "--appendonly", "no") +} + +func startDragonfly(t *testing.T) *server { + return startStandalone(t, dragonflyImage, "--proactor_threads=2", "--maxmemory=512mb") +} + +// startCluster runs a one-node Redis Cluster owning every slot: enough for +// the server to enforce CROSSSLOT on a multi-key command, which is what the +// key schema must survive, and for the cluster client to read the topology. +// It never routes across nodes: a node owning every slot never answers +// MOVED. The node announces 127.0.0.1 on a host port bound to the same +// number, so the address the client learns from CLUSTER SLOTS is dialable +// from the test. +func startCluster(t *testing.T) *server { + t.Helper() + port := freePort(t) + p := network.MustParsePort(port + "/tcp") + ctr, _ := startContainer(t, testcontainers.ContainerRequest{ + Image: redisImage, + Cmd: []string{ + "redis-server", "--port", port, "--cluster-enabled", "yes", "--cluster-port", "16379", + "--cluster-announce-ip", "127.0.0.1", "--save", "", "--appendonly", "no", + }, + ExposedPorts: []string{port + "/tcp"}, + HostConfigModifier: func(hc *container.HostConfig) { + noPersistence(hc) + hc.PortBindings = network.PortMap{p: {{HostPort: port}}} + }, + }, port+"/tcp") + code, out, err := ctr.Exec(context.Background(), []string{"redis-cli", "-p", port, "cluster", "addslotsrange", "0", "16383"}) + require.NoError(t, err) + require.Zero(t, code, "%v", out) + s := &server{ctr: ctr, addr: net.JoinHostPort("127.0.0.1", port), mode: cache.RedisCluster} + waitReady(t, s, func(c rueidis.Client) error { + info, err := c.Do(context.Background(), c.B().ClusterInfo().Build()).ToString() + if err == nil && !strings.Contains(info, "cluster_state:ok") { + err = fmt.Errorf("cluster not ready: %q", info) + } + return err + }) + return s +} + +func freePort(t *testing.T) string { + t.Helper() + var lc net.ListenConfig + for { + ln, err := lc.Listen(context.Background(), "tcp", "127.0.0.1:0") + require.NoError(t, err) + port := strconv.Itoa(ln.Addr().(*net.TCPAddr).Port) + require.NoError(t, ln.Close()) + if port != "16379" { + return port + } + } +} + +// raw opens a plain client on s, for the test to reach under the cache. +func raw(t *testing.T, s *server) rueidis.Client { + t.Helper() + c, err := rueidis.NewClient(rueidis.ClientOption{ + InitAddress: []string{s.addr}, DisableCache: true, ForceSingleClient: s.mode == cache.RedisStandalone, + }) + require.NoError(t, err) + t.Cleanup(c.Close) + return c +} + +// waitReady blocks until s answers — a listening port is not yet a server +// that takes commands — and, given ready, until ready passes too. +func waitReady(t *testing.T, s *server, ready func(rueidis.Client) error) { + t.Helper() + var last error + require.Eventually(t, func() bool { + c, err := rueidis.NewClient(rueidis.ClientOption{ + InitAddress: []string{s.addr}, DisableCache: true, ForceSingleClient: true, + }) + if last = err; err != nil { + return false + } + defer c.Close() + if last = c.Do(context.Background(), c.B().Ping().Build()).Error(); last != nil { + return false + } + if ready != nil { + last = ready(c) + } + return last == nil + }, 60*time.Second, 100*time.Millisecond, "server %s not ready: %v", s.addr, last) +} + +var prefixes atomic.Uint64 + +func uniquePrefix() string { return fmt.Sprintf("t%d", prefixes.Add(1)) } + +// open builds a RedisCache on s. Its timeout is generous: the suite runs +// in parallel under -race, and a timed-out lookup is a miss the conformance +// cases would read as a wrong answer. +func open(t *testing.T, s *server, prefix string, tune ...func(*cache.RedisConfig)) *cache.RedisCache { + t.Helper() + cfg := cache.RedisConfig{ + Addrs: []string{s.addr}, Mode: s.mode, KeyPrefix: prefix, + Timeout: 5 * time.Second, MaxValueBytes: maxValue, CompressMinBytes: cache.DefaultRedisCompressMinBytes, + } + for _, f := range tune { + f(&cfg) + } + c, err := cache.NewRedis(cfg) + require.NoError(t, err) + t.Cleanup(func() { _ = c.Close() }) + return c +} + +func TestRedis_Conformance(t *testing.T) { + t.Parallel() + servers := []struct { + name string + start func(*testing.T) *server + }{ + {"redis", startRedis}, + {"valkey", startValkey}, + {"dragonfly", startDragonfly}, + {"redis cluster", startCluster}, + } + for _, sv := range servers { + t.Run(sv.name, func(t *testing.T) { + t.Parallel() + s := sv.start(t) + cachetest.Run(t, + func(t *testing.T) cache.Cache { return open(t, s, uniquePrefix()) }, + cachetest.Options{ + // The raw-size bound: a value past it is refused before + // compression, and the suite's oversize value is zeros, + // which would compress under the stored-size one. + MaxValueBytes: maxValue * cache.DecodedFactor, + NewPair: func(t *testing.T) (cache.Cache, cache.Cache) { + p := uniquePrefix() + return open(t, s, p), open(t, s, p) + }, + Entries: valueCount(raw(t, s)), + }) + t.Run("cross-tenant invalidation spans slots", func(t *testing.T) { + t.Parallel() + testCrossTenantInvalidate(t, open(t, s, uniquePrefix())) + }) + t.Run("compressed values round-trip", func(t *testing.T) { + t.Parallel() + testCompression(t, s) + }) + t.Run("an incompressible value over the stored limit is not stored", func(t *testing.T) { + t.Parallel() + testIncompressible(t, s) + }) + }) + } +} + +// valueCount counts the values under a cache's own key prefix, scanning +// every node r knows (the test cluster's one node has no replica), and the +// empty key, where a fill under the zero snapshot's empty key would land; +// -1 is a failed read. +func valueCount(r rueidis.Client) func(cache.Cache) int { + return func(c cache.Cache) int { + match := cache.KeyPrefix(c.(*cache.RedisCache)) + ":q:*" + n, err := r.Do(context.Background(), r.B().Exists().Key("").Build()).AsInt64() + if err != nil { + return -1 + } + for _, node := range r.Nodes() { + for cursor := uint64(0); ; { + e, err := node.Do(context.Background(), node.B().Scan().Cursor(cursor).Match(match).Count(1000).Build()).AsScanEntry() + if err != nil { + return -1 + } + if n += int64(len(e.Elements)); e.Cursor == 0 { + break + } + cursor = e.Cursor + } + } + return int(n) + } +} + +// The ingest worker's shared-tables fan-out bumps one table under several +// tenants in one call: tokens in as many slots, one pipeline. +func testCrossTenantInvalidate(t *testing.T, c *cache.RedisCache) { + ctx := context.Background() + var deps [][]cache.Namespace + for i := range 20 { + id := tenantID(i) + d := []cache.Namespace{{Tenant: id, Table: "events"}} + deps = append(deps, d) + _, snap, err := c.Lookup(ctx, id, "q", d) + require.NoError(t, err) + require.NoError(t, c.Set(ctx, snap, []byte("rows"), time.Minute)) + } + var all []cache.Namespace + for _, d := range deps { + all = append(all, d...) + } + n, err := c.Invalidate(ctx, all) + require.NoError(t, err) + assert.Equal(t, uint64(len(all)), n) + for i, d := range deps { + e, _, err := c.Lookup(ctx, tenantID(i), "q", d) + require.NoError(t, err) + assert.Nil(t, e.Value, tenantID(i)) + } +} + +func tenantID(i int) tenant.ID { return tenant.ID(fmt.Sprintf("tenant-%d", i)) } + +func testCompression(t *testing.T, s *server) { + ctx := context.Background() + prefix := uniquePrefix() + c := open(t, s, prefix) + rows := bytes.Repeat([]byte(`{"user_id":"u-1","event":"click","value":42.5},`), 10_000) + require.Greater(t, len(rows), maxValue, "stored only because it compresses under the limit") + deps := []cache.Namespace{{Tenant: "acme", Table: "events"}} + _, snap, err := c.Lookup(ctx, "acme", "big", deps) + require.NoError(t, err) + require.NoError(t, c.Set(ctx, snap, rows, time.Minute)) + e, _, err := c.Lookup(ctx, "acme", "big", deps) + require.NoError(t, err) + assert.Equal(t, rows, e.Value) + + r := raw(t, s) + keys, err := r.Do(ctx, r.B().Keys().Pattern(prefix+":q:*").Build()).AsStrSlice() + require.NoError(t, err) + require.Len(t, keys, 1) + stored, err := r.Do(ctx, r.B().Strlen().Key(keys[0]).Build()).AsInt64() + require.NoError(t, err) + assert.Less(t, stored, int64(len(rows)/10)) +} + +// The stored-size limit, past the raw one: random bytes do not compress, so +// a value between the two is refused only once it is encoded. +func testIncompressible(t *testing.T, s *server) { + ctx := context.Background() + c := open(t, s, uniquePrefix()) + rows := make([]byte, maxValue+1<<10) + _, _ = rand.Read(rows) + require.Less(t, len(rows), maxValue*cache.DecodedFactor, "within the raw limit") + deps := []cache.Namespace{{Tenant: "acme", Table: "events"}} + _, snap, err := c.Lookup(ctx, "acme", "big", deps) + require.NoError(t, err) + require.NoError(t, c.Set(ctx, snap, rows, time.Minute), "declined, not failed") + e, _, err := c.Lookup(ctx, "acme", "big", deps) + require.NoError(t, err) + assert.Nil(t, e.Value) + assert.Zero(t, valueCount(raw(t, s))(c), "nothing stored") +} + +// A token that is lost — evicted, expired, flushed, a restart without +// persistence — is recreated fresh, so a value stored under its predecessor +// can only miss. A counter recreated at its initial value would serve the +// value filed at that value again: this test fails for one. +func TestRedis_LostTokensAreMisses(t *testing.T) { + t.Parallel() + ctx := context.Background() + s := startRedis(t) + r := raw(t, s) + prefix := uniquePrefix() + c := open(t, s, prefix) + deps := []cache.Namespace{{Tenant: "acme", Table: "events", Scope: "org_1"}} + fill := func(t *testing.T, sha string, deps []cache.Namespace) { + t.Helper() + _, snap, err := c.Lookup(ctx, "acme", sha, deps) + require.NoError(t, err) + require.NoError(t, c.Set(ctx, snap, []byte("rows"), time.Minute)) + e, _, err := c.Lookup(ctx, "acme", sha, deps) + require.NoError(t, err) + require.Equal(t, "rows", string(e.Value)) + } + // Twice: the first lookup recreates the lost tokens, and it is the + // second, reading them back, that a recreated counter would fool. + requireMiss := func(t *testing.T, sha string, deps []cache.Namespace) { + t.Helper() + for range 2 { + e, _, err := c.Lookup(ctx, "acme", sha, deps) + require.NoError(t, err) + require.Nil(t, e.Value) + } + } + keys := func(t *testing.T, pattern string) []string { + t.Helper() + keys, err := r.Do(ctx, r.B().Keys().Pattern(pattern).Build()).AsStrSlice() + require.NoError(t, err) + return keys + } + + for _, lost := range []string{"T", "B:events", "S:events:org_1", "*"} { + fill(t, "q", deps) + fill(t, "pipe", nil) + tokens := keys(t, prefix+":{acme}:"+lost) + require.NotEmpty(t, tokens, lost) + require.NoError(t, r.Do(ctx, r.B().Del().Key(tokens...).Build()).Error()) + require.Len(t, keys(t, prefix+":q:*"), 2, "lost %s: the values survive, only their tokens are gone", lost) + requireMiss(t, "q", deps) + if lost == "T" || lost == "*" { + requireMiss(t, "pipe", nil) + } + } + + fill(t, "q", deps) + require.NoError(t, r.Do(ctx, r.B().Flushall().Build()).Error()) + requireMiss(t, "q", deps) + fill(t, "q", deps) +} + +// A token or value key holding something else — another program under the +// prefix, a different token size mid-upgrade — is a reply, not a failure: +// it is replaced, which can only cause misses, and the breaker stays closed. +func TestRedis_ForeignTokenIsReplaced(t *testing.T) { + t.Parallel() + ctx := context.Background() + s := startRedis(t) + r := raw(t, s) + prefix := uniquePrefix() + c := open(t, s, prefix, func(c *cache.RedisConfig) { c.BreakerThreshold = 1 }) + deps := []cache.Namespace{{Tenant: "acme", Table: "events"}} + _, snap, err := c.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + require.NoError(t, c.Set(ctx, snap, []byte("rows"), time.Minute)) + valueKeys, err := r.Do(ctx, r.B().Keys().Pattern(prefix+":q:*").Build()).AsStrSlice() + require.NoError(t, err) + require.Len(t, valueKeys, 1) + + for _, plant := range []struct { + name string + cmd rueidis.Completed + }{ + {"a short string under the table token", r.B().Set().Key(prefix + ":{acme}:B:events").Value("abc").Build()}, + {"", r.B().Del().Key(prefix + ":{acme}:T").Build()}, + {"a list under the tenant token", r.B().Rpush().Key(prefix + ":{acme}:T").Element("x").Build()}, + {"", r.B().Del().Key(valueKeys[0]).Build()}, + {"a hash under the value key", r.B().Hset().Key(valueKeys[0]).FieldValue().FieldValue("f", "v").Build()}, + } { + require.NoError(t, r.Do(ctx, plant.cmd).Error()) + if plant.name == "" { // the first half of a two-step plant + continue + } + e, snap, err := c.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err, plant.name) + assert.Nil(t, e.Value, plant.name) + assert.False(t, cache.Bypassed(c), plant.name) + require.NoError(t, c.Set(ctx, snap, []byte("new rows"), time.Minute), plant.name) + e, _, err = c.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err, plant.name) + assert.Equal(t, "new rows", string(e.Value), plant.name) + } +} + +func dockerClient(t *testing.T) *testcontainers.DockerClient { + t.Helper() + d, err := testcontainers.NewDockerClientWithOpts(context.Background()) + require.NoError(t, err) + t.Cleanup(func() { _ = d.Close() }) + return d +} + +// A server that stops answering costs a lookup a bounded wait — asserted +// under ten times the op timeout, as the suite runs in parallel under -race — +// then nothing: the breaker opens and the cache is bypassed. +// Invalidations made meanwhile are kept and land once it answers again, and +// a process that boots while it is down starts bypassed and connects later. +func TestRedis_ServerStopsAnswering(t *testing.T) { + t.Parallel() + ctx := context.Background() + s := startRedis(t) + d := dockerClient(t) + prefix := uniquePrefix() + const timeout = 100 * time.Millisecond + a := open(t, s, prefix, func(c *cache.RedisConfig) { + c.Timeout, c.BreakerThreshold, c.BreakerOpenFor = timeout, 3, 300*time.Millisecond + }) + deps := []cache.Namespace{{Tenant: "acme", Table: "events"}} + _, snap, err := a.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + require.NoError(t, a.Set(ctx, snap, []byte("pre-write rows"), time.Minute)) + + _, err = d.ContainerPause(ctx, s.ctr.GetContainerID(), client.ContainerPauseOptions{}) + require.NoError(t, err) + paused := true + unpause := func() { + if paused { + paused = false + _, err := d.ContainerUnpause(ctx, s.ctr.GetContainerID(), client.ContainerUnpauseOptions{}) + require.NoError(t, err) + } + } + t.Cleanup(unpause) + + for i := range 3 { + start := time.Now() + e, snap, err := a.Lookup(ctx, "acme", "q", deps) + require.Error(t, err, "lookup %d", i) + assert.Nil(t, e.Value) + assert.Less(t, time.Since(start), 10*timeout, "a lookup returns within ten times the timeout") + require.NoError(t, a.Set(ctx, snap, []byte("rows"), time.Minute), "the failed lookup's snapshot files nothing") + } + require.True(t, cache.Bypassed(a), "three timeouts open the breaker") + start := time.Now() + e, _, err := a.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err, "bypassed is a miss, not a failure") + assert.Nil(t, e.Value) + assert.Less(t, time.Since(start), timeout/2, "bypassed costs no round trip") + + _, err = a.Invalidate(ctx, deps) + require.Error(t, err) + assert.Equal(t, 1, cache.Pending(a)) + + bootStart := time.Now() + late := open(t, s, prefix, func(c *cache.RedisConfig) { c.DialTimeout = 200 * time.Millisecond }) + assert.Less(t, time.Since(bootStart), 5*time.Second, "an unanswering server does not hold boot") + assert.True(t, cache.Bypassed(late)) + + unpause() + require.Eventually(t, func() bool { return cache.Pending(a) == 0 && !cache.Bypassed(a) }, 15*time.Second, 50*time.Millisecond, + "the deferred bump lands once the server answers") + require.Eventually(t, func() bool { return !cache.Bypassed(late) }, 15*time.Second, 50*time.Millisecond, + "the late process connects") + + b := open(t, s, prefix) + e, _, err = b.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + assert.Nil(t, e.Value, "the fill from before the deferred bump is orphaned for every process") +} + +// Close makes its last attempt at the pending bumps past the breaker: an +// open one is why they are pending, and the server may be back by now. +func TestRedis_CloseDeliversPastAnOpenBreaker(t *testing.T) { + t.Parallel() + ctx := context.Background() + s := startRedis(t) + d := dockerClient(t) + prefix := uniquePrefix() + a := open(t, s, prefix, func(c *cache.RedisConfig) { + c.Timeout, c.BreakerThreshold, c.BreakerOpenFor = 100*time.Millisecond, 1, time.Hour + }) + deps := []cache.Namespace{{Tenant: "acme", Table: "events"}} + _, snap, err := a.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + require.NoError(t, a.Set(ctx, snap, []byte("pre-write rows"), time.Minute)) + + _, err = d.ContainerPause(ctx, s.ctr.GetContainerID(), client.ContainerPauseOptions{}) + require.NoError(t, err) + _, _, err = a.Lookup(ctx, "acme", "q", deps) + require.Error(t, err) + require.True(t, cache.Bypassed(a)) + _, err = a.Invalidate(ctx, deps) + require.Error(t, err) + _, err = d.ContainerUnpause(ctx, s.ctr.GetContainerID(), client.ContainerUnpauseOptions{}) + require.NoError(t, err) + + require.True(t, cache.Bypassed(a), "the breaker stays open for its hour") + require.Equal(t, 1, cache.Pending(a)) + require.NoError(t, a.Close()) + assert.Zero(t, cache.Pending(a)) + + e, _, err := open(t, s, prefix).Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + assert.Nil(t, e.Value, "the bump Close delivered orphans the fill") +} + +// A cluster client reads the topology after the handshake; a node that +// answers the handshake and then goes quiet held that read, and so boot and +// Close, for rueidis's 10 s default. It is bounded like a dial now. Any +// server serves: what matters is that the read goes unanswered. +func TestRedis_ClusterTopologyReadIsBounded(t *testing.T) { + t.Parallel() + s := startRedis(t) + opt, err := cache.ClientOption(cache.RedisConfig{Addrs: []string{s.addr}, Mode: cache.RedisCluster, DialTimeout: 300 * time.Millisecond}) + require.NoError(t, err) + opt.DialCtxFn = func(ctx context.Context, addr string, d *net.Dialer, _ *tls.Config) (net.Conn, error) { + c, err := d.DialContext(ctx, "tcp", addr) + return unanswered{c}, err + } + start := time.Now() + c, err := rueidis.NewClient(opt) + if err == nil { + c.Close() + } + require.Error(t, err, "nothing answered the topology read") + assert.Less(t, time.Since(start), 3*time.Second) +} + +// unanswered drops CLUSTER commands unsent, as a node gone quiet would +// leave them unanswered. +type unanswered struct{ net.Conn } + +func (c unanswered) Write(b []byte) (int, error) { + if bytes.Contains(b, []byte("CLUSTER")) { + return len(b), nil + } + return c.Conn.Write(b) +} + +// proxy forwards each connection it accepts to the address target holds at +// that moment, so a switch moves new connections only, as a stable DNS name +// or a proxy does after a failover. Given a TLS config it terminates TLS. +// It holds a new connection for delay before its TLS handshake and again +// before forwarding it, as a slow dial and a slow handshake would, and +// holds everything the server sends for replyDelay, as a slow server would. +type proxy struct { + addr string // to dial + serverTLS *tls.Config + target atomic.Pointer[string] + delay atomic.Int64 // a time.Duration + replyDelay atomic.Int64 // a time.Duration + + mu sync.Mutex + conns []net.Conn +} + +func newProxy(t *testing.T, target string, serverTLS *tls.Config) *proxy { + t.Helper() + var lc net.ListenConfig + ln, err := lc.Listen(context.Background(), "tcp", "127.0.0.1:0") + require.NoError(t, err) + t.Cleanup(func() { _ = ln.Close() }) + p := &proxy{addr: ln.Addr().String(), serverTLS: serverTLS} + p.target.Store(&target) + go func() { + for { + c, err := ln.Accept() + if err != nil { + return + } + p.mu.Lock() + p.conns = append(p.conns, c) + p.mu.Unlock() + go p.serve(c) + } + }() + return p +} + +func (p *proxy) serve(c net.Conn) { + defer func() { _ = c.Close() }() + if p.serverTLS != nil { + time.Sleep(time.Duration(p.delay.Load())) + tc := tls.Server(c, p.serverTLS) + if tc.HandshakeContext(context.Background()) != nil { + return + } + c = tc + } + time.Sleep(time.Duration(p.delay.Load())) + var d net.Dialer + u, err := d.DialContext(context.Background(), "tcp", *p.target.Load()) + if err != nil { + return + } + defer func() { _ = u.Close() }() + go func() { _, _ = io.Copy(u, c); _ = u.Close() }() + buf := make([]byte, 32<<10) + for { + n, err := u.Read(buf) + if n > 0 { + time.Sleep(time.Duration(p.replyDelay.Load())) + if _, err := c.Write(buf[:n]); err != nil { + return + } + } + if err != nil { + return + } + } +} + +// selfSigned returns a server TLS config holding a fresh certificate for +// 127.0.0.1, and a client config that trusts it. +func selfSigned(t *testing.T) (srv, cli *tls.Config) { + t.Helper() + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + require.NoError(t, err) + tmpl := &x509.Certificate{ + SerialNumber: big.NewInt(1), + NotBefore: time.Now().Add(-time.Minute), + NotAfter: time.Now().Add(time.Hour), + IPAddresses: []net.IP{net.IPv4(127, 0, 0, 1)}, + ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, + } + der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key) + require.NoError(t, err) + cert, err := x509.ParseCertificate(der) + require.NoError(t, err) + roots := x509.NewCertPool() + roots.AddCert(cert) + return &tls.Config{Certificates: []tls.Certificate{{Certificate: [][]byte{der}, PrivateKey: key}}, MinVersion: tls.VersionTLS12}, + &tls.Config{RootCAs: roots, MinVersion: tls.VersionTLS12} +} + +// drop closes every connection p carries, so the client must reconnect. +func (p *proxy) drop() { + p.mu.Lock() + defer p.mu.Unlock() + for _, c := range p.conns { + _ = c.Close() + } + p.conns = nil +} + +// A failover behind a stable address: the connections the process holds +// stay on the demoted node, which answers but refuses writes, while new ones +// reach the node promoted in its place. The refusals bypass the cache, and +// replacing connections carries it to the new primary, where the owed bump +// lands before anything it would orphan is served. +func TestRedis_FailoverBehindAStableAddress(t *testing.T) { + t.Parallel() + ctx := context.Background() + start := func() *server { // no delay before a full sync + return startStandalone(t, redisImage, "redis-server", "--save", "", "--appendonly", "no", "--repl-diskless-sync-delay", "0") + } + primary, replica := start(), start() + rPrimary, rReplica := raw(t, primary), raw(t, replica) + ip := func(s *server) string { + t.Helper() + ip, err := s.ctr.ContainerIP(ctx) + require.NoError(t, err) + return ip + } + command(t, rReplica, "REPLICAOF", ip(primary), "6379") + require.Eventually(t, func() bool { + info, err := rReplica.Do(ctx, rReplica.B().Info().Section("replication").Build()).ToString() + return err == nil && strings.Contains(info, "master_link_status:up") + }, 30*time.Second, 50*time.Millisecond, "the replica syncs") + + fwd := newProxy(t, primary.addr, nil) + stable := &server{addr: fwd.addr, mode: cache.RedisStandalone} + prefix := uniquePrefix() + a := open(t, stable, prefix, func(c *cache.RedisConfig) { + c.BreakerThreshold, c.BreakerOpenFor = 1000, 200*time.Millisecond + cache.SetConnLifetime(c, time.Second) + cache.SetOnePipe(c) // every operation uses the connection dialed before the failover + }) + deps := []cache.Namespace{{Tenant: "acme", Table: "events"}} + _, snap, err := a.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + require.NoError(t, a.Set(ctx, snap, []byte("pre-write rows"), time.Minute)) + acked, err := rPrimary.Do(ctx, rPrimary.B().Wait().Numreplicas(1).Timeout(5000).Build()).AsInt64() + require.NoError(t, err) + require.Equal(t, int64(1), acked, "the replica holds the fill") + + command(t, rReplica, "REPLICAOF", "NO", "ONE") + command(t, rPrimary, "REPLICAOF", ip(replica), "6379") + _, err = a.Invalidate(ctx, deps) + require.ErrorContains(t, err, "READONLY") + require.True(t, cache.Bypassed(a)) + fwd.target.Store(&replica.addr) + + deadline := time.Now().Add(15 * time.Second) + for cache.Pending(a) > 0 { + require.True(t, time.Now().Before(deadline), "the owed bump reaches the new primary") + e, _, err := a.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + require.NotEqual(t, "pre-write rows", string(e.Value), "served before the owed bump landed") + time.Sleep(10 * time.Millisecond) + } + e, _, err := open(t, stable, prefix).Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + assert.Nil(t, e.Value, "the bump landed where every process now reads") +} + +// command runs cmd on s through c, for the test to reconfigure the server. +func command(t *testing.T, c rueidis.Client, cmd ...string) { + t.Helper() + require.NoError(t, c.Do(context.Background(), c.B().Arbitrary(cmd[0]).Args(cmd[1:]...).Build()).Error(), "%q", cmd) +} + +// A bump this process owes holds the lookups it would orphan — a bypass +// that files nothing — until it lands, with the breaker closed and every +// other lookup served. The ACL lets the process read the tokens but not +// replace them, so the bump stays owed until the test grants the write. +func TestRedis_OwedBumpHoldsItsLookups(t *testing.T) { + t.Parallel() + ctx := context.Background() + s := startRedis(t) + r := raw(t, s) + prefix := uniquePrefix() + command(t, r, "ACL", "SETUSER", "limited", "on", ">pw", "+@all", "%R~*", "%W~"+prefix+":q:*") + + events := []cache.Namespace{{Tenant: "acme", Table: "events"}} + orders := []cache.Namespace{{Tenant: "acme", Table: "orders"}} + seed := open(t, s, prefix) + for _, deps := range [][]cache.Namespace{events, orders, nil} { + _, snap, err := seed.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + require.NoError(t, seed.Set(ctx, snap, []byte("pre-write rows"), time.Minute)) + } + a := open(t, s, prefix, func(c *cache.RedisConfig) { c.Username, c.Password, c.BreakerThreshold = "limited", "pw", 1000 }) + lookup := func(deps []cache.Namespace) (string, cache.Snapshot) { + t.Helper() + e, snap, err := a.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + return string(e.Value), snap + } + got, _ := lookup(events) + require.Equal(t, "pre-write rows", got) + + _, err := a.Invalidate(ctx, events) + require.ErrorContains(t, err, "NOPERM") + require.Equal(t, 1, cache.Pending(a)) + require.False(t, cache.Bypassed(a), "a refused key is not a refusing server") + + got, snap := lookup(events) + assert.Empty(t, got, "the owed bump would orphan it") + assert.True(t, cache.ZeroSnapshot(snap), "and a fill under the token it replaces would be orphaned too") + for _, deps := range [][]cache.Namespace{orders, nil} { + got, _ = lookup(deps) + assert.Equal(t, "pre-write rows", got, "%v: lookups the bump does not orphan are served", deps) + } + + command(t, r, "ACL", "SETUSER", "limited", "~*") + deadline := time.Now().Add(10 * time.Second) + for cache.Pending(a) > 0 { + require.True(t, time.Now().Before(deadline), "the bump lands once the server takes it") + got, _ = lookup(events) + require.NotEqual(t, "pre-write rows", got, "served before the owed bump landed") + time.Sleep(time.Millisecond) + } + got, _ = lookup(events) + assert.Empty(t, got, "the landed bump orphaned the pre-write rows") +} + +// A server that answers but refuses writes — a primary demoted to a replica +// (READONLY), memory full under noeviction (OOM) — takes no bump, so its +// first refusal opens the breaker whatever the threshold, and the bump stays +// owed. The probe writes, so it keeps the breaker open until the server +// takes writes again; then the owed bump lands before anything it would +// orphan is served, and a process that only invalidates, whose drain loop +// is all that probes for it, recovers at the probe's cadence too. +func TestRedis_RefusedWrites(t *testing.T) { + t.Parallel() + for _, tt := range []struct { + name string + refuse, restore [][]string + reply string + }{ + { + "demoted to a replica", + [][]string{{"REPLICAOF", "127.0.0.1", "1"}}, // nothing listens: it stays a replica, serving reads + [][]string{{"REPLICAOF", "NO", "ONE"}}, + "READONLY", + }, + { + "full under noeviction", + [][]string{{"CONFIG", "SET", "maxmemory-policy", "noeviction"}, {"CONFIG", "SET", "maxmemory", "1"}}, + [][]string{{"CONFIG", "SET", "maxmemory", "0"}}, + "OOM", + }, + } { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + ctx := context.Background() + s := startRedis(t) + r := raw(t, s) + prefix := uniquePrefix() + const openFor = 200 * time.Millisecond + tune := func(c *cache.RedisConfig) { c.BreakerThreshold, c.BreakerOpenFor = 1000, openFor } + a := open(t, s, prefix, tune) + ingest := open(t, s, prefix, tune) + reader := open(t, s, prefix, tune) + events := []cache.Namespace{{Tenant: "acme", Table: "events"}} + orders := []cache.Namespace{{Tenant: "acme", Table: "orders"}} + for _, deps := range [][]cache.Namespace{events, orders} { + _, snap, err := a.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + require.NoError(t, a.Set(ctx, snap, []byte("pre-write rows"), time.Minute)) + } + lookup := func(c *cache.RedisCache, deps []cache.Namespace) string { + t.Helper() + e, _, err := c.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + return string(e.Value) + } + + for _, cmd := range tt.refuse { + command(t, r, cmd...) + } + _, err := a.Invalidate(ctx, events) + require.ErrorContains(t, err, tt.reply) + assert.Equal(t, 1, cache.Pending(a)) + assert.True(t, cache.Bypassed(a), "the first refusal opens the breaker") + wide := slices.Clone(orders) // past one batch: the rest go unsent after the refusal, and drain in two + for i := range 1500 { + wide = append(wide, cache.Namespace{Tenant: "acme", Table: fmt.Sprintf("t%d", i)}) + } + _, err = ingest.Invalidate(ctx, wide) + require.ErrorContains(t, err, tt.reply) + assert.True(t, cache.Bypassed(ingest)) + assert.Equal(t, len(wide), cache.Pending(ingest)) + _, snap, err := reader.Lookup(ctx, "acme", "another query", orders) + require.NoError(t, err) + require.ErrorContains(t, reader.Set(ctx, snap, []byte("rows"), time.Minute), tt.reply) + assert.True(t, cache.Bypassed(reader), "a refused fill opens the breaker too") + + // Long enough for many probes to be refused, and for a drain + // backing off unchecked to be at its 10 s cap. + time.Sleep(7 * time.Second) + assert.True(t, cache.Bypassed(reader), "owing nothing, it is held open by probes that write and are refused") + assert.True(t, cache.Bypassed(a)) + assert.Empty(t, lookup(a, events)) + assert.Equal(t, 1, cache.Pending(a)) + assert.Equal(t, len(wide), cache.Pending(ingest)) + + for _, cmd := range tt.restore { + command(t, r, cmd...) + } + restored := time.Now() + for cache.Pending(a) > 0 { + require.Less(t, time.Since(restored), 10*time.Second, "the bump lands once the server takes writes") + require.NotEqual(t, "pre-write rows", lookup(a, events), "served before the owed bump landed") + time.Sleep(time.Millisecond) + } + require.Eventually(t, func() bool { return cache.Pending(ingest) == 0 }, 10*time.Second, 5*time.Millisecond) + assert.Less(t, time.Since(restored), openFor+3*time.Second, "an ingest-only process probes when due, not at the drain's backoff") + require.Eventually(t, func() bool { return !cache.Bypassed(reader) }, 10*time.Second, 5*time.Millisecond) + + fresh := open(t, s, prefix) + for _, deps := range [][]cache.Namespace{events, orders} { + assert.Empty(t, lookup(fresh, deps), "%v: the landed bumps orphaned the pre-write rows", deps) + } + }) + } +} + +// Restoring a snapshot — RDB, AOF or a backup — is a rollback, not a lost +// token: the old tokens come back with the values filed under them, so what +// was invalidated since is served again, as after a failover to a replica +// that missed the bumps. The documented exception to "lost tokens miss". +func TestRedis_RestoredSnapshotIsARollback(t *testing.T) { + t.Parallel() + ctx := context.Background() + s := startStandalone(t, redisImage, "redis-server", "--save", "", "--appendonly", "no", "--enable-debug-command", "yes") + r := raw(t, s) + prefix := uniquePrefix() + c := open(t, s, prefix) + deps := []cache.Namespace{{Tenant: "acme", Table: "events"}} + _, snap, err := c.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + require.NoError(t, c.Set(ctx, snap, []byte("pre-write rows"), time.Minute)) + command(t, r, "SAVE") + + _, err = c.Invalidate(ctx, deps) + require.NoError(t, err) + e, _, err := c.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + require.Nil(t, e.Value) + + command(t, r, "DEBUG", "RELOAD", "NOSAVE") + e, _, err = open(t, s, prefix).Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + assert.Equal(t, "pre-write rows", string(e.Value), "the restore brought back the pre-write token") +} + +// A reconnect slower than the op timeout (a dial and TLS handshake across +// zones) fails the operations waiting on it, since rueidis dials under their +// context; they open the breaker, and the probe, whose budget covers a +// reconnect, completes it and closes the breaker. rueidis bounds the dial, +// TLS included, by the dial timeout and then the handshake by it again, so +// here each takes most of it: together they outlast one dial timeout plus +// the op timeout. Only the connection the probe lands on reconnects: +// rueidis spreads commands over several, and the rest still reconnect under +// the op timeout. +func TestRedis_ProbeFitsASlowReconnect(t *testing.T) { + t.Parallel() + ctx := context.Background() + s := startRedis(t) + srvTLS, cliTLS := selfSigned(t) + p := newProxy(t, s.addr, srvTLS) + const timeout, dialTimeout = 100 * time.Millisecond, time.Second + a := open(t, &server{addr: p.addr, mode: cache.RedisStandalone}, uniquePrefix(), func(c *cache.RedisConfig) { + c.Timeout, c.DialTimeout, c.TLS = timeout, dialTimeout, cliTLS + c.BreakerThreshold, c.BreakerOpenFor = 1, 200*time.Millisecond + }) + deps := []cache.Namespace{{Tenant: "acme", Table: "events"}} + _, _, err := a.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + + p.delay.Store(int64(dialTimeout * 7 / 10)) + p.drop() + _, _, err = a.Lookup(ctx, "acme", "q", deps) + require.Error(t, err, "the reconnect does not fit in the lookup's timeout") + require.True(t, cache.Bypassed(a)) + require.Eventually(t, func() bool { return !cache.Bypassed(a) }, 20*time.Second, 10*time.Millisecond, + "the probe reconnects within twice the dial timeout") +} + +// A server that answers, but slower than the op timeout, fails every +// operation, so the probe's reconnect allowance must not close the breaker +// on it: a probe write slower than the op timeout is repeated under it, and +// only a repeat that lands closes the breaker. +func TestRedis_ProbeKeepsASlowServerBypassed(t *testing.T) { + t.Parallel() + ctx := context.Background() + s := startRedis(t) + p := newProxy(t, s.addr, nil) + const timeout = 100 * time.Millisecond + a := open(t, &server{addr: p.addr, mode: cache.RedisStandalone}, uniquePrefix(), func(c *cache.RedisConfig) { + c.Timeout, c.DialTimeout = timeout, time.Second + c.BreakerThreshold, c.BreakerOpenFor = 1, 200*time.Millisecond + }) + deps := []cache.Namespace{{Tenant: "acme", Table: "events"}} + _, _, err := a.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + + p.replyDelay.Store(int64(3 * timeout)) + _, _, err = a.Lookup(ctx, "acme", "q", deps) + require.Error(t, err, "a reply slower than the timeout fails the lookup") + require.True(t, cache.Bypassed(a)) + // Several probes, each answered within the reconnect allowance. + for end := time.Now().Add(4 * time.Second); time.Now().Before(end); time.Sleep(5 * time.Millisecond) { + require.True(t, cache.Bypassed(a), "a probe answered slower than the op timeout closed the breaker") + } + p.replyDelay.Store(0) + require.Eventually(t, func() bool { return !cache.Bypassed(a) }, 10*time.Second, 10*time.Millisecond, + "a probe answered in time closes it") +} + +// A password rotated under a running process refuses its next connection's +// handshake, and so every operation: one such reply opens the breaker, +// whatever the threshold, and the probe closes it once the credentials work. +func TestRedis_RejectedCredentialsOpenTheBreaker(t *testing.T) { + t.Parallel() + ctx := context.Background() + s := startRedis(t) + r := raw(t, s) + command(t, r, "ACL", "SETUSER", "rotating", "on", ">old", "+@all", "~*") + a := open(t, s, uniquePrefix(), func(c *cache.RedisConfig) { + c.Username, c.Password = "rotating", "old" + c.BreakerThreshold, c.BreakerOpenFor = 1000, 200*time.Millisecond + }) + deps := []cache.Namespace{{Tenant: "acme", Table: "events"}} + _, _, err := a.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + + command(t, r, "ACL", "SETUSER", "rotating", "resetpass", ">new") + command(t, r, "CLIENT", "KILL", "USER", "rotating") // as the connection lifetime would + require.Eventually(t, func() bool { + _, _, err = a.Lookup(ctx, "acme", "q", deps) + return err != nil && strings.Contains(err.Error(), "WRONGPASS") + }, 5*time.Second, 10*time.Millisecond, "the reconnect is refused") + assert.True(t, cache.Bypassed(a), "one refused handshake opens the breaker") + time.Sleep(time.Second) // several refused probes + assert.True(t, cache.Bypassed(a)) + + command(t, r, "ACL", "SETUSER", "rotating", ">old") + require.Eventually(t, func() bool { return !cache.Bypassed(a) }, 10*time.Second, 10*time.Millisecond, + "the probe closes it once the credentials work") +} diff --git a/internal/cache/redis_test.go b/internal/cache/redis_test.go new file mode 100644 index 000000000..5f9d5cec4 --- /dev/null +++ b/internal/cache/redis_test.go @@ -0,0 +1,354 @@ +package cache + +import ( + "context" + "errors" + "fmt" + "log/slog" + "net" + "strings" + "testing" + "time" + + "github.com/redis/rueidis" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/attribute" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" + + "github.com/Wave-RF/WaveHouse/internal/tenant" + "github.com/Wave-RF/WaveHouse/internal/testutil/logtest" +) + +func TestRedisConfig_Validation(t *testing.T) { + t.Parallel() + ok := RedisConfig{Addrs: []string{"redis:6379"}} + tests := []struct { + name string + mutate func(c *RedisConfig) + wantErr string + }{ + {"no address", func(c *RedisConfig) { c.Addrs = nil }, "at least one address"}, + {"address without a port", func(c *RedisConfig) { c.Addrs = []string{"redis"} }, `address "redis"`}, + {"unknown mode", func(c *RedisConfig) { c.Mode = "ring" }, `mode "ring"`}, + {"cluster with a db", func(c *RedisConfig) { c.Mode, c.DB = RedisCluster, 1 }, "only database 0"}, + {"sentinel without a master", func(c *RedisConfig) { c.Mode = RedisSentinel }, "master set name"}, + {"negative db", func(c *RedisConfig) { c.DB = -1 }, "negative"}, + {"hash tag in the prefix", func(c *RedisConfig) { c.KeyPrefix = "{wh}" }, "brace"}, + {"negative timeout", func(c *RedisConfig) { c.Timeout = -time.Second }, "timeout is negative"}, + {"negative size", func(c *RedisConfig) { c.MaxValueBytes = -1 }, "max value bytes is negative"}, + {"version ttl under EX's resolution", func(c *RedisConfig) { c.VersionTTL = 1500 * time.Millisecond }, "under 2s"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + c := ok + tt.mutate(&c) + _, err := NewRedis(c) + require.ErrorContains(t, err, tt.wantErr) + }) + } +} + +func TestRedisConfig_Defaults(t *testing.T) { + t.Parallel() + c, err := RedisConfig{Addrs: []string{"redis:6379"}}.withDefaults() + require.NoError(t, err) + assert.Equal(t, RedisStandalone, c.Mode) + assert.Equal(t, DefaultRedisKeyPrefix, c.KeyPrefix) + assert.Equal(t, DefaultRedisTimeout, c.Timeout) + assert.Equal(t, DefaultRedisMaxValueBytes, c.MaxValueBytes) + assert.Equal(t, DefaultRedisVersionTTL, c.VersionTTL) + assert.Zero(t, c.CompressMinBytes, "0 means never compress, not the default") + assert.True(t, c.clientOption().ForceSingleClient) + assert.Equal(t, DefaultRedisDialTimeout, c.clientOption().ConnWriteTimeout) + assert.Equal(t, defaultConnLifetime, c.clientOption().ConnLifetime) + slow := c + slow.Timeout = 5 * time.Second + assert.Equal(t, slow.Timeout, slow.clientOption().ConnWriteTimeout, "never under the op timeout") + + c.Mode, c.SentinelMaster = RedisSentinel, "mymaster" + assert.Equal(t, "mymaster", c.clientOption().Sentinel.MasterSet) + c.Mode = RedisCluster + assert.False(t, c.clientOption().ForceSingleClient) +} + +// closedAddr is an address nothing listens on: dials are refused at once. +func closedAddr(t *testing.T) string { + t.Helper() + var lc net.ListenConfig + ln, err := lc.Listen(context.Background(), "tcp", "127.0.0.1:0") + require.NoError(t, err) + addr := ln.Addr().String() + require.NoError(t, ln.Close()) + return addr +} + +// An unreachable server does not fail construction: the cache is bypassed +// — a miss that files nothing, a no-op fill, a deferred invalidation — and +// keeps dialing. +func TestRedis_UnreachableIsBypassed(t *testing.T) { + t.Parallel() + ctx := context.Background() + r, err := NewRedis(RedisConfig{Addrs: []string{closedAddr(t)}, DialTimeout: 100 * time.Millisecond, PendingMax: 2}) + require.NoError(t, err) + t.Cleanup(func() { _ = r.Close() }) + assert.True(t, r.bypassed()) + + deps := []Namespace{{Tenant: "acme", Table: "events"}} + e, snap, err := r.Lookup(ctx, "acme", "q", deps) + require.NoError(t, err) + assert.Nil(t, e.Value) + assert.Empty(t, snap.key, "nothing to file under") + + _, _, err = r.Lookup(ctx, "acme", "q", []Namespace{{Tenant: "globex", Table: "events"}}) + require.ErrorIs(t, err, ErrForeignDependency) + + require.NoError(t, r.Set(ctx, Snapshot{key: "k"}, []byte("rows"), time.Minute)) + + n, err := r.Invalidate(ctx, []Namespace{{Tenant: "acme", Table: "events", Scope: "org_1"}}) + require.ErrorIs(t, err, errBypassed) + assert.Equal(t, uint64(1), n) + assert.Equal(t, 2, r.pending.len()) + require.ErrorIs(t, r.InvalidateTenant(ctx, "globex"), errBypassed) + owed := r.pending.snapshot() + assert.Len(t, owed, 2, "past PendingMax: one bump per tenant") + assert.Contains(t, owed, "wh:{acme}:T") + assert.Contains(t, owed, "wh:{globex}:T") + + require.NoError(t, r.Close()) + require.NoError(t, r.Close(), "idempotent") +} + +// Each size limit is counted: the raw one before compression, the stored one +// after it, which a server-less Set cannot otherwise tell from a bypass. +func TestRedis_SetDeclinesWithoutTouchingTheServer(t *testing.T) { + // No t.Parallel(): swaps the global meter provider. + reader := meterReader(t) + r, err := NewRedis(RedisConfig{Addrs: []string{closedAddr(t)}, DialTimeout: 100 * time.Millisecond, MaxValueBytes: 64}) + require.NoError(t, err) + t.Cleanup(func() { _ = r.Close() }) + ctx := context.Background() + snap := Snapshot{key: "k", tokens: newToken()} + for _, tt := range []struct { + name string + snap Snapshot + value []byte + ttl time.Duration + }{ + {"zero snapshot", Snapshot{}, []byte("rows"), time.Minute}, + {"zero ttl", snap, []byte("rows"), 0}, + {"over the stored limit", snap, randomBytes(100), time.Minute}, + {"over the decoded limit", snap, make([]byte, 64*decodedFactor+1), time.Minute}, + } { + require.NoError(t, r.Set(ctx, tt.snap, tt.value, tt.ttl), tt.name) + } + assert.Equal(t, int64(2), sumOf(t, collect(t, reader), "wavehouse_cache_oversize_total", "", "")) +} + +func TestRedis_Record(t *testing.T) { + t.Parallel() + r := &RedisCache{breaker: newBreaker(1, time.Hour, time.Now)} + live, cancelled := context.Background(), cancelledCtx() + + r.record(live, rueidis.Nil) + r.record(live, &rueidis.RedisError{}) + r.record(cancelled, context.Canceled) + r.record(live, fmt.Errorf("%w: token is 3 bytes", errMalformedReply)) + assert.False(t, r.breaker.isOpen(), "a reply, even an error reply, or the caller giving up says nothing against the server") + + r.record(live, context.DeadlineExceeded) + assert.True(t, r.breaker.isOpen()) +} + +// Each opening of the breaker logs one WARN naming why; the operations that +// fail while it is open log nothing. Not parallel: it captures the default +// logger. +func TestRedis_BreakerOpeningLogsOnce(t *testing.T) { + buf := logtest.Capture(t, slog.LevelWarn) + clock := &fakeClock{t: time.Unix(0, 0)} + r := &RedisCache{breaker: newBreaker(2, time.Second, clock.now)} + live := context.Background() + warns := func() int { return strings.Count(buf.String(), `"level":"WARN"`) } + + r.recordReply(live, "READONLY You can't write against a read only replica.") + r.recordReply(live, "READONLY You can't write against a read only replica.") + r.record(live, context.DeadlineExceeded) + r.record(live, context.DeadlineExceeded) + assert.Equal(t, 1, warns(), buf.String()) + assert.Contains(t, buf.String(), `"reply":"READONLY You can't write against a read only replica."`) + + clock.t = clock.t.Add(time.Second) + _, probe := r.breaker.allow() + require.True(t, probe) + r.record(live, nil) + require.False(t, r.breaker.isOpen()) + + r.record(live, context.DeadlineExceeded) + assert.Equal(t, 1, warns(), "under the threshold") + r.record(live, context.DeadlineExceeded) + r.record(live, context.DeadlineExceeded) + assert.Equal(t, 2, warns(), buf.String()) + assert.Contains(t, buf.String(), "not answering") + + clock.t = clock.t.Add(time.Second) + _, probe = r.breaker.allow() + require.True(t, probe) + r.record(live, context.DeadlineExceeded) + assert.Equal(t, 3, warns(), "a failed probe opens it again") +} + +func TestRejectsCredentials(t *testing.T) { + t.Parallel() + assert.True(t, rejectsCredentials("WRONGPASS invalid username-password pair or user is disabled.")) + assert.True(t, rejectsCredentials("NOAUTH Authentication required.")) + assert.False(t, rejectsCredentials("NOPERM No permissions to access a key"), "about a key, not the credentials") + assert.False(t, rejectsCredentials("READONLY You can't write against a read only replica.")) + assert.False(t, rejectsCredentials("ERR WRONGPASS")) +} + +func TestRefusesWork(t *testing.T) { + t.Parallel() + for _, msg := range []string{ + "READONLY You can't write against a read only replica.", + "OOM command not allowed when used memory > 'maxmemory'.", + "MASTERDOWN Link with MASTER is down and replica-serve-stale-data is set to 'no'.", + "NOREPLICAS Not enough good replicas to write.", + "MISCONF Errors writing to the AOF file: No space left on device", + "LOADING Redis is loading the dataset in memory", + "BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE.", + "CLUSTERDOWN The cluster is down", + } { + assert.True(t, refusesWork(msg), msg) + } + for _, msg := range []string{ + "WRONGTYPE Operation against a key holding the wrong kind of value", + "NOPERM this user has no permissions to access one of the keys used as arguments", + "TRYAGAIN Multiple keys request during rehashing of slot", + "BUSYKEY Target key name already exists.", + "ERR unknown command", + "", + } { + assert.False(t, refusesWork(msg), msg) + } +} + +// The first bump owed wakes the drain at once; later ones ride the retry +// already under way rather than resetting its backoff. +func TestRedis_FirstDeferralWakesTheDrain(t *testing.T) { + t.Parallel() + r := &RedisCache{breaker: newBreaker(1, time.Hour, time.Now), pending: newPendingBumps("wh", 10), wake: make(chan struct{}, 1)} + var err error + r.metrics, err = newMetrics("redis", r.bypassed, r.pending.len) + require.NoError(t, err) + t.Cleanup(r.metrics.close) + + r.deferBumps(map[string]tenant.ID{"wh:{acme}:B:events": "acme"}) + assert.Len(t, r.wake, 1) + <-r.wake + r.deferBumps(map[string]tenant.ID{"wh:{acme}:B:orders": "acme"}) + assert.Empty(t, r.wake) +} + +func cancelledCtx() context.Context { + ctx, cancel := context.WithCancel(context.Background()) + cancel() + return ctx +} + +func TestSetFailureReason(t *testing.T) { + t.Parallel() + assert.Equal(t, "timeout", setFailureReason(context.DeadlineExceeded)) + assert.Equal(t, "other", setFailureReason(errors.New("broken pipe"))) +} + +func TestIsAuthError(t *testing.T) { + t.Parallel() + assert.True(t, isAuthError(errors.New("WRONGPASS invalid username-password pair"))) + assert.True(t, isAuthError(errors.New("NOAUTH authentication required"))) + assert.False(t, isAuthError(errors.New("dial tcp: connection refused"))) +} + +func TestReadTokens_NotAnArrayIsMalformed(t *testing.T) { + t.Parallel() + _, _, _, err := readTokens(rueidis.RedisResult{}) + require.ErrorIs(t, err, errMalformedReply) +} + +// meterReader makes a manual reader the global meter provider's for the +// test's duration, which must then not run in parallel. +func meterReader(t *testing.T) *sdkmetric.ManualReader { + t.Helper() + saved := otel.GetMeterProvider() + reader := sdkmetric.NewManualReader() + mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader)) + otel.SetMeterProvider(mp) + t.Cleanup(func() { + _ = mp.Shutdown(context.Background()) + otel.SetMeterProvider(saved) + }) + return reader +} + +// collect reads every instrument's data from reader, by name. +func collect(t *testing.T, reader *sdkmetric.ManualReader) map[string]metricdata.Aggregation { + t.Helper() + var rm metricdata.ResourceMetrics + require.NoError(t, reader.Collect(context.Background(), &rm)) + got := map[string]metricdata.Aggregation{} + for _, sm := range rm.ScopeMetrics { + for _, m := range sm.Metrics { + got[m.Name] = m.Data + } + } + return got +} + +// sumOf adds up an int64 counter's or gauge's points whose attribute key +// is value; key "" takes every point. +func sumOf(t *testing.T, got map[string]metricdata.Aggregation, name, key, value string) int64 { + t.Helper() + var n int64 + switch d := got[name].(type) { + case metricdata.Sum[int64]: + for _, dp := range d.DataPoints { + if v, ok := dp.Attributes.Value(attribute.Key(key)); key == "" || ok && v.AsString() == value { + n += dp.Value + } + } + case metricdata.Gauge[int64]: + for _, dp := range d.DataPoints { + n += dp.Value + } + default: + t.Fatalf("%s: %T", name, got[name]) + } + return n +} + +func TestRedisMetrics(t *testing.T) { + // No t.Parallel(): swaps the global meter provider. + reader := meterReader(t) + r, err := NewRedis(RedisConfig{Addrs: []string{closedAddr(t)}, DialTimeout: 100 * time.Millisecond}) + require.NoError(t, err) + t.Cleanup(func() { _ = r.Close() }) + ctx := context.Background() + _, _, _ = r.Lookup(ctx, "acme", "q", nil) + _, _ = r.Invalidate(ctx, []Namespace{{Tenant: "acme", Table: "events"}}) + r.metrics.op("set", time.Now()) + r.metrics.stored(10) + r.metrics.tooLarge() + r.metrics.setFailed("oom") + + got := collect(t, reader) + assert.Equal(t, int64(1), sumOf(t, got, "wavehouse_cache_lookups_total", "result", resultBypass)) + assert.Equal(t, int64(1), sumOf(t, got, "wavehouse_cache_invalidations_total", "result", "deferred")) + assert.Equal(t, int64(1), sumOf(t, got, "wavehouse_cache_invalidations_pending", "", "")) + assert.Equal(t, int64(1), sumOf(t, got, "wavehouse_cache_breaker_open", "", "")) + assert.Equal(t, int64(1), sumOf(t, got, "wavehouse_cache_oversize_total", "", "")) + assert.Equal(t, int64(1), sumOf(t, got, "wavehouse_cache_set_failures_total", "reason", "oom")) + assert.Contains(t, got, "wavehouse_cache_op_duration_seconds") + assert.Contains(t, got, "wavehouse_cache_value_bytes") +} diff --git a/internal/cache/version_manager.go b/internal/cache/version_manager.go index f6a763315..6b1de3b5f 100644 --- a/internal/cache/version_manager.go +++ b/internal/cache/version_manager.go @@ -1,7 +1,6 @@ package cache import ( - "fmt" "sort" "strconv" "strings" @@ -11,27 +10,45 @@ import ( "github.com/Wave-RF/WaveHouse/internal/tenant" ) -// VersionManager handles the safe tracking of table + scope versioning. -// It uses a standard map because versions must NEVER be evicted under memory pressure. -// TODO: this potentially could be bad/dangerous with a low amount of RAM available/high memory pressure AND a TON of tables/scopes per table... will need to work out eventually +// VersionManager is the invalidation index: one version per tenant, per +// (tenant, table) and per (tenant, table, scope), in maps keyed by name +// alone, never by another version (#262). A bump overwrites a version in +// place, so the index holds one entry per live tenant, table and scope +// however often each is bumped, and forgetting a tenant releases all of it. +// +// An entry's key (`QueryKey`) folds all three versions of each dependency, +// which gives the lattice: a table bump orphans every scope, a scope bump +// that scope and the whole-table view, and a tenant bump everything of the +// tenant's. type VersionManager struct { mu sync.RWMutex - // tenantVersions leads every key of a tenant, so BumpTenant orphans the - // tenant's every namespace and query in one step — the ones no bump ever - // keyed included, which is what an enumeration of the maps would miss. - tenantVersions map[tenant.ID]uint64 // -> tenant_version - tableVersions map[string]uint64 // ..
-> table_version - namespaceVersions map[string]uint64 // ..
.. -> namespace_version + // tenants holds each tenant's index from the first entry key (`QueryKey`) + // built for it until the tenant is bumped or pruned. + tenants map[tenant.ID]*tenantVersions + + // lastGen is the last generation handed to a tenant; see tenantVersions.gen. + lastGen uint64 } -// NewVersionManager initializes the thread-safe version store. -func NewVersionManager() *VersionManager { - return &VersionManager{ - tenantVersions: make(map[tenant.ID]uint64), - tableVersions: make(map[string]uint64), - namespaceVersions: make(map[string]uint64), - } +// tenantVersions is one tenant's slice of the index. +type tenantVersions struct { + // gen is the tenant's version: unique within the process, so a tenant + // forgotten and recreated can never fold a generation an entry was + // cached under. That is what makes dropping the tenant's whole index a + // safe bump. + gen uint64 + tables map[string]*tableVersions +} + +// tableVersions is one table's version and its scopes'. A missing table or +// scope reads as 0: an entry is only ever removed together with a bump of +// the version above it (a table bump clears the scopes, a tenant bump +// drops the tables), so a 0 read after a removal never matches an entry +// cached before it. +type tableVersions struct { + version uint64 + scopes map[string]uint64 } // Namespace is one (tenant, table, scope) a cached result depends on. The @@ -46,79 +63,103 @@ type Namespace struct { Scope string } -// tableKeyLocked renders the table-versions key, -// "..
", its fields joined by keyenc so no -// dot in a table name can run into the next field; caller must hold vm.mu. -func (vm *VersionManager) tableKeyLocked(id tenant.ID, table string) string { - return keyenc.Join('.', string(id), strconv.FormatUint(vm.tenantVersions[id], 10), table) -} - -// namespaceKeyLocked builds the namespace-table key: the table key, then -// the table version and the scope, one more level of the same join; caller -// must hold vm.mu. -func (vm *VersionManager) namespaceKeyLocked(ns Namespace) string { - tk := vm.tableKeyLocked(ns.Tenant, ns.Table) - return string(keyenc.AppendJoin([]byte(tk+"."), '.', strconv.FormatUint(vm.tableVersions[tk], 10), ns.Scope)) -} - -// NamespaceKey renders the namespace-table key for ns at its tenant's and -// table's current versions: -// "..
..", each field -// escaped (scopeless scope is "", so e.g. ".0.
.."). -func (vm *VersionManager) NamespaceKey(ns Namespace) string { - vm.mu.RLock() - defer vm.mu.RUnlock() - return vm.namespaceKeyLocked(ns) +// NewVersionManager initializes the thread-safe version store. +func NewVersionManager() *VersionManager { + return &VersionManager{tenants: make(map[tenant.ID]*tenantVersions)} } // QueryKey builds the queries-table key for tenant id's result that depends // on deps: the query's sha (hash of SQL+params) folded with the tenant's -// version and every dependency's namespace key AND its namespace version, so -// a bump of the tenant or of any dependency misses the key — a result with no -// deps (a pipe) is orphaned by BumpTenant too. A structured query passes one -// Namespace; a pipe passes none yet (#343). Deps are sorted so their order -// never changes the key. The key nests two levels: the escaped sha and the -// '.'-joined tenant and dependency segments, separated by '|', which no -// escaped field or '.' join ever holds. +// version and, for every dependency, its tenant's, table's and scope's +// versions, so a bump of the tenant or of any dependency misses the key — a +// result with no deps (a pipe) is orphaned by BumpTenant too. A structured +// query passes one Namespace, a pipe none yet (#343). Deps are sorted so +// their order never changes the key, and every version is read under one +// lock, so the key is one consistent snapshot. The key nests two levels: +// the escaped sha and the '.'-joined tenant and dependency segments, +// separated by '|', which no escaped field or '.' join ever holds. +// +// The first key built for a tenant creates its index at a fresh generation. func (vm *VersionManager) QueryKey(id tenant.ID, sha string, deps []Namespace) string { + vm.mu.RLock() + key, ok := vm.queryKeyLocked(id, sha, deps, false) + vm.mu.RUnlock() + if ok { + return key + } + vm.mu.Lock() + defer vm.mu.Unlock() + key, _ = vm.queryKeyLocked(id, sha, deps, true) + return key +} + +// queryKeyLocked renders QueryKey with vm.mu held — for writing when create +// is set, which creates the index of each tenant the key names that has +// none; otherwise such a tenant reports !ok. +func (vm *VersionManager) queryKeyLocked(id tenant.ID, sha string, deps []Namespace, create bool) (string, bool) { + index := func(id tenant.ID) (*tenantVersions, bool) { + tv := vm.tenants[id] + if tv == nil && create { + tv = vm.newTenantLocked(id) + } + return tv, tv != nil + } + own, ok := index(id) + if !ok { + return "", false + } segs := make([]string, len(deps)) - // Lock per dependency rather than across the whole loop: each dep's table + - // namespace versions are read together (consistent for that dep), but we don't - // hold the lock across all deps. A concurrent bump can land between deps; the - // caller files its fill under this key (a Snapshot), so a bump that lands - // anywhere after the read of a version orphans it. The sort/join run with - // no lock held. for i, d := range deps { - vm.mu.RLock() - nsKey := vm.namespaceKeyLocked(d) - segs[i] = fmt.Sprintf("%s.%d", nsKey, vm.namespaceVersions[nsKey]) - vm.mu.RUnlock() + tv, ok := index(d.Tenant) + if !ok { + return "", false + } + var table, scope uint64 + if t := tv.tables[d.Table]; t != nil { + table, scope = t.version, t.scopes[d.Scope] + } + segs[i] = keyenc.Join('.', string(d.Tenant), strconv.FormatUint(tv.gen, 10), d.Table, strconv.FormatUint(table, 10), d.Scope, strconv.FormatUint(scope, 10)) } - vm.mu.RLock() - tv := vm.tenantVersions[id] - vm.mu.RUnlock() sort.Strings(segs) - return keyenc.Escape(sha) + "|" + keyenc.Join('.', string(id), strconv.FormatUint(tv, 10)) + "|" + strings.Join(segs, "|") + return keyenc.Escape(sha) + "|" + keyenc.Join('.', string(id), strconv.FormatUint(own.gen, 10)) + "|" + strings.Join(segs, "|"), true +} + +func (vm *VersionManager) newTenantLocked(id tenant.ID) *tenantVersions { + vm.lastGen++ + tv := &tenantVersions{gen: vm.lastGen, tables: make(map[string]*tableVersions)} + vm.tenants[id] = tv + return tv +} + +// tableLocked is the entry for a tenant's table, created at version 0, or +// nil when the tenant has no index: no key folds its current generation +// yet, so there is nothing a bump could orphan. Caller holds vm.mu for +// writing. +func (vm *VersionManager) tableLocked(id tenant.ID, table string) *tableVersions { + tv := vm.tenants[id] + if tv == nil { + return nil + } + t := tv.tables[table] + if t == nil { + t = &tableVersions{} + tv.tables[table] = t + } + return t } // BumpTable advances a tenant's table version, orphaning every namespace — and // every cached query — that depends on the table, in one step (the whole-table -// nuke). The same table under another tenant is untouched. +// nuke). The table's scope versions are dropped with it: every key they were +// folded into also folds the old table version. The same table under another +// tenant is untouched. func (vm *VersionManager) BumpTable(id tenant.ID, table string) { vm.mu.Lock() defer vm.mu.Unlock() - vm.tableVersions[vm.tableKeyLocked(id, table)]++ -} - -// BumpTenant advances a tenant's version, orphaning its every namespace — -// and every cached query, whatever its deps — in one step (the whole-tenant -// nuke): every namespace and query key of the tenant carries the version, so -// nothing has to be enumerated, and a table no bump ever keyed is orphaned -// like the rest. Other tenants are untouched. -func (vm *VersionManager) BumpTenant(id tenant.ID) { - vm.mu.Lock() - defer vm.mu.Unlock() - vm.tenantVersions[id]++ + if t := vm.tableLocked(id, table); t != nil { + t.version++ + t.scopes = nil + } } // BumpNamespace advances one (tenant, table, scope) namespace plus the table's @@ -127,8 +168,54 @@ func (vm *VersionManager) BumpTenant(id tenant.ID) { func (vm *VersionManager) BumpNamespace(ns Namespace) { vm.mu.Lock() defer vm.mu.Unlock() - vm.namespaceVersions[vm.namespaceKeyLocked(ns)]++ + t := vm.tableLocked(ns.Tenant, ns.Table) + if t == nil { + return + } + if t.scopes == nil { + t.scopes = make(map[string]uint64) + } + t.scopes[ns.Scope]++ if ns.Scope != "" { - vm.namespaceVersions[vm.namespaceKeyLocked(Namespace{Tenant: ns.Tenant, Table: ns.Table})]++ + t.scopes[""]++ + } +} + +// BumpTenant orphans every cached query of a tenant, whatever its deps, in +// one step (the whole-tenant nuke), by dropping the tenant's index: the next +// key built for it gets a fresh generation, which no cached entry folds. +// Nothing has to be enumerated, a table no bump ever keyed is orphaned like +// the rest, and the index the tenant held is released. Other tenants are +// untouched. +func (vm *VersionManager) BumpTenant(id tenant.ID) { + vm.mu.Lock() + defer vm.mu.Unlock() + delete(vm.tenants, id) +} + +// Prune drops the index of every tenant keep rejects, as BumpTenant would, +// so a tenant that stops being served stops holding memory; one served again +// starts over at a fresh generation. +func (vm *VersionManager) Prune(keep func(tenant.ID) bool) { + vm.mu.Lock() + defer vm.mu.Unlock() + for id := range vm.tenants { + if !keep(id) { + delete(vm.tenants, id) + } + } +} + +// size is the number of versions the index holds, for tests. +func (vm *VersionManager) size() int { + vm.mu.RLock() + defer vm.mu.RUnlock() + n := len(vm.tenants) + for _, tv := range vm.tenants { + n += len(tv.tables) + for _, t := range tv.tables { + n += len(t.scopes) + } } + return n } diff --git a/internal/cache/version_manager_test.go b/internal/cache/version_manager_test.go index 91b59db52..079453060 100644 --- a/internal/cache/version_manager_test.go +++ b/internal/cache/version_manager_test.go @@ -8,56 +8,28 @@ import ( "github.com/Wave-RF/WaveHouse/internal/tenant" ) -func TestVersionManager_NamespaceKey(t *testing.T) { - t.Parallel() - vm := NewVersionManager() - - // The tenant leads at its default version (0), then the table at its - // default version (0); a scopeless namespace renders a trailing dot. The - // flat directory's tenant is "0". - assert.Equal(t, "acme.0.users.0.", vm.NamespaceKey(Namespace{Tenant: "acme", Table: "users"})) - assert.Equal(t, "acme.0.users.0.org_1", vm.NamespaceKey(Namespace{Tenant: "acme", Table: "users", Scope: "org_1"})) - assert.Equal(t, "0.0.users.0.", vm.NamespaceKey(Namespace{Tenant: tenant.Default, Table: "users"})) - - // Names arrive raw and are escaped into the key, so a dot or a space in - // one is never read as the separator. - assert.Equal(t, "acme.0.default%2Eclicks.0.org%2E1", vm.NamespaceKey(Namespace{Tenant: "acme", Table: "default.clicks", Scope: "org.1"})) - assert.Equal(t, "acme.0.my%20table.0.", vm.NamespaceKey(Namespace{Tenant: "acme", Table: "my table"})) - - // The table version is embedded in every namespace key for that tenant's - // table, so a BumpTable is reflected across all its scopes at once — and - // nowhere else: the same table under another tenant keeps its version. - vm.BumpTable("acme", "users") - assert.Equal(t, "acme.0.users.1.", vm.NamespaceKey(Namespace{Tenant: "acme", Table: "users"})) - assert.Equal(t, "acme.0.users.1.org_1", vm.NamespaceKey(Namespace{Tenant: "acme", Table: "users", Scope: "org_1"})) - assert.Equal(t, "globex.0.users.0.", vm.NamespaceKey(Namespace{Tenant: "globex", Table: "users"})) - - // The tenant version leads every key of the tenant, so a BumpTenant moves - // every table of acme's — the never-bumped orders table included — to a - // fresh key space, at table version 0 again, and no other tenant's. - vm.BumpTenant("acme") - assert.Equal(t, "acme.1.users.0.", vm.NamespaceKey(Namespace{Tenant: "acme", Table: "users"})) - assert.Equal(t, "acme.1.orders.0.", vm.NamespaceKey(Namespace{Tenant: "acme", Table: "orders"})) - assert.Equal(t, "globex.0.users.0.", vm.NamespaceKey(Namespace{Tenant: "globex", Table: "users"})) -} - func TestVersionManager_QueryKey(t *testing.T) { t.Parallel() vm := NewVersionManager() - // One dependency at default versions: - // sha | . | ..
.... + // sha | . | ..
...; + // acme's index is created by its first key, at generation 1. key := vm.QueryKey("acme", "hash123", []Namespace{{Tenant: "acme", Table: "users", Scope: "org_1"}}) - assert.Equal(t, "hash123|acme.0|acme.0.users.0.org_1.0", key) + assert.Equal(t, "hash123|acme.1|acme.1.users.0.org_1.0", key) // No deps (a pipe) still folds the tenant version. - assert.Equal(t, "hash123|acme.0|", vm.QueryKey("acme", "hash123", nil)) + assert.Equal(t, "hash123|acme.1|", vm.QueryKey("acme", "hash123", nil)) // The sha is a field like any other: escaped, so no '|' in it can pass // for the separator. - assert.Equal(t, "acme%3Aquery%3Aab|acme.0|acme.0.my%20table.0..0", + assert.Equal(t, "acme%3Aquery%3Aab|acme.1|acme.1.my%20table.0..0", vm.QueryKey("acme", "acme:query:ab", []Namespace{{Tenant: "acme", Table: "my table"}})) + // Names arrive raw and are escaped into the key, so a dot or a space in + // one is never read as the separator. + assert.Equal(t, "h2|acme.1|acme.1.default%2Eclicks.0.org%2E1.0", + vm.QueryKey("acme", "h2", []Namespace{{Tenant: "acme", Table: "default.clicks", Scope: "org.1"}})) + // Dependency order must not change the key (segments are sorted). deps1 := []Namespace{{Tenant: "acme", Table: "a"}, {Tenant: "acme", Table: "b"}} deps2 := []Namespace{{Tenant: "acme", Table: "b"}, {Tenant: "acme", Table: "a"}} @@ -69,6 +41,9 @@ func TestVersionManager_QueryKey(t *testing.T) { vm.QueryKey("acme", "h", []Namespace{{Tenant: "acme", Table: "users"}}), vm.QueryKey("globex", "h", []Namespace{{Tenant: "globex", Table: "users"}})) assert.NotEqual(t, vm.QueryKey("acme", "h", nil), vm.QueryKey("globex", "h", nil)) + + // Reading keys is stable: nothing but a bump moves a version. + assert.Equal(t, key, vm.QueryKey("acme", "hash123", []Namespace{{Tenant: "acme", Table: "users", Scope: "org_1"}})) } func TestVersionManager_BumpTable(t *testing.T) { @@ -79,16 +54,16 @@ func TestVersionManager_BumpTable(t *testing.T) { orders := []Namespace{{Tenant: "acme", Table: "orders", Scope: "org_1"}} globexUsers := []Namespace{{Tenant: "globex", Table: "users", Scope: "org_1"}} - usersBefore := vm.QueryKey(users[0].Tenant, "h", users) - ordersBefore := vm.QueryKey(orders[0].Tenant, "h", orders) - globexBefore := vm.QueryKey(globexUsers[0].Tenant, "h", globexUsers) + usersBefore := vm.QueryKey("acme", "h", users) + ordersBefore := vm.QueryKey("acme", "h", orders) + globexBefore := vm.QueryKey("globex", "h", globexUsers) // Bumping a table changes the key for that tenant's table but leaves other // tables — and the same table under another tenant — alone. vm.BumpTable("acme", "users") - assert.NotEqual(t, usersBefore, vm.QueryKey(users[0].Tenant, "h", users)) - assert.Equal(t, ordersBefore, vm.QueryKey(orders[0].Tenant, "h", orders)) - assert.Equal(t, globexBefore, vm.QueryKey(globexUsers[0].Tenant, "h", globexUsers)) + assert.NotEqual(t, usersBefore, vm.QueryKey("acme", "h", users)) + assert.Equal(t, ordersBefore, vm.QueryKey("acme", "h", orders)) + assert.Equal(t, globexBefore, vm.QueryKey("globex", "h", globexUsers)) } func TestVersionManager_BumpNamespace(t *testing.T) { @@ -100,24 +75,44 @@ func TestVersionManager_BumpNamespace(t *testing.T) { otherScope := []Namespace{{Tenant: "acme", Table: "users", Scope: "org_2"}} otherTenant := []Namespace{{Tenant: "globex", Table: "users", Scope: "org_1"}} - scopedBefore := vm.QueryKey(scoped[0].Tenant, "h", scoped) - wholeBefore := vm.QueryKey(wholeTable[0].Tenant, "h", wholeTable) - otherBefore := vm.QueryKey(otherScope[0].Tenant, "h", otherScope) - otherTenantBefore := vm.QueryKey(otherTenant[0].Tenant, "h", otherTenant) + scopedBefore := vm.QueryKey("acme", "h", scoped) + wholeBefore := vm.QueryKey("acme", "h", wholeTable) + otherBefore := vm.QueryKey("acme", "h", otherScope) + otherTenantBefore := vm.QueryKey("globex", "h", otherTenant) // Bumping (acme, users, org_1) changes that scope AND the whole-table view, // but leaves every other scope — and the same scope under another tenant — // valid. vm.BumpNamespace(Namespace{Tenant: "acme", Table: "users", Scope: "org_1"}) - assert.NotEqual(t, scopedBefore, vm.QueryKey(scoped[0].Tenant, "h", scoped)) - assert.NotEqual(t, wholeBefore, vm.QueryKey(wholeTable[0].Tenant, "h", wholeTable)) - assert.Equal(t, otherBefore, vm.QueryKey(otherScope[0].Tenant, "h", otherScope)) - assert.Equal(t, otherTenantBefore, vm.QueryKey(otherTenant[0].Tenant, "h", otherTenant)) + assert.NotEqual(t, scopedBefore, vm.QueryKey("acme", "h", scoped)) + assert.NotEqual(t, wholeBefore, vm.QueryKey("acme", "h", wholeTable)) + assert.Equal(t, otherBefore, vm.QueryKey("acme", "h", otherScope)) + assert.Equal(t, otherTenantBefore, vm.QueryKey("globex", "h", otherTenant)) +} + +// A table bump drops the table's scope versions, which then read as 0 again +// — safe only because every key a scope version was folded into also folds +// the table version the bump moved. Pinned so a table bump that forgot to +// advance the table version would revive the scoped entry. +func TestVersionManager_BumpTableDropsScopes(t *testing.T) { + t.Parallel() + vm := NewVersionManager() + scoped := []Namespace{{Tenant: "acme", Table: "users", Scope: "org_1"}} + + fresh := vm.QueryKey("acme", "h", scoped) + vm.BumpNamespace(scoped[0]) + bumped := vm.QueryKey("acme", "h", scoped) + vm.BumpTable("acme", "users") + after := vm.QueryKey("acme", "h", scoped) + + assert.NotEqual(t, fresh, after) + assert.NotEqual(t, bumped, after) + assert.Equal(t, 2, vm.size(), "the tenant and its table; the scopes went with the table bump") } // TestVersionManager_BumpTenant: a tenant's every namespace is orphaned in -// one step — a table that was never bumped (so has no key of its own to bump) -// included — and no other tenant's is touched. +// one step — a table that was never bumped included — and no other tenant's +// is touched. func TestVersionManager_BumpTenant(t *testing.T) { t.Parallel() vm := NewVersionManager() @@ -125,18 +120,123 @@ func TestVersionManager_BumpTenant(t *testing.T) { users := []Namespace{{Tenant: "acme", Table: "users", Scope: "org_1"}} orders := []Namespace{{Tenant: "acme", Table: "orders"}} globexUsers := []Namespace{{Tenant: "globex", Table: "users", Scope: "org_1"}} + vm.QueryKey("acme", "h", users) vm.BumpTable("acme", "users") - usersBefore := vm.QueryKey(users[0].Tenant, "h", users) - ordersBefore := vm.QueryKey(orders[0].Tenant, "h", orders) - globexBefore := vm.QueryKey(globexUsers[0].Tenant, "h", globexUsers) + usersBefore := vm.QueryKey("acme", "h", users) + ordersBefore := vm.QueryKey("acme", "h", orders) + globexBefore := vm.QueryKey("globex", "h", globexUsers) vm.BumpTenant("acme") - assert.NotEqual(t, usersBefore, vm.QueryKey(users[0].Tenant, "h", users)) - assert.NotEqual(t, ordersBefore, vm.QueryKey(orders[0].Tenant, "h", orders), "a table no bump ever keyed is orphaned too") - assert.Equal(t, globexBefore, vm.QueryKey(globexUsers[0].Tenant, "h", globexUsers)) + assert.NotEqual(t, usersBefore, vm.QueryKey("acme", "h", users)) + assert.NotEqual(t, ordersBefore, vm.QueryKey("acme", "h", orders), "a table no bump ever keyed is orphaned too") + assert.Equal(t, globexBefore, vm.QueryKey("globex", "h", globexUsers)) pipeBefore := vm.QueryKey("acme", "h", nil) vm.BumpTenant("acme") assert.NotEqual(t, pipeBefore, vm.QueryKey("acme", "h", nil), "a result with no deps is orphaned too") } + +// Dropping a tenant's index is a bump only because the index it gets back +// never repeats a generation: every key built before any of these drops must +// differ from every key built after it. A counter per tenant restarting at 0 +// fails this, reviving the first entry. +func TestVersionManager_GenerationsNeverRepeat(t *testing.T) { + t.Parallel() + vm := NewVersionManager() + deps := []Namespace{{Tenant: "acme", Table: "users"}} + seen := map[string]bool{} + for i := range 100 { + key := vm.QueryKey("acme", "h", deps) + assert.False(t, seen[key], "round %d revived %s", i, key) + seen[key] = true + if i%2 == 0 { + vm.BumpTenant("acme") + } else { + vm.Prune(func(tenant.ID) bool { return false }) + } + } +} + +// A bump of a tenant with no index is a no-op: no key folds its next +// generation yet, so nothing needs orphaning — and an insert still in flight +// for a tenant just pruned does not bring its index back. +func TestVersionManager_BumpWithoutIndex(t *testing.T) { + t.Parallel() + vm := NewVersionManager() + + vm.BumpTable("acme", "users") + assert.Zero(t, vm.size(), "a table bump for a tenant with no index creates nothing") + + vm.BumpNamespace(Namespace{Tenant: "acme", Table: "users", Scope: "org_1"}) + assert.Zero(t, vm.size(), "a namespace bump for a tenant with no index creates nothing") + + vm.BumpTenant("acme") + assert.Zero(t, vm.size(), "bumping a tenant with no index is a no-op") + + // An insert still in flight for a tenant just pruned must not bring its + // index back: a write racing the prune sees the tenant gone and bumps + // blind, same as above. + vm.QueryKey("acme", "h", nil) + vm.Prune(func(tenant.ID) bool { return false }) + assert.Zero(t, vm.size(), "prune released the tenant's index") + + vm.BumpTable("acme", "users") + vm.BumpNamespace(Namespace{Tenant: "acme", Table: "users", Scope: "org_1"}) + assert.Zero(t, vm.size(), "a bump for a tenant just pruned must not recreate its index") +} + +func TestVersionManager_Prune(t *testing.T) { + t.Parallel() + vm := NewVersionManager() + acme := []Namespace{{Tenant: "acme", Table: "users"}} + globex := []Namespace{{Tenant: "globex", Table: "users"}} + acmeBefore := vm.QueryKey("acme", "h", acme) + globexBefore := vm.QueryKey("globex", "h", globex) + vm.BumpTable("acme", "users") + vm.BumpTable("globex", "users") + acmeBumped := vm.QueryKey("acme", "h", acme) + globexBumped := vm.QueryKey("globex", "h", globex) + + vm.Prune(func(id tenant.ID) bool { return id == "globex" }) + assert.Equal(t, 2, vm.size(), "globex and its table; acme released") + assert.Equal(t, globexBumped, vm.QueryKey("globex", "h", globex), "a kept tenant is untouched") + + back := vm.QueryKey("acme", "h", acme) + assert.NotEqual(t, acmeBefore, back, "a pruned tenant never revives what it cached") + assert.NotEqual(t, acmeBumped, back) + assert.NotEqual(t, globexBefore, globexBumped) +} + +// The index holds one version per live tenant, table and scope, however often +// each is bumped (#262): the nested index this replaced kept every table and +// scope under every tenant version it had seen. +func TestVersionManager_SizeDoesNotGrowWithBumps(t *testing.T) { + t.Parallel() + vm := NewVersionManager() + touch := func() { + for _, id := range []tenant.ID{"acme", "globex"} { + for _, table := range []string{"users", "orders"} { + for _, scope := range []string{"", "org_1", "org_2"} { + vm.QueryKey(id, "h", []Namespace{{Tenant: id, Table: table, Scope: scope}}) + vm.BumpNamespace(Namespace{Tenant: id, Table: table, Scope: scope}) + } + } + } + } + touch() + settled := vm.size() + assert.Equal(t, 2+2*2+2*2*3, settled, "two tenants, two tables each, three scopes each") + + for i := range 10_000 { + switch i % 3 { + case 0: + vm.BumpTable("acme", []string{"users", "orders"}[i%2]) + case 1: + vm.BumpTenant("globex") + } + touch() + assert.LessOrEqual(t, vm.size(), settled) + } + assert.Equal(t, settled, vm.size()) +} diff --git a/internal/config/backends.go b/internal/config/backends.go index fab927460..239e95334 100644 --- a/internal/config/backends.go +++ b/internal/config/backends.go @@ -35,21 +35,34 @@ func (m MQ) validate() error { // CacheBackend names the query-result cache implementation. type CacheBackend string -// CacheLocal is the in-process Ristretto cache, sized by cache.l1_max_cost. -const CacheLocal CacheBackend = "local" +const ( + // CacheLocal is the in-process Ristretto cache, sized by + // cache.l1_max_cost. + CacheLocal CacheBackend = "local" + // CacheRedis is one Redis-compatible server shared by every process, + // configured by cache.redis. + CacheRedis CacheBackend = "redis" +) -var cacheBackends = []CacheBackend{CacheLocal} +var cacheBackends = []CacheBackend{CacheLocal, CacheRedis} // Cache selects and sizes the query-result cache. The time-range bucket // structured queries normalize to is a settings-directory key // (query.timestamp_bucket_seconds) — query shaping, not process memory. type Cache struct { - Backend CacheBackend `yaml:"backend" env:"WH_CACHE_BACKEND"` - L1MaxCost int64 `yaml:"l1_max_cost" env:"WH_CACHE_L1_MAX_COST"` + Backend CacheBackend `yaml:"backend" env:"WH_CACHE_BACKEND"` + L1MaxCost int64 `yaml:"l1_max_cost" env:"WH_CACHE_L1_MAX_COST"` + Redis CacheRedisConfig `yaml:"redis"` } func (c Cache) validate() error { - return checkBackend("cache.backend", "WH_CACHE_BACKEND", c.Backend, cacheBackends) + if err := checkBackend("cache.backend", "WH_CACHE_BACKEND", c.Backend, cacheBackends); err != nil { + return err + } + if c.Backend == CacheRedis { + return c.Redis.validate() + } + return nil } // DedupeBackend names where ingest dedupe keeps the ids it has seen. @@ -126,19 +139,26 @@ func (c *Config) NeedsDataDir() bool { } // Warnings returns what a valid configuration is still likely to get wrong, -// one line each, for boot to log at WARN. They are not errors because each is -// correct for a single replica, and one process cannot count its replicas. +// one line each, for boot to log at WARN. The shared-queue ones are not +// errors because each is correct for a single replica, and one process +// cannot count its replicas. func (c *Config) Warnings() []string { - if !c.Distributed() { - return nil - } - // Both are the api role's: a process without it opens neither a cache it - // reads nor a dedupe store (a split that would need the cache shared is - // refused, validateTopology). + // All are the api role's: a process without it opens no cache it reads + // and no dedupe store, and a split's Deployments differ only in roles, so + // the API's warnings cover the others'. if !c.Has(RoleAPI) { return nil } var out []string + if c.Cache.Backend == CacheRedis && c.Cache.Redis.TLS.InsecureSkipVerify { + out = append(out, "cache.redis.tls.insecure_skip_verify is on: the cache accepts any certificate, so whoever can intercept the connection can read and replace cached query results") + } + if c.Cache.Backend != CacheRedis && c.Cache.Redis.hasAddrs() { + out = append(out, "cache.redis.addrs is set but cache.backend is "+string(c.Cache.Backend)+": the redis block is not read; set cache.backend=redis to share the cache") + } + if !c.Distributed() { + return out + } if c.Cache.Backend == CacheLocal { out = append(out, "cache.backend=local with a shared mq.backend is correct for one replica only: an event ingested on another replica never invalidates this one's cache, so its reads stay stale until the cached entry expires") } diff --git a/internal/config/backends_test.go b/internal/config/backends_test.go index 36103d12c..50d346d21 100644 --- a/internal/config/backends_test.go +++ b/internal/config/backends_test.go @@ -112,7 +112,7 @@ func TestValidate_UnknownBackend(t *testing.T) { want string }{ {"mq", func(c *Config) { c.MQ.Backend = "kafka" }, `mq.backend (WH_MQ_BACKEND) "kafka" is not a backend this build has; valid: embedded`}, - {"cache", func(c *Config) { c.Cache.Backend = "redis" }, `cache.backend (WH_CACHE_BACKEND) "redis" is not a backend this build has; valid: local`}, + {"cache", func(c *Config) { c.Cache.Backend = "memcached" }, `cache.backend (WH_CACHE_BACKEND) "memcached" is not a backend this build has; valid: local, redis`}, {"dedupe", func(c *Config) { c.Dedupe.Backend = "dynamodb" }, `dedupe.backend (WH_DEDUPE_BACKEND) "dynamodb" is not a backend this build has; valid: pebble`}, {"coord", func(c *Config) { c.Coord.Backend = "nats" }, `coord.backend (WH_COORD_BACKEND) "nats" is not a backend this build has; valid: local`}, // The zero value, which a Config built without Load carries. diff --git a/internal/config/cache_redis.go b/internal/config/cache_redis.go new file mode 100644 index 000000000..456006670 --- /dev/null +++ b/internal/config/cache_redis.go @@ -0,0 +1,178 @@ +package config + +import ( + "crypto/tls" + "crypto/x509" + "errors" + "fmt" + "net" + "os" + "strconv" + "strings" + "time" +) + +// Redis deployment modes for cache.redis.mode. RedisSentinel is refused +// until the backend supports it (#656). +const ( + RedisStandalone = "standalone" + RedisCluster = "cluster" + RedisSentinel = "sentinel" +) + +// maxRedisTimeout caps cache.redis.timeout and cache.redis.dial_timeout. +// Boot and the cache's Close each wait out a dial in flight: a connect and +// a handshake, each bounded by dial_timeout, then for a cluster a topology +// read bounded by the larger of the two. At the caps that is at most 3s; +// Close then spends up to 1s delivering owed invalidations, so at most 4s, +// inside the 5s budget it shares with the stores released after it. +const maxRedisTimeout = time.Second + +// CacheRedisConfig configures cache.backend=redis: one Redis-compatible server +// (Redis, Valkey, Dragonfly, ElastiCache, MemoryDB) shared by every process. +// Read only when that backend is selected. +type CacheRedisConfig struct { + // Addrs are host:port pairs: the server, or seeds for a cluster. + Addrs []string `yaml:"addrs" env:"WH_CACHE_REDIS_ADDRS"` + Mode string `yaml:"mode" env:"WH_CACHE_REDIS_MODE"` + Username string `yaml:"username" env:"WH_CACHE_REDIS_USERNAME"` + Password string `yaml:"password" env:"WH_CACHE_REDIS_PASSWORD"` + DB int `yaml:"db" env:"WH_CACHE_REDIS_DB"` + TLS CacheRedisTLS `yaml:"tls"` + // KeyPrefix leads every key, so deployments can share one server. + KeyPrefix string `yaml:"key_prefix" env:"WH_CACHE_REDIS_KEY_PREFIX"` + Timeout time.Duration `yaml:"timeout" env:"WH_CACHE_REDIS_TIMEOUT"` + DialTimeout time.Duration `yaml:"dial_timeout" env:"WH_CACHE_REDIS_DIAL_TIMEOUT"` + // MaxValueBytes is the largest value stored, after compression. + MaxValueBytes int `yaml:"max_value_bytes" env:"WH_CACHE_REDIS_MAX_VALUE_BYTES"` + // CompressMinBytes is the smallest value zstd-compressed; 0 never + // compresses. + CompressMinBytes int `yaml:"compress_min_bytes" env:"WH_CACHE_REDIS_COMPRESS_MIN_BYTES"` + // VersionTTL is how long a version token outlives its last bump. + VersionTTL time.Duration `yaml:"version_ttl" env:"WH_CACHE_REDIS_VERSION_TTL"` +} + +// CacheRedisTLS is cache.redis.tls. The files are paths, read at boot. +type CacheRedisTLS struct { + Enabled bool `yaml:"enabled" env:"WH_CACHE_REDIS_TLS_ENABLED"` + CAFile string `yaml:"ca_file" env:"WH_CACHE_REDIS_TLS_CA_FILE"` + CertFile string `yaml:"cert_file" env:"WH_CACHE_REDIS_TLS_CERT_FILE"` + KeyFile string `yaml:"key_file" env:"WH_CACHE_REDIS_TLS_KEY_FILE"` + ServerName string `yaml:"server_name" env:"WH_CACHE_REDIS_TLS_SERVER_NAME"` + InsecureSkipVerify bool `yaml:"insecure_skip_verify" env:"WH_CACHE_REDIS_TLS_INSECURE_SKIP_VERIFY"` +} + +// hasAddrs reports whether any address is set; a YAML `addrs: [""]` is none. +func (r CacheRedisConfig) hasAddrs() bool { + return len(r.Addrs) > 1 || len(r.Addrs) == 1 && r.Addrs[0] != "" +} + +func (r CacheRedisConfig) validate() error { + if !r.hasAddrs() { + return errors.New("cache.backend=redis needs cache.redis.addrs (WH_CACHE_REDIS_ADDRS): the server's host:port, or a cluster's seeds") + } + for i, a := range r.Addrs { + // A redis:// URL, or user:pass@host, may carry a password: refuse it + // without echoing it into the boot error and the logs. + if strings.Contains(a, "://") || strings.Contains(a, "@") { + return fmt.Errorf("cache.redis.addrs (WH_CACHE_REDIS_ADDRS) entry %d is a URL or holds credentials (not echoed): give host:port, and set the user and password with WH_CACHE_REDIS_USERNAME and WH_CACHE_REDIS_PASSWORD, and TLS (rediss://) with WH_CACHE_REDIS_TLS_ENABLED", i+1) + } + if strings.TrimSpace(a) != a { + return fmt.Errorf("cache.redis.addrs (WH_CACHE_REDIS_ADDRS) %q: no spaces around an address", a) + } + _, port, err := net.SplitHostPort(a) + if err != nil { + return fmt.Errorf("cache.redis.addrs (WH_CACHE_REDIS_ADDRS) %q: want host:port: %w", a, err) + } + if n, err := strconv.Atoi(port); err != nil || n < 1 || n > 65535 { + return fmt.Errorf("cache.redis.addrs (WH_CACHE_REDIS_ADDRS) %q: want host:port with a port from 1 to 65535", a) + } + } + switch r.Mode { + case RedisStandalone, RedisCluster: + case RedisSentinel: + return fmt.Errorf("cache.redis.mode (WH_CACHE_REDIS_MODE) %q is not supported yet: the cache neither authenticates to the sentinels nor refreshes their topology (https://github.com/Wave-RF/WaveHouse/issues/656); valid: %s, %s", r.Mode, RedisStandalone, RedisCluster) + default: + return fmt.Errorf("cache.redis.mode (WH_CACHE_REDIS_MODE) %q: valid: %s, %s", r.Mode, RedisStandalone, RedisCluster) + } + // Standalone dials the first address only, so a second one (a replica, + // say) would be silently ignored rather than failed over to. + if r.Mode == RedisStandalone && len(r.Addrs) > 1 { + return fmt.Errorf("cache.redis.addrs (WH_CACHE_REDIS_ADDRS) has %d addresses: mode standalone connects to one server; several are a cluster's seeds (mode cluster)", len(r.Addrs)) + } + if r.DB < 0 { + return fmt.Errorf("cache.redis.db (WH_CACHE_REDIS_DB) %d is negative", r.DB) + } + if r.Mode == RedisCluster && r.DB != 0 { + return fmt.Errorf("cache.redis.db (WH_CACHE_REDIS_DB) %d: a Redis cluster has only database 0", r.DB) + } + if r.KeyPrefix == "" || strings.ContainsAny(r.KeyPrefix, "{}") { + return fmt.Errorf("cache.redis.key_prefix (WH_CACHE_REDIS_KEY_PREFIX) %q: want a non-empty prefix without a hash-tag brace", r.KeyPrefix) + } + for _, d := range []struct { + key string + v time.Duration + }{ + {"cache.redis.timeout (WH_CACHE_REDIS_TIMEOUT)", r.Timeout}, + {"cache.redis.dial_timeout (WH_CACHE_REDIS_DIAL_TIMEOUT)", r.DialTimeout}, + } { + if d.v <= 0 { + return fmt.Errorf("%s %s must be positive", d.key, d.v) + } + if d.v > maxRedisTimeout { + return fmt.Errorf("%s %s is over %s: boot and shutdown each wait out a connection attempt, which both bound", d.key, d.v, maxRedisTimeout) + } + } + if r.VersionTTL < 2*time.Second { + return fmt.Errorf("cache.redis.version_ttl (WH_CACHE_REDIS_VERSION_TTL) %s is under 2s", r.VersionTTL) + } + if r.MaxValueBytes <= 0 { + return fmt.Errorf("cache.redis.max_value_bytes (WH_CACHE_REDIS_MAX_VALUE_BYTES) %d must be positive", r.MaxValueBytes) + } + if r.CompressMinBytes < 0 { + return fmt.Errorf("cache.redis.compress_min_bytes (WH_CACHE_REDIS_COMPRESS_MIN_BYTES) %d is negative: want a size, or 0 to never compress", r.CompressMinBytes) + } + if _, err := r.TLS.Config(); err != nil { + return err + } + return nil +} + +// Config builds the tls.Config the block describes, reading its files, or +// nil when TLS is off. A file set while TLS is off is an error rather than +// a silently plaintext connection. +func (t CacheRedisTLS) Config() (*tls.Config, error) { + if !t.Enabled { + if t != (CacheRedisTLS{}) { + return nil, errors.New("cache.redis.tls: files, server_name or insecure_skip_verify are set but cache.redis.tls.enabled (WH_CACHE_REDIS_TLS_ENABLED) is off") + } + return nil, nil + } + if (t.CertFile == "") != (t.KeyFile == "") { + return nil, errors.New("cache.redis.tls: cert_file and key_file must be set together") + } + cfg := &tls.Config{ + MinVersion: tls.VersionTLS12, + ServerName: t.ServerName, + InsecureSkipVerify: t.InsecureSkipVerify, //nolint:gosec // G402: the operator's cache.redis.tls.insecure_skip_verify, warned about at boot + } + if t.CAFile != "" { + pemBytes, err := os.ReadFile(t.CAFile) + if err != nil { + return nil, fmt.Errorf("cache.redis.tls.ca_file: %w", err) + } + pool := x509.NewCertPool() + if !pool.AppendCertsFromPEM(pemBytes) { + return nil, fmt.Errorf("cache.redis.tls.ca_file: no certificates in %s", t.CAFile) + } + cfg.RootCAs = pool + } + if t.CertFile != "" { + cert, err := tls.LoadX509KeyPair(t.CertFile, t.KeyFile) + if err != nil { + return nil, fmt.Errorf("cache.redis.tls.cert_file: %w", err) + } + cfg.Certificates = []tls.Certificate{cert} + } + return cfg, nil +} diff --git a/internal/config/cache_redis_test.go b/internal/config/cache_redis_test.go new file mode 100644 index 000000000..3c1f7c05d --- /dev/null +++ b/internal/config/cache_redis_test.go @@ -0,0 +1,326 @@ +package config + +import ( + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/x509" + "crypto/x509/pkix" + "encoding/pem" + "math/big" + "os" + "path/filepath" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// redisBackend is what Load produces for cache.backend=redis with only the +// address set. +func redisBackend() Config { + c := defaultBackends() + c.Cache.Backend = CacheRedis + c.Cache.Redis = CacheRedisConfig{ + Addrs: []string{"redis:6379"}, Mode: RedisStandalone, KeyPrefix: "wh", + Timeout: 100 * time.Millisecond, DialTimeout: time.Second, + MaxValueBytes: 1 << 20, CompressMinBytes: 1 << 10, VersionTTL: 168 * time.Hour, + } + return c +} + +func TestLoad_CacheRedisDefaults(t *testing.T) { + t.Setenv("WH_CACHE_BACKEND", "redis") + t.Setenv("WH_CACHE_REDIS_ADDRS", "redis:6379") + cfg, err := Load("nonexistent.yaml") + require.NoError(t, err) + want := redisBackend() + assert.Equal(t, want.Cache.Redis, cfg.Cache.Redis) + assert.Empty(t, cfg.Warnings()) +} + +func TestLoad_CacheRedisFromEnv(t *testing.T) { + dir := t.TempDir() + caFile, certFile, keyFile := writeTestPKI(t, dir) + for k, v := range map[string]string{ + "WH_CACHE_BACKEND": "redis", + "WH_CACHE_REDIS_ADDRS": "r1:6379", + "WH_CACHE_REDIS_MODE": "standalone", + "WH_CACHE_REDIS_USERNAME": "wavehouse", + "WH_CACHE_REDIS_PASSWORD": "s3cret", + "WH_CACHE_REDIS_DB": "2", + "WH_CACHE_REDIS_TLS_ENABLED": "true", + "WH_CACHE_REDIS_TLS_CA_FILE": caFile, + "WH_CACHE_REDIS_TLS_CERT_FILE": certFile, + "WH_CACHE_REDIS_TLS_KEY_FILE": keyFile, + "WH_CACHE_REDIS_TLS_SERVER_NAME": "redis.internal", + "WH_CACHE_REDIS_KEY_PREFIX": "staging", + "WH_CACHE_REDIS_TIMEOUT": "250ms", + "WH_CACHE_REDIS_DIAL_TIMEOUT": "500ms", + "WH_CACHE_REDIS_MAX_VALUE_BYTES": "2048", + "WH_CACHE_REDIS_COMPRESS_MIN_BYTES": "0", + "WH_CACHE_REDIS_VERSION_TTL": "24h", + "WH_CACHE_REDIS_TLS_INSECURE_SKIP_VERIFY": "false", + } { + t.Setenv(k, v) + } + cfg, err := Load("nonexistent.yaml") + require.NoError(t, err) + assert.Equal(t, CacheRedisConfig{ + Addrs: []string{"r1:6379"}, Mode: RedisStandalone, + Username: "wavehouse", Password: "s3cret", DB: 2, + TLS: CacheRedisTLS{ + Enabled: true, CAFile: caFile, CertFile: certFile, KeyFile: keyFile, ServerName: "redis.internal", + }, + KeyPrefix: "staging", Timeout: 250 * time.Millisecond, DialTimeout: 500 * time.Millisecond, + MaxValueBytes: 2048, CompressMinBytes: 0, VersionTTL: 24 * time.Hour, + }, cfg.Cache.Redis) + tc, err := cfg.Cache.Redis.TLS.Config() + require.NoError(t, err) + assert.Equal(t, "redis.internal", tc.ServerName) + assert.NotNil(t, tc.RootCAs) + assert.Len(t, tc.Certificates, 1) +} + +func TestLoad_CacheRedisFromYAML(t *testing.T) { + t.Parallel() + path := filepath.Join(t.TempDir(), "config.yaml") + require.NoError(t, os.WriteFile(path, []byte(` +settings: + dir: ./settings +cache: + backend: redis + redis: + addrs: ["n1:6379", "n2:6379"] + mode: cluster + key_prefix: prod + timeout: 50ms + version_ttl: 72h +`), 0o600)) + cfg, err := Load(path) + require.NoError(t, err) + r := cfg.Cache.Redis + assert.Equal(t, CacheRedis, cfg.Cache.Backend) + assert.Equal(t, []string{"n1:6379", "n2:6379"}, r.Addrs) + assert.Equal(t, RedisCluster, r.Mode) + assert.Equal(t, "prod", r.KeyPrefix) + assert.Equal(t, 50*time.Millisecond, r.Timeout) + assert.Equal(t, 72*time.Hour, r.VersionTTL) + assert.Equal(t, time.Second, r.DialTimeout, "an unset key takes its default") + assert.Equal(t, 1024, r.CompressMinBytes) +} + +// A 0 in the file is kept, and is the backend's "never compress". +func TestLoad_CacheRedisCompressZeroInYAMLIsNever(t *testing.T) { + t.Parallel() + path := filepath.Join(t.TempDir(), "config.yaml") + require.NoError(t, os.WriteFile(path, []byte(` +settings: + dir: ./settings +cache: + backend: redis + redis: + addrs: ["r:6379"] + compress_min_bytes: 0 +`), 0o600)) + cfg, err := Load(path) + require.NoError(t, err) + assert.Zero(t, cfg.Cache.Redis.CompressMinBytes) +} + +func TestLoad_CacheRedisRefusesUnknownKeys(t *testing.T) { + t.Parallel() + path := filepath.Join(t.TempDir(), "config.yaml") + require.NoError(t, os.WriteFile(path, []byte(` +settings: + dir: ./settings +cache: + backend: redis + redis: + addr: r:6379 + near_cache: + max_cost: 1 + sentinel_master: mymaster + tls: + ca: /x + memcached: + addrs: ["m:11211"] +`), 0o600)) + _, err := Load(path) + require.Error(t, err) + assert.Contains(t, err.Error(), "cache.memcached, cache.redis.addr, cache.redis.near_cache, cache.redis.sentinel_master, cache.redis.tls.ca") +} + +// The documented env file lists WH_CACHE_REDIS_ADDRS blank: that is no +// address, not an unread block to warn about, and not a valid redis one. +func TestLoad_CacheRedisBlankAddrs(t *testing.T) { + t.Setenv("WH_CACHE_REDIS_ADDRS", "") + cfg, err := Load("nonexistent.yaml") + require.NoError(t, err) + assert.Empty(t, cfg.Warnings()) + + t.Setenv("WH_CACHE_BACKEND", "redis") + _, err = Load("nonexistent.yaml") + require.ErrorContains(t, err, "cache.backend=redis needs cache.redis.addrs") +} + +// A URL-style address carries its password, and a boot error reaches the +// logs: the refusal names the entry, never its value. +func TestLoad_CacheRedisRefusesURLAddrsWithoutEchoingThem(t *testing.T) { + for _, addr := range []string{ + "redis://default:s3cret@redis:6379", + "rediss://default:s3cret@redis:6380", + "default:s3cret@redis:6379", + "redis://redis:6379", + } { + t.Run(addr, func(t *testing.T) { + t.Setenv("WH_CACHE_BACKEND", "redis") + t.Setenv("WH_CACHE_REDIS_ADDRS", "ok:6379,"+addr) + _, err := Load("nonexistent.yaml") + require.ErrorContains(t, err, "cache.redis.addrs (WH_CACHE_REDIS_ADDRS) entry 2 is a URL or holds credentials") + assert.Contains(t, err.Error(), "WH_CACHE_REDIS_USERNAME and WH_CACHE_REDIS_PASSWORD") + assert.NotContains(t, err.Error(), "s3cret") + assert.NotContains(t, err.Error(), addr) + }) + } +} + +func TestUnboundEnv_KnowsTheCacheRedisVariables(t *testing.T) { + t.Parallel() + assert.Empty(t, unboundEnv([]string{ + "WH_CACHE_REDIS_ADDRS=r:6379", "WH_CACHE_REDIS_PASSWORD=x", "WH_CACHE_REDIS_TLS_CA_FILE=/ca.pem", + "WH_CACHE_REDIS_VERSION_TTL=1h", "WH_CACHE_REDIS_COMPRESS_MIN_BYTES=0", + })) + assert.Equal(t, []string{"WH_CACHE_REDIS_ADDR"}, unboundEnv([]string{"WH_CACHE_REDIS_ADDR=r:6379"})) + assert.Equal(t, []string{"WH_CACHE_REDIS_SENTINEL_MASTER"}, unboundEnv([]string{"WH_CACHE_REDIS_SENTINEL_MASTER=m"}), "no sentinel mode until #656") +} + +func TestValidate_CacheRedis(t *testing.T) { + t.Parallel() + dir := t.TempDir() + caFile, certFile, keyFile := writeTestPKI(t, dir) + notPEM := filepath.Join(dir, "not.pem") + require.NoError(t, os.WriteFile(notPEM, []byte("hello"), 0o600)) + cases := []struct { + name string + set func(*CacheRedisConfig) + want string // "" = valid + }{ + {"defaults", func(*CacheRedisConfig) {}, ""}, + {"no addrs", func(r *CacheRedisConfig) { r.Addrs = nil }, "cache.backend=redis needs cache.redis.addrs (WH_CACHE_REDIS_ADDRS)"}, + {"addr with space", func(r *CacheRedisConfig) { r.Addrs = []string{"a:6379", " b:6379"} }, "no spaces around an address"}, + {"addr without port", func(r *CacheRedisConfig) { r.Addrs = []string{"redis"} }, `cache.redis.addrs (WH_CACHE_REDIS_ADDRS) "redis": want host:port`}, + {"addr empty port", func(r *CacheRedisConfig) { r.Addrs = []string{"redis:"} }, `"redis:": want host:port with a port from 1 to 65535`}, + {"addr port not a number", func(r *CacheRedisConfig) { r.Addrs = []string{"redis:637x"} }, "a port from 1 to 65535"}, + {"addr port out of range", func(r *CacheRedisConfig) { r.Addrs = []string{"redis:99999"} }, "a port from 1 to 65535"}, + {"standalone with two addrs", func(r *CacheRedisConfig) { r.Addrs = []string{"a:6379", "b:6379"} }, "has 2 addresses: mode standalone connects to one server"}, + {"cluster with two seeds", func(r *CacheRedisConfig) { r.Mode, r.Addrs = RedisCluster, []string{"a:6379", "b:6379"} }, ""}, + {"mode", func(r *CacheRedisConfig) { r.Mode = "replica" }, `cache.redis.mode (WH_CACHE_REDIS_MODE) "replica": valid: standalone, cluster`}, + {"sentinel refused", func(r *CacheRedisConfig) { r.Mode = RedisSentinel }, `cache.redis.mode (WH_CACHE_REDIS_MODE) "sentinel" is not supported yet: the cache neither authenticates to the sentinels nor refreshes their topology (https://github.com/Wave-RF/WaveHouse/issues/656)`}, + {"cluster", func(r *CacheRedisConfig) { r.Mode = RedisCluster }, ""}, + {"cluster db", func(r *CacheRedisConfig) { r.Mode, r.DB = RedisCluster, 1 }, "a Redis cluster has only database 0"}, + {"standalone db", func(r *CacheRedisConfig) { r.DB = 3 }, ""}, + {"negative db", func(r *CacheRedisConfig) { r.DB = -1 }, "is negative"}, + {"empty prefix", func(r *CacheRedisConfig) { r.KeyPrefix = "" }, "cache.redis.key_prefix"}, + {"brace prefix", func(r *CacheRedisConfig) { r.KeyPrefix = "a{b}" }, "hash-tag brace"}, + {"zero timeout", func(r *CacheRedisConfig) { r.Timeout = 0 }, "cache.redis.timeout (WH_CACHE_REDIS_TIMEOUT) 0s must be positive"}, + {"negative dial timeout", func(r *CacheRedisConfig) { r.DialTimeout = -time.Second }, "cache.redis.dial_timeout"}, + {"timeouts at the cap", func(r *CacheRedisConfig) { r.Timeout, r.DialTimeout = time.Second, time.Second }, ""}, + {"timeout over the cap", func(r *CacheRedisConfig) { r.Timeout = time.Second + time.Millisecond }, "cache.redis.timeout (WH_CACHE_REDIS_TIMEOUT) 1.001s is over 1s"}, + {"dial timeout over the cap", func(r *CacheRedisConfig) { r.DialTimeout = time.Second + time.Millisecond }, "cache.redis.dial_timeout (WH_CACHE_REDIS_DIAL_TIMEOUT) 1.001s is over 1s"}, + {"short version ttl", func(r *CacheRedisConfig) { r.VersionTTL = time.Second }, "cache.redis.version_ttl (WH_CACHE_REDIS_VERSION_TTL) 1s is under 2s"}, + {"zero max value", func(r *CacheRedisConfig) { r.MaxValueBytes = 0 }, "cache.redis.max_value_bytes"}, + {"compress never", func(r *CacheRedisConfig) { r.CompressMinBytes = 0 }, ""}, + {"compress negative", func(r *CacheRedisConfig) { r.CompressMinBytes = -1 }, "-1 is negative: want a size, or 0 to never compress"}, + {"tls files while off", func(r *CacheRedisConfig) { r.TLS.CAFile = caFile }, "cache.redis.tls.enabled (WH_CACHE_REDIS_TLS_ENABLED) is off"}, + {"tls system roots", func(r *CacheRedisConfig) { r.TLS.Enabled = true }, ""}, + {"tls full", func(r *CacheRedisConfig) { + r.TLS = CacheRedisTLS{Enabled: true, CAFile: caFile, CertFile: certFile, KeyFile: keyFile} + }, ""}, + {"tls cert without key", func(r *CacheRedisConfig) { r.TLS = CacheRedisTLS{Enabled: true, CertFile: certFile} }, "cert_file and key_file must be set together"}, + {"tls missing ca", func(r *CacheRedisConfig) { + r.TLS = CacheRedisTLS{Enabled: true, CAFile: filepath.Join(dir, "missing.pem")} + }, "cache.redis.tls.ca_file"}, + {"tls ca not pem", func(r *CacheRedisConfig) { r.TLS = CacheRedisTLS{Enabled: true, CAFile: notPEM} }, "no certificates in"}, + {"tls bad pair", func(r *CacheRedisConfig) { + r.TLS = CacheRedisTLS{Enabled: true, CertFile: certFile, KeyFile: notPEM} + }, "cache.redis.tls.cert_file"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + cfg := redisBackend() + tc.set(&cfg.Cache.Redis) + err := cfg.Validate() + if tc.want == "" { + require.NoError(t, err) + return + } + require.Error(t, err) + assert.Contains(t, err.Error(), tc.want) + }) + } +} + +// The redis block is read only when selected: an invalid one under +// backend=local does not refuse boot, it warns that it is ignored. +func TestValidate_CacheRedisIgnoredUnlessSelected(t *testing.T) { + t.Parallel() + cfg := defaultBackends() + cfg.Cache.Redis.Addrs = []string{"no-port"} + require.NoError(t, cfg.Validate()) + got := cfg.Warnings() + require.Len(t, got, 1) + assert.Contains(t, got[0], "cache.redis.addrs is set but cache.backend is local") +} + +func TestWarnings_CacheRedis(t *testing.T) { + t.Parallel() + cfg := redisBackend() + assert.Empty(t, cfg.Warnings()) + cfg.Cache.Redis.TLS = CacheRedisTLS{Enabled: true, InsecureSkipVerify: true} + require.NoError(t, cfg.Validate()) + got := cfg.Warnings() + require.Len(t, got, 1) + assert.Contains(t, got[0], "cache.redis.tls.insecure_skip_verify is on") + + // A shared cache clears the shared-queue warning about a local one. + cfg = redisBackend() + cfg.MQ.Backend = "shared" + got = cfg.Warnings() + require.Len(t, got, 1) + assert.Contains(t, got[0], "dedupe.backend=pebble") +} + +// writeTestPKI writes a self-signed authority and a client certificate it +// signed, returning the three paths cache.redis.tls names. +func writeTestPKI(t *testing.T, dir string) (caFile, certFile, keyFile string) { + t.Helper() + caKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + require.NoError(t, err) + ca := &x509.Certificate{ + SerialNumber: big.NewInt(1), Subject: pkix.Name{CommonName: "test ca"}, + NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(time.Hour), + IsCA: true, BasicConstraintsValid: true, KeyUsage: x509.KeyUsageCertSign, + } + caDER, err := x509.CreateCertificate(rand.Reader, ca, ca, &caKey.PublicKey, caKey) + require.NoError(t, err) + leafKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + require.NoError(t, err) + leaf := &x509.Certificate{ + SerialNumber: big.NewInt(2), Subject: pkix.Name{CommonName: "wavehouse"}, + NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(time.Hour), + KeyUsage: x509.KeyUsageDigitalSignature, ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}, + } + leafDER, err := x509.CreateCertificate(rand.Reader, leaf, ca, &leafKey.PublicKey, caKey) + require.NoError(t, err) + keyDER, err := x509.MarshalECPrivateKey(leafKey) + require.NoError(t, err) + write := func(name, typ string, der []byte) string { + path := filepath.Join(dir, name) + require.NoError(t, os.WriteFile(path, pem.EncodeToMemory(&pem.Block{Type: typ, Bytes: der}), 0o600)) + return path + } + return write("ca.pem", "CERTIFICATE", caDER), write("client.pem", "CERTIFICATE", leafDER), write("client.key", "EC PRIVATE KEY", keyDER) +} diff --git a/internal/config/config.go b/internal/config/config.go index 95bfacb15..0bc70f2ab 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -7,6 +7,7 @@ import ( "os" "slices" "strings" + "time" "github.com/ilyakaznacheev/cleanenv" ) @@ -217,7 +218,7 @@ func (c *Config) validateTopology() error { return fmt.Errorf("roles %s with mq.backend=embedded: the embedded MQ lives inside this process, and a process without it cannot reach its queue — run every role (%s), or set a shared mq.backend", joinRoles(c.Roles), joinRoles(allRoles)) } if c.splitsCache() && c.Cache.Backend == CacheLocal { - return fmt.Errorf("roles %s with cache.backend=local: api and ingest run in different processes, and the ingest worker's cache invalidation would never reach the API's cache — run api and ingest together, or set a shared cache.backend", joinRoles(c.Roles)) + return fmt.Errorf("roles %s with cache.backend=local: api and ingest run in different processes, and the ingest worker's cache invalidation would never reach the API's cache — run api and ingest together, or set cache.backend=redis, one cache every process shares", joinRoles(c.Roles)) } return nil } @@ -255,9 +256,16 @@ func defaults() Config { Roles: AllRoles(), Server: Server{Port: 8080, ShutdownTimeout: 10}, MQ: MQ{Backend: MQEmbedded}, - Cache: Cache{Backend: CacheLocal, L1MaxCost: 64 << 20}, - Dedupe: Dedupe{Backend: DedupePebble}, - Coord: Coord{Backend: CoordLocal}, + Cache: Cache{ + Backend: CacheLocal, L1MaxCost: 64 << 20, + Redis: CacheRedisConfig{ + Mode: RedisStandalone, KeyPrefix: "wh", + Timeout: 100 * time.Millisecond, DialTimeout: time.Second, + MaxValueBytes: 1 << 20, CompressMinBytes: 1 << 10, VersionTTL: 168 * time.Hour, + }, + }, + Dedupe: Dedupe{Backend: DedupePebble}, + Coord: Coord{Backend: CoordLocal}, OTel: OTel{ Traces: OTelTraces{Enabled: true, SampleRate: 1.0}, Metrics: OTelMetrics{Enabled: true}, diff --git a/internal/config/defaults_test.go b/internal/config/defaults_test.go index 6877be4bb..1a8fba337 100644 --- a/internal/config/defaults_test.go +++ b/internal/config/defaults_test.go @@ -9,6 +9,7 @@ import ( "strconv" "strings" "testing" + "time" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" @@ -37,6 +38,15 @@ var zeroCases = []zeroCase{ {"cache.l1_max_cost", "WH_CACHE_L1_MAX_COST", int64(0), int64(64 << 20), "1024", int64(1024), func(c *Config) any { return c.Cache.L1MaxCost }}, {"prometheus.path", "WH_PROMETHEUS_PATH", "", "/metrics", "/prom", "/prom", func(c *Config) any { return c.Prometheus.Path }}, {"data_dir", "WH_DATA_DIR", "", "./data", "/var/lib/wh", "/var/lib/wh", func(c *Config) any { return c.DataDir }}, + // The cache.redis block is validated only under backend=redis, so under + // the default backend its zeros load as written. + {"cache.redis.mode", "WH_CACHE_REDIS_MODE", "", RedisStandalone, RedisCluster, RedisCluster, func(c *Config) any { return c.Cache.Redis.Mode }}, + {"cache.redis.key_prefix", "WH_CACHE_REDIS_KEY_PREFIX", "", "wh", "staging", "staging", func(c *Config) any { return c.Cache.Redis.KeyPrefix }}, + {"cache.redis.timeout", "WH_CACHE_REDIS_TIMEOUT", time.Duration(0), 100 * time.Millisecond, "250ms", 250 * time.Millisecond, func(c *Config) any { return c.Cache.Redis.Timeout }}, + {"cache.redis.dial_timeout", "WH_CACHE_REDIS_DIAL_TIMEOUT", time.Duration(0), time.Second, "500ms", 500 * time.Millisecond, func(c *Config) any { return c.Cache.Redis.DialTimeout }}, + {"cache.redis.max_value_bytes", "WH_CACHE_REDIS_MAX_VALUE_BYTES", 0, 1 << 20, "2048", 2048, func(c *Config) any { return c.Cache.Redis.MaxValueBytes }}, + {"cache.redis.compress_min_bytes", "WH_CACHE_REDIS_COMPRESS_MIN_BYTES", 0, 1 << 10, "2048", 2048, func(c *Config) any { return c.Cache.Redis.CompressMinBytes }}, + {"cache.redis.version_ttl", "WH_CACHE_REDIS_VERSION_TTL", time.Duration(0), 168 * time.Hour, "1h", time.Hour, func(c *Config) any { return c.Cache.Redis.VersionTTL }}, } // refusedZeros are the non-zero defaults whose zero Validate refuses: written @@ -296,11 +306,15 @@ func parseDocDefault(t *testing.T, key, cell string, like any) any { v, err = strconv.ParseInt(cell, 10, 64) case float64: v, err = strconv.ParseFloat(cell, 64) + case time.Duration: + v, err = time.ParseDuration(cell) default: rt := reflect.TypeOf(like) switch { case rt.Kind() == reflect.String: v = reflect.ValueOf(cell).Convert(rt).Interface() + case rt.Kind() == reflect.Slice && rt.Elem().Kind() == reflect.String && cell == "": + v = reflect.Zero(rt).Interface() // cache.redis.addrs: nil case rt.Kind() == reflect.Slice && rt.Elem().Kind() == reflect.String: // roles: a comma-separated cell parts := strings.Split(cell, ",") sv := reflect.MakeSlice(rt, len(parts), len(parts)) diff --git a/internal/config/roles_test.go b/internal/config/roles_test.go index 18b28267c..914815bf9 100644 --- a/internal/config/roles_test.go +++ b/internal/config/roles_test.go @@ -114,12 +114,14 @@ func TestValidate_RoleSplits(t *testing.T) { {"every role, shared queue", all, "shared", CacheLocal, ""}, {"api+ingest, shared queue", []Role{RoleAPI, RoleIngest}, "shared", CacheLocal, ""}, {"sweeper, shared queue", []Role{RoleSweeper}, "shared", CacheLocal, ""}, - {"api, local cache", []Role{RoleAPI}, "shared", CacheLocal, "roles api with cache.backend=local: api and ingest run in different processes"}, + {"api, local cache", []Role{RoleAPI}, "shared", CacheLocal, "roles api with cache.backend=local: api and ingest run in different processes, and the ingest worker's cache invalidation would never reach the API's cache — run api and ingest together, or set cache.backend=redis"}, {"ingest, local cache", []Role{RoleIngest}, "shared", CacheLocal, "roles ingest with cache.backend=local"}, {"api+sweeper, local cache", []Role{RoleAPI, RoleSweeper}, "shared", CacheLocal, "roles api,sweeper with cache.backend=local"}, {"ingest+sweeper, local cache", []Role{RoleIngest, RoleSweeper}, "shared", CacheLocal, "roles ingest,sweeper with cache.backend=local"}, {"api, shared cache", []Role{RoleAPI}, "shared", "shared", ""}, {"ingest, shared cache", []Role{RoleIngest}, "shared", "shared", ""}, + {"api, redis cache", []Role{RoleAPI}, "shared", CacheRedis, ""}, + {"ingest, redis cache", []Role{RoleIngest}, "shared", CacheRedis, ""}, } { t.Run(tc.name, func(t *testing.T) { t.Parallel() diff --git a/internal/ingest/worker.go b/internal/ingest/worker.go index 365b5ae56..5ec26ac6a 100644 --- a/internal/ingest/worker.go +++ b/internal/ingest/worker.go @@ -906,7 +906,9 @@ func (w *IngestWorker) invalidate(ctx context.Context, id tenant.ID, tableName s } invCtx := trace.ContextWithSpanContext(context.WithoutCancel(ctx), trace.SpanContextFromContext(ctx)) if _, err := w.cache.Invalidate(invCtx, namespaces); err != nil { - slog.ErrorContext(invCtx, "failed to invalidate cache after insert - your cache is holding stale data now!", "tenant", id, "table", tableName, "error", err) + // WARN, not ERROR: a shared backend defers and retries the bump, and + // an outage would otherwise log an ERROR for every batch. + slog.WarnContext(invCtx, "cache invalidation after insert did not land; the table's cached results may be stale until it does", "tenant", id, "table", tableName, "error", err) } } diff --git a/scripts/orchestrator/main.go b/scripts/orchestrator/main.go index a128d402e..10eacba89 100644 --- a/scripts/orchestrator/main.go +++ b/scripts/orchestrator/main.go @@ -1,10 +1,12 @@ // E2E orchestrator — drives a clean, isolated E2E test session against -// one ClickHouse + one WaveHouse, then runs the vitest suite once. +// one ClickHouse + one Redis + one WaveHouse, then runs the vitest suite +// once. // // Lifecycle: // -// 1. Start ClickHouse via testcontainers-go (random host ports — no -// conflict with `make dev` or other compose stacks). +// 1. Start ClickHouse and Redis (the fixture's shared cache) via +// testcontainers-go (random host ports — no conflict with `make dev` or +// other compose stacks). // 2. Pick a random free TCP port on 127.0.0.1 and start bin/wavehouse-cov // bound to it (WH_SERVER_PORT) with auth enabled. Random port avoids // conflicts with `make dev`, dev servers, and previous runs that may @@ -44,6 +46,7 @@ import ( "syscall" "time" + "github.com/moby/moby/api/types/container" "github.com/testcontainers/testcontainers-go" "github.com/testcontainers/testcontainers-go/wait" ) @@ -181,6 +184,43 @@ func run() error { return fmt.Errorf("settings dir: %w", err) } + // The shared cache the fixture's cache.backend=redis names: the suite + // runs the backend a multi-instance deployment runs. No persistence, and + // /data on tmpfs so the image's VOLUME leaves no anonymous volume behind. + log.Println("→ starting Redis testcontainer...") + redis, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{ + ContainerRequest: testcontainers.ContainerRequest{ + // Pinned to match internal/cache's integration suite. + Image: "redis:8.10.2-alpine", + Cmd: []string{"redis-server", "--save", "", "--appendonly", "no"}, + ExposedPorts: []string{"6379/tcp"}, + HostConfigModifier: func(hc *container.HostConfig) { + hc.Tmpfs = map[string]string{"/data": ""} + }, + WaitingFor: wait.ForLog("Ready to accept connections").WithStartupTimeout(60 * time.Second), + }, + Started: true, + }) + if err != nil { + return fmt.Errorf("redis start: %w", err) + } + defer func() { + log.Println("→ terminating Redis testcontainer...") + if err := redis.Terminate(context.Background()); err != nil { + log.Printf(" redis terminate: %v", err) + } + }() + redisHost, err := redis.Host(ctx) + if err != nil { + return fmt.Errorf("redis host: %w", err) + } + redisPort, err := redis.MappedPort(ctx, "6379") + if err != nil { + return fmt.Errorf("redis port: %w", err) + } + redisAddr := net.JoinHostPort(redisHost, redisPort.Port()) + log.Printf("✓ Redis ready: %s", redisAddr) + whPort, err := pickFreePort(ctx) if err != nil { return fmt.Errorf("pick free port: %w", err) @@ -198,14 +238,15 @@ func run() error { // tests/e2e/fixtures/config.yaml and the tunables, policy, roles, and // pipes in tests/e2e/fixtures/settings — edit them there, not here. The // vars below are the per-run dynamic overrides (port, scratch paths, the - // patched settings copy) plus GOCOVERDIR and WH_CONFIG, which can't live - // in YAML. + // patched settings copy, the Redis address) plus GOCOVERDIR and + // WH_CONFIG, which can't live in YAML. whCmd.Env = append(os.Environ(), "GOCOVERDIR="+coverDir, "WH_CONFIG="+filepath.Join(repoRoot, "tests", "e2e", "fixtures", "config.yaml"), "WH_SERVER_PORT="+strconv.Itoa(whPort), "WH_SETTINGS_DIR="+settingsDir, "WH_DATA_DIR="+dataDir, + "WH_CACHE_REDIS_ADDRS="+redisAddr, ) if os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT") == "" { diff --git a/tests/e2e/fixtures/config.yaml b/tests/e2e/fixtures/config.yaml index 4ce30280b..69e8cb33e 100644 --- a/tests/e2e/fixtures/config.yaml +++ b/tests/e2e/fixtures/config.yaml @@ -1,8 +1,9 @@ # WaveHouse config for the e2e harness (scripts/orchestrator + tests/e2e). # -# Dynamic values — ClickHouse addr/HTTP port (testcontainer), WaveHouse -# server port (free port), WH_DATA_DIR (per-run scratch) — are injected by -# the orchestrator via env vars. Everything below is pinned here so the +# Dynamic values — ClickHouse addr/HTTP port (testcontainer), the Redis +# address (testcontainer, WH_CACHE_REDIS_ADDRS), WaveHouse server port (free +# port), WH_DATA_DIR (per-run scratch) — are injected by the orchestrator via +# env vars. Everything below is pinned here so the # rig config is visible and editable without recompiling Go. # JWT validation. The suite signs test tokens with this fixed dev secret; @@ -13,6 +14,15 @@ auth: # operator_key: "" # non-JWT full-access operator credential (Authorization: Operator, or X-Operator-Key); unset in the rig +# The shared cache, as a multi-instance deployment runs it; the in-process +# backend is covered by the unit and integration suites. The timeout is ten +# times the default so a loaded runner's slow round trip is not a bypassed +# lookup that a HIT assertion reads as a failure. +cache: + backend: redis + redis: + timeout: 1s + # Tenant tunables (schema refresh_interval 5s so schema-discovery tests don't # wait a minute; dedupe enabled + id_field; DLQ on; CORS "*"; a 1 GiB NATS # stream budget so the testcontainer stays tiny), the policy, and the pipes diff --git a/tests/integration/shared_cache_test.go b/tests/integration/shared_cache_test.go new file mode 100644 index 000000000..6ace9a764 --- /dev/null +++ b/tests/integration/shared_cache_test.go @@ -0,0 +1,315 @@ +//go:build integration + +package tests + +import ( + "context" + "fmt" + "io" + "net" + "net/http" + "net/url" + "strings" + "sync/atomic" + "testing" + "time" + + "github.com/moby/moby/api/types/container" + "github.com/moby/moby/client" + "github.com/redis/rueidis" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "github.com/testcontainers/testcontainers-go" + "github.com/testcontainers/testcontainers-go/wait" + + "github.com/Wave-RF/WaveHouse/internal/app" + "github.com/Wave-RF/WaveHouse/internal/cache" + "github.com/Wave-RF/WaveHouse/internal/config" +) + +// Pinned, as internal/cache's integration suite pins it. +const redisImage = "redis:8.10.2-alpine" + +// minCacheTTL is cache.QueryTimeToTTL's floor: a fill made less than this +// long ago cannot have expired, so a miss inside it is an invalidation. +var minCacheTTL = cache.QueryTimeToTTL(0) + +var cachePrefixes atomic.Uint64 + +// startRedis runs a throwaway Redis with no persistence, its /data on tmpfs +// so the image's VOLUME leaves no anonymous volume behind. +func startRedis(t *testing.T) (testcontainers.Container, string) { + t.Helper() + ctx := context.Background() + ctr, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{ + ContainerRequest: testcontainers.ContainerRequest{ + Image: redisImage, + Cmd: []string{"redis-server", "--save", "", "--appendonly", "no"}, + ExposedPorts: []string{"6379/tcp"}, + HostConfigModifier: func(hc *container.HostConfig) { + hc.Tmpfs = map[string]string{"/data": ""} + }, + WaitingFor: wait.ForLog("Ready to accept connections").WithStartupTimeout(90 * time.Second), + }, + Started: true, + }) + testcontainers.CleanupContainer(t, ctr) + require.NoError(t, err) + host, err := ctr.Host(ctx) + require.NoError(t, err) + port, err := ctr.MappedPort(ctx, "6379/tcp") + require.NoError(t, err) + return ctr, net.JoinHostPort(host, port.Port()) +} + +// bootRedisApp runs a second, independent WaveHouse — its own embedded NATS, +// ingest worker and data_dir — against the suite's ClickHouse, with +// cache.backend=redis. It returns the instance's base URL. +func bootRedisApp(t *testing.T, redisAddr, prefix string, timeout time.Duration) string { + t.Helper() + e := env(t) + ctx := context.Background() + settingsDir, err := writeTestSettings(e.ch) + require.NoError(t, err) + var lc net.ListenConfig + ln, err := lc.Listen(ctx, "tcp", "127.0.0.1:0") + require.NoError(t, err) + cfg := &config.Config{ + DataDir: t.TempDir(), + Server: config.Server{ShutdownTimeout: 10}, + ClickHouse: config.ClickHouse{Password: testCHPassword}, + MQ: config.MQ{Backend: config.MQEmbedded}, + Cache: config.Cache{Backend: config.CacheRedis, Redis: config.CacheRedisConfig{ + Addrs: []string{redisAddr}, Mode: config.RedisStandalone, KeyPrefix: prefix, + Timeout: timeout, DialTimeout: time.Second, + MaxValueBytes: 1 << 20, CompressMinBytes: 1 << 10, VersionTTL: time.Hour, + }}, + Dedupe: config.Dedupe{Backend: config.DedupePebble}, + Coord: config.Coord{Backend: config.CoordLocal}, + Roles: config.AllRoles(), + Settings: config.Settings{Dir: settingsDir}, + } + a, err := app.New(ctx, app.Options{Config: cfg, Listener: ln}) + require.NoError(t, err) + runCtx, stop := context.WithCancel(ctx) + runDone := make(chan error, 1) + go func() { runDone <- a.Run(runCtx) }() + t.Cleanup(func() { + stop() + assert.NoError(t, <-runDone) + closeCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + assert.NoError(t, a.Close(closeCtx)) + }) + baseURL := "http://" + ln.Addr().String() + require.NoError(t, waitForLive(ctx, baseURL, 30*time.Second)) + return baseURL +} + +// structuredQuery posts a select-all structured query and returns the +// status, the X-Cache header and the body. +func structuredQuery(t *testing.T, baseURL, table string) (int, string, string) { + t.Helper() + status, xc, body, err := tryStructuredQuery(baseURL, table) + require.NoError(t, err) + return status, xc, body +} + +// tryStructuredQuery is structuredQuery for an Eventually condition, which +// runs off the test goroutine and so must not call require. +func tryStructuredQuery(baseURL, table string) (int, string, string, error) { + req, err := http.NewRequestWithContext(context.Background(), http.MethodPost, + baseURL+"/v1/query?table="+url.QueryEscape(table), strings.NewReader(`{"select_all":true}`)) + if err != nil { + return 0, "", "", err + } + req.Header.Set("Content-Type", "application/json") + resp, err := http.DefaultClient.Do(req) + if err != nil { + return 0, "", "", err + } + defer func() { _ = resp.Body.Close() }() + body, err := io.ReadAll(resp.Body) + return resp.StatusCode, resp.Header.Get("X-Cache"), string(body), err +} + +func ingestRow(t *testing.T, baseURL, table, user string) { + t.Helper() + resp, err := http.Post(baseURL+"/v1/ingest?table="+url.QueryEscape(table), "application/json", + strings.NewReader(fmt.Sprintf(`{"user_id":%q,"value":1}`, user))) + require.NoError(t, err) + defer func() { _ = resp.Body.Close() }() + require.Equal(t, http.StatusOK, resp.StatusCode) +} + +// Two WaveHouse processes share one Redis and one ClickHouse. A result one +// fills is a hit for the other, and an insert one process's worker makes +// invalidates what the other cached: the other's next query is a miss that +// returns the new row, well inside the TTL the stale entry was filed with. +func TestSharedCache_IngestOnOneInstanceInvalidatesAnother(t *testing.T) { + table := createTable(t, "user_id String, value Float64", "ORDER BY user_id") + _, redisAddr := startRedis(t) + prefix := fmt.Sprintf("it%d", cachePrefixes.Add(1)) + a := bootRedisApp(t, redisAddr, prefix, time.Second) + b := bootRedisApp(t, redisAddr, prefix, time.Second) + + rc, err := rueidis.NewClient(rueidis.ClientOption{InitAddress: []string{redisAddr}, DisableCache: true, ForceSingleClient: true}) + require.NoError(t, err) + t.Cleanup(rc.Close) + tableToken := func() string { + v, err := rc.Do(context.Background(), rc.B().Get().Key(prefix+":{0}:B:"+table).Build()).ToString() + if rueidis.IsRedisNil(err) { + return "" + } + require.NoError(t, err) + return v + } + + status, xc, body := structuredQuery(t, b, table) + require.Equal(t, http.StatusOK, status, body) + require.Equal(t, "MISS", xc) + status, xc, _ = structuredQuery(t, a, table) + require.Equal(t, http.StatusOK, status) + require.Equal(t, "HIT", xc, "a fills, b hits: one cache") + + // An insert's worker and its invalidation run on whichever process took + // the ingest, so the discriminating window is the stale entry's TTL: if + // the batch window and load push the new row past it, the round proves + // nothing and runs again with a fresh fill. + for round := 1; ; round++ { + user := fmt.Sprintf("user-%d", round) + status, xc, body = structuredQuery(t, b, table) + require.Equal(t, http.StatusOK, status, body) + require.NotContains(t, body, user) + filled := time.Now() + if xc == "HIT" { + // The previous round's fill: refresh it so the TTL window starts now. + _, err := rc.Do(context.Background(), rc.B().Flushdb().Build()).ToString() + require.NoError(t, err) + status, xc, body = structuredQuery(t, b, table) + require.Equal(t, http.StatusOK, status, body) + require.Equal(t, "MISS", xc) + filled = time.Now() + } + before := tableToken() + + ingestRow(t, a, table, user) + var seenAt time.Time + require.Eventually(t, func() bool { + var err error + status, xc, body, err = tryStructuredQuery(b, table) + if err == nil && status == http.StatusOK && strings.Contains(body, user) { + seenAt = time.Now() + return true + } + return false + }, 30*time.Second, 100*time.Millisecond, "b never served the row ingested through a") + assert.NotEqual(t, before, tableToken(), "a's worker bumped the table token in the shared server") + if seenAt.Sub(filled) < minCacheTTL-time.Second { + assert.Equal(t, "MISS", xc, "the first answer carrying the new row is b's refill") + status, xc, _ = structuredQuery(t, b, table) + require.Equal(t, http.StatusOK, status) + assert.Equal(t, "HIT", xc, "b's refill is cached again") + return + } + require.Less(t, round, 3, "b served the new row only once its stale entry could have expired, in every round: the invalidation never reached it, or ingest is too slow here to tell") + t.Logf("round %d: row landed %s after the fill, past the TTL floor; retrying", round, seenAt.Sub(filled)) + } +} + +// The same lifecycle on the default backend, against the suite's own app +// (cache.backend=local), which e2e no longer runs: a fill is a hit, and an +// ingest invalidates it, so the first answer carrying the new row is a miss +// well inside the stale entry's TTL, and the refill is a hit again. +func TestLocalCache_IngestInvalidates(t *testing.T) { + base := env(t).baseURL + // As above: a round whose row lands past the TTL floor proves nothing, + // and runs again on a fresh table, so a fresh fill. + for round := 1; ; round++ { + table := createTable(t, "user_id String, value Float64", "ORDER BY user_id") + status, xc, body := structuredQuery(t, base, table) + require.Equal(t, http.StatusOK, status, body) + require.Equal(t, "MISS", xc) + filled := time.Now() + status, xc, _ = structuredQuery(t, base, table) + require.Equal(t, http.StatusOK, status) + require.Equal(t, "HIT", xc) + + ingestRow(t, base, table, "u1") + var seenAt time.Time + require.Eventually(t, func() bool { + var err error + status, xc, body, err = tryStructuredQuery(base, table) + if err == nil && status == http.StatusOK && strings.Contains(body, "u1") { + seenAt = time.Now() + return true + } + return false + }, 30*time.Second, 100*time.Millisecond, "the ingested row is never served") + if seenAt.Sub(filled) < minCacheTTL-time.Second { + assert.Equal(t, "MISS", xc, "the first answer carrying the new row is a refill") + status, xc, body = structuredQuery(t, base, table) + require.Equal(t, http.StatusOK, status) + assert.Equal(t, "HIT", xc, "the refill is cached again") + assert.Contains(t, body, "u1") + return + } + require.Less(t, round, 3, "the new row was served only once the stale entry could have expired, in every round: the ingest never invalidated it, or ingest is too slow here to tell") + t.Logf("round %d: row landed %s after the fill, past the TTL floor; retrying", round, seenAt.Sub(filled)) + } +} + +// A Redis that stops answering costs queries nothing but the cache: they +// keep succeeding, straight from ClickHouse, each a miss; an ingest made +// meanwhile is visible at once. Once it answers again, the cache serves hits. +func TestSharedCache_RedisDownQueriesBypass(t *testing.T) { + ctx := context.Background() + table := createTable(t, "user_id String, value Float64", "ORDER BY user_id") + ctr, redisAddr := startRedis(t) + const timeout = 200 * time.Millisecond + a := bootRedisApp(t, redisAddr, fmt.Sprintf("it%d", cachePrefixes.Add(1)), timeout) + + status, xc, _ := structuredQuery(t, a, table) + require.Equal(t, http.StatusOK, status) + require.Equal(t, "MISS", xc) + status, xc, _ = structuredQuery(t, a, table) + require.Equal(t, http.StatusOK, status) + require.Equal(t, "HIT", xc) + + d, err := testcontainers.NewDockerClientWithOpts(ctx) + require.NoError(t, err) + t.Cleanup(func() { _ = d.Close() }) + _, err = d.ContainerPause(ctx, ctr.GetContainerID(), client.ContainerPauseOptions{}) + require.NoError(t, err) + paused := true + unpause := func() { + if paused { + paused = false + _, err := d.ContainerUnpause(ctx, ctr.GetContainerID(), client.ContainerUnpauseOptions{}) + require.NoError(t, err) + } + } + t.Cleanup(unpause) + + for range 8 { + start := time.Now() + status, xc, body := structuredQuery(t, a, table) + require.Equal(t, http.StatusOK, status, body) + assert.Equal(t, "MISS", xc) + // A lookup and a fill each wait at most the timeout; the query itself + // is a few ms. Generous for -race under load. + assert.Less(t, time.Since(start), 5*timeout+2*time.Second) + } + + ingestRow(t, a, table, "while-down") + require.Eventually(t, func() bool { + status, _, body, err := tryStructuredQuery(a, table) + return err == nil && status == http.StatusOK && strings.Contains(body, "while-down") + }, 30*time.Second, 200*time.Millisecond, "a row ingested while the cache is down is served") + + unpause() + require.Eventually(t, func() bool { + _, xc, body, err := tryStructuredQuery(a, table) + return err == nil && xc == "HIT" && strings.Contains(body, "while-down") + }, 30*time.Second, 200*time.Millisecond, "the cache serves hits again once the server answers") +}