From 7632e3e3f7c3533532b10deb2e4dcff1f4996205 Mon Sep 17 00:00:00 2001 From: taitelee Date: Fri, 18 Sep 2026 09:25:32 -0400 Subject: [PATCH 01/11] feat(tenant): resolve X-Tenant-ID before auth and thread the tenant --- AGENTS.md | 6 +- CHANGELOG.md | 2 + docs/src/content/docs/api.md | 23 ++ docs/src/content/docs/architecture.md | 10 +- docs/src/content/docs/sdk/index.mdx | 2 +- internal/api/boot_chain_test.go | 3 +- internal/api/errors_test.go | 14 +- internal/api/ingest.go | 31 ++- internal/api/ingest_seams_test.go | 20 +- internal/api/ingest_test.go | 243 +++++++++++----------- internal/api/pipes.go | 29 ++- internal/api/pipes_test.go | 80 +++---- internal/api/router.go | 68 +++--- internal/api/router_test.go | 54 +++-- internal/api/stream_test.go | 9 +- internal/api/structured_query.go | 33 +-- internal/api/structured_query_test.go | 49 ++--- internal/api/tenant.go | 79 +++++++ internal/api/tenant_helpers_test.go | 34 +++ internal/api/tenant_test.go | 164 +++++++++++++++ internal/app/app.go | 4 + internal/app/app_test.go | 31 +++ internal/app/wire.go | 43 +++- internal/discovery/discovery.go | 21 +- internal/discovery/discovery_test.go | 28 +-- internal/discovery/timestamp_test.go | 4 +- internal/ingest/sweeper.go | 18 +- internal/ingest/sweeper_test.go | 9 +- internal/ingest/worker.go | 20 +- internal/ingest/worker_test.go | 23 +- internal/settings/registry.go | 22 ++ internal/settings/registry_test.go | 24 +++ internal/stream/hub.go | 17 +- internal/stream/hub_test.go | 81 ++++---- internal/stream/roweval_test.go | 7 +- internal/stream/tenant_test.go | 11 + internal/tenant/tenant.go | 50 +++++ internal/tenant/tenant_test.go | 53 +++++ internal/testutil/testutil.go | 3 +- tests/integration/boot_resilience_test.go | 5 +- tests/integration/query_limits_test.go | 10 +- 41 files changed, 1027 insertions(+), 410 deletions(-) create mode 100644 internal/api/tenant.go create mode 100644 internal/api/tenant_helpers_test.go create mode 100644 internal/api/tenant_test.go create mode 100644 internal/settings/registry.go create mode 100644 internal/settings/registry_test.go create mode 100644 internal/stream/tenant_test.go create mode 100644 internal/tenant/tenant.go create mode 100644 internal/tenant/tenant_test.go diff --git a/AGENTS.md b/AGENTS.md index cc888039..85a39dc5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ One binary: - **`cmd/wavehouse/`** — Standalone mode (all-in-one with embedded NATS, optional Pebble dedup): argv dispatch, the logger, `config.Load`, and the signal context; everything else is `internal/app` -Seventeen internal packages under `internal/` (plus `internal/testutil/` for shared test helpers): +Eighteen internal packages under `internal/` (plus `internal/testutil/` for shared test helpers): - **`api/`** — Chi HTTP router, JWT/JWKS middleware (from `auth/`), ingest/query/structured-query/SSE/schema/DLQ/pipes handlers - **`app/`** — the process wiring: `New` builds every component from the boot config and the settings directory (each one wired in one place — what it opens, what it loops, what it releases — with the settings store handed to its wiring function whole, the injection point the per-tenant registry of #583 lands on), `Run` drives the long-lived ones under one `errgroup` until the context is cancelled or one fails, `Close` releases them in reverse order. `cmd/wavehouse` and `tests/integration` both boot through it @@ -43,8 +43,9 @@ Seventeen internal packages under `internal/` (plus `internal/testutil/` for sha - **`pipes/`** — Named query pipes: `NamedQuery` type + `BindParams` + `Source` (read per request; `settings.Store` in production, `Static(q...)` in tests) - **`policy/`** — Hasura-style access control, **role-first**: `TablePolicy` is `map[string]RolePermissions`, and a role's grant splits by operation into `SelectPermissions` (columns, row `filter`, aggregations, the `max_*` limits) and `InsertPermissions` (columns, `check`) — so a field only one side honors does not exist on the other. `Evaluate()` resolves ONE operation and leaves the other side **nil** (`Select *ResolvedSelect` / `Insert *ResolvedInsert`), which every accessor fails closed on — nil is "not resolved", distinct from an empty side, which is "unrestricted" (what the admin return builds). Claim templating (`{{ jwt.claim.path }}`) resolves during that call. Policies come from `Source`, a `func() *Policy` read per call (`settings.Store.Policy` in production, `Static(p)` in tests) - **`query/`** — Structured query AST types + SQL builder with schema validation, structural policy predicate/limit emission, timestamp bucketing -- **`settings/`** — the settings directory: `Validate` (strict JSON, per-file rules, cross-file role references), `Store` (the adopted snapshot + serialized `Reload`, typed accessors read per call, `AfterAdopt` hooks), the fsnotify `Watch`, and the `go:embed`ded seed `wavehouse bootstrap` writes +- **`settings/`** — the settings directory: `Validate` (strict JSON, per-file rules, cross-file role references), `Store` (the adopted snapshot + serialized `Reload`, typed accessors read per call, `AfterAdopt` hooks), `Registry` (tenant id → `Store`; holds the one store under `tenant.Default`), the fsnotify `Watch`, and the `go:embed`ded seed `wavehouse bootstrap` writes - **`stream/`** — SSE fan-out: rows travel POSITIONALLY, so each connection is told its projected column list in an `event: schema` frame before its first row and again on drift — **not** guaranteed after a gap-fill across a column change, which can leave a connection reading live rows against a stale list until it reconnects ([#543](https://github.com/Wave-RF/WaveHouse/issues/543)) — (tracked per connection; replay tracks its own). The event `Hub` (registers subscribers by `(topic, role)`; `Broadcast` projects + serializes each event once per role, the #294 delivery hot path — a role carrying a row-level `filter` keeps the shared projection but delivers per subscriber, each subscriber's claims evaluated against the row, #319), `Subscriber` (per-connection outbound `Frame` queue, `Send`/`Frames`; claims fixed at construction, immutable), the `Bucket` fan-out set (`subscriberSet`, one per `(topic, role)`), the `Heartbeater` keepalive wheel, and `Metrics` (the `wavehouse_sse_*` stream instruments) +- **`tenant/`** — the tenant identifier ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)): `ID` (a validated string), `Parse` (letters, digits, `_`, `-`; ≤ 64 bytes — safe as a folder name and as an MQ subject token), `Default` (`"0"`), and `Header` (`X-Tenant-ID`). Imports nothing from the rest of the repo. `api.TenantMW` resolves the header against `settings.Registry` before auth on every `/v1` route outside `/v1/ops/*` (`400` malformed, `404` unknown) and puts the resolved `*settings.Store` in the request context; handlers read it once (`api.StoreFromContext`) and pass it down as an argument, and nothing below a handler reads context. The async paths (ingest worker, sweeper, stream hub, schema registry) are constructed with a `tenant.ID` and their getters take it ## Key Design Decisions @@ -440,6 +441,7 @@ internal/policy/ → Access control policies (types, evaluation, Source) internal/query/ → Structured query AST + SQL builder internal/settings/ → Settings directory (validate, adopted snapshot + reload, watcher, embedded seed) internal/stream/ → SSE fan-out (event Hub: project once per role, Subscriber outbound queue, Bucket fan-out, keepalive Heartbeater wheel) +internal/tenant/ → Tenant id (type, grammar, reserved default, request header name) internal/testutil/ → Shared test helpers (NopLogger, etc.) tests/ → Integration & E2E tests tests/integration/ → Go integration tests (//go:build integration; ClickHouse testcontainer) diff --git a/CHANGELOG.md b/CHANGELOG.md index 80729a8f..33909991 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added +- **Requests resolve to a tenant before authentication, and the tenant is threaded through every settings read** (`internal/tenant/` (new, + tests), `internal/settings/registry.go` (new, + tests), `internal/api/tenant.go` (new, + tests), `internal/api/{router,ingest,structured_query,pipes}.go`, `internal/ingest/{worker,sweeper}.go`, `internal/stream/hub.go`, `internal/discovery/discovery.go`, `internal/app/{app,wire}.go`): story 1 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)), with no behavior change for a deployment that sends no tenant header. `internal/tenant` defines the id — a validated string (letters, digits, `_`, `-`; at most 64 bytes, so it is safe as a folder name and as an MQ subject token), the reserved default `0`, and the `X-Tenant-ID` header name — and imports nothing from the rest of the repository. `api.TenantMW` runs ahead of the auth middleware on every `/v1` route outside `/v1/ops/*`: an absent or empty header is tenant `0`, a malformed id or a repeated header is a `400`, a well-formed id the new `settings.Registry` does not hold is a `404`, and the resolved `*settings.Store` rides the request context. The registry holds the one store `settings.Open` adopted, keyed `0`. Handlers read the store once and pass it down as an argument — the ingest, structured-query, and pipe getters (`PolicySource`, `DedupeSettings`, `bucketSecs`, `defaultMaxRows`, the pipes source) now take it as a parameter — and a tenant route reached without a resolved tenant answers `500` rather than fall back to one. The ingest worker, sweeper, stream hub, and schema registry are constructed with a `tenant.ID` (`tenant.Default` in `internal/app`) and their settings getters take it. The probes, `/version`, the metrics path, and `/v1/ops/*` stay tenant-exempt, with the ops tree behind the auth middleware and the admin gate exactly as before. `X-Tenant-ID` joins the CORS `Access-Control-Allow-Headers` list so a browser client can send it; the SDK needs no change (`options.headers`). + - **Schema discovery captures each table's DDL, its columns' ordinals and default expressions, and the server version** (`internal/discovery/discovery.go`, `internal/testutil/testutil.go`): `Column` gains `DefaultExpression` and `Position` (both from a widened `system.columns` select), `TableSchema` gains `DDL` from `system.tables.create_table_query`, and `SchemaRegistry` gains `ServerVersion()` from a `SELECT version()` probe next to the existing `SELECT timezone()`. Groundwork for the native type layer, captured on the same refresh as the columns so a stale version cannot outlive the schemas it describes. That is a publication guarantee, not a same-server one: `chconn.Manager` resolves the connection per call, so a reload changing `clickhouse.addr` mid-refresh can still pair a version from one server with schemas from another — narrow, and self-correcting on the next refresh. `DDL` is `json:"-"` and does **not** appear in `/v1/ops/schema`: that endpoint marshals `TableSchema` straight to the client, and an external-engine table (S3, MySQL, PostgreSQL, Kafka) renders its wiring there unconditionally — endpoint, bucket or host, database, username, S3 access key id. ClickHouse masks the password itself as `[HIDDEN]` from ~23.9 (verified on 26.7.3), so the exposure is the topology rather than the secret — except on an older server, or one with `display_secrets_in_show_and_select` enabled. `position` and `default_expression` are additive fields in the response. A table listed in `system.tables` with no `system.columns` rows is skipped rather than published column-less, and both new queries fail the refresh on error exactly as `timezone()` and `system.columns` do — callers keep the prior cache and retry. - **Settings-directory hot reload — boot loading, three reload triggers, and the config-key migration** (`internal/settings/` (new: `store.go`, `watch.go`, + tests), `internal/api/settings.go` (new, + tests), `internal/api/{router,ingest,structured_query}.go`, `internal/discovery/discovery.go`, `internal/config/config.go`, `cmd/wavehouse/main.go`, `config.yaml`, `deployments/compose/standalone.yaml`, `docs/src/content/docs/settings-directory.mdx` (new — the hot-reloadable half of configuration gets its own page; `configuration.mdx` is boot config only); closes the loop [#500](https://github.com/Wave-RF/WaveHouse/pull/500) opened, tracked by [#48](https://github.com/Wave-RF/WaveHouse/issues/48)): the server now *consumes* the settings directory instead of only validating it. `settings.Store` owns the adopted snapshot: `settings.dir` / `WH_SETTINGS_DIR` is now **required**, boot validates and adopts the directory (missing or invalid refuses to start); a running instance then re-validates and re-adopts on any of three triggers — a **directory watch** (fsnotify on the directory, not the files, so atomic-writer replaces and Kubernetes ConfigMap symlink swaps aren't lost; bursts debounce into one reload), **`SIGHUP`**, and **`POST /v1/ops/settings/reload`** (admin-gated; returns `{"adopted", "findings"}`, `200` adopted / `422` rejected) — all funneling through one serialized reload path. A reload that fails validation keeps the previous good snapshot (an operator mid-edit degrades to a log line, never a broken server); warnings don't block adoption, matching `wavehouse validate`. The tenant tunables **migrate out of boot config** into the directory's `config.json`: `dedupe.id_field` / `dedupe.require_id` (now with the per-table overrides under `dedupe.tables` that [#222](https://github.com/Wave-RF/WaveHouse/issues/222) asked for, resolved per record through the table → global cascade in one atomic snapshot read, so a reload lands at a record boundary and never mixes documents within one record), `query.default_max_rows` and `query.timestamp_bucket_seconds` (read per query), `schema.refresh_interval` (re-read after each tick, so a change applies from the next cycle), `stream.keepalive_interval` / `stream.keepalive_buckets` (a reload calls the new `Heartbeater.Reconfigure`, which rebuilds the keepalive wheel in place with every live subscriber carried over and re-times the running ticker) and `stream.gap_window_minutes` (the sweeper re-reads it every sweep), `mq.max_bytes_gb` (an after-adopt hook updates the `WAVEHOUSE` and `WAVEHOUSE_DLQ` stream limits in place via `EmbeddedNATS.Resize` — shrinking below the buffered size backpressures until the worker drains, nothing is dropped), `dlq.enabled` with per-table overrides under `dlq.tables` (resolved by the ingest worker at the moment a poison row is isolated: on → park it on `WAVEHOUSE_DLQ` and ack; off → leave it unacked for redelivery, never dropped; the DLQ stream and `GET /v1/ops/dlq/stats` now always exist, so the switch is purely behavioral), the **ClickHouse wiring** (`clickhouse.addr` / `http_port` / `http_scheme` / `database` / `username` / `query_timeout`: the new `chconn.Manager` is the one `driver.Conn` every consumer holds and swaps the connection behind it on reload — unconditionally, since the adopted settings are the authority and reachability already surfaces through schema discovery and `/readyz`; the replaced one closes after a `query_timeout` grace; the ingest worker, raw-SQL proxy, and schema registry read the HTTP target, timeout, and database per call), the **auth verifier wiring** (`auth.jwks_url` / `auth.role_claim`: the new `auth.Authenticator` swaps a whole verifier — key source plus its pinned algorithm allowlist — atomically per reload, unconditionally, so an unreachable JWKS fails closed until it can be fetched; `auth.Middleware` is gone — `Authenticator` is the one constructor), and the CORS allowlist (`cors.allowed_origins`, resolved per request). The corresponding YAML/env keys are **removed**: `server.cors_allowed_origins`, `query.default_max_rows`, `schema.refresh_interval`, `dedupe.enabled`, `dedupe.id_field`, `dedupe.require_id`, `stream.keepalive_interval`, `stream.keepalive_buckets`, `mq.gap_window_minutes`, `cache.timestamp_bucket_seconds`, `mq.max_bytes_gb`, `dlq.enabled`, `clickhouse.addr`, `clickhouse.http_port`, `clickhouse.http_scheme`, `clickhouse.database`, `clickhouse.username`, `clickhouse.query_timeout`, `auth.jwks_url`, `auth.role_claim` (and `WH_SERVER_CORS_ALLOWED_ORIGINS`, `WH_QUERY_DEFAULT_MAX_ROWS`, `WH_SCHEMA_REFRESH_INTERVAL`, `WH_DEDUPE_ENABLED`, `WH_DEDUPE_ID_FIELD`, `WH_DEDUPE_REQUIRE_ID`, `WH_STREAM_KEEPALIVE_INTERVAL`, `WH_STREAM_KEEPALIVE_BUCKETS`, `WH_MQ_GAP_WINDOW_MINUTES`, `WH_CACHE_TIMESTAMP_BUCKET_SECONDS`, `WH_MQ_MAX_BYTES_GB`, `WH_DLQ_ENABLED`, `WH_CH_ADDR`, `WH_CH_HTTP_PORT`, `WH_CH_HTTP_SCHEME`, `WH_CH_DATABASE`, `WH_CH_USERNAME`, `WH_CH_QUERY_TIMEOUT`, `WH_AUTH_JWKS_URL`, `WH_AUTH_ROLE_CLAIM`); the secrets — `clickhouse.password`, `auth.jwt_secret`, `auth.operator_key` — stay boot config on purpose (never in a tracked JSON file; combined with the adopted wiring on every reconnect, rotating one is a restart), and boot config is now **strict**: `config.Load` re-reads the YAML against the struct's tags and refuses to start naming every undeclared key, so a `dlq:` or `clickhouse: addr:` left behind can't be read, ignored, and believed; the binary carries **no compiled defaults** — every `config.json` key is required (validation names each missing one), so the adopted snapshot is what the files say, and once adopted it outlives its files (a deleted file or vanished directory is just a rejected reload). Defaults live in one checked-in seed directory (`internal/settings/seed/`, `go:embed`ded): the new **`wavehouse bootstrap [dir]`** writes it (refusing a non-empty directory, the `initdb` contract; the directory resolves exactly as it does for `validate` — the argument, else `WH_SETTINGS_DIR`, usage error with neither — so the two commands are interchangeable on one path and a bare `bootstrap` inside the container images seeds `/app/settings`), the dev `config.yaml` points at a gitignored `./settings` that `make dev` seeds from it, and the e2e fixture ships a copy. The container images ship **no** settings directory: `WH_SETTINGS_DIR` is preset to `/app/settings`, the operator mounts a directory there (`standalone.yaml` bind-mounts the checked-in `deployments/compose/settings/`), and a missing mount refuses to boot rather than running on defaults nobody chose. `dedupe.enabled` moves too: the new `dedupe.Managed` wraps the Pebble store and a `Store.AfterAdopt` hook opens or closes it after every adoption, so flipping the switch is a reload, not a restart (seen ids persist across an off/on cycle; a failed open on reload is logged and ingest fails closed with `500` until the next reload, since the files asked for dedupe — at boot it still refuses to start; a record caught in the instant of the flip is published un-deduped and counted by `wavehouse_ingest_dedupe_disabled_total` rather than failed, and the hook is registered before the boot apply so a reload can never leave the settings and the store out of step). The watcher reloads once as soon as its watch exists, closing the gap between the boot read and the watch — an edit landing in between (a ConfigMap update during a rolling restart) is adopted, not silently missed. `dedupe.enabled` / `WH_DEDUPE_ENABLED` are removed from boot config alongside the other keys. What stays in boot config is only what cannot change under a running process — resource sizing (`data_dir`, `cache.l1_max_cost`), the listeners, the observability exporters — and the secrets. The compose stack now bind-mounts a checked-in `deployments/compose/settings/` (the seed with `clickhouse.addr` pointed at the `clickhouse` service) instead of a volume seeded with `bootstrap`, so the quickstart is `up -d` again; the e2e orchestrator copies the fixture settings per run and patches the testcontainer's ClickHouse ports into `config.json`, since that wiring no longer has an env override. Every after-adopt hook (dedupe, keepalive wheel) is registered before the reload triggers start, so the watcher's first reload can never be missed by a hook. Consumers take functions, not values (`IngestHandler.DedupeSettings`, the structured-query handler's `defaultMaxRows` / `bucketSecs func() int`, the ingest worker's `dlqEnabled func(table) bool`, the sweeper's `gapWindow func() time.Duration`, `corsMiddleware`'s origins getter, `SchemaRegistry`'s database and refresh-interval sources, the query handlers' timeout sources), so `internal/api` stays testable without materializing settings directories. The settings directory is also the **runtime authority for access control and named pipes** (`internal/settings/store.go`, `internal/policy/source.go` (new), `internal/pipes/pipes.go`, `internal/api/{policy,pipes,router}.go`, `internal/stream/hub.go`, `internal/auth/auth.go`, `cmd/wavehouse/main.go`, `Makefile`, `deployments/compose/settings/{policies,roles}.json`, `clients/ts/src/settings.ts` (new); closes [#229](https://github.com/Wave-RF/WaveHouse/issues/229), [#33](https://github.com/Wave-RF/WaveHouse/issues/33), [#461](https://github.com/Wave-RF/WaveHouse/issues/461), [#514](https://github.com/Wave-RF/WaveHouse/issues/514), [#460](https://github.com/Wave-RF/WaveHouse/issues/460), [#363](https://github.com/Wave-RF/WaveHouse/issues/363); advances [#48](https://github.com/Wave-RF/WaveHouse/issues/48) and [#214](https://github.com/Wave-RF/WaveHouse/issues/214)): `roles.json`, `policies.json`, and `pipes.json` are adopted with `config.json` as one snapshot and re-adopted on the same three triggers, and **files are the only write path** — standalone, the operator edits them on the host; on WaveHouse Cloud the control plane writes them — so there is no stored copy that can skip validation: every adoption runs the current rules (strict decode rejecting unknown and duplicate keys, the full policy validation including the claim-template grammar, pipe name/SQL/parameter-type rules, and the cross-file check that every role a grant or `allowed_roles` names is declared in `roles.json`), and a rejected edit keeps the previous good policy and pipes in effect. `policies.json` is one policy document (`{}` = no policy, adopted fail-closed with a warning); `pipes.json` carries full definitions (`allowed_roles`, `parameters`, `description`), so a file-defined pipe is no longer admin-only by construction. Consumers read the adopted snapshot per request through `policy.Source` (a `func() *policy.Policy`; `settings.Store.Policy` in production, `policy.Static(p)` in tests) and `pipes.Source` (`settings.Store`; `pipes.Static(q...)` in tests), so a reload applies to the very next request, including the SSE hub's per-event policy read. `GET /v1/ops/policy`, `POST /v1/ops/policy/validate`, `GET /v1/ops/pipes`, `GET /v1/ops/pipes/{name}`, and pipe execution are unchanged; the operator key still passes the `/v1/ops/*` gate under no policy, now as the break-glass that inspects the policy and triggers `POST /v1/ops/settings/reload` after `policies.json` is fixed. The SDK gains `wh.settings.reload()` (`POST /v1/ops/settings/reload`, returning `{ adopted, findings }`). The compose stack's trial `public` policy moves into the bind-mounted `deployments/compose/settings/policies.json` + `roles.json`, and `make dev` copies the same two files into its seeded `./settings` so a fresh dev server works tokenless. **Removed** — the write endpoints `PUT /v1/ops/policy`, `PUT /v1/ops/pipes/{name}`, and `DELETE /v1/ops/pipes/{name}`; the NATS KV buckets `WAVEHOUSE_POLICY` and `WAVEHOUSE_PIPES` and their KV Watch sync (`internal/policy/store.go`, the pipes KV store); the boot-config keys `policy.file_path` / `WH_POLICY_FILE_PATH` and `pipes.dir` / `WH_PIPES_DIR` (a leftover `policy:` or `pipes:` YAML block now refuses boot by name, like the other moved keys) and the `.sql`-directory pipes bootstrap; `deployments/compose/dev-policy.yaml`; the SDK methods `wh.policy.set`, `wh.pipes.set`, and `wh.pipes.delete`; and the test helpers `policy.NewMemoryStore`, `pipes.NewMemoryStore`, and `testutil/natsjs.go`. diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index 369e4365..ab54f255 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -50,6 +50,29 @@ WaveHouse extracts the role from a configurable JWT claim path (`auth.role_claim Policies support Hasura-style row-level and column-level permissions with JWT claim templating (e.g., `{{ jwt.app_metadata.tenant_id }}`). +## Tenant Selection + +Every `/v1` route outside `/v1/ops/*` resolves a tenant before it authenticates the request. The tenant comes from the `X-Tenant-ID` request header: + +```text +X-Tenant-ID: 0 +``` + +A request without the header, or with an empty one, resolves to tenant `0`, the default tenant, whose settings are the [settings directory](/settings-directory). Setting the header on every request is the client's or the fronting proxy's job; WaveHouse never derives it from the token. + +A tenant id is 1–64 characters of ASCII letters, digits, `_`, and `-`. It is a string, not a number, so a long numeric id keeps every digit. + +| Status | Body | When | +| ------ | ---- | ---- | +| `400` | `{"error": "invalid X-Tenant-ID: …"}` | The id breaks the grammar above, or the header was sent more than once | +| `404` | `{"error": "unknown tenant: "}` | The id is well formed but no such tenant exists | + +Both are decided before authentication, so they are returned whatever token the request carries. + +The probes (`/livez`, `/readyz`, `/healthz`), `/version`, the Prometheus metrics path, and `/v1/ops/*` are tenant-exempt: they ignore the header entirely. + +`X-Tenant-ID` is in the CORS `Access-Control-Allow-Headers` list, so a browser client can send it cross-origin. The SDK sends it through [`options.headers`](/sdk#custom-headers). + ## Response Format ### Error Responses diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 8e73a691..9c309bcb 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -67,7 +67,8 @@ internal/ ├── policy/ Hasura-style access control (policy types, evaluation, Source) ├── query/ Structured query AST, SQL builder, and timestamp bucketing ├── settings/ Settings directory: validate the JSON files, hold the adopted snapshot, reload on watch / SIGHUP / API -└── stream/ SSE fan-out: event Hub (project once per role), Subscriber queue, Bucket fan-out, keepalive Heartbeater wheel +├── stream/ SSE fan-out: event Hub (project once per role), Subscriber queue, Bucket fan-out, keepalive Heartbeater wheel +└── tenant/ Tenant id: the type, its grammar, the reserved default, the request header name ``` ### `api/` — HTTP Layer @@ -76,6 +77,7 @@ The API layer uses [Chi](https://github.com/go-chi/chi) for routing with Request - **router.go** — Route definitions. Public: `/livez`, `/readyz`, and the content-free `/v1/health` SDK ping (plus the permanent `/healthz` alias and the deprecated `/health`, `/ready` aliases). Policy-gated: `/v1/ingest?table={table}`, `/v1/query?table={table}` (structured), `/v1/pipes/{name}` (named pipes), `/v1/stream`. Admin-only (`RequireAdmin` — role == `policy.admin_role`, or a request bearing the operator key's operator bit, which passes even under a nil policy): `/v1/ops/schema/*`, `/v1/ops/dlq/stats`, `GET /v1/ops/pipes[/{name}]`, `/v1/ops/settings/reload`, `/v1/ops/query` (raw SQL — same gate as the rest of `/v1/ops/*`). - **auth middleware** — the JWT/JWKS authentication middleware is its own package, [`auth/`](#auth--authentication); the router runs it on every `/v1/*` route. +- **tenant.go** — `TenantMW` resolves the request's tenant ahead of the auth middleware on every `/v1` route outside `/v1/ops/*`: the [`X-Tenant-ID`](/api#tenant-selection) header (absent means `tenant.Default`), validated by `tenant.Parse` (`400`), looked up in the `settings.Registry` (`404` on a miss), and the resolved `*settings.Store` stored in the request context (`WithStore` / `StoreFromContext` — here rather than in `tenant/`, because `settings` names `tenant.ID`). A handler reads the store once and passes it down as an argument — the per-tenant getters it holds take it as a parameter (`(*settings.Store).Policy`, `.DedupeFor`, `.DefaultMaxRows`, … in production) — and nothing below a handler reads the context; a tenant route reached without a resolved store answers `500` rather than fall back to a tenant. The probes, `/version`, the metrics path, and `/v1/ops/*` are tenant-exempt; the ops pipe reads serve the default tenant. - **pipes.go** — Named query pipe handlers: admin listing (`GET /v1/ops/pipes[/{name}]`, read per request from its `pipes.Source`) and execution with parameter binding. `pipes.json` is the only write path. - **structured_query.go** — Handler for `POST /v1/query?table={table}`: validates query AST, enforces permissions, builds and executes SQL. - **ingest.go** — Accepts `POST /v1/ingest?table={table}` in three body shapes: one flat JSON object, a JSON array of them, or NDJSON. The **required** `Content-Type` chooses the format *family* — `application/json` versus the four NDJSON spellings — and within the JSON family the body's first non-whitespace byte picks array versus single object; the bytes never choose the family. Anything that is not exactly one readable media type is a `415`, decided before the body is read: the header is parsed per RFC 9110 §8.3, and because `Content-Type` is a singleton field, repeated header lines must all resolve to the same format and a value carrying a comma is refused unless the value as a whole parses as one media type — a comma inside a *quoted* parameter value is data, so `application/json; a=", application/x-ndjson; b="` is accepted. It then reads the whole (`MaxBytesReader`-capped) body into a pooled buffer and runs the per-format record readers over those bytes, so the `413` lands before any record is processed and peak memory per request is O(body) rather than O(record). Then it validates each record against the discovered schema, optional dedup, and publishes each row through `mq.Publisher` on `mq.Topic{Table, Scope}` (raw names — the subject it becomes is `internal/mq`'s; a full queue comes back as `mq.ErrQueueFull`, which is the `503` + `Retry-After`). When dedup is on, a row missing the configured `id_field` can't be deduped: it is logged at `WARN` and counted by `wavehouse_ingest_dedupe_missing_id_total` (labeled by `table`), then published un-deduped — or rejected when `dedupe.require_id` is set ([#219](https://github.com/Wave-RF/WaveHouse/issues/219)). @@ -183,6 +185,12 @@ The hot-reloadable half of configuration: a directory of four JSON files (`confi - **watch.go** — fsnotify on the *directory* (not the files, so atomic-writer replaces and Kubernetes ConfigMap symlink swaps aren't lost), debounced into one reload; reloads once as soon as the watch exists so an edit between the boot read and the watch is never missed. `SIGHUP` and the reload endpoint funnel through the same serialized `Reload`. - **seed.go** / **seed/** — The `go:embed`ded starter directory with every key at its default. The binary carries no compiled defaults: `wavehouse bootstrap [dir]` writes this seed, and the compose stack and e2e fixture ship copies of it. +- **registry.go** — `Registry` maps a tenant id to its `Store` (`For(id)`). It holds the one store `Open` adopted, under `tenant.Default`; reload and the watcher stay on the `Store`. + +### `tenant/` — Tenant Identifier + +- **tenant.go** — `ID`, a validated string (never a number: a 19-digit id already rounds as a float64), and `Parse`, the one grammar that makes an id safe both as a folder name and as a message-queue subject token: ASCII letters, digits, `_`, `-`, at most `MaxLen` (64) bytes. `Default` (`"0"`) is the tenant a request without the header resolves to; `Header` is `X-Tenant-ID`. The package imports nothing from the rest of the repository, so any package can name a tenant. HTTP handlers receive the tenant as its resolved `*settings.Store`; the asynchronous paths — ingest worker, sweeper, stream hub, schema registry — are constructed with a `tenant.ID` and their settings getters take it as a parameter, which `internal/app` resolves through the registry. + ### `chconn/` — ClickHouse Connection Manager - **chconn.go** — `Manager` is a `driver.Conn` whose backing connection is swapped by `Reconfigure(Params)` after a settings reload changes the ClickHouse wiring (`clickhouse.addr` / `http_port` / `http_scheme` / `database` / `username` / `query_timeout`, combined with the boot-config password). Like `clickhouse.Open` it never dials, so boot tolerates an unreachable ClickHouse (schema discovery retries) and a bad address surfaces where reachability is already handled (`/readyz`, query errors). The replaced connection closes after a `query_timeout` grace so in-flight queries finish. `Target()` / `Database()` / `QueryTimeout()` expose the current wiring for the HTTP-interface consumers (ingest INSERTs, raw-SQL proxy). diff --git a/docs/src/content/docs/sdk/index.mdx b/docs/src/content/docs/sdk/index.mdx index 90dbab9f..4779f204 100644 --- a/docs/src/content/docs/sdk/index.mdx +++ b/docs/src/content/docs/sdk/index.mdx @@ -377,7 +377,7 @@ Your headers are applied *underneath* the SDK's own, and a collision means yours Nothing is ever comma-joined: on a collision the SDK's value stands alone, and two of your own entries differing only in case collapse to the last one — a header joined rather than replaced is how you end up sending `Content-Type: application/json, image/png`. -From a **browser**, a cross-origin custom header must also survive CORS preflight, and WaveHouse allow-lists a fixed set (`Accept`, `Authorization`, `Content-Type`, `Last-Event-ID`, `X-Request-ID`) with no config knob. So custom headers work server-side, or from a browser when the proxy in front owns CORS — which is the same proxy the header is usually for. +From a **browser**, a cross-origin custom header must also survive CORS preflight, and WaveHouse allow-lists a fixed set (`Accept`, `Authorization`, `Content-Type`, `Last-Event-ID`, `X-Request-ID`, `X-Tenant-ID`) with no config knob. So any other custom header works server-side, or from a browser when the proxy in front owns CORS — which is the same proxy the header is usually for. [`X-Tenant-ID`](/api#tenant-selection) is on the list, so a browser can select a tenant with `headers: { "X-Tenant-ID": "…" }` directly. Headers are static. For a credential that rotates per request, wrap the transport with [`options.fetch`](#supplying-your-own-fetch); a callback form is tracked in [#459](https://github.com/Wave-RF/WaveHouse/issues/459). diff --git a/internal/api/boot_chain_test.go b/internal/api/boot_chain_test.go index ea0c6c6b..088557b4 100644 --- a/internal/api/boot_chain_test.go +++ b/internal/api/boot_chain_test.go @@ -18,6 +18,7 @@ import ( "github.com/stretchr/testify/require" "github.com/Wave-RF/WaveHouse/internal/discovery" + "github.com/Wave-RF/WaveHouse/internal/tenant" "github.com/Wave-RF/WaveHouse/internal/testutil" ) @@ -89,7 +90,7 @@ func TestBoot_Chain_DegradedThenRecovers(t *testing.T) { conn := &errsThenSuccessConn{errs: []error{connRefused, connRefused, dbMissing}} logger := slog.New(slog.NewJSONHandler(io.Discard, nil)) - registry := discovery.NewSchemaRegistry(conn, func() string { return "test" }, func() time.Duration { return time.Hour }, logger) + registry := discovery.NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, logger) // Phase 0 — synchronous boot Refresh fails. internal/app records the // diagnostic in BootState and proceeds with the retry loop in a diff --git a/internal/api/errors_test.go b/internal/api/errors_test.go index 03875c79..f1e23f24 100644 --- a/internal/api/errors_test.go +++ b/internal/api/errors_test.go @@ -14,6 +14,7 @@ import ( "github.com/Wave-RF/WaveHouse/internal/pipes" "github.com/Wave-RF/WaveHouse/internal/policy" "github.com/Wave-RF/WaveHouse/internal/stream" + "github.com/Wave-RF/WaveHouse/internal/tenant" "github.com/Wave-RF/WaveHouse/internal/testutil" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" @@ -133,15 +134,15 @@ func TestRequireAdmin_InvalidTokenDenialLogsFailLoudReason(t *testing.T) { func TestPipesHandler_Execute_DenialLogsAllowedRoles(t *testing.T) { t.Parallel() logger, buf := warnBufLogger() - store := pipes.Static( + store := staticPipes( &pipes.NamedQuery{Name: "report", SQL: "SELECT * FROM clicks", AllowedRoles: []string{"analyst", "viewer"}}, ) - h := NewPipesHandler(store, policy.Static(&policy.Policy{}), nil, nil, noTimeout, logger) + h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout, logger) r := pipesRequest(t, http.MethodPost, "/v1/pipes/report/execute", "report", nil) r = r.WithContext(auth.WithRole(r.Context(), "guest")) w := httptest.NewRecorder() - h.Execute(w, r) + h.Execute(w, withTenant(r)) require.Equal(t, http.StatusForbidden, w.Code) out := buf.String() @@ -162,7 +163,7 @@ func TestIngest_DenialLogsPolicyGate(t *testing.T) { t.Parallel() logger, buf := warnBufLogger() h := NewIngestHandler(testRegistry(t), &testutil.MockPublisher{}, logger) - h.PolicySource = policy.Static(&policy.Policy{ + h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": {"viewer": {Select: &policy.SelectPermissions{}}}, // no insert for viewer }, @@ -171,7 +172,7 @@ func TestIngest_DenialLogsPolicyGate(t *testing.T) { req := ingestRequest(t, "clicks", map[string]any{"page": "/home"}) req = req.WithContext(auth.WithRole(req.Context(), "viewer")) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) require.Equal(t, http.StatusForbidden, w.Code) out := buf.String() @@ -192,9 +193,10 @@ func TestAuthzDenied_LogsChiRoutePattern(t *testing.T) { logger, buf := warnBufLogger() reg := testutil.NewTestSchemaRegistry(t, nil) router := NewRouter(Dependencies{ + Tenants: testTenants(), Ingest: NewIngestHandler(reg, &testutil.MockPublisher{}, logger), Query: &QueryHandler{}, - SSE: NewStreamHandler(stream.NewHub(nil, nil, nil), nil), + SSE: NewStreamHandler(stream.NewHub(tenant.Default, nil, nil, nil), nil), Health: &HealthHandler{}, Schema: NewSchemaHandler(reg), AuthMW: func(next http.Handler) http.Handler { return next }, diff --git a/internal/api/ingest.go b/internal/api/ingest.go index ee25d9c3..56f64b0e 100644 --- a/internal/api/ingest.go +++ b/internal/api/ingest.go @@ -18,6 +18,7 @@ import ( "github.com/Wave-RF/WaveHouse/internal/ingest" "github.com/Wave-RF/WaveHouse/internal/mq" "github.com/Wave-RF/WaveHouse/internal/policy" + "github.com/Wave-RF/WaveHouse/internal/settings" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/attribute" @@ -39,12 +40,13 @@ type IngestHandler struct { Registry *discovery.SchemaRegistry Dedup dedupe.Deduplicator // nil when no dedupe store is wired (tests) // DedupeSettings resolves the effective dedupe id_field/require_id for a - // table (settings.Store.DedupeFor in production). Called once per record so - // a settings reload lands at a record boundary — one record never mixes two - // documents' values. Dedup is skipped when nil. - DedupeSettings func(table string) (enabled bool, idField string, requireID bool) + // table of the request's tenant ((*settings.Store).DedupeFor in + // production). Called once per record so a settings reload lands at a + // record boundary — one record never mixes two documents' values. Dedup is + // skipped when nil. + DedupeSettings func(store *settings.Store, table string) (enabled bool, idField string, requireID bool) Publisher mq.Publisher - PolicySource policy.Source + PolicySource PolicySource logger *slog.Logger // Validator and Checker are the per-record seams a native type layer will @@ -134,6 +136,10 @@ type requestAbort struct { } func (h *IngestHandler) Handle(w http.ResponseWriter, r *http.Request) { + store, ok := requestStore(w, r) + if !ok { + return + } now := time.Now().UTC() table := r.URL.Query().Get("table") @@ -174,7 +180,7 @@ func (h *IngestHandler) Handle(w http.ResponseWriter, r *http.Request) { var role string if h.PolicySource != nil { - p := h.PolicySource() + p := h.PolicySource(store) role = policy.ResolveRole(p, auth.RoleFromContext(ctx)) claims, _ := auth.ClaimsFromContext(ctx) perms = policy.Evaluate(p, role, table, "insert", claims) @@ -265,10 +271,10 @@ func (h *IngestHandler) Handle(w http.ResponseWriter, r *http.Request) { } if batch { - h.handleBatch(ctx, w, rr, reqCap, table, scope, schema, perms, role, now, h.policyCheckGuard(ctx, table, role, schema, perms)) + h.handleBatch(ctx, w, rr, reqCap, store, table, scope, schema, perms, role, now, h.policyCheckGuard(ctx, table, role, schema, perms)) return } - h.handleSingle(ctx, w, rr, reqCap, table, scope, schema, perms, role, now, h.policyCheckGuard(ctx, table, role, schema, perms)) + h.handleSingle(ctx, w, rr, reqCap, store, table, scope, schema, perms, role, now, h.policyCheckGuard(ctx, table, role, schema, perms)) } // handleSingle ingests a lone flat JSON object and preserves the GA response @@ -279,6 +285,7 @@ func (h *IngestHandler) handleSingle( w http.ResponseWriter, rr recordReader, reqCap int64, + store *settings.Store, table, scope string, schema *discovery.TableSchema, perms *policy.ResolvedPermissions, @@ -299,7 +306,7 @@ func (h *IngestHandler) handleSingle( return } - dup, reject, abort := h.processRecord(ctx, table, scope, schema, perms, role, data, now, checkGuard) + dup, reject, abort := h.processRecord(ctx, store, table, scope, schema, perms, role, data, now, checkGuard) if abort != nil { writeAbort(w, abort) return @@ -331,6 +338,7 @@ func (h *IngestHandler) handleBatch( w http.ResponseWriter, rr recordReader, reqCap int64, + store *settings.Store, table, scope string, schema *discovery.TableSchema, perms *policy.ResolvedPermissions, @@ -369,7 +377,7 @@ func (h *IngestHandler) handleBatch( result.Total++ idx := result.Total - dup, reject, abort := h.processRecord(ctx, table, scope, schema, perms, role, data, now, checkGuard) + dup, reject, abort := h.processRecord(ctx, store, table, scope, schema, perms, role, data, now, checkGuard) if abort != nil { // Whole-request failure: surface the status rather than recording a // request-scoped condition as per-record loss (see requestAbort). @@ -513,6 +521,7 @@ func (h *IngestHandler) policyCheckGuard( // - abort non-nil: a whole-request failure; the caller stops and returns it. func (h *IngestHandler) processRecord( ctx context.Context, + store *settings.Store, table, scope string, schema *discovery.TableSchema, perms *policy.ResolvedPermissions, @@ -628,7 +637,7 @@ func (h *IngestHandler) processRecord( // lands at a record boundary. A Deduplicator without a settings source is // a wiring bug, not a mode — main wires both or neither. if h.Dedup != nil && h.DedupeSettings != nil { - if enabled, idField, requireID := h.DedupeSettings(table); enabled { + if enabled, idField, requireID := h.DedupeSettings(store, table); enabled { idVal, ok := data[idField] if !ok { dedupeMissingIDCounter.Add(ctx, 1, metric.WithAttributes(attribute.String("table", table))) diff --git a/internal/api/ingest_seams_test.go b/internal/api/ingest_seams_test.go index 683e9ff6..f365335a 100644 --- a/internal/api/ingest_seams_test.go +++ b/internal/api/ingest_seams_test.go @@ -52,7 +52,7 @@ func TestIngest_RecordValidatorSeam_IsUsed(t *testing.T) { h.Validator = v w := httptest.NewRecorder() - h.Handle(w, ingestRequest(t, "clicks", map[string]any{"page": "/home"})) + h.Handle(w, withTenant(ingestRequest(t, "clicks", map[string]any{"page": "/home"}))) assert.Equal(t, http.StatusBadRequest, w.Code) testutil.AssertJSONErrorResponse(t, w) @@ -70,7 +70,7 @@ func TestIngest_RecordValidatorSeam_IsUsed(t *testing.T) { h.Validator = v w := httptest.NewRecorder() - h.Handle(w, ingestRequest(t, "clicks", map[string]any{"page": "/home"})) + h.Handle(w, withTenant(ingestRequest(t, "clicks", map[string]any{"page": "/home"}))) require.Equal(t, http.StatusOK, w.Code, "body=%s", w.Body.String()) assert.Equal(t, 1, v.validated) @@ -90,7 +90,7 @@ func TestIngest_DefaultValidator_WhenUnwired(t *testing.T) { assert.IsType(t, discoveryValidator{}, h.validator()) w := httptest.NewRecorder() - h.Handle(w, ingestRequest(t, "clicks", map[string]any{"nonexistent_field": 1})) + h.Handle(w, withTenant(ingestRequest(t, "clicks", map[string]any{"nonexistent_field": 1}))) assert.Equal(t, http.StatusBadRequest, w.Code) testutil.AssertJSONErrorResponse(t, w) assert.Empty(t, pub.Messages) @@ -130,7 +130,7 @@ func TestIngest_InsertCheckerSeam_IsUsed(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) - h.PolicySource = policy.Static(p) + h.PolicySource = staticPolicy(p) h.Checker = alwaysChecker{matches: tt.matches} value := "org-something-else" @@ -138,7 +138,7 @@ func TestIngest_InsertCheckerSeam_IsUsed(t *testing.T) { value = required // the default checker would accept this } w := httptest.NewRecorder() - h.Handle(w, viewerIngestRequest(t, "clicks", map[string]any{"page": "/a", "org_id": value})) + h.Handle(w, withTenant(viewerIngestRequest(t, "clicks", map[string]any{"page": "/a", "org_id": value}))) assert.Equal(t, tt.want, w.Code, "body=%s", w.Body.String()) if tt.want == http.StatusForbidden { testutil.AssertJSONErrorResponse(t, w) @@ -178,7 +178,7 @@ func TestIngest_InsertCheckerSeam_InSet_IsUsed(t *testing.T) { req = req.WithContext(ctx) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, tt.want, w.Code, "body=%s", w.Body.String()) if tt.want == http.StatusForbidden { testutil.AssertJSONErrorResponse(t, w) @@ -199,12 +199,12 @@ func TestIngest_DefaultChecker_WhenUnwired(t *testing.T) { }} pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) - h.PolicySource = policy.Static(p) + h.PolicySource = staticPolicy(p) require.Nil(t, h.Checker) assert.IsType(t, canonicalChecker{}, h.checker()) w := httptest.NewRecorder() - h.Handle(w, viewerIngestRequest(t, "clicks", map[string]any{"page": "/a", "org_id": "wrong"})) + h.Handle(w, withTenant(viewerIngestRequest(t, "clicks", map[string]any{"page": "/a", "org_id": "wrong"}))) assert.Equal(t, http.StatusForbidden, w.Code) testutil.AssertJSONErrorResponse(t, w) assert.Empty(t, pub.Messages) @@ -225,12 +225,12 @@ func TestIngest_SeamOrdering_ChecksSitBetweenValidateAndCanonicalize(t *testing. pub := &testutil.MockPublisher{} v := &recordingValidator{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) - h.PolicySource = policy.Static(p) + h.PolicySource = staticPolicy(p) h.Validator = v // A failing check must land AFTER Validate and BEFORE canonicalization. w := httptest.NewRecorder() - h.Handle(w, viewerIngestRequest(t, "clicks", map[string]any{"page": "/a", "org_id": "wrong"})) + h.Handle(w, withTenant(viewerIngestRequest(t, "clicks", map[string]any{"page": "/a", "org_id": "wrong"}))) require.Equal(t, http.StatusForbidden, w.Code) testutil.AssertJSONErrorResponse(t, w) assert.Equal(t, 1, v.validated, "validation runs before the check clauses") diff --git a/internal/api/ingest_test.go b/internal/api/ingest_test.go index 0881735d..8cf57a78 100644 --- a/internal/api/ingest_test.go +++ b/internal/api/ingest_test.go @@ -21,6 +21,7 @@ import ( "github.com/Wave-RF/WaveHouse/internal/ingest" "github.com/Wave-RF/WaveHouse/internal/mq" "github.com/Wave-RF/WaveHouse/internal/policy" + "github.com/Wave-RF/WaveHouse/internal/settings" "github.com/Wave-RF/WaveHouse/internal/testutil" "github.com/golang-jwt/jwt/v5" "github.com/stretchr/testify/assert" @@ -60,7 +61,7 @@ func TestIngest_ValidPayload(t *testing.T) { req := ingestRequest(t, "clicks", map[string]any{"page": "/home", "count": 1}) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) var resp map[string]bool @@ -116,7 +117,7 @@ func TestIngest_MissingTable(t *testing.T) { ) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) // Assertions remain identical for all error cases assert.Equal(t, http.StatusBadRequest, w.Code) @@ -133,7 +134,7 @@ func TestIngest_UnknownTable(t *testing.T) { req := ingestRequest(t, "nonexistent", map[string]any{"x": 1}) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusNotFound, w.Code) assert.Contains(t, w.Body.String(), "unknown table") @@ -148,7 +149,7 @@ func TestIngest_InvalidJSON(t *testing.T) { r := rawIngestRequest(t, "clicks", "application/json", "not json") w := httptest.NewRecorder() - h.Handle(w, r) + h.Handle(w, withTenant(r)) assert.Equal(t, http.StatusBadRequest, w.Code) assert.Contains(t, w.Body.String(), "invalid json") testutil.AssertJSONErrorResponse(t, w) @@ -161,7 +162,7 @@ func TestIngest_SchemaValidation_UnknownField(t *testing.T) { req := ingestRequest(t, "clicks", map[string]any{"page": "/home", "nonexistent_field": 42}) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusBadRequest, w.Code) assert.Contains(t, w.Body.String(), "error") @@ -173,11 +174,11 @@ func TestIngest_Dedup_FirstTime(t *testing.T) { dedup := testutil.NewMockDeduplicator() h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.Dedup = dedup - h.DedupeSettings = func(string) (bool, string, bool) { return true, "event_id", false } + h.DedupeSettings = func(*settings.Store, string) (bool, string, bool) { return true, "event_id", false } req := ingestRequest(t, "clicks", map[string]any{"page": "/home", "event_id": "evt-1"}) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) assert.NotNil(t, pub.LastMessage(), "should have published") @@ -189,18 +190,18 @@ func TestIngest_Dedup_Duplicate(t *testing.T) { dedup := testutil.NewMockDeduplicator() h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.Dedup = dedup - h.DedupeSettings = func(string) (bool, string, bool) { return true, "event_id", false } + h.DedupeSettings = func(*settings.Store, string) (bool, string, bool) { return true, "event_id", false } // First call. req := ingestRequest(t, "clicks", map[string]any{"page": "/home", "event_id": "dup-1"}) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) // Second call — duplicate. req = ingestRequest(t, "clicks", map[string]any{"page": "/home", "event_id": "dup-1"}) w = httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) var resp map[string]bool @@ -217,7 +218,7 @@ func TestIngest_PublishError_503(t *testing.T) { req := ingestRequest(t, "clicks", map[string]any{"page": "/home"}) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusServiceUnavailable, w.Code) assert.Equal(t, "30", w.Header().Get("Retry-After")) @@ -231,7 +232,7 @@ func TestIngest_PublishError_500(t *testing.T) { req := ingestRequest(t, "clicks", map[string]any{"page": "/home"}) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusInternalServerError, w.Code) assert.Contains(t, w.Body.String(), "publish failed") @@ -242,7 +243,7 @@ func TestIngest_Policy_Forbidden(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) - h.PolicySource = policy.Static(&policy.Policy{ + h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": { "viewer": {Select: &policy.SelectPermissions{}}, @@ -256,7 +257,7 @@ func TestIngest_Policy_Forbidden(t *testing.T) { req = req.WithContext(ctx) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusForbidden, w.Code) assert.Contains(t, w.Body.String(), "forbidden") @@ -267,7 +268,7 @@ func TestIngest_Policy_ColumnDenied(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) - h.PolicySource = policy.Static(&policy.Policy{ + h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": { "writer": {Insert: &policy.InsertPermissions{AllowColumns: []string{"page"}}}, @@ -281,7 +282,7 @@ func TestIngest_Policy_ColumnDenied(t *testing.T) { req = req.WithContext(ctx) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusForbidden, w.Code) assert.Contains(t, w.Body.String(), "not allowed for insert") @@ -292,7 +293,7 @@ func TestIngest_Policy_CheckClause_Mismatch(t *testing.T) { pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) orgTemplate := "{{ jwt.org_id }}" - h.PolicySource = policy.Static(&policy.Policy{ + h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": { "user": {Insert: &policy.InsertPermissions{Check: map[string]policy.Filter{ @@ -309,7 +310,7 @@ func TestIngest_Policy_CheckClause_Mismatch(t *testing.T) { req = req.WithContext(ctx) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusForbidden, w.Code) assert.Contains(t, w.Body.String(), "check failed") @@ -320,7 +321,7 @@ func TestIngest_Policy_CheckClause_Match(t *testing.T) { pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) orgTemplate := "{{ jwt.org_id }}" - h.PolicySource = policy.Static(&policy.Policy{ + h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": { "user": {Insert: &policy.InsertPermissions{Check: map[string]policy.Filter{ @@ -338,7 +339,7 @@ func TestIngest_Policy_CheckClause_Match(t *testing.T) { req = req.WithContext(ctx) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) assert.NotNil(t, pub.LastMessage(), "should have published") @@ -354,7 +355,7 @@ func TestIngest_Policy_CheckClause_NumericSpellingMatch(t *testing.T) { pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) countTemplate := "{{ jwt.max_count }}" - h.PolicySource = policy.Static(&policy.Policy{ + h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": { "user": {Insert: &policy.InsertPermissions{Check: map[string]policy.Filter{ @@ -371,7 +372,7 @@ func TestIngest_Policy_CheckClause_NumericSpellingMatch(t *testing.T) { req = req.WithContext(ctx) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) assert.NotNil(t, pub.LastMessage(), "should have published") @@ -398,7 +399,7 @@ func TestIngest_Policy_CheckClause_StaticNumericSpelling(t *testing.T) { pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) staticCount := "1.0" - h.PolicySource = policy.Static(&policy.Policy{ + h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": { "user": {Insert: &policy.InsertPermissions{Check: map[string]policy.Filter{ @@ -414,7 +415,7 @@ func TestIngest_Policy_CheckClause_StaticNumericSpelling(t *testing.T) { req = req.WithContext(ctx) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) assert.NotNil(t, pub.LastMessage(), "should have published") @@ -443,7 +444,7 @@ func TestIngest_Policy_CheckClause_StringClaimStrictEquality(t *testing.T) { pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) orgTemplate := "{{ jwt.org_id }}" - h.PolicySource = policy.Static(&policy.Policy{ + h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": { "user": {Insert: &policy.InsertPermissions{Check: map[string]policy.Filter{ @@ -459,7 +460,7 @@ func TestIngest_Policy_CheckClause_StringClaimStrictEquality(t *testing.T) { req = req.WithContext(ctx) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, tt.want, w.Code) }) @@ -476,7 +477,7 @@ func TestIngest_Policy_CheckClause_NullValue_FailsClosed(t *testing.T) { pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) orgTemplate := "{{ jwt.org_id }}" - h.PolicySource = policy.Static(&policy.Policy{ + h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": { "user": {Insert: &policy.InsertPermissions{Check: map[string]policy.Filter{ @@ -492,7 +493,7 @@ func TestIngest_Policy_CheckClause_NullValue_FailsClosed(t *testing.T) { req = req.WithContext(ctx) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusForbidden, w.Code) assert.Contains(t, w.Body.String(), "check failed") @@ -505,7 +506,7 @@ func TestIngest_Policy_CheckClause_AutoInject(t *testing.T) { pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) orgTemplate := "{{ jwt.org_id }}" - h.PolicySource = policy.Static(&policy.Policy{ + h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": { "user": {Insert: &policy.InsertPermissions{Check: map[string]policy.Filter{ @@ -523,7 +524,7 @@ func TestIngest_Policy_CheckClause_AutoInject(t *testing.T) { req = req.WithContext(ctx) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) // Verify the published message has org_id injected. @@ -535,9 +536,9 @@ func TestIngest_Policy_CheckClause_AutoInject(t *testing.T) { // checkInStore builds a policy whose insert check restricts org_id to the set // carried by the token's `orgs` claim (an _in check) — the multi-tenant // "a writer may only insert rows for tenants they belong to" case (#224). -func checkInStore() policy.Source { +func checkInStore() PolicySource { orgsTemplate := "{{ jwt.orgs }}" - return policy.Static(&policy.Policy{ + return staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": { "user": {Insert: &policy.InsertPermissions{Check: map[string]policy.Filter{"org_id": {In: &orgsTemplate}}}}, @@ -559,7 +560,7 @@ func TestIngest_Policy_CheckIn_InSet(t *testing.T) { req = req.WithContext(ctx) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) assert.NotNil(t, pub.LastMessage(), "an in-set value should publish") @@ -578,7 +579,7 @@ func TestIngest_Policy_CheckIn_NotInSet(t *testing.T) { req = req.WithContext(ctx) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusForbidden, w.Code) assert.Contains(t, w.Body.String(), "check failed") @@ -602,7 +603,7 @@ func TestIngest_Policy_CheckIn_NullValue_FailsClosed(t *testing.T) { req = req.WithContext(ctx) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusForbidden, w.Code) assert.Contains(t, w.Body.String(), "check failed") @@ -623,7 +624,7 @@ func TestIngest_Policy_CheckIn_Absent_FailsClosed(t *testing.T) { req = req.WithContext(ctx) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusForbidden, w.Code) assert.Contains(t, w.Body.String(), "check failed") @@ -651,7 +652,7 @@ func TestIngest_Policy_CheckIn_AbsentClaim_FailsClosed(t *testing.T) { req = req.WithContext(ctx) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusForbidden, w.Code) assert.Contains(t, w.Body.String(), "check failed") @@ -665,13 +666,13 @@ func TestIngest_Dedup_MissingIDField(t *testing.T) { dedup := testutil.NewMockDeduplicator() h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.Dedup = dedup - h.DedupeSettings = func(string) (bool, string, bool) { return true, "event_id", false } + h.DedupeSettings = func(*settings.Store, string) (bool, string, bool) { return true, "event_id", false } // Payload omits event_id and require_id is off: the row skips // dedup and is still published — the warn+counter path, not a rejection (#219). req := ingestRequest(t, "clicks", map[string]any{"page": "/home"}) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) assert.NotNil(t, pub.LastMessage(), "should have published even without dedup ID") @@ -684,17 +685,17 @@ func TestIngest_Dedup_RequireID_Rejects(t *testing.T) { pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.Dedup = testutil.NewMockDeduplicator() - h.DedupeSettings = func(string) (bool, string, bool) { return true, "event_id", true } + h.DedupeSettings = func(*settings.Store, string) (bool, string, bool) { return true, "event_id", true } w := httptest.NewRecorder() - h.Handle(w, ingestRequest(t, "clicks", map[string]any{"page": "/home"})) + h.Handle(w, withTenant(ingestRequest(t, "clicks", map[string]any{"page": "/home"}))) assert.Equal(t, http.StatusBadRequest, w.Code) assert.Contains(t, w.Body.String(), "missing dedupe id field") testutil.AssertJSONErrorResponse(t, w) assert.Nil(t, pub.LastMessage(), "must not publish a row missing the dedupe id under require_id") w = httptest.NewRecorder() - h.Handle(w, ingestRequest(t, "clicks", map[string]any{"page": "/home", "event_id": "ok-1"})) + h.Handle(w, withTenant(ingestRequest(t, "clicks", map[string]any{"page": "/home", "event_id": "ok-1"}))) assert.Equal(t, http.StatusOK, w.Code) assert.NotNil(t, pub.LastMessage(), "a record carrying the id is still accepted") } @@ -706,7 +707,7 @@ func TestIngest_NDJSON_RequireID_Rejects(t *testing.T) { pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.Dedup = testutil.NewMockDeduplicator() - h.DedupeSettings = func(string) (bool, string, bool) { return true, "event_id", true } + h.DedupeSettings = func(*settings.Store, string) (bool, string, bool) { return true, "event_id", true } req := ndjsonRequest(t, "clicks", jsonLine(t, map[string]any{"page": "/a", "event_id": "e1"}), @@ -714,7 +715,7 @@ func TestIngest_NDJSON_RequireID_Rejects(t *testing.T) { jsonLine(t, map[string]any{"page": "/c", "event_id": "e2"}), ) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) resp := decodeBatchResult(t, w) @@ -732,7 +733,7 @@ func TestIngest_Policy_DenyColumns(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) - h.PolicySource = policy.Static(&policy.Policy{ + h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": { "writer": {Insert: &policy.InsertPermissions{DenyColumns: []string{"count"}}}, @@ -745,7 +746,7 @@ func TestIngest_Policy_DenyColumns(t *testing.T) { req = req.WithContext(ctx) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusForbidden, w.Code) assert.Contains(t, w.Body.String(), "not allowed for insert") @@ -755,7 +756,7 @@ func TestIngest_AdminRole_NoPolicy(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) - h.PolicySource = policy.Static(&policy.Policy{ + h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": {}, }, @@ -766,7 +767,7 @@ func TestIngest_AdminRole_NoPolicy(t *testing.T) { req = req.WithContext(ctx) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) } @@ -826,7 +827,7 @@ func TestIngest_NDJSON_AllValid(t *testing.T) { jsonLine(t, map[string]any{"page": "/c", "count": 3}), ) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) resp := decodeBatchResult(t, w) @@ -854,7 +855,7 @@ func TestIngest_NDJSON_PartialFailure_Validation(t *testing.T) { jsonLine(t, map[string]any{"page": "/c"}), ) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) resp := decodeBatchResult(t, w) @@ -880,7 +881,7 @@ func TestIngest_NDJSON_MalformedLine(t *testing.T) { jsonLine(t, map[string]any{"page": "/c"}), ) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) resp := decodeBatchResult(t, w) @@ -910,7 +911,7 @@ func TestIngest_NDJSON_BlankLinesSkipped(t *testing.T) { "\t", ) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) resp := decodeBatchResult(t, w) @@ -938,7 +939,7 @@ func TestIngest_NDJSON_EmptyBody(t *testing.T) { req := ndjsonRequest(t, "clicks", tt.lines...) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusBadRequest, w.Code) assert.Contains(t, w.Body.String(), "empty ndjson body") @@ -954,7 +955,7 @@ func TestIngest_NDJSON_Dedup(t *testing.T) { dedup := testutil.NewMockDeduplicator() h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.Dedup = dedup - h.DedupeSettings = func(string) (bool, string, bool) { return true, "event_id", false } + h.DedupeSettings = func(*settings.Store, string) (bool, string, bool) { return true, "event_id", false } req := ndjsonRequest(t, "clicks", jsonLine(t, map[string]any{"page": "/a", "event_id": "e1"}), @@ -962,7 +963,7 @@ func TestIngest_NDJSON_Dedup(t *testing.T) { jsonLine(t, map[string]any{"page": "/c", "event_id": "e2"}), ) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) resp := decodeBatchResult(t, w) @@ -988,7 +989,7 @@ func TestIngest_NDJSON_Backpressure_503(t *testing.T) { jsonLine(t, map[string]any{"page": "/b"}), ) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusServiceUnavailable, w.Code) assert.Equal(t, "30", w.Header().Get("Retry-After")) @@ -1002,7 +1003,7 @@ func TestIngest_NDJSON_PublishError_500(t *testing.T) { req := ndjsonRequest(t, "clicks", jsonLine(t, map[string]any{"page": "/a"})) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusInternalServerError, w.Code) assert.Contains(t, w.Body.String(), "publish failed") @@ -1013,7 +1014,7 @@ func TestIngest_NDJSON_Policy_ColumnDenied_PerLine(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) - h.PolicySource = policy.Static(&policy.Policy{ + h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": { "writer": {Insert: &policy.InsertPermissions{AllowColumns: []string{"page"}}}, @@ -1029,7 +1030,7 @@ func TestIngest_NDJSON_Policy_ColumnDenied_PerLine(t *testing.T) { req = req.WithContext(ctx) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) resp := decodeBatchResult(t, w) @@ -1046,7 +1047,7 @@ func TestIngest_NDJSON_Policy_TableForbidden(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) - h.PolicySource = policy.Static(&policy.Policy{ + h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": { "viewer": {Select: &policy.SelectPermissions{}}, @@ -1060,7 +1061,7 @@ func TestIngest_NDJSON_Policy_TableForbidden(t *testing.T) { req = req.WithContext(ctx) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) // Table-level denial happens before any record is read — whole-request 403. assert.Equal(t, http.StatusForbidden, w.Code) @@ -1074,7 +1075,7 @@ func TestIngest_NDJSON_Policy_CheckClause_PerLineAndAutoInject(t *testing.T) { pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) orgTemplate := "{{ jwt.org_id }}" - h.PolicySource = policy.Static(&policy.Policy{ + h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": { "user": {Insert: &policy.InsertPermissions{Check: map[string]policy.Filter{"org_id": {Eq: &orgTemplate}}}}, @@ -1093,7 +1094,7 @@ func TestIngest_NDJSON_Policy_CheckClause_PerLineAndAutoInject(t *testing.T) { req = req.WithContext(ctx) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) resp := decodeBatchResult(t, w) @@ -1118,7 +1119,7 @@ func TestIngest_NDJSON_ContentTypeWithCharset(t *testing.T) { req.Header.Set("Content-Type", "application/x-ndjson; charset=utf-8") w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) resp := decodeBatchResult(t, w) @@ -1138,7 +1139,7 @@ func TestIngest_NDJSON_ErrorsTruncated(t *testing.T) { } req := ndjsonRequest(t, "clicks", lines...) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) resp := decodeBatchResult(t, w) @@ -1315,7 +1316,7 @@ func TestIngest_UndeclaredOrUnsupportedContentType_415(t *testing.T) { pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) w := httptest.NewRecorder() - h.Handle(w, rawIngestRequest(t, "clicks", tt.ct, `{"page":"/a"}`)) + h.Handle(w, withTenant(rawIngestRequest(t, "clicks", tt.ct, `{"page":"/a"}`))) assert.Equal(t, http.StatusUnsupportedMediaType, w.Code) testutil.AssertJSONErrorResponse(t, w) @@ -1365,7 +1366,7 @@ func TestIngest_ContentTypeRefusalBeatsEmptyBody(t *testing.T) { pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) w := httptest.NewRecorder() - h.Handle(w, rawIngestRequest(t, "clicks", ct, "")) + h.Handle(w, withTenant(rawIngestRequest(t, "clicks", ct, ""))) assert.Equal(t, http.StatusUnsupportedMediaType, w.Code, "the header is resolved before the body is read, so this is a 415 and not an empty-body 400") @@ -1384,7 +1385,7 @@ func TestIngest_DeclaredNDJSON_ArrayBodyIsNotReframed(t *testing.T) { pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) w := httptest.NewRecorder() - h.Handle(w, rawIngestRequest(t, "clicks", "application/x-ndjson", `[{"page":"/a"},{"page":"/b"}]`)) + h.Handle(w, withTenant(rawIngestRequest(t, "clicks", "application/x-ndjson", `[{"page":"/a"},{"page":"/b"}]`))) assert.Equal(t, http.StatusOK, w.Code) resp := decodeBatchResult(t, w) @@ -1434,7 +1435,7 @@ func TestIngest_DuplicateContentTypeHeaders(t *testing.T) { req.Header.Add("Content-Type", "application/x-ndjson") w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusUnsupportedMediaType, w.Code) testutil.AssertJSONErrorResponse(t, w) @@ -1459,7 +1460,7 @@ func TestIngest_DuplicateContentTypeHeaders(t *testing.T) { req.Header.Add("Content-Type", "text/csv") w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusUnsupportedMediaType, w.Code) testutil.AssertJSONErrorResponse(t, w) @@ -1478,7 +1479,7 @@ func TestIngest_DuplicateContentTypeHeaders(t *testing.T) { req.Header.Add("Content-Type", "application/ndjson; charset=utf-8") w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) assert.Len(t, pub.Messages, 2, "same format, different spelling — not ambiguous") @@ -1505,7 +1506,7 @@ func TestIngest_DuplicateContentTypeHeaders(t *testing.T) { pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) w := httptest.NewRecorder() - h.Handle(w, rawIngestRequest(t, "clicks", ct, ndjson)) + h.Handle(w, withTenant(rawIngestRequest(t, "clicks", ct, ndjson))) assert.Equal(t, http.StatusUnsupportedMediaType, w.Code) testutil.AssertJSONErrorResponse(t, w) @@ -1549,7 +1550,7 @@ func TestIngest_DuplicateContentTypeHeaders(t *testing.T) { t.Parallel() wJ := httptest.NewRecorder() NewIngestHandler(testRegistry(t), &testutil.MockPublisher{}, testutil.NopLogger()). - Handle(wJ, rawIngestRequest(t, "clicks", tc.joined, `{"page":"/a"}`)) + Handle(wJ, withTenant(rawIngestRequest(t, "clicks", tc.joined, `{"page":"/a"}`))) assert.Equal(t, tc.wJoined, wJ.Code, "joined") req := rawIngestRequest(t, "clicks", "", `{"page":"/a"}`) @@ -1558,7 +1559,7 @@ func TestIngest_DuplicateContentTypeHeaders(t *testing.T) { } wR := httptest.NewRecorder() NewIngestHandler(testRegistry(t), &testutil.MockPublisher{}, testutil.NopLogger()). - Handle(wR, req) + Handle(wR, withTenant(req)) assert.Equal(t, tc.wRepeat, wR.Code, "repeated") }) } @@ -1569,7 +1570,7 @@ func TestIngest_DuplicateContentTypeHeaders(t *testing.T) { pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) w := httptest.NewRecorder() - h.Handle(w, rawIngestRequest(t, "clicks", `application/json; profile="a,b"`, `{"page":"/a"}`)) + h.Handle(w, withTenant(rawIngestRequest(t, "clicks", `application/json; profile="a,b"`, `{"page":"/a"}`))) assert.Equal(t, http.StatusOK, w.Code, "the comma is inside a quoted value, so this parses cleanly") assert.Len(t, pub.Messages, 1) @@ -1584,7 +1585,7 @@ func TestIngest_DuplicateContentTypeHeaders(t *testing.T) { req.Header.Add("Content-Type", "application/x-ndjson") w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusUnsupportedMediaType, w.Code) testutil.AssertJSONErrorResponse(t, w) @@ -1611,7 +1612,7 @@ func TestIngest_DuplicateContentTypeHeaders(t *testing.T) { } w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusUnsupportedMediaType, w.Code, "empty first=%v", first) testutil.AssertJSONErrorResponse(t, w) @@ -1631,7 +1632,7 @@ func TestIngest_DuplicateContentTypeHeaders(t *testing.T) { req.Header.Add("Content-Type", "text/plain") w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusUnsupportedMediaType, w.Code) testutil.AssertJSONErrorResponse(t, w) @@ -1650,7 +1651,7 @@ func TestIngest_DuplicateContentTypeHeaders(t *testing.T) { req.Header.Add("Content-Type", "application/x-ndjson") w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) assert.Len(t, pub.Messages, 2, "both NDJSON records ride through") @@ -1669,7 +1670,7 @@ func TestIngest_JSONArray_AllValid(t *testing.T) { {"page": "/b", "count": 2}, }) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) resp := decodeBatchResult(t, w) @@ -1691,7 +1692,7 @@ func TestIngest_JSONArray_SingleElement(t *testing.T) { // the single-object {"ok":true}). req := ingestRequest(t, "clicks", []map[string]any{{"page": "/solo"}}) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) resp := decodeBatchResult(t, w) @@ -1713,7 +1714,7 @@ func TestIngest_JSONArray_PartialValidationFailure(t *testing.T) { {"page": "/c"}, }) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) // The bad element is reported per-record; the request itself is 200. assert.Equal(t, http.StatusOK, w.Code) @@ -1743,7 +1744,7 @@ func TestIngest_JSONArray_ScalarElements(t *testing.T) { map[string]any{"page": "/b"}, }) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) resp := decodeBatchResult(t, w) @@ -1768,7 +1769,7 @@ func TestIngest_JSONArray_SyntaxError_Fatal(t *testing.T) { // already published (at-least-once on retry). req := rawIngestRequest(t, "clicks", "application/json", `[{"page":"/a"}, {bad]`) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusBadRequest, w.Code) assert.Contains(t, w.Body.String(), "invalid json") @@ -1799,7 +1800,7 @@ func TestIngest_JSONArray_Truncated_Fatal(t *testing.T) { req := rawIngestRequest(t, "clicks", "application/json", tt.body) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusBadRequest, w.Code) assert.Contains(t, w.Body.String(), "invalid json") @@ -1816,7 +1817,7 @@ func TestIngest_JSONArray_Empty(t *testing.T) { // An explicit empty array is a valid, record-less batch → 200 with no rows. req := rawIngestRequest(t, "clicks", "application/json", `[]`) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) resp := decodeBatchResult(t, w) @@ -1834,7 +1835,7 @@ func TestIngest_SingleObject_PrettyPrinted(t *testing.T) { // NDJSON — it's one record on the single-object path. req := rawIngestRequest(t, "clicks", "application/json", "{\n \"page\": \"/a\",\n \"count\": 1\n}") w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) var resp map[string]bool @@ -1853,7 +1854,7 @@ func TestIngest_DeclaredJSON_ConcatenatedObjects_FirstOnly(t *testing.T) { // behavior — send application/x-ndjson to batch them). req := rawIngestRequest(t, "clicks", "application/json", `{"page":"/first"}{"page":"/second"}`) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) var resp map[string]bool @@ -1883,7 +1884,7 @@ func TestIngest_LeadingWhitespace_Sniff(t *testing.T) { req := rawIngestRequest(t, "clicks", "application/json", tt.body) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusOK, w.Code) if tt.batch { @@ -1921,7 +1922,7 @@ func TestIngest_EmptyBody(t *testing.T) { req := rawIngestRequest(t, "clicks", tt.contentType, tt.body) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusBadRequest, w.Code) assert.Contains(t, w.Body.String(), tt.wantMsg) @@ -1955,7 +1956,7 @@ func TestIngest_BodyReadFailure_400(t *testing.T) { req.Header.Set("Content-Type", ct) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusBadRequest, w.Code) testutil.AssertJSONErrorResponse(t, w) @@ -2005,7 +2006,7 @@ func TestIngest_BodyCap_413(t *testing.T) { req := rawIngestRequest(t, "clicks", tt.ct, tt.body) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusRequestEntityTooLarge, w.Code) assert.Contains(t, w.Body.String(), "request body exceeded") @@ -2041,7 +2042,7 @@ func TestIngest_ContentTypeResolvesBeforeTheBodyIsRead(t *testing.T) { req.Header.Set("Content-Type", "text/csv") w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusUnsupportedMediaType, w.Code, "reading the body first would answer 400 invalid request body") @@ -2058,7 +2059,7 @@ func TestIngest_ContentTypeResolvesBeforeTheBodyIsRead(t *testing.T) { req := rawIngestRequest(t, "clicks", "text/csv", `{"page":"/`+strings.Repeat("a", 200)+`"}`) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusUnsupportedMediaType, w.Code, "reading the body first would answer 413 request body exceeded") @@ -2124,7 +2125,7 @@ func TestIngest_TimestampsCanonicalized(t *testing.T) { "ts_ms": 1782014400500, // integer number = ClickHouse ticks at the column scale (ms here) }) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) require.Equal(t, http.StatusOK, w.Code) data := publishedData(t, pub) @@ -2146,7 +2147,7 @@ func TestIngest_AutoInjectedLiteralTimestampCanonicalized(t *testing.T) { pub := &testutil.MockPublisher{} h := NewIngestHandler(tsRegistry(t), pub, testutil.NopLogger()) staticTS := "2026-06-21 04:00:00" - h.PolicySource = policy.Static(&policy.Policy{ + h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "events": { "user": {Insert: &policy.InsertPermissions{Check: map[string]policy.Filter{ @@ -2164,7 +2165,7 @@ func TestIngest_AutoInjectedLiteralTimestampCanonicalized(t *testing.T) { req = req.WithContext(ctx) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) require.Equal(t, http.StatusOK, w.Code) assert.Equal(t, "2026-06-21T04:00:00Z", publishedData(t, pub)["ts"], @@ -2180,7 +2181,7 @@ func TestIngest_TimestampGarbage_PassesThrough(t *testing.T) { req := ingestRequest(t, "events", map[string]any{"name": "e", "ts": "banana"}) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) require.Equal(t, http.StatusOK, w.Code) assert.Equal(t, "banana", publishedData(t, pub)["ts"], "unparseable value published verbatim") @@ -2199,7 +2200,7 @@ func TestIngest_Batch_MixedTimestampSpellings(t *testing.T) { {"name": "c", "ts": float64(1782014400)}, }) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) require.Equal(t, http.StatusOK, w.Code) var result struct { @@ -2245,10 +2246,10 @@ func TestIngest_Dedup_DisabledBySettings(t *testing.T) { dedup.Err = errors.New("must not be called while disabled") h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.Dedup = dedup - h.DedupeSettings = func(string) (bool, string, bool) { return false, "event_id", true } + h.DedupeSettings = func(*settings.Store, string) (bool, string, bool) { return false, "event_id", true } w := httptest.NewRecorder() - h.Handle(w, ingestRequest(t, "clicks", tt.body)) + h.Handle(w, withTenant(ingestRequest(t, "clicks", tt.body))) assert.Equal(t, http.StatusOK, w.Code) assert.Len(t, pub.Messages, 1, "record publishes, neither deduped nor rejected") }) @@ -2265,10 +2266,10 @@ func TestIngest_Dedup_DisabledMidReload(t *testing.T) { dedup.Err = dedupe.ErrDisabled h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.Dedup = dedup - h.DedupeSettings = func(string) (bool, string, bool) { return true, "event_id", true } + h.DedupeSettings = func(*settings.Store, string) (bool, string, bool) { return true, "event_id", true } w := httptest.NewRecorder() - h.Handle(w, ingestRequest(t, "clicks", map[string]any{"event_id": "e1", "page": "/home"})) + h.Handle(w, withTenant(ingestRequest(t, "clicks", map[string]any{"event_id": "e1", "page": "/home"}))) assert.Equal(t, http.StatusOK, w.Code) assert.Len(t, pub.Messages, 1, "published without idempotency, not 500") } @@ -2311,7 +2312,7 @@ func TestProcessRecord_UnresolvedInsertSideAborts(t *testing.T) { "all-nullable/defaulted columns accept an empty record — this is what makes the read reachable") dup, reject, abort := h.processRecord( - context.Background(), "loose", "", schema, selectResolved, "viewer", map[string]any{}, time.Now(), nil) + context.Background(), testStore, "loose", "", schema, selectResolved, "viewer", map[string]any{}, time.Now(), nil) assert.False(t, dup) assert.Nil(t, reject, "a request-scoped condition must not be reported per record") @@ -2356,7 +2357,7 @@ func TestIngest_ContentTypeEchoIsBounded(t *testing.T) { pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) w := httptest.NewRecorder() - h.Handle(w, build(t)) + h.Handle(w, withTenant(build(t))) require.Equal(t, http.StatusUnsupportedMediaType, w.Code) testutil.AssertJSONErrorResponse(t, w) @@ -2382,7 +2383,7 @@ func TestIngest_ContentTypeEchoIsBounded(t *testing.T) { req.Header.Add("Content-Type", fmt.Sprintf("application/%04d", i)+strings.Repeat("\xff", 112)) } w := httptest.NewRecorder() - NewIngestHandler(testRegistry(t), &testutil.MockPublisher{}, testutil.NopLogger()).Handle(w, req) + NewIngestHandler(testRegistry(t), &testutil.MockPublisher{}, testutil.NopLogger()).Handle(w, withTenant(req)) require.Equal(t, http.StatusUnsupportedMediaType, w.Code) return w.Body.Len() } @@ -2414,7 +2415,7 @@ func TestIngest_ConflictMessageNamesTheDisagreement(t *testing.T) { req.Header.Add("Content-Type", "application/x-ndjson") w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) require.Equal(t, http.StatusUnsupportedMediaType, w.Code) msg := jsonErrorMessage(t, w) @@ -2449,7 +2450,7 @@ func TestIngest_ConflictMessageNamesADifferentSpelling(t *testing.T) { } w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) require.Equal(t, http.StatusUnsupportedMediaType, w.Code) assert.Contains(t, jsonErrorMessage(t, w), `"application/x-ndjson"`, @@ -2481,7 +2482,7 @@ func TestIngest_ConflictLogNamesTheDisagreement(t *testing.T) { req.Header.Add("Content-Type", ct) } w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) require.Equal(t, http.StatusUnsupportedMediaType, w.Code) assert.Contains(t, buf.String(), "application/x-ndjson", @@ -2497,10 +2498,10 @@ func TestIngest_CheckColumnNotInSchema_Rejected(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) - h.PolicySource = policy.Static(checkColumnPolicy(t, "tenant_id", "acme")) + h.PolicySource = staticPolicy(checkColumnPolicy(t, "tenant_id", "acme")) w := httptest.NewRecorder() - h.Handle(w, viewerIngestRequest(t, "clicks", map[string]any{"page": "/a"})) + h.Handle(w, withTenant(viewerIngestRequest(t, "clicks", map[string]any{"page": "/a"}))) assert.Equal(t, http.StatusForbidden, w.Code) assert.Contains(t, w.Body.String(), "tenant_id", "the message names the offending column") @@ -2530,10 +2531,10 @@ func TestIngest_CheckOnComputedColumn_Rejected(t *testing.T) { }} pub := &testutil.MockPublisher{} h := NewIngestHandler(computedRegistry(t), pub, testutil.NopLogger()) - h.PolicySource = policy.Static(p) + h.PolicySource = staticPolicy(p) w := httptest.NewRecorder() - h.Handle(w, viewerIngestRequest(t, "clicks", map[string]any{"page": "/a"})) + h.Handle(w, withTenant(viewerIngestRequest(t, "clicks", map[string]any{"page": "/a"}))) assert.Equal(t, http.StatusForbidden, w.Code) testutil.AssertJSONErrorResponse(t, w) @@ -2553,10 +2554,10 @@ func TestIngest_CheckColumnInSchema_StillInjects(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) - h.PolicySource = policy.Static(checkColumnPolicy(t, "org_id", "org-42")) + h.PolicySource = staticPolicy(checkColumnPolicy(t, "org_id", "org-42")) w := httptest.NewRecorder() - h.Handle(w, viewerIngestRequest(t, "clicks", map[string]any{"page": "/a"})) + h.Handle(w, withTenant(viewerIngestRequest(t, "clicks", map[string]any{"page": "/a"}))) require.Equal(t, http.StatusOK, w.Code, "body=%s", w.Body.String()) require.Len(t, pub.Messages, 1) @@ -2579,7 +2580,7 @@ func TestIngest_CheckGuardLogsOncePerRequest(t *testing.T) { var buf bytes.Buffer pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, slog.New(slog.NewJSONHandler(&buf, nil))) - h.PolicySource = policy.Static(checkColumnPolicy(t, "tenant_id", "acme")) + h.PolicySource = staticPolicy(checkColumnPolicy(t, "tenant_id", "acme")) const n = 25 body := "[" + strings.Repeat(`{"page":"/a"},`, n-1) + `{"page":"/z"}]` @@ -2587,7 +2588,7 @@ func TestIngest_CheckGuardLogsOncePerRequest(t *testing.T) { req = req.WithContext(auth.WithRole(req.Context(), "viewer")) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) resp := decodeBatchResult(t, w) require.Equal(t, n, resp.Total) @@ -2611,14 +2612,14 @@ func TestIngest_CheckColumnNotInSchema_BatchRejectsPerRecord(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) - h.PolicySource = policy.Static(checkColumnPolicy(t, "tenant_id", "acme")) + h.PolicySource = staticPolicy(checkColumnPolicy(t, "tenant_id", "acme")) req := rawIngestRequest(t, "clicks", "application/json", `[{"page":"/a"},{"page":"/b","tenant_id":"acme"},{"page":"/c"}]`) req = req.WithContext(auth.WithRole(req.Context(), "viewer")) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) require.Equal(t, http.StatusOK, w.Code, "a misconfigured policy must not abort the request") resp := decodeBatchResult(t, w) @@ -2670,10 +2671,10 @@ func TestIngest_CheckOnEphemeralColumn_Rejected(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} h := NewIngestHandler(computedRegistry(t), pub, testutil.NopLogger()) - h.PolicySource = policy.Static(checkColumnPolicy(t, "raw", "anything")) + h.PolicySource = staticPolicy(checkColumnPolicy(t, "raw", "anything")) w := httptest.NewRecorder() - h.Handle(w, viewerIngestRequest(t, "clicks", map[string]any{"page": "/a"})) + h.Handle(w, withTenant(viewerIngestRequest(t, "clicks", map[string]any{"page": "/a"}))) assert.Equal(t, http.StatusForbidden, w.Code, "body=%s", w.Body.String()) testutil.AssertJSONErrorResponse(t, w) diff --git a/internal/api/pipes.go b/internal/api/pipes.go index 2de2da76..494b6647 100644 --- a/internal/api/pipes.go +++ b/internal/api/pipes.go @@ -12,6 +12,7 @@ import ( "github.com/Wave-RF/WaveHouse/internal/cache" "github.com/Wave-RF/WaveHouse/internal/pipes" "github.com/Wave-RF/WaveHouse/internal/policy" + "github.com/Wave-RF/WaveHouse/internal/settings" "github.com/go-chi/chi/v5" "golang.org/x/sync/singleflight" ) @@ -20,11 +21,15 @@ import ( // listing for admins. Pipes are defined in the settings directory's // pipes.json and read per request, so a reload applies immediately. type PipesHandler struct { - Source pipes.Source - PolicySource policy.Source // resolves empty role to default_role; may be nil - CHConn driver.Conn - Cache cache.Cache - sf singleflight.Group + // Source yields a tenant's pipes (the store itself in production). + Source func(*settings.Store) pipes.Source + PolicySource PolicySource // resolves empty role to default_role; may be nil + // OpsStore is the store the admin reads (List, Get) serve: /v1/ops is + // tenant-exempt, so they carry no request tenant and read the default one. + OpsStore *settings.Store + CHConn driver.Conn + Cache cache.Cache + sf singleflight.Group // queryTimeout bounds each pipe execution, read per request // (chconn.Manager.QueryTimeout in production) so a settings reload // applies without a restart. @@ -39,14 +44,14 @@ type PipesHandler struct { maxRequestBytes int64 } -func NewPipesHandler(source pipes.Source, policySource policy.Source, conn driver.Conn, c cache.Cache, queryTimeout func() time.Duration, logger *slog.Logger) *PipesHandler { +func NewPipesHandler(source func(*settings.Store) pipes.Source, policySource PolicySource, conn driver.Conn, c cache.Cache, queryTimeout func() time.Duration, logger *slog.Logger) *PipesHandler { return &PipesHandler{Source: source, PolicySource: policySource, CHConn: conn, Cache: c, queryTimeout: queryTimeout, logger: logger} } // List returns all named queries (admin endpoint). func (h *PipesHandler) List(w http.ResponseWriter, _ *http.Request) { w.Header().Set("Content-Type", "application/json") - q := h.Source.Pipes() + q := h.Source(h.OpsStore).Pipes() if q == nil { q = []*pipes.NamedQuery{} } @@ -56,7 +61,7 @@ func (h *PipesHandler) List(w http.ResponseWriter, _ *http.Request) { // Get returns a specific named query (admin endpoint). func (h *PipesHandler) Get(w http.ResponseWriter, r *http.Request) { name := chi.URLParam(r, "name") - q := h.Source.Pipe(name) + q := h.Source(h.OpsStore).Pipe(name) if q == nil { writeJSONError(w, http.StatusNotFound, "pipe not found") return @@ -67,8 +72,12 @@ func (h *PipesHandler) Get(w http.ResponseWriter, r *http.Request) { // Execute runs a named query with the provided parameters. func (h *PipesHandler) Execute(w http.ResponseWriter, r *http.Request) { + store, ok := requestStore(w, r) + if !ok { + return + } name := chi.URLParam(r, "name") - q := h.Source.Pipe(name) + q := h.Source(store).Pipe(name) if q == nil { writeJSONError(w, http.StatusNotFound, "pipe not found") return @@ -85,7 +94,7 @@ func (h *PipesHandler) Execute(w http.ResponseWriter, r *http.Request) { // allowed_roles therefore authorizes nobody but admin (fails closed). var p *policy.Policy if h.PolicySource != nil { - p = h.PolicySource() + p = h.PolicySource(store) } role := policy.ResolveRole(p, auth.RoleFromContext(r.Context())) if !policy.RoleAllowed(p, role, q.AllowedRoles) { diff --git a/internal/api/pipes_test.go b/internal/api/pipes_test.go index 35c63cd5..4621f513 100644 --- a/internal/api/pipes_test.go +++ b/internal/api/pipes_test.go @@ -43,7 +43,7 @@ func noTimeout() time.Duration { return 0 } func TestPipesHandler_List(t *testing.T) { t.Parallel() - store := pipes.Static( + store := staticPipes( &pipes.NamedQuery{Name: "top_pages", SQL: "SELECT page, count(*) FROM clicks GROUP BY page"}, &pipes.NamedQuery{Name: "recent", SQL: "SELECT * FROM clicks ORDER BY ts DESC LIMIT 10"}, ) @@ -61,7 +61,7 @@ func TestPipesHandler_List(t *testing.T) { func TestPipesHandler_Get_Found(t *testing.T) { t.Parallel() - store := pipes.Static( + store := staticPipes( &pipes.NamedQuery{Name: "top_pages", SQL: "SELECT page FROM clicks"}, ) h := NewPipesHandler(store, nil, nil, nil, noTimeout, testutil.NopLogger()) @@ -78,7 +78,7 @@ func TestPipesHandler_Get_Found(t *testing.T) { func TestPipesHandler_Get_NotFound(t *testing.T) { t.Parallel() - store := pipes.Static() + store := staticPipes() h := NewPipesHandler(store, nil, nil, nil, noTimeout, testutil.NopLogger()) w := httptest.NewRecorder() @@ -92,7 +92,7 @@ func TestPipesHandler_Get_NotFound(t *testing.T) { func TestPipesHandler_List_Empty(t *testing.T) { t.Parallel() - store := pipes.Static() + store := staticPipes() h := NewPipesHandler(store, nil, nil, nil, noTimeout, testutil.NopLogger()) w := httptest.NewRecorder() @@ -108,12 +108,12 @@ func TestPipesHandler_List_Empty(t *testing.T) { func TestPipesHandler_Execute_NotFound(t *testing.T) { t.Parallel() - store := pipes.Static() + store := staticPipes() h := NewPipesHandler(store, nil, nil, nil, noTimeout, testutil.NopLogger()) w := httptest.NewRecorder() r := pipesRequest(t, http.MethodPost, "/v1/pipes/nope/execute", "nope", nil) - h.Execute(w, r) + h.Execute(w, withTenant(r)) assert.Equal(t, http.StatusNotFound, w.Code) assert.Contains(t, w.Body.String(), "pipe not found") @@ -123,7 +123,7 @@ func TestPipesHandler_Execute_NotFound(t *testing.T) { func TestPipesHandler_Execute_RoleAuthorization(t *testing.T) { t.Parallel() testutil.RunRoleMatrix(t, testutil.StandardRoleMatrix(), func(t *testing.T, tc testutil.RoleCase) *httptest.ResponseRecorder { - store := pipes.Static( + store := staticPipes( &pipes.NamedQuery{ Name: "report", SQL: "SELECT * FROM clicks", @@ -133,7 +133,7 @@ func TestPipesHandler_Execute_RoleAuthorization(t *testing.T) { // A real (non-nil) policy so the default admin role ("admin") is defined // and bypasses the allowlist, per the matrix. With a nil policy nobody is // admin (total lockout) — covered separately in internal/policy tests. - h := NewPipesHandler(store, policy.Static(&policy.Policy{}), nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout, testutil.NopLogger()) w := httptest.NewRecorder() r := pipesRequest(t, http.MethodPost, "/v1/pipes/report/execute", "report", nil) @@ -146,14 +146,14 @@ func TestPipesHandler_Execute_RoleAuthorization(t *testing.T) { // safeHandle recovers the nil-backend panic on the allowed path so a // served request surfaces as a clean non-403 rather than crashing the // parallel test binary; a forbidden request returns a real 403 first. - safeHandle(h.Execute, w, r) + safeHandle(h.Execute, w, withTenant(r)) return w }) } func TestPipesHandler_Execute_RestrictedPipe_EmptyRoleDenied(t *testing.T) { t.Parallel() - store := pipes.Static( + store := staticPipes( &pipes.NamedQuery{ Name: "admin_report", SQL: "SELECT * FROM clicks", @@ -166,7 +166,7 @@ func TestPipesHandler_Execute_RestrictedPipe_EmptyRoleDenied(t *testing.T) { // No ContextKeyRole set, which simulates no token or a JWT without the role claim. r := pipesRequest(t, http.MethodPost, "/v1/pipes/admin_report/execute", "admin_report", nil) - safeHandle(h.Execute, w, r) + safeHandle(h.Execute, w, withTenant(r)) assert.Equal(t, http.StatusForbidden, w.Code, "pipe restricted to %v must reject a request with no role in context", []string{"admin"}) @@ -178,17 +178,17 @@ func TestPipesHandler_Execute_RestrictedPipe_EmptyRoleDenied(t *testing.T) { // in context) resolves to the policy default_role, which is in AllowedRoles. func TestPipesHandler_Execute_DefaultRoleGrantsAccess(t *testing.T) { t.Parallel() - store := pipes.Static( + store := staticPipes( &pipes.NamedQuery{Name: "report", SQL: "SELECT * FROM clicks", AllowedRoles: []string{"viewer"}}, ) h := NewPipesHandler(store, nil, nil, nil, noTimeout, testutil.NopLogger()) - h.PolicySource = policy.Static(&policy.Policy{DefaultRole: "viewer"}) + h.PolicySource = staticPolicy(&policy.Policy{DefaultRole: "viewer"}) w := httptest.NewRecorder() // No role in context (tokenless / JWT without role claim). r := pipesRequest(t, http.MethodPost, "/v1/pipes/report/execute", "report", nil) - safeHandle(h.Execute, w, r) + safeHandle(h.Execute, w, withTenant(r)) assert.NotEqual(t, http.StatusForbidden, w.Code, "empty role should resolve to default_role 'viewer', which is in AllowedRoles") @@ -199,16 +199,16 @@ func TestPipesHandler_Execute_DefaultRoleGrantsAccess(t *testing.T) { // is still gated by AllowedRoles — a default that isn't listed is denied. func TestPipesHandler_Execute_DefaultRoleNotInAllowedRolesDenied(t *testing.T) { t.Parallel() - store := pipes.Static( + store := staticPipes( &pipes.NamedQuery{Name: "admin_report", SQL: "SELECT * FROM clicks", AllowedRoles: []string{"admin"}}, ) h := NewPipesHandler(store, nil, nil, nil, noTimeout, testutil.NopLogger()) - h.PolicySource = policy.Static(&policy.Policy{DefaultRole: "viewer"}) + h.PolicySource = staticPolicy(&policy.Policy{DefaultRole: "viewer"}) w := httptest.NewRecorder() r := pipesRequest(t, http.MethodPost, "/v1/pipes/admin_report/execute", "admin_report", nil) - safeHandle(h.Execute, w, r) + safeHandle(h.Execute, w, withTenant(r)) assert.Equal(t, http.StatusForbidden, w.Code, "default_role 'viewer' is not in AllowedRoles [admin] → denied") @@ -217,7 +217,7 @@ func TestPipesHandler_Execute_DefaultRoleNotInAllowedRolesDenied(t *testing.T) { func TestPipesHandler_Execute_MissingParam(t *testing.T) { t.Parallel() - store := pipes.Static( + store := staticPipes( &pipes.NamedQuery{ Name: "by_page", SQL: "SELECT * FROM clicks WHERE page = {{page}}", @@ -226,14 +226,14 @@ func TestPipesHandler_Execute_MissingParam(t *testing.T) { }, }, ) - h := NewPipesHandler(store, policy.Static(&policy.Policy{}), nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout, testutil.NopLogger()) w := httptest.NewRecorder() // No query params or body — missing "page". r := pipesRequest(t, http.MethodGet, "/v1/pipes/by_page/execute", "by_page", nil) r = r.WithContext(auth.WithRole(r.Context(), "admin")) - h.Execute(w, r) + h.Execute(w, withTenant(r)) assert.Equal(t, http.StatusBadRequest, w.Code) assert.Contains(t, w.Body.String(), "missing required parameter") @@ -242,7 +242,7 @@ func TestPipesHandler_Execute_MissingParam(t *testing.T) { func TestPipesHandler_Execute_ParamsFromQuery(t *testing.T) { t.Parallel() - store := pipes.Static( + store := staticPipes( &pipes.NamedQuery{ Name: "by_page", SQL: "SELECT * FROM clicks WHERE page = {{page}}", @@ -251,7 +251,7 @@ func TestPipesHandler_Execute_ParamsFromQuery(t *testing.T) { }, }, ) - h := NewPipesHandler(store, policy.Static(&policy.Policy{}), nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout, testutil.NopLogger()) w := httptest.NewRecorder() r := httptest.NewRequestWithContext(context.Background(), http.MethodGet, "/v1/pipes/by_page/execute?page=/home", nil) @@ -260,7 +260,7 @@ func TestPipesHandler_Execute_ParamsFromQuery(t *testing.T) { r = r.WithContext(context.WithValue(r.Context(), chi.RouteCtxKey, rctx)) r = r.WithContext(auth.WithRole(r.Context(), "admin")) - safeHandle(h.Execute, w, r) + safeHandle(h.Execute, w, withTenant(r)) // Should pass param binding — will fail later at executeQuery (nil conn). assert.NotEqual(t, http.StatusBadRequest, w.Code) @@ -273,14 +273,14 @@ func TestPipesHandler_Execute_ParamsFromQuery(t *testing.T) { // is tiny so we don't allocate 1 MiB per run. func TestPipesHandler_Execute_RequestBodyCap(t *testing.T) { t.Parallel() - store := pipes.Static( + store := staticPipes( &pipes.NamedQuery{ Name: "by_page", SQL: "SELECT * FROM clicks WHERE page = {{page}}", Parameters: []pipes.ParamDef{{Name: "page", Type: "string", Required: true}}, }, ) - h := NewPipesHandler(store, policy.Static(&policy.Policy{}), nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout, testutil.NopLogger()) h.maxRequestBytes = 64 w := httptest.NewRecorder() @@ -289,7 +289,7 @@ func TestPipesHandler_Execute_RequestBodyCap(t *testing.T) { }) r = r.WithContext(auth.WithRole(r.Context(), "admin")) - h.Execute(w, r) + h.Execute(w, withTenant(r)) assert.Equal(t, http.StatusRequestEntityTooLarge, w.Code, "oversized body must 413") assert.Contains(t, w.Body.String(), "request body exceeded") @@ -298,7 +298,7 @@ func TestPipesHandler_Execute_RequestBodyCap(t *testing.T) { func TestPipesHandler_Execute_PostBodyParams(t *testing.T) { t.Parallel() - store := pipes.Static( + store := staticPipes( &pipes.NamedQuery{ Name: "by_page", SQL: "SELECT * FROM clicks WHERE page = {{page}}", @@ -307,14 +307,14 @@ func TestPipesHandler_Execute_PostBodyParams(t *testing.T) { }, }, ) - h := NewPipesHandler(store, policy.Static(&policy.Policy{}), nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout, testutil.NopLogger()) w := httptest.NewRecorder() body := map[string]any{"page": "/about"} r := pipesRequest(t, http.MethodPost, "/v1/pipes/by_page/execute", "by_page", body) r = r.WithContext(auth.WithRole(r.Context(), "admin")) - safeHandle(h.Execute, w, r) + safeHandle(h.Execute, w, withTenant(r)) // Should pass param binding — will fail at executeQuery (nil conn). assert.NotEqual(t, http.StatusBadRequest, w.Code) @@ -327,7 +327,7 @@ func TestPipesHandler_Execute_PostBodyParams(t *testing.T) { // empty allowlist matches no role. func TestPipesHandler_Execute_NoAllowedRoles_NonAdminDenied(t *testing.T) { t.Parallel() - store := pipes.Static( + store := staticPipes( &pipes.NamedQuery{Name: "open", SQL: "SELECT * FROM clicks"}, // no AllowedRoles ) h := NewPipesHandler(store, nil, nil, nil, noTimeout, testutil.NopLogger()) @@ -338,7 +338,7 @@ func TestPipesHandler_Execute_NoAllowedRoles_NonAdminDenied(t *testing.T) { ctx = auth.WithRole(ctx, "viewer") r = r.WithContext(ctx) - safeHandle(h.Execute, w, r) + safeHandle(h.Execute, w, withTenant(r)) assert.Equal(t, http.StatusForbidden, w.Code, "a pipe with no allowed_roles must reject a non-admin role") @@ -349,21 +349,21 @@ func TestPipesHandler_Execute_NoAllowedRoles_NonAdminDenied(t *testing.T) { // IN list and passes binding (failing only later at the nil ClickHouse conn). func TestPipesHandler_Execute_ArrayParamBinds(t *testing.T) { t.Parallel() - store := pipes.Static( + store := staticPipes( &pipes.NamedQuery{ Name: "by_ids", SQL: "SELECT * FROM clicks WHERE id IN {{ids}}", Parameters: []pipes.ParamDef{{Name: "ids", Type: "array", Required: true}}, }, ) - h := NewPipesHandler(store, policy.Static(&policy.Policy{}), nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout, testutil.NopLogger()) w := httptest.NewRecorder() body := map[string]any{"ids": []any{"a", "b"}} r := pipesRequest(t, http.MethodPost, "/v1/pipes/by_ids/execute", "by_ids", body) r = r.WithContext(auth.WithRole(r.Context(), "admin")) - safeHandle(h.Execute, w, r) + safeHandle(h.Execute, w, withTenant(r)) // Binding succeeded — the only failure left is the nil conn, never a 400. assert.NotEqual(t, http.StatusBadRequest, w.Code) @@ -374,17 +374,17 @@ func TestPipesHandler_Execute_ArrayParamBinds(t *testing.T) { // form and is refused with a 400 before any query runs (#317). func TestPipesHandler_Execute_ObjectParamRejected(t *testing.T) { t.Parallel() - store := pipes.Static( + store := staticPipes( &pipes.NamedQuery{Name: "by_col", SQL: "SELECT * FROM clicks WHERE col = {{p}}"}, ) - h := NewPipesHandler(store, policy.Static(&policy.Policy{}), nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout, testutil.NopLogger()) w := httptest.NewRecorder() body := map[string]any{"p": map[string]any{"k": "v"}} r := pipesRequest(t, http.MethodPost, "/v1/pipes/by_col/execute", "by_col", body) r = r.WithContext(auth.WithRole(r.Context(), "admin")) - h.Execute(w, r) + h.Execute(w, withTenant(r)) assert.Equal(t, http.StatusBadRequest, w.Code) assert.Contains(t, w.Body.String(), "unsupported parameter type object") @@ -395,16 +395,16 @@ func TestPipesHandler_Execute_ObjectParamRejected(t *testing.T) { // roles bypass the allowlist, so admin can run a pipe with no allowed_roles. func TestPipesHandler_Execute_NoAllowedRoles_AdminAllowed(t *testing.T) { t.Parallel() - store := pipes.Static( + store := staticPipes( &pipes.NamedQuery{Name: "open", SQL: "SELECT * FROM clicks"}, ) - h := NewPipesHandler(store, policy.Static(&policy.Policy{}), nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout, testutil.NopLogger()) w := httptest.NewRecorder() r := pipesRequest(t, http.MethodPost, "/v1/pipes/open/execute", "open", nil) r = r.WithContext(auth.WithRole(r.Context(), "admin")) - safeHandle(h.Execute, w, r) + safeHandle(h.Execute, w, withTenant(r)) assert.NotEqual(t, http.StatusForbidden, w.Code, "admin bypasses the allowlist on a pipe with no allowed_roles") diff --git a/internal/api/router.go b/internal/api/router.go index 39a9c239..6288135e 100644 --- a/internal/api/router.go +++ b/internal/api/router.go @@ -9,6 +9,8 @@ import ( "github.com/Wave-RF/WaveHouse/internal/auth" "github.com/Wave-RF/WaveHouse/internal/policy" + "github.com/Wave-RF/WaveHouse/internal/settings" + "github.com/Wave-RF/WaveHouse/internal/tenant" "github.com/go-chi/chi/v5" "github.com/go-chi/chi/v5/middleware" @@ -31,6 +33,10 @@ type Dependencies struct { // directory is configured (nothing to reload). Settings *SettingsHandler AuthMW func(http.Handler) http.Handler + // Tenants resolves the tenant.Header of every tenant route to that + // tenant's settings store (TenantMW), ahead of AuthMW. The probes, + // /version, the metrics path, and /v1/ops/* are tenant-exempt. + Tenants *settings.Registry // PolicySource backs the RequireAdmin gate: the admin role (policy.AdminRole) // is read live from the adopted policy, so admin_role changes apply on reload. PolicySource policy.Source @@ -120,45 +126,49 @@ func NewRouter(deps Dependencies) http.Handler { r.Method(http.MethodGet, deps.MetricsPath, deps.MetricsHandler) } - // API v1 endpoints. The JWT auth middleware always runs (no enable/disable switch). + // API v1 endpoints. The JWT auth middleware always runs (no enable/disable + // switch) on both halves: the tenant routes, which resolve their tenant + // first, and the tenant-exempt ops tree. r.Route("/v1", func(r chi.Router) { - r.Use(deps.AuthMW) + r.Group(func(r chi.Router) { + r.Use(TenantMW(deps.Tenants)) + r.Use(deps.AuthMW) - // Public content-free liveness ping. Lives under /v1 deliberately: - // it's documented API surface the SDK relies on to check "is this - // server reachable" before sending data, so it must stay public even - // in deployments that filter the bare /livez|/readyz|/healthz probe - // paths at the reverse proxy. AuthMW runs but never rejects, so no - // token is required and there's no authz gate. Mirrors /livez under - // the hood (200 past boot, 503 while degraded), no body. - r.Get("/health", deps.Health.Online) + // Public content-free liveness ping. Lives under /v1 deliberately: + // it's documented API surface the SDK relies on to check "is this + // server reachable" before sending data, so it must stay public even + // in deployments that filter the bare /livez|/readyz|/healthz probe + // paths at the reverse proxy. AuthMW runs but never rejects, so no + // token is required and there's no authz gate. Mirrors /livez under + // the hood (200 past boot, 503 while degraded), no body. + r.Get("/health", deps.Health.Online) - // Single admin gate for every admin-equivalent surface. The admin role - // is policy.AdminRole (configurable via admin_role, "admin" by default), - // read live from the policy store so changes apply without a restart. - // Declaring it once keeps the gate consistent across the tree. - requireAdmin := RequireAdmin(deps.PolicySource, deps.Logger) + r.Post("/ingest", deps.Ingest.Handle) + r.Get("/stream", deps.SSE.Handle) - r.Post("/ingest", deps.Ingest.Handle) - r.Get("/stream", deps.SSE.Handle) - - // Structured query endpoint. - if deps.StructuredQuery != nil { - r.Post("/query", deps.StructuredQuery.Handle) - } + // Structured query endpoint. + if deps.StructuredQuery != nil { + r.Post("/query", deps.StructuredQuery.Handle) + } - // Named query pipes. - if deps.Pipes != nil { - r.Get("/pipes/{name}", deps.Pipes.Execute) - r.Post("/pipes/{name}", deps.Pipes.Execute) - } + // Named query pipes. + if deps.Pipes != nil { + r.Get("/pipes/{name}", deps.Pipes.Execute) + r.Post("/pipes/{name}", deps.Pipes.Execute) + } + }) // Ops routes — every admin-gated surface lives under /v1/ops. The // requireAdmin gate covers the whole tree; every surface below — // including raw-SQL passthrough — shares the same admin principal // set (policy.AdminRole). r.Route("/ops", func(r chi.Router) { - r.Use(requireAdmin) + // Single admin gate for every admin-equivalent surface. The admin + // role is policy.AdminRole (configurable via admin_role, "admin" by + // default), read live from the policy store so changes apply + // without a restart. + r.Use(deps.AuthMW) + r.Use(RequireAdmin(deps.PolicySource, deps.Logger)) // Schema discovery. r.Get("/schema", deps.Schema.Get) @@ -342,7 +352,7 @@ func corsMiddleware(origins func() []string) func(http.Handler) http.Handler { w.Header().Set("Access-Control-Allow-Methods", "GET, POST, PUT, DELETE, OPTIONS") // Last-Event-ID lets a cross-origin SSE client resume a stream // (read by StreamHandler.Handle); without it the preflight fails. - w.Header().Set("Access-Control-Allow-Headers", "Accept, Authorization, Content-Type, Last-Event-ID, X-Request-ID") + w.Header().Set("Access-Control-Allow-Headers", "Accept, Authorization, Content-Type, Last-Event-ID, X-Request-ID, "+tenant.Header) w.Header().Set("Access-Control-Expose-Headers", "X-Cache, X-Request-ID") w.Header().Set("Access-Control-Max-Age", "3600") } diff --git a/internal/api/router_test.go b/internal/api/router_test.go index fc1337c6..e664025b 100644 --- a/internal/api/router_test.go +++ b/internal/api/router_test.go @@ -11,9 +11,9 @@ import ( "github.com/Wave-RF/WaveHouse/internal/auth" "github.com/Wave-RF/WaveHouse/internal/discovery" "github.com/Wave-RF/WaveHouse/internal/mq" - "github.com/Wave-RF/WaveHouse/internal/pipes" "github.com/Wave-RF/WaveHouse/internal/policy" "github.com/Wave-RF/WaveHouse/internal/stream" + "github.com/Wave-RF/WaveHouse/internal/tenant" "github.com/Wave-RF/WaveHouse/internal/testutil" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" @@ -173,6 +173,7 @@ func TestCORSMiddleware_Preflight(t *testing.T) { // Last-Event-ID is the SSE resumption header (issue #215): cross-origin // fetch-based stream clients that resume via it must clear preflight. assert.Contains(t, w.Header().Get("Access-Control-Allow-Headers"), "Last-Event-ID") + assert.Contains(t, w.Header().Get("Access-Control-Allow-Headers"), tenant.Header, "a browser client must be allowed to send the tenant header") } func TestCORSMiddleware_NormalRequest(t *testing.T) { @@ -321,13 +322,14 @@ func TestNewRouter_RoutesRegistered(t *testing.T) { {Name: "events", Columns: []discovery.Column{{Name: "id", Type: "String"}}}, }) pub := &testutil.MockPublisher{} - hub := stream.NewHub(nil, nil, nil) + hub := stream.NewHub(tenant.Default, nil, nil, nil) emb, err := mq.NewEmbedded(t.TempDir(), 1024*1024, testutil.NopLogger()) require.NoError(t, err) t.Cleanup(func() { _ = emb.Close() }) deps := Dependencies{ + Tenants: testTenants(), Ingest: NewIngestHandler(reg, pub, testutil.NopLogger()), Query: &QueryHandler{}, SSE: NewStreamHandler(hub, nil), @@ -335,7 +337,7 @@ func TestNewRouter_RoutesRegistered(t *testing.T) { Version: NewVersionHandler("test", "test", "test"), Schema: NewSchemaHandler(reg), DLQ: NewDLQHandler(emb, testutil.NopLogger()), - Pipes: NewPipesHandler(pipes.Static(), policy.Static(&policy.Policy{}), nil, nil, nil, testutil.NopLogger()), + Pipes: NewPipesHandler(staticPipes(), staticPolicy(&policy.Policy{}), nil, nil, nil, testutil.NopLogger()), AuthMW: func(next http.Handler) http.Handler { return next }, PolicySource: policy.Static(&policy.Policy{}), Logger: testutil.NopLogger(), @@ -411,8 +413,9 @@ func TestNewRouter_RoutesRegistered(t *testing.T) { func TestNewRouter_CORSOnStream(t *testing.T) { t.Parallel() - hub := stream.NewHub(nil, nil, nil) + hub := stream.NewHub(tenant.Default, nil, nil, nil) router := NewRouter(Dependencies{ + Tenants: testTenants(), SSE: NewStreamHandler(hub, nil), Health: &HealthHandler{}, AuthMW: func(next http.Handler) http.Handler { return next }, @@ -483,9 +486,10 @@ func TestNewRouter_RawSQLAdminGate(t *testing.T) { reg := testutil.NewTestSchemaRegistry(t, nil) pub := &testutil.MockPublisher{} - hub := stream.NewHub(nil, nil, nil) + hub := stream.NewHub(tenant.Default, nil, nil, nil) router := NewRouter(Dependencies{ + Tenants: testTenants(), Ingest: NewIngestHandler(reg, pub, testutil.NopLogger()), Query: &QueryHandler{}, SSE: NewStreamHandler(hub, nil), @@ -544,9 +548,10 @@ func TestNewRouter_OptionalDepsNil(t *testing.T) { reg := testutil.NewTestSchemaRegistry(t, nil) pub := &testutil.MockPublisher{} - hub := stream.NewHub(nil, nil, nil) + hub := stream.NewHub(tenant.Default, nil, nil, nil) deps := Dependencies{ + Tenants: testTenants(), Ingest: NewIngestHandler(reg, pub, testutil.NopLogger()), Query: &QueryHandler{}, SSE: NewStreamHandler(hub, nil), @@ -623,15 +628,16 @@ func TestNewRouter_NotFoundEmitsJSON(t *testing.T) { reg := testutil.NewTestSchemaRegistry(t, nil) pub := &testutil.MockPublisher{} - hub := stream.NewHub(nil, nil, nil) + hub := stream.NewHub(tenant.Default, nil, nil, nil) deps := Dependencies{ - Ingest: NewIngestHandler(reg, pub, testutil.NopLogger()), - Query: &QueryHandler{}, - SSE: NewStreamHandler(hub, nil), - Health: &HealthHandler{}, - Schema: NewSchemaHandler(reg), - AuthMW: func(next http.Handler) http.Handler { return next }, - Logger: testutil.NopLogger(), + Tenants: testTenants(), + Ingest: NewIngestHandler(reg, pub, testutil.NopLogger()), + Query: &QueryHandler{}, + SSE: NewStreamHandler(hub, nil), + Health: &HealthHandler{}, + Schema: NewSchemaHandler(reg), + AuthMW: func(next http.Handler) http.Handler { return next }, + Logger: testutil.NopLogger(), } router := NewRouter(deps) @@ -648,15 +654,16 @@ func TestNewRouter_MethodNotAllowedEmitsJSON(t *testing.T) { reg := testutil.NewTestSchemaRegistry(t, nil) pub := &testutil.MockPublisher{} - hub := stream.NewHub(nil, nil, nil) + hub := stream.NewHub(tenant.Default, nil, nil, nil) deps := Dependencies{ - Ingest: NewIngestHandler(reg, pub, testutil.NopLogger()), - Query: &QueryHandler{}, - SSE: NewStreamHandler(hub, nil), - Health: &HealthHandler{}, - Schema: NewSchemaHandler(reg), - AuthMW: func(next http.Handler) http.Handler { return next }, - Logger: testutil.NopLogger(), + Tenants: testTenants(), + Ingest: NewIngestHandler(reg, pub, testutil.NopLogger()), + Query: &QueryHandler{}, + SSE: NewStreamHandler(hub, nil), + Health: &HealthHandler{}, + Schema: NewSchemaHandler(reg), + AuthMW: func(next http.Handler) http.Handler { return next }, + Logger: testutil.NopLogger(), } router := NewRouter(deps) @@ -747,9 +754,10 @@ func TestNewRouter_SchemaAdminOnly(t *testing.T) { {Name: "events", Columns: []discovery.Column{{Name: "id", Type: "String"}}}, }) pub := &testutil.MockPublisher{} - hub := stream.NewHub(nil, nil, nil) + hub := stream.NewHub(tenant.Default, nil, nil, nil) router := NewRouter(Dependencies{ + Tenants: testTenants(), Ingest: NewIngestHandler(reg, pub, testutil.NopLogger()), Query: &QueryHandler{}, SSE: NewStreamHandler(hub, nil), diff --git a/internal/api/stream_test.go b/internal/api/stream_test.go index 88db1d99..d41351d4 100644 --- a/internal/api/stream_test.go +++ b/internal/api/stream_test.go @@ -11,6 +11,7 @@ import ( "time" "github.com/Wave-RF/WaveHouse/internal/stream" + "github.com/Wave-RF/WaveHouse/internal/tenant" "github.com/Wave-RF/WaveHouse/internal/testutil" "github.com/stretchr/testify/assert" ) @@ -22,7 +23,7 @@ import ( func TestSSE_RejectsMissingOrInvalidTable(t *testing.T) { t.Parallel() - h := &StreamHandler{Hub: stream.NewHub(nil, nil, nil)} + h := &StreamHandler{Hub: stream.NewHub(tenant.Default, nil, nil, nil)} cases := []struct { name string @@ -50,7 +51,7 @@ func TestSSE_RejectsMissingOrInvalidTable(t *testing.T) { func TestSSE_AcceptsSafeTableName(t *testing.T) { t.Parallel() - h := &StreamHandler{Hub: stream.NewHub(nil, nil, nil)} + h := &StreamHandler{Hub: stream.NewHub(tenant.Default, nil, nil, nil)} // Use a request context that's already cancelled so the handler exits // the live-stream select loop immediately instead of blocking the test. @@ -75,7 +76,7 @@ func TestSSE_EmitsHeartbeatsWhenIdle(t *testing.T) { hb := stream.NewHeartbeater(20*time.Millisecond, 1) go hb.Run(t.Context()) - h := &StreamHandler{Hub: stream.NewHub(nil, nil, nil), Heartbeater: hb} + h := &StreamHandler{Hub: stream.NewHub(tenant.Default, nil, nil, nil), Heartbeater: hb} ctx, cancel := context.WithCancel(context.Background()) req := httptest.NewRequestWithContext(ctx, http.MethodGet, "/v1/stream?table=clicks", nil) @@ -114,7 +115,7 @@ func TestSSE_WheelTickRacesHandlerTeardown(t *testing.T) { defer cancel() go hb.Run(ctx) - h := &StreamHandler{Hub: stream.NewHub(nil, nil, nil), Heartbeater: hb} + h := &StreamHandler{Hub: stream.NewHub(tenant.Default, nil, nil, nil), Heartbeater: hb} const conns = 40 var wg sync.WaitGroup diff --git a/internal/api/structured_query.go b/internal/api/structured_query.go index a31fde75..1486b9c9 100644 --- a/internal/api/structured_query.go +++ b/internal/api/structured_query.go @@ -15,6 +15,7 @@ import ( "github.com/Wave-RF/WaveHouse/internal/discovery" "github.com/Wave-RF/WaveHouse/internal/policy" "github.com/Wave-RF/WaveHouse/internal/query" + "github.com/Wave-RF/WaveHouse/internal/settings" "golang.org/x/sync/singleflight" ) @@ -23,22 +24,22 @@ type StructuredQueryHandler struct { CHConn driver.Conn Cache cache.Cache Registry *discovery.SchemaRegistry - PolicySource policy.Source + PolicySource PolicySource sf singleflight.Group // queryTimeout bounds each query, read per request // (chconn.Manager.QueryTimeout in production) so a settings reload // applies without a restart. queryTimeout func() time.Duration - // bucketSecs returns the current time-range bucket - // (settings.Store.TimestampBucketSeconds in production) and defaultMaxRows - // the current fallback result LIMIT (settings.Store.DefaultMaxRows) — - // funcs, not ints, so a settings reload takes effect on the next query - // without a restart. A nil bucketSecs means no bucketing; a nil + // bucketSecs returns the request tenant's current time-range bucket + // ((*settings.Store).TimestampBucketSeconds in production) and + // defaultMaxRows its current fallback result LIMIT + // ((*settings.Store).DefaultMaxRows) — funcs, not ints, so a settings + // reload takes effect on the next query without a restart. A nil bucketSecs means no bucketing; a nil // defaultMaxRows or a non-positive return means the builder's compiled // constant. - bucketSecs func() int - defaultMaxRows func() int + bucketSecs func(*settings.Store) int + defaultMaxRows func(*settings.Store) int logger *slog.Logger // maxRequestBytes optionally overrides the default inbound request body @@ -53,10 +54,10 @@ func NewStructuredQueryHandler( conn driver.Conn, c cache.Cache, registry *discovery.SchemaRegistry, - policyStore policy.Source, - bucketSecs func() int, + policyStore PolicySource, + bucketSecs func(*settings.Store) int, queryTimeout func() time.Duration, - defaultMaxRows func() int, + defaultMaxRows func(*settings.Store) int, logger *slog.Logger, ) *StructuredQueryHandler { return &StructuredQueryHandler{ @@ -72,6 +73,10 @@ func NewStructuredQueryHandler( } func (h *StructuredQueryHandler) Handle(w http.ResponseWriter, r *http.Request) { + store, ok := requestStore(w, r) + if !ok { + return + } table := r.URL.Query().Get("table") if table == "" { writeJSONError(w, http.StatusBadRequest, "missing table") @@ -105,7 +110,7 @@ func (h *StructuredQueryHandler) Handle(w http.ResponseWriter, r *http.Request) } // Resolve permissions. - p := h.PolicySource() + p := h.PolicySource(store) role := policy.ResolveRole(p, auth.RoleFromContext(r.Context())) claims, _ := auth.ClaimsFromContext(r.Context()) perms := policy.Evaluate(p, role, table, "select", claims) @@ -128,11 +133,11 @@ func (h *StructuredQueryHandler) Handle(w http.ResponseWriter, r *http.Request) // map to 403; a malformed query maps to 400. maxRows := 0 // non-positive → the builder's compiled constant if h.defaultMaxRows != nil { - maxRows = h.defaultMaxRows() + maxRows = h.defaultMaxRows(store) } bucketSecs := 0 if h.bucketSecs != nil { - bucketSecs = h.bucketSecs() + bucketSecs = h.bucketSecs(store) } result, err := query.Build(table, &sq, schema, perms, bucketSecs, maxRows) if err != nil { diff --git a/internal/api/structured_query_test.go b/internal/api/structured_query_test.go index 733140ed..223eda1e 100644 --- a/internal/api/structured_query_test.go +++ b/internal/api/structured_query_test.go @@ -15,6 +15,7 @@ import ( "github.com/Wave-RF/WaveHouse/internal/discovery" "github.com/Wave-RF/WaveHouse/internal/policy" "github.com/Wave-RF/WaveHouse/internal/query" + "github.com/Wave-RF/WaveHouse/internal/settings" "github.com/Wave-RF/WaveHouse/internal/testutil" "github.com/golang-jwt/jwt/v5" "github.com/stretchr/testify/assert" @@ -40,7 +41,7 @@ func newStructuredQueryHandler(t testing.TB) *StructuredQueryHandler { }, }, }) - return NewStructuredQueryHandler(nil, nil, reg, nil, func() int { return 60 }, func() time.Duration { return 5 * time.Second }, nil, testutil.NopLogger()) + return NewStructuredQueryHandler(nil, nil, reg, nil, func(*settings.Store) int { return 60 }, func() time.Duration { return 5 * time.Second }, nil, testutil.NopLogger()) } func TestStructuredQuery_MissingTable(t *testing.T) { @@ -89,7 +90,7 @@ func TestStructuredQuery_MissingTable(t *testing.T) { ) w := httptest.NewRecorder() - h.Handle(w, req) + h.Handle(w, withTenant(req)) assert.Equal(t, http.StatusBadRequest, w.Code) assert.Contains(t, w.Body.String(), "missing table") @@ -103,7 +104,7 @@ func TestStructuredQuery_UnknownTable(t *testing.T) { h := newStructuredQueryHandler(t) r := structuredQueryRequest(t, "nope", query.StructuredQuery{Columns: []string{"x"}}) w := httptest.NewRecorder() - h.Handle(w, r) + h.Handle(w, withTenant(r)) assert.Equal(t, http.StatusNotFound, w.Code) assert.Contains(t, w.Body.String(), "unknown table") @@ -115,7 +116,7 @@ func TestStructuredQuery_InvalidJSON(t *testing.T) { h := newStructuredQueryHandler(t) r := httptest.NewRequestWithContext(context.Background(), http.MethodPost, "/v1/query?table=clicks", bytes.NewReader([]byte(`{bad}`))) w := httptest.NewRecorder() - h.Handle(w, r) + h.Handle(w, withTenant(r)) assert.Equal(t, http.StatusBadRequest, w.Code) assert.Contains(t, w.Body.String(), "invalid json") @@ -145,7 +146,7 @@ func TestStructuredQuery_RequestBodyCap(t *testing.T) { r := httptest.NewRequestWithContext(context.Background(), http.MethodPost, "/v1/query?table=clicks", bytes.NewReader(body)) w := httptest.NewRecorder() - h.Handle(w, r) + h.Handle(w, withTenant(r)) require.Equal(t, http.StatusRequestEntityTooLarge, w.Code, "oversized request must 413, not 400") assert.Contains(t, w.Body.String(), "request body exceeded") @@ -162,7 +163,7 @@ func TestStructuredQuery_PolicyForbidden(t *testing.T) { }, } h := newStructuredQueryHandler(t) - h.PolicySource = policy.Static(p) + h.PolicySource = staticPolicy(p) sq := query.StructuredQuery{Columns: []string{"page"}} r := structuredQueryRequest(t, "clicks", sq) @@ -171,7 +172,7 @@ func TestStructuredQuery_PolicyForbidden(t *testing.T) { r = r.WithContext(ctx) w := httptest.NewRecorder() - h.Handle(w, r) + h.Handle(w, withTenant(r)) assert.Equal(t, http.StatusForbidden, w.Code) assert.Contains(t, w.Body.String(), "forbidden") @@ -188,7 +189,7 @@ func TestStructuredQuery_ColumnNotAllowed(t *testing.T) { }, } h := newStructuredQueryHandler(t) - h.PolicySource = policy.Static(p) + h.PolicySource = staticPolicy(p) // Request "count" column which is not in AllowColumns. sq := query.StructuredQuery{Columns: []string{"count"}} @@ -198,7 +199,7 @@ func TestStructuredQuery_ColumnNotAllowed(t *testing.T) { r = r.WithContext(ctx) w := httptest.NewRecorder() - h.Handle(w, r) + h.Handle(w, withTenant(r)) assert.Equal(t, http.StatusForbidden, w.Code) assert.Contains(t, w.Body.String(), "column") @@ -219,7 +220,7 @@ func TestStructuredQuery_AggregationNotAllowed(t *testing.T) { }, } h := newStructuredQueryHandler(t) - h.PolicySource = policy.Static(p) + h.PolicySource = staticPolicy(p) sq := query.StructuredQuery{ Aggregations: []query.Aggregation{ @@ -232,7 +233,7 @@ func TestStructuredQuery_AggregationNotAllowed(t *testing.T) { r = r.WithContext(ctx) w := httptest.NewRecorder() - h.Handle(w, r) + h.Handle(w, withTenant(r)) assert.Equal(t, http.StatusForbidden, w.Code) assert.Contains(t, w.Body.String(), "aggregation") @@ -246,11 +247,11 @@ func TestStructuredQuery_NilPolicyFailsClosed(t *testing.T) { // An adopted-but-empty policies.json yields a nil policy: total lockout, // nobody passes (AGENTS.md invariant 11). A PolicySource is always wired // in production; this pins the value it returns, not its absence. - h.PolicySource = policy.Static(nil) + h.PolicySource = staticPolicy(nil) sq := query.StructuredQuery{Columns: []string{"page"}} r := structuredQueryRequest(t, "clicks", sq) w := httptest.NewRecorder() - h.Handle(w, r) + h.Handle(w, withTenant(r)) assert.Equal(t, http.StatusForbidden, w.Code) } @@ -289,7 +290,7 @@ func newCapturingHandler(t *testing.T, conn driver.Conn, p *policy.Policy) *Stru }, }, }) - return NewStructuredQueryHandler(conn, nil, reg, policy.Static(p), func() int { return 60 }, func() time.Duration { return 5 * time.Second }, nil, testutil.NopLogger()) + return NewStructuredQueryHandler(conn, nil, reg, staticPolicy(p), func(*settings.Store) int { return 60 }, func() time.Duration { return 5 * time.Second }, nil, testutil.NopLogger()) } func viewerRequest(t *testing.T, sq query.StructuredQuery) *http.Request { @@ -317,7 +318,7 @@ func TestStructuredQuery_SelectAll_RestrictedRoleGetsAllowedProjection(t *testin h := newCapturingHandler(t, conn, policyWithViewer(policy.SelectPermissions{AllowColumns: []string{"page", "ts"}})) w := httptest.NewRecorder() - h.Handle(w, viewerRequest(t, query.StructuredQuery{SelectAll: true})) + h.Handle(w, withTenant(viewerRequest(t, query.StructuredQuery{SelectAll: true}))) require.Equal(t, http.StatusOK, w.Code, "body=%s", w.Body.String()) assert.Equal(t, "SELECT `page`, `ts` FROM `clicks` LIMIT 10000", conn.lastSQL) @@ -349,7 +350,7 @@ func TestStructuredQuery_RowFilterAndMaxRows_ReachClickHouse(t *testing.T) { ctx := auth.WithClaims(auth.WithRole(r.Context(), "viewer"), jwt.MapClaims{"org_id": "org-1"}) w := httptest.NewRecorder() - h.Handle(w, r.WithContext(ctx)) + h.Handle(w, withTenant(r.WithContext(ctx))) require.Equal(t, http.StatusOK, w.Code, "body=%s", w.Body.String()) assert.Equal(t, "SELECT `page` FROM `clicks` WHERE (`user_id` = ?) AND `page` = ? LIMIT 100", conn.lastSQL) @@ -366,7 +367,7 @@ func TestStructuredQuery_OmittedColumns_ReturnsNothing(t *testing.T) { h := newCapturingHandler(t, conn, policyWithViewer(policy.SelectPermissions{AllowColumns: []string{"page", "ts"}})) w := httptest.NewRecorder() - h.Handle(w, viewerRequest(t, query.StructuredQuery{})) + h.Handle(w, withTenant(viewerRequest(t, query.StructuredQuery{}))) require.Equal(t, http.StatusOK, w.Code, "body=%s", w.Body.String()) assert.JSONEq(t, "[]", w.Body.String()) @@ -381,7 +382,7 @@ func TestStructuredQuery_SelectAll_DenyListExpands(t *testing.T) { h := newCapturingHandler(t, conn, policyWithViewer(policy.SelectPermissions{DenyColumns: []string{"payload"}})) w := httptest.NewRecorder() - h.Handle(w, viewerRequest(t, query.StructuredQuery{SelectAll: true})) + h.Handle(w, withTenant(viewerRequest(t, query.StructuredQuery{SelectAll: true}))) require.Equal(t, http.StatusOK, w.Code, "body=%s", w.Body.String()) assert.Equal(t, "SELECT `page`, `user_id`, `ts` FROM `clicks` LIMIT 10000", conn.lastSQL) @@ -397,7 +398,7 @@ func TestStructuredQuery_LiteralStarColumn_Unknown(t *testing.T) { h := newCapturingHandler(t, conn, policyWithViewer(policy.SelectPermissions{AllowColumns: []string{"*"}})) w := httptest.NewRecorder() - h.Handle(w, viewerRequest(t, query.StructuredQuery{Columns: []string{"*"}})) + h.Handle(w, withTenant(viewerRequest(t, query.StructuredQuery{Columns: []string{"*"}}))) require.Equal(t, http.StatusBadRequest, w.Code, "body=%s", w.Body.String()) assert.Empty(t, conn.lastSQL) @@ -412,7 +413,7 @@ func TestStructuredQuery_UnrestrictedRoleKeepsSelectStar(t *testing.T) { h := newCapturingHandler(t, conn, policyWithViewer(policy.SelectPermissions{AllowColumns: []string{"*"}})) w := httptest.NewRecorder() - h.Handle(w, viewerRequest(t, query.StructuredQuery{SelectAll: true})) + h.Handle(w, withTenant(viewerRequest(t, query.StructuredQuery{SelectAll: true}))) require.Equal(t, http.StatusOK, w.Code, "body=%s", w.Body.String()) assert.Equal(t, "SELECT * FROM `clicks` LIMIT 10000", conn.lastSQL) @@ -458,7 +459,7 @@ func TestStructuredQuery_DeniedColumnInAnyClause_Returns403(t *testing.T) { h := newCapturingHandler(t, conn, policyWithViewer(policy.SelectPermissions{AllowColumns: []string{"page", "ts"}})) w := httptest.NewRecorder() - h.Handle(w, viewerRequest(t, tt.sq)) + h.Handle(w, withTenant(viewerRequest(t, tt.sq))) assert.Equal(t, http.StatusForbidden, w.Code, "body=%s", w.Body.String()) assert.Contains(t, w.Body.String(), "not allowed") @@ -477,7 +478,7 @@ func TestStructuredQuery_NoReadableColumns_Returns403(t *testing.T) { h := newCapturingHandler(t, conn, policyWithViewer(policy.SelectPermissions{AllowColumns: []string{"nonexistent"}})) w := httptest.NewRecorder() - h.Handle(w, viewerRequest(t, query.StructuredQuery{SelectAll: true})) + h.Handle(w, withTenant(viewerRequest(t, query.StructuredQuery{SelectAll: true}))) assert.Equal(t, http.StatusForbidden, w.Code, "body=%s", w.Body.String()) assert.Empty(t, conn.lastSQL) @@ -497,7 +498,7 @@ func TestStructuredQuery_UnauthenticatedUsesDefaultRoleProjection(t *testing.T) // No role on the context — a tokenless request. r := structuredQueryRequest(t, "clicks", query.StructuredQuery{SelectAll: true, Limit: 2}) w := httptest.NewRecorder() - h.Handle(w, r) + h.Handle(w, withTenant(r)) require.Equal(t, http.StatusOK, w.Code, "body=%s", w.Body.String()) assert.Equal(t, "SELECT `page` FROM `clicks` LIMIT 2", conn.lastSQL) @@ -524,7 +525,7 @@ func TestStructuredQuery_CacheKeyIsolatesColumnVisibility(t *testing.T) { r := structuredQueryRequest(t, "clicks", query.StructuredQuery{SelectAll: true}) r = r.WithContext(auth.WithClaims(auth.WithRole(r.Context(), role), jwt.MapClaims{})) w := httptest.NewRecorder() - h.Handle(w, r) + h.Handle(w, withTenant(r)) require.Equal(t, http.StatusOK, w.Code, "role=%s body=%s", role, w.Body.String()) return conn.lastSQL } diff --git a/internal/api/tenant.go b/internal/api/tenant.go new file mode 100644 index 00000000..e608ec44 --- /dev/null +++ b/internal/api/tenant.go @@ -0,0 +1,79 @@ +package api + +import ( + "context" + "log/slog" + "net/http" + + "github.com/Wave-RF/WaveHouse/internal/policy" + "github.com/Wave-RF/WaveHouse/internal/settings" + "github.com/Wave-RF/WaveHouse/internal/tenant" +) + +// PolicySource yields the access-control policy of the request's tenant, +// read per request so a settings reload applies to the next one +// ((*settings.Store).Policy in production). A nil policy from a wired source +// is a deliberate lockout; a nil source means policy filtering is not wired +// at all, which only tests do. +type PolicySource func(*settings.Store) *policy.Policy + +type tenantStoreKey struct{} + +// WithStore returns ctx carrying the request's resolved tenant store. These +// helpers live here rather than in internal/tenant because settings names +// tenant.ID, so tenant cannot name settings.Store back. +func WithStore(ctx context.Context, store *settings.Store) context.Context { + return context.WithValue(ctx, tenantStoreKey{}, store) +} + +// StoreFromContext returns the store TenantMW resolved for this request. +// Handlers call it once and pass the store down as an argument; nothing +// below a handler reads it from the context. +func StoreFromContext(ctx context.Context) (*settings.Store, bool) { + store, _ := ctx.Value(tenantStoreKey{}).(*settings.Store) + return store, store != nil +} + +// requestStore is a handler's single read of the tenant store. A request +// that reaches a tenant route without one skipped TenantMW — a routing bug, +// answered with a 500 rather than by serving some other tenant's settings. +func requestStore(w http.ResponseWriter, r *http.Request) (*settings.Store, bool) { + store, ok := StoreFromContext(r.Context()) + if !ok { + slog.ErrorContext(r.Context(), "tenant route reached without a resolved tenant", "path", r.URL.Path) + writeJSONError(w, http.StatusInternalServerError, "internal server error") + } + return store, ok +} + +// TenantMW resolves the request's tenant before authentication runs: the +// tenant.Header value, tenant.Default when absent. A malformed id is a 400 +// and a well-formed id the registry does not hold is a 404. A repeated +// header is refused rather than picked from, so a value a proxy sets can +// never be shadowed by one the client sent. +func TenantMW(tenants *settings.Registry) func(http.Handler) http.Handler { + return func(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + id := tenant.Default + values := r.Header.Values(tenant.Header) + if len(values) > 1 { + writeJSONError(w, http.StatusBadRequest, "invalid "+tenant.Header+": sent more than once") + return + } + if len(values) == 1 && values[0] != "" { + parsed, err := tenant.Parse(values[0]) + if err != nil { + writeJSONError(w, http.StatusBadRequest, "invalid "+tenant.Header+": "+err.Error()) + return + } + id = parsed + } + store, ok := tenants.For(id) + if !ok { + writeJSONError(w, http.StatusNotFound, "unknown tenant: "+id.String()) + return + } + next.ServeHTTP(w, r.WithContext(WithStore(r.Context(), store))) + }) + } +} diff --git a/internal/api/tenant_helpers_test.go b/internal/api/tenant_helpers_test.go new file mode 100644 index 00000000..4f991efd --- /dev/null +++ b/internal/api/tenant_helpers_test.go @@ -0,0 +1,34 @@ +package api + +import ( + "net/http" + + "github.com/Wave-RF/WaveHouse/internal/pipes" + "github.com/Wave-RF/WaveHouse/internal/policy" + "github.com/Wave-RF/WaveHouse/internal/settings" +) + +// testStore stands in for the store TenantMW resolves. It holds no document: +// handler tests inject fixed getters that ignore it, so a handler that read +// it directly would panic rather than pass. +var testStore = &settings.Store{} + +// withTenant attaches testStore to r the way TenantMW would, for tests that +// call a tenant-route handler without the router. +func withTenant(r *http.Request) *http.Request { + return r.WithContext(WithStore(r.Context(), testStore)) +} + +// testTenants is a registry whose default tenant is testStore. +func testTenants() *settings.Registry { return settings.NewRegistry(testStore) } + +// staticPolicy is a PolicySource fixed to p, whatever the tenant. +func staticPolicy(p *policy.Policy) PolicySource { + return func(*settings.Store) *policy.Policy { return p } +} + +// staticPipes is a PipesHandler source fixed to queries, whatever the tenant. +func staticPipes(queries ...*pipes.NamedQuery) func(*settings.Store) pipes.Source { + src := pipes.Static(queries...) + return func(*settings.Store) pipes.Source { return src } +} diff --git a/internal/api/tenant_test.go b/internal/api/tenant_test.go new file mode 100644 index 00000000..f34c84c8 --- /dev/null +++ b/internal/api/tenant_test.go @@ -0,0 +1,164 @@ +package api + +import ( + "net/http" + "net/http/httptest" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/Wave-RF/WaveHouse/internal/policy" + "github.com/Wave-RF/WaveHouse/internal/settings" + "github.com/Wave-RF/WaveHouse/internal/stream" + "github.com/Wave-RF/WaveHouse/internal/tenant" + "github.com/Wave-RF/WaveHouse/internal/testutil" +) + +func TestTenantMW(t *testing.T) { + t.Parallel() + tests := []struct { + name string + header []string // every X-Tenant-ID line sent + wantStatus int + wantBody string + }{ + {name: "no header resolves to the default tenant", wantStatus: http.StatusOK}, + {name: "empty header resolves to the default tenant", header: []string{""}, wantStatus: http.StatusOK}, + {name: "explicit default tenant", header: []string{"0"}, wantStatus: http.StatusOK}, + {name: "well-formed unknown tenant", header: []string{"acme"}, wantStatus: http.StatusNotFound, wantBody: "unknown tenant: acme"}, + {name: "malformed id", header: []string{"../etc"}, wantStatus: http.StatusBadRequest, wantBody: "invalid X-Tenant-ID"}, + {name: "subject wildcard", header: []string{">"}, wantStatus: http.StatusBadRequest, wantBody: "invalid X-Tenant-ID"}, + {name: "over the length cap", header: []string{strings.Repeat("a", tenant.MaxLen+1)}, wantStatus: http.StatusBadRequest, wantBody: "invalid X-Tenant-ID"}, + {name: "repeated header, even agreeing", header: []string{"0", "0"}, wantStatus: http.StatusBadRequest, wantBody: "sent more than once"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + var resolved *settings.Store + h := TenantMW(testTenants())(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + resolved, _ = StoreFromContext(r.Context()) + w.WriteHeader(http.StatusOK) + })) + req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/v1/health", nil) + for _, v := range tt.header { + req.Header.Add(tenant.Header, v) + } + w := httptest.NewRecorder() + h.ServeHTTP(w, req) + + require.Equal(t, tt.wantStatus, w.Code, "body: %s", w.Body.String()) + if tt.wantStatus == http.StatusOK { + assert.Same(t, testStore, resolved, "the resolved store rides the request context") + return + } + assert.Nil(t, resolved, "a refused request must not reach the handler") + assert.Contains(t, w.Body.String(), tt.wantBody) + assert.Equal(t, "application/json", w.Header().Get("Content-Type")) + }) + } +} + +func TestStoreFromContext_Absent(t *testing.T) { + t.Parallel() + store, ok := StoreFromContext(httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil).Context()) + assert.False(t, ok) + assert.Nil(t, store) +} + +// A tenant-route handler served without TenantMW must fail closed, never +// fall back to some tenant's settings. +func TestTenantRouteHandlers_NoResolvedTenantIs500(t *testing.T) { + t.Parallel() + reg := testRegistry(t) + handlers := map[string]http.HandlerFunc{ + "ingest": NewIngestHandler(reg, &testutil.MockPublisher{}, testutil.NopLogger()).Handle, + "structured query": newStructuredQueryHandler(t).Handle, + "pipe execute": NewPipesHandler(staticPipes(), nil, nil, nil, noTimeout, testutil.NopLogger()).Execute, + } + for name, handle := range handlers { + t.Run(name, func(t *testing.T) { + t.Parallel() + w := httptest.NewRecorder() + handle(w, httptest.NewRequestWithContext(t.Context(), http.MethodPost, "/v1/x?table=clicks", strings.NewReader(`{}`))) + testutil.AssertJSONContains(t, w, http.StatusInternalServerError, map[string]any{"error": "internal server error"}) + }) + } +} + +// tenantProbeRouter is a router whose AuthMW records whether the tenant +// store was already resolved when authentication ran. +func tenantProbeRouter(t *testing.T, sawStore *[]bool) http.Handler { + t.Helper() + reg := testRegistry(t) + return NewRouter(Dependencies{ + Tenants: testTenants(), + Ingest: NewIngestHandler(reg, &testutil.MockPublisher{}, testutil.NopLogger()), + Query: &QueryHandler{}, + SSE: NewStreamHandler(stream.NewHub(tenant.Default, nil, nil, nil), nil), + Health: &HealthHandler{}, + Version: NewVersionHandler("test", "test", "test"), + Schema: NewSchemaHandler(reg), + AuthMW: func(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, ok := StoreFromContext(r.Context()) + *sawStore = append(*sawStore, ok) + next.ServeHTTP(w, r) + }) + }, + PolicySource: policy.Static(&policy.Policy{}), + Logger: testutil.NopLogger(), + }) +} + +func TestNewRouter_TenantResolvesBeforeAuth(t *testing.T) { + t.Parallel() + var sawStore []bool + router := tenantProbeRouter(t, &sawStore) + + w := httptest.NewRecorder() + router.ServeHTTP(w, httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/v1/health", nil)) + require.Equal(t, http.StatusOK, w.Code) + assert.Equal(t, []bool{true}, sawStore, "AuthMW must run after the tenant is resolved") + + // An unknown tenant is refused before authentication ever runs. + sawStore = nil + req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/v1/health", nil) + req.Header.Set(tenant.Header, "acme") + w = httptest.NewRecorder() + router.ServeHTTP(w, req) + assert.Equal(t, http.StatusNotFound, w.Code) + assert.Empty(t, sawStore) +} + +// The probes, /version, and /v1/ops/* never look at the tenant header: a +// value that would 404 or 400 on a tenant route changes nothing there, and +// the ops tree stays behind AuthMW and the admin gate. +func TestNewRouter_TenantExemptRoutes(t *testing.T) { + t.Parallel() + for _, header := range []string{"acme", "../etc"} { + for _, path := range []string{"/livez", "/readyz", "/healthz", "/version"} { + t.Run(header+" "+path, func(t *testing.T) { + t.Parallel() + var sawStore []bool + req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, path, nil) + req.Header.Set(tenant.Header, header) + w := httptest.NewRecorder() + tenantProbeRouter(t, &sawStore).ServeHTTP(w, req) + assert.Equal(t, http.StatusOK, w.Code) + }) + } + t.Run(header+" /v1/ops/schema", func(t *testing.T) { + t.Parallel() + var sawStore []bool + req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/v1/ops/schema", nil) + req.Header.Set(tenant.Header, header) + w := httptest.NewRecorder() + tenantProbeRouter(t, &sawStore).ServeHTTP(w, req) + // Roleless, so the admin gate answers — not the tenant middleware. + assert.Equal(t, http.StatusForbidden, w.Code, "body: %s", w.Body.String()) + assert.Equal(t, []bool{false}, sawStore, "ops runs AuthMW with no tenant resolved") + }) + } +} diff --git a/internal/app/app.go b/internal/app/app.go index 7651b4d4..3c8b21df 100644 --- a/internal/app/app.go +++ b/internal/app/app.go @@ -86,7 +86,11 @@ type App struct { logLevel *slog.LevelVar listener net.Listener + // store is the default tenant's settings, which the process-wide + // resources (ClickHouse, dedupe, MQ, auth, CORS, reload) still follow; + // tenants is the registry every tenant-aware path resolves through. store *settings.Store + tenants *settings.Registry policies policy.Source promHandler http.Handler ch *chconn.Manager diff --git a/internal/app/app_test.go b/internal/app/app_test.go index 88ba47ba..dec5bbc5 100644 --- a/internal/app/app_test.go +++ b/internal/app/app_test.go @@ -24,6 +24,7 @@ import ( "github.com/Wave-RF/WaveHouse/internal/config" "github.com/Wave-RF/WaveHouse/internal/mq" "github.com/Wave-RF/WaveHouse/internal/settings" + "github.com/Wave-RF/WaveHouse/internal/tenant" ) // None of these tests run in parallel: New installs a process-wide default @@ -143,6 +144,36 @@ func TestNew_DegradedBootServesDiagnostics(t *testing.T) { assert.NoError(t, a.Close(context.Background()), "Close is idempotent") } +// The wired registry holds the default tenant only: no header and "0" reach +// the route, any other well-formed id is a 404, a malformed one a 400, and +// the ops tree never looks at the header. A 503 is the handler's own answer — +// boot is degraded without ClickHouse — so it proves the tenant resolved. +func TestNew_TenantHeaderResolvesAgainstTheRegistry(t *testing.T) { + a := newApp(t, testConfig(t, writeSettings(t, nil)), Options{}) + + tests := []struct { + name, path, header string + want int + }{ + {name: "no header", path: "/v1/health", want: http.StatusServiceUnavailable}, + {name: "default tenant", path: "/v1/health", header: "0", want: http.StatusServiceUnavailable}, + {name: "unknown tenant", path: "/v1/health", header: "acme", want: http.StatusNotFound}, + {name: "malformed tenant", path: "/v1/health", header: "a.b", want: http.StatusBadRequest}, + {name: "ops ignores the header", path: "/v1/ops/schema", header: "acme", want: http.StatusForbidden}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, tt.path, nil) + if tt.header != "" { + req.Header.Set(tenant.Header, tt.header) + } + rec := httptest.NewRecorder() + a.Handler().ServeHTTP(rec, req) + assert.Equal(t, tt.want, rec.Code, "body: %s", rec.Body.String()) + }) + } +} + func TestNew_DedupeFollowsSettings(t *testing.T) { tests := []struct { name string diff --git a/internal/app/wire.go b/internal/app/wire.go index 55f1963a..81401975 100644 --- a/internal/app/wire.go +++ b/internal/app/wire.go @@ -25,9 +25,11 @@ import ( "github.com/Wave-RF/WaveHouse/internal/ingest" "github.com/Wave-RF/WaveHouse/internal/mq" "github.com/Wave-RF/WaveHouse/internal/observability" + "github.com/Wave-RF/WaveHouse/internal/pipes" "github.com/Wave-RF/WaveHouse/internal/policy" "github.com/Wave-RF/WaveHouse/internal/settings" "github.com/Wave-RF/WaveHouse/internal/stream" + "github.com/Wave-RF/WaveHouse/internal/tenant" ) const ( @@ -57,6 +59,7 @@ func (a *App) wireSettings() error { return fmt.Errorf("settings directory %s invalid, refusing to start — findings above; `wavehouse validate` reproduces them, `wavehouse bootstrap` writes a starter directory", a.cfg.Settings.Dir) } a.store = store + a.tenants = settings.NewRegistry(store) a.policies = policy.Source(store.Policy) if store.Policy() == nil { slog.Warn("no policy adopted — every token-based request is denied until policies.json defines one (fail closed)") @@ -72,6 +75,22 @@ func (a *App) wireSettings() error { // malformed header is logged and skipped by the SDK (fail-soft); // InitProvider's own error is likewise non-fatal — logged, stdout-only from // there on. +// perTenant adapts a store accessor to the tenant-keyed getter the async +// paths take: they hold a tenant id (tenant.Default today, the MQ subject's +// from #583 story 5), not a request's resolved store. Only tenant.Default +// exists, so a miss is a wiring bug and reads as T's zero value; what a +// removed tenant means to each async path is story 3's to decide. +func perTenant[T any](tenants *settings.Registry, get func(*settings.Store) T) func(tenant.ID) T { + return func(id tenant.ID) T { + store, ok := tenants.For(id) + if !ok { + var zero T + return zero + } + return get(store) + } +} + func (a *App) wireObservability(ctx context.Context) { cfg := a.cfg if !cfg.OTel.Enabled && !cfg.Prometheus.Enabled { @@ -162,7 +181,7 @@ func (a *App) wireDiscovery(ctx context.Context) { a.bootState = api.NewBootState(nil) // Both sources are read per refresh, so a settings reload retunes the // cadence and a ClickHouse reconfigure moves the database without a restart. - registry := discovery.NewSchemaRegistry(a.ch, a.ch.Database, a.store.SchemaRefreshInterval, slog.Default()) + registry := discovery.NewSchemaRegistry(a.ch, a.ch.Database, tenant.Default, perTenant(a.tenants, (*settings.Store).SchemaRefreshInterval), slog.Default()) a.registry = registry bootErr := registry.Refresh(ctx) if bootErr != nil { @@ -284,7 +303,7 @@ func (a *App) wireCache() error { // written to ClickHouse and older than the SSE gap window // (stream.gap_window_minutes, re-read every sweep). Runs every minute. func (a *App) wireSweeper() { - sweeper := ingest.NewSweeper(a.mq, a.store.GapWindow, slog.Default()) + sweeper := ingest.NewSweeper(a.mq, tenant.Default, perTenant(a.tenants, (*settings.Store).GapWindow), slog.Default()) a.add(component{name: "sweeper", run: func(ctx context.Context) error { sweeper.Start(ctx) return nil @@ -297,7 +316,7 @@ func (a *App) wireSweeper() { // that role's subscribers; the MQ → Hub bridge; and the keepalive wheel. func (a *App) wireStreaming() { a.sseMetrics = stream.NewMetrics() - a.hub = stream.NewHub(a.policies, a.registry, a.sseMetrics) + a.hub = stream.NewHub(tenant.Default, perTenant(a.tenants, (*settings.Store).Policy), a.registry, a.sseMetrics) // Hub bridge: MQ → broadcast to connected SSE clients. The Hub decodes and // projects each event itself (skipping malformed payloads), so the bridge @@ -335,7 +354,11 @@ func (a *App) wireStreaming() { // drain within the shutdown timeout. func (a *App) wireIngestWorker() { a.add(component{name: "ingest worker", run: func(ctx context.Context) error { - stop, failed, err := ingest.StartIngestWorker(ctx, a.mq, a.cache, a.ch.Target, a.store.DLQFor) + dlqEnabled := func(id tenant.ID, table string) bool { + store, ok := a.tenants.For(id) + return ok && store.DLQFor(table) + } + stop, failed, err := ingest.StartIngestWorker(ctx, a.mq, a.cache, a.ch.Target, tenant.Default, dlqEnabled) if err != nil { return err } @@ -457,9 +480,9 @@ func (a *App) wireHTTP(authMW func(http.Handler) http.Handler) { logger := slog.Default() ingestHandler := api.NewIngestHandler(a.registry, a.mq, logger) - ingestHandler.PolicySource = a.policies + ingestHandler.PolicySource = (*settings.Store).Policy ingestHandler.Dedup = a.dedup - ingestHandler.DedupeSettings = a.store.DedupeFor + ingestHandler.DedupeSettings = (*settings.Store).DedupeFor healthHandler := api.NewHealthHandler(a.ch) healthHandler.Boot = a.bootState @@ -472,6 +495,9 @@ func (a *App) wireHTTP(authMW func(http.Handler) http.Handler) { closing := make(chan struct{}) streamHandler.Closing = closing + pipesHandler := api.NewPipesHandler(func(s *settings.Store) pipes.Source { return s }, (*settings.Store).Policy, a.ch, a.cache, a.ch.QueryTimeout, logger) + pipesHandler.OpsStore = a.store + deps := api.Dependencies{ Ingest: ingestHandler, // /v1/ops/query proxies straight to ClickHouse over HTTP — no native @@ -483,10 +509,11 @@ func (a *App) wireHTTP(authMW func(http.Handler) http.Handler) { Version: api.NewVersionHandler(a.build.Version, a.build.GitCommit, a.build.BuildTime), Schema: api.NewSchemaHandler(a.registry), DLQ: api.NewDLQHandler(a.mq, logger), - Pipes: api.NewPipesHandler(a.store, a.policies, a.ch, a.cache, a.ch.QueryTimeout, logger), - StructuredQuery: api.NewStructuredQueryHandler(a.ch, a.cache, a.registry, a.policies, a.store.TimestampBucketSeconds, a.ch.QueryTimeout, a.store.DefaultMaxRows, logger), + Pipes: pipesHandler, + StructuredQuery: api.NewStructuredQueryHandler(a.ch, a.cache, a.registry, (*settings.Store).Policy, (*settings.Store).TimestampBucketSeconds, a.ch.QueryTimeout, (*settings.Store).DefaultMaxRows, logger), AuthMW: authMW, + Tenants: a.tenants, PolicySource: a.policies, Logger: logger, CORSOrigins: a.store.CORSOrigins, diff --git a/internal/discovery/discovery.go b/internal/discovery/discovery.go index 9c134c71..6ae58ad7 100644 --- a/internal/discovery/discovery.go +++ b/internal/discovery/discovery.go @@ -8,6 +8,7 @@ import ( "time" "github.com/ClickHouse/clickhouse-go/v2/lib/driver" + "github.com/Wave-RF/WaveHouse/internal/tenant" "go.opentelemetry.io/otel" ) @@ -177,10 +178,12 @@ type SchemaRegistry struct { // ClickHouse reconfigure that changes clickhouse.database is honored by // the next refresh (chconn.Manager.Database in production). database func() string - // refreshInterval supplies the auto-refresh interval on each tick, so a - // settings reload retunes the cadence without restarting the loop - // (settings.Store.SchemaRefreshInterval in production). - refreshInterval func() time.Duration + // tenant is whose tables the registry discovers. + tenant tenant.ID + // refreshInterval supplies the tenant's auto-refresh interval on each + // tick, so a settings reload retunes the cadence without restarting the + // loop (settings.Store.SchemaRefreshInterval in production). + refreshInterval func(tenant.ID) time.Duration logger *slog.Logger mu sync.RWMutex tables map[string]*TableSchema @@ -189,11 +192,13 @@ type SchemaRegistry struct { serverVersion string } -// NewSchemaRegistry creates a registry that discovers schemas from system.columns. -func NewSchemaRegistry(conn driver.Conn, database func() string, refreshInterval func() time.Duration, logger *slog.Logger) *SchemaRegistry { +// NewSchemaRegistry creates the registry of tenant id, which discovers +// schemas from system.columns. +func NewSchemaRegistry(conn driver.Conn, database func() string, id tenant.ID, refreshInterval func(tenant.ID) time.Duration, logger *slog.Logger) *SchemaRegistry { return &SchemaRegistry{ conn: conn, database: database, + tenant: id, refreshInterval: refreshInterval, logger: logger, tables: make(map[string]*TableSchema), @@ -429,7 +434,7 @@ func (sr *SchemaRegistry) RetryRefresh(ctx context.Context, initialBackoff, maxB // — an in-flight wait finishes at the old cadence rather than resetting, // which keeps a reload from ever deferring an imminent refresh. func (sr *SchemaRegistry) StartAutoRefresh(ctx context.Context) { - interval := sr.refreshInterval() + interval := sr.refreshInterval(sr.tenant) ticker := time.NewTicker(interval) defer ticker.Stop() for { @@ -440,7 +445,7 @@ func (sr *SchemaRegistry) StartAutoRefresh(ctx context.Context) { if err := sr.Refresh(ctx); err != nil { sr.logger.Error("schema auto-refresh failed", "error", err) } - if next := sr.refreshInterval(); next != interval { + if next := sr.refreshInterval(sr.tenant); next != interval { interval = next ticker.Reset(interval) } diff --git a/internal/discovery/discovery_test.go b/internal/discovery/discovery_test.go index 6ba50a3c..63859d18 100644 --- a/internal/discovery/discovery_test.go +++ b/internal/discovery/discovery_test.go @@ -16,6 +16,8 @@ import ( "github.com/ClickHouse/clickhouse-go/v2/lib/driver" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" + + "github.com/Wave-RF/WaveHouse/internal/tenant" ) func TestTableSchema_ColumnNames(t *testing.T) { @@ -51,10 +53,10 @@ func TestTableSchema_ColumnNames(t *testing.T) { func TestNewSchemaRegistry_ConstructorDefaults(t *testing.T) { t.Parallel() logger := slog.New(slog.NewTextHandler(io.Discard, nil)) - sr := NewSchemaRegistry(nil, func() string { return "wavehouse" }, func() time.Duration { return 30 * time.Second }, logger) + sr := NewSchemaRegistry(nil, func() string { return "wavehouse" }, tenant.Default, func(tenant.ID) time.Duration { return 30 * time.Second }, logger) require.NotNil(t, sr) assert.Equal(t, "wavehouse", sr.database()) - assert.Equal(t, 30*time.Second, sr.refreshInterval()) + assert.Equal(t, 30*time.Second, sr.refreshInterval(tenant.Default)) assert.Same(t, logger, sr.logger) assert.NotNil(t, sr.tables) assert.Empty(t, sr.List()) @@ -80,7 +82,7 @@ func TestRefresh_PopulatesAndLookups(t *testing.T) { {"ghost", "CREATE TABLE test.ghost (`x` String) ENGINE = MergeTree"}, }, } - sr := NewSchemaRegistry(conn, func() string { return "test" }, func() time.Duration { return time.Hour }, discardLogger()) + sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, discardLogger()) require.NoError(t, sr.Refresh(context.Background())) clicks := sr.Get("clicks") @@ -113,7 +115,7 @@ func TestRefresh_DDLIsNotSerialized(t *testing.T) { // topology is not. The field is withheld for the topology. tables: [][2]string{{"clicks", "CREATE TABLE test.clicks (`id` String) ENGINE = S3('https://acme-private.s3.amazonaws.com/events.csv', 'AKIAEXAMPLEKEY', '[HIDDEN]', 'CSV')"}}, } - sr := NewSchemaRegistry(conn, func() string { return "test" }, func() time.Duration { return time.Hour }, discardLogger()) + sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, discardLogger()) require.NoError(t, sr.Refresh(context.Background())) encoded, err := json.Marshal(sr.Get("clicks")) @@ -133,7 +135,7 @@ func TestRefresh_ServerVersionQueryFails(t *testing.T) { version: "25.3.2.2", columns: []fakeColumn{{table: "clicks", name: "id", chType: "String", position: 1}}, } - sr := NewSchemaRegistry(conn, func() string { return "test" }, func() time.Duration { return time.Hour }, discardLogger()) + sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, discardLogger()) require.NoError(t, sr.Refresh(context.Background())) require.Equal(t, "25.3.2.2", sr.ServerVersion()) @@ -154,7 +156,7 @@ func TestRefresh_TablesQueryFails(t *testing.T) { columns: []fakeColumn{{table: "clicks", name: "id", chType: "String", position: 1}}, tables: [][2]string{{"clicks", "CREATE TABLE test.clicks (id String) ENGINE = MergeTree"}}, } - sr := NewSchemaRegistry(conn, func() string { return "test" }, func() time.Duration { return time.Hour }, discardLogger()) + sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, discardLogger()) require.NoError(t, sr.Refresh(context.Background())) require.NotNil(t, sr.Get("clicks")) require.NotEmpty(t, sr.Get("clicks").DDL) @@ -357,7 +359,7 @@ func newFakeRegistry(t *testing.T, errs []error) (*SchemaRegistry, *fakeConn) { t.Helper() conn := &fakeConn{errsThenSuccess: errs} logger := slog.New(slog.NewJSONHandler(io.Discard, nil)) - return NewSchemaRegistry(conn, func() string { return "test" }, func() time.Duration { return time.Hour }, logger), conn + return NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, logger), conn } // TestRefresh_UnresolvableServerTimezone_NotFatal: an unresolvable server zone @@ -365,7 +367,7 @@ func newFakeRegistry(t *testing.T, errs []error) (*SchemaRegistry, *fakeConn) { func TestRefresh_UnresolvableServerTimezone_NotFatal(t *testing.T) { t.Parallel() conn := &fakeConn{tz: "Not/AZone"} - sr := NewSchemaRegistry(conn, func() string { return "test" }, func() time.Duration { return time.Hour }, discardLogger()) + sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, discardLogger()) require.NoError(t, sr.Refresh(context.Background())) } @@ -379,7 +381,7 @@ func TestRefresh_RowsIterationError_Fails(t *testing.T) { columns: []fakeColumn{{table: "events", name: "id", chType: "String", position: 1}}, iterErr: errors.New("network drop mid-stream"), } - sr := NewSchemaRegistry(conn, func() string { return "test" }, func() time.Duration { return time.Hour }, discardLogger()) + sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, discardLogger()) err := sr.Refresh(context.Background()) require.ErrorContains(t, err, "network drop mid-stream") require.Nil(t, sr.Get("events"), "truncated scan must not be published") @@ -578,7 +580,7 @@ func TestClampBackoff(t *testing.T) { func TestStartAutoRefresh_ExitsOnContextCancel(t *testing.T) { t.Parallel() // Long interval so the ticker never fires before cancel. - sr := NewSchemaRegistry(nil, func() string { return "test" }, func() time.Duration { return time.Hour }, discardLogger()) + sr := NewSchemaRegistry(nil, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, discardLogger()) ctx, cancel := context.WithCancel(context.Background()) @@ -638,7 +640,7 @@ func TestStartAutoRefresh_LogsAndContinuesOnError(t *testing.T) { var buf concurrentBuffer logger := slog.New(slog.NewJSONHandler(&buf, &slog.HandlerOptions{Level: slog.LevelDebug})) - sr := NewSchemaRegistry(conn, func() string { return "test" }, func() time.Duration { return 5 * time.Millisecond }, logger) + sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return 5 * time.Millisecond }, logger) ctx, cancel := context.WithCancel(context.Background()) done := make(chan struct{}) @@ -692,7 +694,7 @@ func TestRefresh_DatabaseSnapshottedForWholeRefresh(t *testing.T) { } return "old" } - sr := NewSchemaRegistry(conn, db, func() time.Duration { return time.Hour }, discardLogger()) + sr := NewSchemaRegistry(conn, db, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, discardLogger()) require.NoError(t, sr.Refresh(context.Background())) require.Len(t, seen, 2, "both scans should be parameterised by a database") @@ -765,7 +767,7 @@ func TestRefresh_CapturesDefaultKind(t *testing.T) { {table: "t", name: "mat", chType: "String", defaultKind: "MATERIALIZED", defaultExpr: "concat('m', id)", position: 2}, {table: "t", name: "page", chType: "String", defaultKind: "DEFAULT", defaultExpr: "'/'", position: 3}, }} - sr := NewSchemaRegistry(conn, func() string { return "test" }, func() time.Duration { return time.Hour }, discardLogger()) + sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, discardLogger()) require.NoError(t, sr.Refresh(context.Background())) ts := sr.Get("t") diff --git a/internal/discovery/timestamp_test.go b/internal/discovery/timestamp_test.go index aa1f1382..7a5573bf 100644 --- a/internal/discovery/timestamp_test.go +++ b/internal/discovery/timestamp_test.go @@ -10,6 +10,8 @@ import ( "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" + + "github.com/Wave-RF/WaveHouse/internal/tenant" ) func TestIsTimestampType(t *testing.T) { @@ -301,7 +303,7 @@ func TestResolveTimestampSpecs(t *testing.T) { func TestRefresh_PrecomputesSpecs(t *testing.T) { t.Parallel() conn := &fakeConn{columns: []fakeColumn{{table: "t", name: "ts", chType: "DateTime", position: 1}}} - reg := NewSchemaRegistry(conn, func() string { return "test" }, func() time.Duration { return time.Hour }, discardLogger()) + reg := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, discardLogger()) require.NoError(t, reg.Refresh(context.Background())) col := reg.Get("t").Columns[0] require.NotNil(t, col.tsSpec) diff --git a/internal/ingest/sweeper.go b/internal/ingest/sweeper.go index eebd1751..1e0f037b 100644 --- a/internal/ingest/sweeper.go +++ b/internal/ingest/sweeper.go @@ -7,6 +7,7 @@ import ( "time" "github.com/Wave-RF/WaveHouse/internal/mq" + "github.com/Wave-RF/WaveHouse/internal/tenant" ) // Sweeper implements the Active Sweeper pattern. It runs every minute and @@ -20,18 +21,21 @@ import ( // finds the purge point is its own business (see mq.Purger). type Sweeper struct { purger mq.Purger - // gapWindow is read on every sweep (settings.Store.GapWindow in - // production) so a reload of stream.gap_window_minutes applies from the - // next sweep without a restart. - gapWindow func() time.Duration + tenant tenant.ID + // gapWindow is the tenant's gap window, read on every sweep + // (settings.Store.GapWindow in production) so a reload of + // stream.gap_window_minutes applies from the next sweep without a restart. + gapWindow func(tenant.ID) time.Duration logger *slog.Logger } -// NewSweeper creates an Active Sweeper. gapWindow is resolved per sweep. +// NewSweeper creates the Active Sweeper of tenant id. gapWindow is resolved +// per sweep. // TODO: (future) need leader election or shared lock to only run one instance of the sweeper in clustered mode -func NewSweeper(purger mq.Purger, gapWindow func() time.Duration, logger *slog.Logger) *Sweeper { +func NewSweeper(purger mq.Purger, id tenant.ID, gapWindow func(tenant.ID) time.Duration, logger *slog.Logger) *Sweeper { return &Sweeper{ purger: purger, + tenant: id, gapWindow: gapWindow, logger: logger, } @@ -52,7 +56,7 @@ func (s *Sweeper) Start(ctx context.Context) { } func (s *Sweeper) sweep(ctx context.Context) { - _, err := s.purger.PurgeAcked(ctx, BufferConsumerName, time.Now().Add(-s.gapWindow())) + _, err := s.purger.PurgeAcked(ctx, BufferConsumerName, time.Now().Add(-s.gapWindow(s.tenant))) if err != nil { if errors.Is(err, mq.ErrConsumerNotFound) { // Consumer may not exist yet if no messages have been ingested. diff --git a/internal/ingest/sweeper_test.go b/internal/ingest/sweeper_test.go index 6399562d..ebb76c94 100644 --- a/internal/ingest/sweeper_test.go +++ b/internal/ingest/sweeper_test.go @@ -7,6 +7,7 @@ import ( "time" "github.com/Wave-RF/WaveHouse/internal/mq" + "github.com/Wave-RF/WaveHouse/internal/tenant" "github.com/Wave-RF/WaveHouse/internal/testutil" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" @@ -19,7 +20,7 @@ func TestSweep_AsksForTheBufferConsumerAndTheGapWindow(t *testing.T) { t.Parallel() gapWindow := 5 * time.Minute purger := &testutil.MockPurger{Purged: true} - s := NewSweeper(purger, func() time.Duration { return gapWindow }, testutil.NopLogger()) + s := NewSweeper(purger, tenant.Default, func(tenant.ID) time.Duration { return gapWindow }, testutil.NopLogger()) before := time.Now() s.sweep(context.Background()) @@ -36,7 +37,7 @@ func TestSweep_RereadsTheGapWindowEverySweep(t *testing.T) { t.Parallel() gapWindow := time.Minute purger := &testutil.MockPurger{} - s := NewSweeper(purger, func() time.Duration { return gapWindow }, testutil.NopLogger()) + s := NewSweeper(purger, tenant.Default, func(tenant.ID) time.Duration { return gapWindow }, testutil.NopLogger()) s.sweep(context.Background()) gapWindow = time.Hour // a settings reload @@ -50,7 +51,7 @@ func TestSweep_ErrorsDoNotPanic(t *testing.T) { t.Parallel() for _, err := range []error{mq.ErrConsumerNotFound, errors.New("broker unavailable")} { purger := &testutil.MockPurger{Err: err} - s := NewSweeper(purger, func() time.Duration { return time.Minute }, testutil.NopLogger()) + s := NewSweeper(purger, tenant.Default, func(tenant.ID) time.Duration { return time.Minute }, testutil.NopLogger()) s.sweep(context.Background()) assert.Len(t, purger.Calls, 1) } @@ -62,7 +63,7 @@ func TestSweep_ErrorsDoNotPanic(t *testing.T) { func TestStart_ContextCancellation(t *testing.T) { t.Parallel() - s := NewSweeper(&testutil.MockPurger{}, func() time.Duration { return 5 * time.Minute }, testutil.NopLogger()) + s := NewSweeper(&testutil.MockPurger{}, tenant.Default, func(tenant.ID) time.Duration { return 5 * time.Minute }, testutil.NopLogger()) ctx, cancel := context.WithCancel(context.Background()) cancel() // Cancel immediately. diff --git a/internal/ingest/worker.go b/internal/ingest/worker.go index ae64c747..bc039d3b 100644 --- a/internal/ingest/worker.go +++ b/internal/ingest/worker.go @@ -19,6 +19,7 @@ import ( "github.com/Wave-RF/WaveHouse/internal/chsql" "github.com/Wave-RF/WaveHouse/internal/mq" "github.com/Wave-RF/WaveHouse/internal/query" + "github.com/Wave-RF/WaveHouse/internal/tenant" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/metric" @@ -64,11 +65,14 @@ type IngestWorker struct { target func() chconn.Target maxBatch int maxWait time.Duration - // dlqEnabled reports, per table, whether a row that still fails after - // row-by-row isolation is parked on the DLQ (settings.Store.DLQFor in - // production; nil means always). Resolved at the moment of the failure, so + // tenant is whose events the worker writes; every event is its tenant's + // until the MQ subject carries one (#583 story 5). + tenant tenant.ID + // dlqEnabled reports, per tenant table, whether a row that still fails + // after row-by-row isolation is parked on the DLQ (settings.Store.DLQFor + // in production; nil means always). Resolved at the moment of the failure, so // a settings reload applies to the next poison row without a restart. - dlqEnabled func(table string) bool + dlqEnabled func(id tenant.ID, table string) bool // wg tracks the dispatch loop; ackWg tracks backgrounded DoubleAck goroutines. // Separate so shutdown can drain inserts (wg → tableWg) before waiting on the @@ -128,7 +132,8 @@ const ( func StartIngestWorker( ctx context.Context, queue Queue, cache cache.Cache, target func() chconn.Target, - dlqEnabled func(table string) bool, + id tenant.ID, + dlqEnabled func(id tenant.ID, table string) bool, ) (stop func(context.Context) error, failed <-chan error, err error) { if queue == nil { return nil, nil, fmt.Errorf("message queue is nil") @@ -177,6 +182,7 @@ func StartIngestWorker( target: target, maxBatch: defaultMaxBatch, maxWait: defaultMaxWait, + tenant: id, dlqEnabled: dlqEnabled, } @@ -570,7 +576,7 @@ func (w *IngestWorker) flushGroup(ctx context.Context, tableName string, group [ for _, pm := range group { singleErr := w.insertToClickHouse(ctx, tableName, cols, []parsedMsg{pm}) if singleErr != nil { - if w.dlqEnabled != nil && !w.dlqEnabled(tableName) { + if w.dlqEnabled != nil && !w.dlqEnabled(w.tenant, tableName) { w.logger.ErrorContext(ctx, "isolated bad row, DLQ disabled for table — left unacked, NATS will redeliver it until it inserts or dlq is enabled", "table", tableName, "error", singleErr) continue } @@ -704,7 +710,7 @@ func (w *IngestWorker) handleSuccess(ctx context.Context, tableName string, msgs // publish that FAILS leaves the message unacked, exactly as the isolation path // does: that is a transient DLQ outage, and retrying beats destroying the row. func (w *IngestWorker) rejectPoison(ctx context.Context, m *mq.Message, tableName, reason, detail string) { - if w.dlqEnabled == nil || w.dlqEnabled(tableName) { + if w.dlqEnabled == nil || w.dlqEnabled(w.tenant, tableName) { // Backgrounded on ackWg for the same reason handleSuccess backgrounds its // acks: parkOnDLQ does a DLQ publish AND an fsync-bound DoubleAck, // and parseMsg runs on the dispatchLoop goroutine. The scenario this whole diff --git a/internal/ingest/worker_test.go b/internal/ingest/worker_test.go index 21e04a3a..c34f5b2e 100644 --- a/internal/ingest/worker_test.go +++ b/internal/ingest/worker_test.go @@ -28,6 +28,7 @@ import ( "github.com/Wave-RF/WaveHouse/internal/chconn" "github.com/Wave-RF/WaveHouse/internal/discovery" "github.com/Wave-RF/WaveHouse/internal/mq" + "github.com/Wave-RF/WaveHouse/internal/tenant" "github.com/Wave-RF/WaveHouse/internal/testutil" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" @@ -134,7 +135,7 @@ func TestStartIngestWorker_Validation(t *testing.T) { t.Parallel() q, c := tt.setup(t) _, _, err := StartIngestWorker(context.Background(), q, c, - func() chconn.Target { return chconn.Target{URL: "http://localhost:8123"} }, nil) + func() chconn.Target { return chconn.Target{URL: "http://localhost:8123"} }, tenant.Default, nil) require.Error(t, err) assert.Contains(t, err.Error(), tt.wantErrSub) }) @@ -270,7 +271,7 @@ func TestStartIngestWorker_StopFunc_RespectsShutdownDeadline(t *testing.T) { stopFn, _, err := StartIngestWorker(ctx, emb, &testutil.MockCache{}, func() chconn.Target { return chconn.Target{URL: fmt.Sprintf("http://%s:%s", host, port), Username: "u", Password: "p", Database: "db"} - }, nil) + }, tenant.Default, nil) require.NoError(t, err) // Publish so there's an in-flight insert blocking on `release`. @@ -304,7 +305,7 @@ func TestStartIngestWorker_StopFunc_CleanShutdown(t *testing.T) { // chURL is never dialed: with no messages there is no flush, so a dummy // host/port is fine. stopFn, _, err := StartIngestWorker(context.Background(), emb, &testutil.MockCache{}, - func() chconn.Target { return chconn.Target{URL: "http://localhost:8123"} }, nil) + func() chconn.Target { return chconn.Target{URL: "http://localhost:8123"} }, tenant.Default, nil) require.NoError(t, err) // Nothing to flush, so shutdown drains immediately and returns nil before the @@ -887,7 +888,7 @@ func TestFlushTable_BadRow_DLQDisabledForTable_LeftUnacked(t *testing.T) { }, } w, pub, _, wait := newTestWorker(rt) - w.dlqEnabled = func(table string) bool { return table != "events" } + w.dlqEnabled = func(_ tenant.ID, table string) bool { return table != "events" } good := newIngestMsg(t, "events", "", map[string]any{"id": 1, "poison": false}) poison := newIngestMsg(t, "events", "", map[string]any{"id": 2, "poison": true}) @@ -1317,7 +1318,7 @@ func TestParseMsg_DuplicateColumn_Unpairable(t *testing.T) { func TestParseMsg_PoisonEnvelope_DLQDisabled_AckedAndDropped(t *testing.T) { t.Parallel() w, pub, _, _ := newTestWorker(&testutil.MockRoundTripper{}) - w.dlqEnabled = func(string) bool { return false } + w.dlqEnabled = func(tenant.ID, string) bool { return false } m := &testutil.MockMessage{ MsgTopic: mq.Topic{Table: "events"}, @@ -1485,7 +1486,7 @@ func TestRejectPoison_CountedByDisposition(t *testing.T) { // DLQ off for the table: acked and dropped, counted separately. w2, _, _, _ := newTestWorker(&testutil.MockRoundTripper{}) - w2.dlqEnabled = func(string) bool { return false } + w2.dlqEnabled = func(tenant.ID, string) bool { return false } _, ok = w2.parseMsg(context.Background(), poison().Message()) require.False(t, ok) w2.ackWg.Wait() @@ -1506,7 +1507,7 @@ func TestRejectPoison_CountedByDisposition(t *testing.T) { // counting before the ack would report a row as gone forever — the meaning // deployment.md gives "dropped" — once per redelivery, while it is still there. w4, _, _, _ := newTestWorker(&testutil.MockRoundTripper{}) - w4.dlqEnabled = func(string) bool { return false } + w4.dlqEnabled = func(tenant.ID, string) bool { return false } unackable := poison() unackable.DoubleAckErr = errors.New("ack timed out") _, ok = w4.parseMsg(context.Background(), unackable.Message()) @@ -1599,8 +1600,8 @@ func TestDispatchLoop_DeliveryEndedFailsLoud(t *testing.T) { rt := &testutil.MockRoundTripper{} w, _, _, _ := newTestWorker(rt) w.maxBatch = 100 - w.maxWait = time.Hour // only the shutdown flush can write the row - w.dlqEnabled = func(string) bool { return false } // the sentinel is acked-and-dropped, no publish + w.maxWait = time.Hour // only the shutdown flush can write the row + w.dlqEnabled = func(tenant.ID, string) bool { return false } // the sentinel is acked-and-dropped, no publish held := newIngestMsg(t, "events", "", map[string]any{"id": 1}) sentinel := &testutil.MockMessage{MsgTopic: mq.Topic{Table: "events"}, MsgData: []byte("not json")} @@ -1651,8 +1652,8 @@ func TestDispatchLoop_HandoffReturnsOnCancel(t *testing.T) { t.Parallel() w, _, _, _ := newTestWorker(&testutil.MockRoundTripper{}) - w.maxBatch = 1 // msgChan holds 2 - w.dlqEnabled = func(string) bool { return false } // any message parsed is acked-and-dropped, no publish + w.maxBatch = 1 // msgChan holds 2 + w.dlqEnabled = func(tenant.ID, string) bool { return false } // any message parsed is acked-and-dropped, no publish ctx, cancel := context.WithCancel(context.Background()) cancel() // already stopping: the loop exits at its first ctx.Done pick, leaving msgChan full diff --git a/internal/settings/registry.go b/internal/settings/registry.go new file mode 100644 index 00000000..007c592b --- /dev/null +++ b/internal/settings/registry.go @@ -0,0 +1,22 @@ +package settings + +import "github.com/Wave-RF/WaveHouse/internal/tenant" + +// Registry maps a tenant id to the Store holding that tenant's adopted +// settings. It holds exactly one store, under tenant.Default: the settings +// directory is one tenant's four files, and Open, Reload, and Watch stay on +// the Store behind it. +type Registry struct { + stores map[tenant.ID]*Store +} + +// NewRegistry returns a Registry serving store as tenant.Default. +func NewRegistry(store *Store) *Registry { + return &Registry{stores: map[tenant.ID]*Store{tenant.Default: store}} +} + +// For returns the store of tenant id, or false when no such tenant exists. +func (r *Registry) For(id tenant.ID) (*Store, bool) { + s, ok := r.stores[id] + return s, ok +} diff --git a/internal/settings/registry_test.go b/internal/settings/registry_test.go new file mode 100644 index 00000000..733cf65b --- /dev/null +++ b/internal/settings/registry_test.go @@ -0,0 +1,24 @@ +package settings + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/Wave-RF/WaveHouse/internal/tenant" +) + +func TestRegistry_For(t *testing.T) { + t.Parallel() + store := newLoadedStore(t, nil) + reg := NewRegistry(store) + + got, ok := reg.For(tenant.Default) + require.True(t, ok) + assert.Same(t, store, got) + + got, ok = reg.For(tenant.ID("acme")) + assert.False(t, ok, "only the default tenant exists") + assert.Nil(t, got) +} diff --git a/internal/stream/hub.go b/internal/stream/hub.go index 2517a1d2..997b6648 100644 --- a/internal/stream/hub.go +++ b/internal/stream/hub.go @@ -10,6 +10,7 @@ import ( "github.com/Wave-RF/WaveHouse/internal/discovery" "github.com/Wave-RF/WaveHouse/internal/ingest" "github.com/Wave-RF/WaveHouse/internal/policy" + "github.com/Wave-RF/WaveHouse/internal/tenant" ) // Hub fans live events out to SSE subscribers. Column projection is serialized ONCE @@ -27,7 +28,8 @@ import ( type Hub struct { mu sync.RWMutex topics map[string]*topicRoutes - policy policy.Source // nil ⇒ policy filtering not configured (legacy passthrough) + tenant tenant.ID // whose policy every event and subscriber is evaluated against + policy PolicySource // nil ⇒ policy filtering not configured (legacy passthrough) registry *discovery.SchemaRegistry // nil ⇒ no column types; row-filter comparison degrades fail-closed (see columnSpecs) metric *Metrics // nil-safe @@ -74,14 +76,19 @@ type topicRoutes struct { roles map[string]Bucket // role -> Bucket of Subscribers } -// NewHub builds an event hub. A nil policy store passes every event through +// PolicySource yields a tenant's access-control policy, read per event so a +// settings reload applies to the next one. A nil policy from a wired source +// is a deliberate lockout. +type PolicySource func(tenant.ID) *policy.Policy + +// NewHub builds the event hub of tenant id. A nil policy store passes every event through // unfiltered (the unwired-tests case); a non-nil store whose Get returns nil is a // total lockout (a deleted/absent policy denies everyone). A nil registry leaves // every column's type unknown, so row-filter comparison degrades FAIL-CLOSED: // equality/set predicates admit only a byte-identical value and ordering/!= admit // nothing (see policy.ColumnKind); metric may be nil. -func NewHub(policyStore policy.Source, registry *discovery.SchemaRegistry, metric *Metrics) *Hub { - return &Hub{topics: make(map[string]*topicRoutes), policy: policyStore, registry: registry, metric: metric} +func NewHub(id tenant.ID, policyStore PolicySource, registry *discovery.SchemaRegistry, metric *Metrics) *Hub { + return &Hub{topics: make(map[string]*topicRoutes), tenant: id, policy: policyStore, registry: registry, metric: metric} } // Add registers sub to receive events for (topic, role), creating the role bucket @@ -426,7 +433,7 @@ func (h *Hub) snapshotPolicy() (p *policy.Policy, filter bool) { if h.policy == nil { return nil, false } - return h.policy(), true + return h.policy(h.tenant), true } // ReplayProjector returns the projection function for one connection's gap-fill: diff --git a/internal/stream/hub_test.go b/internal/stream/hub_test.go index 40a08185..acd70224 100644 --- a/internal/stream/hub_test.go +++ b/internal/stream/hub_test.go @@ -17,6 +17,7 @@ import ( "github.com/Wave-RF/WaveHouse/internal/discovery" "github.com/Wave-RF/WaveHouse/internal/ingest" "github.com/Wave-RF/WaveHouse/internal/policy" + "github.com/Wave-RF/WaveHouse/internal/tenant" "github.com/Wave-RF/WaveHouse/internal/testutil" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" @@ -76,7 +77,7 @@ func rawEvent(tb testing.TB, table, ts string, data map[string]any) []byte { func TestBroadcast_DuplicateColumnWithheld(t *testing.T) { t.Parallel() const topic = "clicks" - hub := NewHub(nil, nil, nil) // nil store ⇒ passthrough, so nothing else withholds + hub := NewHub(tenant.Default, nil, nil, nil) // nil store ⇒ passthrough, so nothing else withholds sub := NewSubscriber(nil, nil) hub.Add(topic, "viewer", sub) @@ -109,7 +110,7 @@ func TestBroadcast_DuplicateColumnWithheld(t *testing.T) { // The pairable control is load-bearing: without it a projector that returned no // frames for EVERY envelope would pass. func TestReplayProjector_UnpairableWithheld(t *testing.T) { - hub := NewHub(nil, nil, NewMetrics()) + hub := NewHub(tenant.Default, nil, nil, NewMetrics()) project := hub.ReplayProjector("viewer", NewSubscriber(nil, nil)) bad, err := json.Marshal(ingest.EventMessage{ @@ -139,7 +140,7 @@ func TestReplayProjector_UnpairableWithheld(t *testing.T) { // and the positive control proves the withholding is the format's doing. func TestEventView_UnknownFormatWithheld(t *testing.T) { const topic = "clicks" - hub := NewHub(nil, nil, NewMetrics()) + hub := NewHub(tenant.Default, nil, nil, NewMetrics()) sub := NewSubscriber(nil, nil) hub.Add(topic, "viewer", sub) @@ -314,7 +315,7 @@ func recvEventCols(t *testing.T, sub *Subscriber, cols []string) (Frame, map[str func TestHub_ProjectsOncePerRole_FanOutToAllSubscribers(t *testing.T) { t.Parallel() - hub := NewHub(nil, nil, nil) // nil store ⇒ passthrough, no filtering + hub := NewHub(tenant.Default, nil, nil, nil) // nil store ⇒ passthrough, no filtering const topic = "clicks" a, b := NewSubscriber(nil, nil), NewSubscriber(nil, nil) @@ -347,7 +348,7 @@ func TestHub_ProjectsPerRole_ColumnFilterAndDenial(t *testing.T) { }, }, } - hub := NewHub(policy.Static(p), nil, nil) + hub := NewHub(tenant.Default, staticPolicy(p), nil, nil) const topic = "clicks" viewer := NewSubscriber(nil, nil) @@ -427,7 +428,7 @@ func TestHub_ProjectsPerRole_DistinctRolesGetDistinctFrames(t *testing.T) { }, }, } - hub := NewHub(policy.Static(p), nil, nil) + hub := NewHub(tenant.Default, staticPolicy(p), nil, nil) const topic = "clicks" viewer, editor := NewSubscriber(nil, nil), NewSubscriber(nil, nil) @@ -480,7 +481,7 @@ func rowFilterPolicy() *policy.Policy { // matching the constant-false predicate the query path binds for it. func TestHub_RowFilter_PerSubscriberIsolation(t *testing.T) { t.Parallel() - hub := NewHub(policy.Static(rowFilterPolicy()), nil, nil) + hub := NewHub(tenant.Default, staticPolicy(rowFilterPolicy()), nil, nil) const topic = "clicks" acme := NewSubscriber(jwtClaims(t, map[string]any{"tenant": "acme"}), nil) @@ -524,7 +525,7 @@ func TestHub_RowFilter_ClaimsSnapshotImmuneToCallerMutation(t *testing.T) { "clicks": {"viewer": {Select: &policy.SelectPermissions{Filter: map[string]policy.Filter{"tenant_id": {Eq: new("{{ jwt.org.tenant }}")}}}}}, }, } - hub := NewHub(policy.Static(p), nil, nil) + hub := NewHub(tenant.Default, staticPolicy(p), nil, nil) const topic = "clicks" org := map[string]any{"tenant": "globex"} @@ -550,7 +551,7 @@ func TestHub_RowFilter_ClaimsSnapshotImmuneToCallerMutation(t *testing.T) { "clicks": {"viewer": {Select: &policy.SelectPermissions{Filter: map[string]policy.Filter{"tenant_id": {In: new("{{ jwt.tenants }}")}}}}}, }, } - inHub := NewHub(policy.Static(inPolicy), nil, nil) + inHub := NewHub(tenant.Default, staticPolicy(inPolicy), nil, nil) tenants := []any{"globex"} inSub := NewSubscriber(map[string]any{"tenants": tenants}, nil) inHub.Add(topic, "viewer", inSub) @@ -570,7 +571,7 @@ func TestHub_RowFilter_ClaimsSnapshotImmuneToCallerMutation(t *testing.T) { // column can't be proven visible, so it is withheld rather than leaked. func TestHub_RowFilter_MissingColumn_FailsClosed(t *testing.T) { t.Parallel() - hub := NewHub(policy.Static(rowFilterPolicy()), nil, nil) + hub := NewHub(tenant.Default, staticPolicy(rowFilterPolicy()), nil, nil) const topic = "clicks" acme := NewSubscriber(map[string]any{"tenant": "acme"}, nil) @@ -587,7 +588,7 @@ func TestHub_RowFilter_MissingColumn_FailsClosed(t *testing.T) { // per-subscriber, not the serialization. func TestHub_RowFilter_SharedProjectionAcrossSameClaims(t *testing.T) { t.Parallel() - hub := NewHub(policy.Static(rowFilterPolicy()), nil, nil) + hub := NewHub(tenant.Default, staticPolicy(rowFilterPolicy()), nil, nil) const topic = "clicks" a := NewSubscriber(map[string]any{"tenant": "acme"}, nil) @@ -625,7 +626,7 @@ func TestHub_RowFilter_NumericOrdering_SchemaInformed(t *testing.T) { "clicks": {"viewer": {Select: &policy.SelectPermissions{Filter: map[string]policy.Filter{"amount": {Gt: new("100")}}}}}, }, } - hub := NewHub(policy.Static(p), reg, nil) + hub := NewHub(tenant.Default, staticPolicy(p), reg, nil) const topic = "clicks" sub := NewSubscriber(nil, nil) // constant filter value ⇒ no claims needed @@ -641,7 +642,7 @@ func TestHub_RowFilter_NumericOrdering_SchemaInformed(t *testing.T) { // Same policy, no schema registry: an ordering predicate can't be proven either // way, so both rows are withheld — including the one the schema-informed path // delivers above. - noSchema := NewHub(policy.Static(p), nil, nil) + noSchema := NewHub(tenant.Default, staticPolicy(p), nil, nil) blind := NewSubscriber(nil, nil) noSchema.Add(topic, "viewer", blind) noSchema.Broadcast(topic, rawEvent(t, "clicks", "t1", map[string]any{"amount": float64(9), "page": "/a"})) @@ -665,7 +666,7 @@ func TestHub_RowFilter_FloatNarrowing_SchemaInformed(t *testing.T) { "clicks": {"viewer": {Select: &policy.SelectPermissions{Filter: map[string]policy.Filter{"score": {Gt: new("16777216")}}}}}, }, } - hub := NewHub(policy.Static(p), reg, nil) + hub := NewHub(tenant.Default, staticPolicy(p), reg, nil) const topic = "clicks" sub := NewSubscriber(nil, nil) hub.Add(topic, "viewer", sub) @@ -680,7 +681,7 @@ func TestHub_RowFilter_FloatNarrowing_SchemaInformed(t *testing.T) { func TestHub_TopicIsolation(t *testing.T) { t.Parallel() - hub := NewHub(nil, nil, nil) + hub := NewHub(tenant.Default, nil, nil, nil) clicks, views := NewSubscriber(nil, nil), NewSubscriber(nil, nil) hub.Add("clicks", "public", clicks) hub.Add("views", "public", views) @@ -706,7 +707,7 @@ func TestHub_PassthroughAndFailClosed(t *testing.T) { t.Parallel() tests := []struct { name string - store policy.Source + store PolicySource payload string wantData string // "" ⇒ expect no frame (event skipped) }{ @@ -723,14 +724,14 @@ func TestHub_PassthroughAndFailClosed(t *testing.T) { }, { name: "non-EventMessage is dropped (fail closed) when a policy store is wired", - store: policy.Static(&policy.Policy{}), + store: staticPolicy(&policy.Policy{}), payload: `{"custom":"data","value":42}`, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { t.Parallel() - hub := NewHub(tt.store, nil, nil) + hub := NewHub(tenant.Default, tt.store, nil, nil) const topic = "custom" sub := NewSubscriber(nil, nil) hub.Add(topic, "public", sub) @@ -751,7 +752,7 @@ func TestHub_PassthroughAndFailClosed(t *testing.T) { func TestHub_AddRemoveGCsBucketsAndTopics(t *testing.T) { t.Parallel() - hub := NewHub(nil, nil, nil) + hub := NewHub(tenant.Default, nil, nil, nil) const topic = "clicks" sub := NewSubscriber(nil, nil) @@ -772,7 +773,7 @@ func TestHub_AddRemoveGCsBucketsAndTopics(t *testing.T) { func TestHub_BroadcastNoSubscribers_NoOp(t *testing.T) { t.Parallel() - hub := NewHub(nil, nil, nil) + hub := NewHub(tenant.Default, nil, nil, nil) assert.NotPanics(t, func() { hub.Broadcast("nobody", rawEvent(t, "clicks", "t", map[string]any{"a": float64(1)})) }) @@ -790,7 +791,7 @@ func TestHub_SlowConsumerDropIncrementsMetric(t *testing.T) { }) m := NewMetrics() - hub := NewHub(nil, nil, m) + hub := NewHub(tenant.Default, nil, nil, m) const topic = "clicks" // cap-2: the first broadcast fills it with the schema frame plus the row, so // the second undrained broadcast's row drops (its column list is unchanged, so @@ -821,7 +822,7 @@ func TestHub_ReplayProjector(t *testing.T) { "clicks": {"viewer": {Select: &policy.SelectPermissions{AllowColumns: []string{"page"}}}}, }, } - hub := NewHub(policy.Static(p), nil, nil) + hub := NewHub(tenant.Default, staticPolicy(p), nil, nil) raw := rawEvent(t, "clicks", "2026-06-26T00:00:00Z", map[string]any{"page": "/home", "secret": "x"}) tests := []struct { @@ -866,7 +867,7 @@ func TestHub_ReplayProjector(t *testing.T) { // gap-fill event is projected only when the connection's claims satisfy the filter. func TestHub_ReplayProjector_RowFilter(t *testing.T) { t.Parallel() - hub := NewHub(policy.Static(rowFilterPolicy()), nil, nil) + hub := NewHub(tenant.Default, staticPolicy(rowFilterPolicy()), nil, nil) raw := rawEvent(t, "clicks", "2026-06-26T00:00:00Z", map[string]any{"tenant_id": "acme", "page": "/a", "secret": "x"}) @@ -898,7 +899,7 @@ func TestHub_ReplayProjector_RowFilter(t *testing.T) { func TestHub_ConcurrentAddRemoveBroadcast_Race(t *testing.T) { t.Parallel() - hub := NewHub(nil, nil, nil) + hub := NewHub(tenant.Default, nil, nil, nil) const topic = "clicks" raw := rawEvent(t, "clicks", "t", map[string]any{"a": float64(1)}) @@ -929,7 +930,7 @@ func TestHub_ConcurrentAddRemoveBroadcast_Race(t *testing.T) { // racing silently on a security decision. func TestHub_ConcurrentRowFilteredBroadcast_Race(t *testing.T) { t.Parallel() - hub := NewHub(policy.Static(rowFilterPolicy()), nil, nil) + hub := NewHub(tenant.Default, staticPolicy(rowFilterPolicy()), nil, nil) const topic = "clicks" raw := rawEvent(t, "clicks", "t", map[string]any{"tenant_id": "acme", "page": "/a"}) @@ -978,7 +979,7 @@ func TestHub_RowFilter_BigIntegerExact(t *testing.T) { "clicks": {"viewer": {Select: &policy.SelectPermissions{Filter: map[string]policy.Filter{"tenant_id": {Eq: new("{{ jwt.tenant }}")}}}}}, }, } - hub := NewHub(policy.Static(p), reg, nil) + hub := NewHub(tenant.Default, staticPolicy(p), reg, nil) const topic = "clicks" // Claims come from real signed tokens through the production middleware, so a @@ -1023,7 +1024,7 @@ func TestHub_RowFilter_TimestampInstantMatch(t *testing.T) { }, }, } - hub := NewHub(policy.Static(p), reg, nil) + hub := NewHub(tenant.Default, staticPolicy(p), reg, nil) const topic = "clicks" sub := NewSubscriber(nil, nil) @@ -1057,7 +1058,7 @@ func TestHub_RowFilterWithheldIncrementsMetric(t *testing.T) { otel.SetMeterProvider(savedMP) }) - hub := NewHub(policy.Static(rowFilterPolicy()), nil, NewMetrics()) + hub := NewHub(tenant.Default, staticPolicy(rowFilterPolicy()), nil, NewMetrics()) const topic = "clicks" acme := NewSubscriber(map[string]any{"tenant": "acme"}, nil) globex := NewSubscriber(map[string]any{"tenant": "globex"}, nil) @@ -1092,7 +1093,7 @@ func BenchmarkBroadcast_RowFilteredFanout(b *testing.B) { for _, n := range []int{100, 1_000, 10_000} { b.Run(fmt.Sprintf("subscribers=%d", n), func(b *testing.B) { - hub := NewHub(policy.Static(rowFilterPolicy()), nil, nil) + hub := NewHub(tenant.Default, staticPolicy(rowFilterPolicy()), nil, nil) subs := make([]*Subscriber, n) for i := range n { tenant := "acme" @@ -1223,7 +1224,7 @@ func sumByName(rm metricdata.ResourceMetrics, name string) int64 { // stream. func TestHub_SchemaFrame_AnnouncedOncePerConnection(t *testing.T) { t.Parallel() - hub := NewHub(nil, nil, nil) + hub := NewHub(tenant.Default, nil, nil, nil) const topic = "clicks" sub := NewSubscriber(nil, nil) hub.Add(topic, "public", sub) @@ -1249,7 +1250,7 @@ func TestHub_SchemaFrame_AnnouncedOncePerConnection(t *testing.T) { // it — otherwise a client zips values under the wrong names. func TestHub_SchemaFrame_ReannouncedOnDrift(t *testing.T) { t.Parallel() - hub := NewHub(nil, nil, nil) + hub := NewHub(tenant.Default, nil, nil, nil) const topic = "clicks" sub := NewSubscriber(nil, nil) hub.Add(topic, "public", sub) @@ -1280,7 +1281,7 @@ func TestHub_SchemaFrame_ReannouncedOnDrift(t *testing.T) { // told the column list even though an earlier subscriber already was. func TestHub_SchemaFrame_PerConnectionNotPerRole(t *testing.T) { t.Parallel() - hub := NewHub(nil, nil, nil) + hub := NewHub(tenant.Default, nil, nil, nil) const topic = "clicks" early := NewSubscriber(nil, nil) hub.Add(topic, "public", early) @@ -1316,7 +1317,7 @@ func TestHub_SubscribeSchemaFrame(t *testing.T) { "clicks": {"viewer": {Select: &policy.SelectPermissions{AllowColumns: []string{"page"}}}}, }, } - hub := NewHub(policy.Static(p), reg, nil) + hub := NewHub(tenant.Default, staticPolicy(p), reg, nil) sub := NewSubscriber(nil, nil) f, ok := hub.SubscribeSchemaFrame("clicks", "viewer", sub) @@ -1352,9 +1353,9 @@ func TestHub_SubscribeSchemaFrame_NothingToAnnounce(t *testing.T) { table string role string }{ - {"no registry", NewHub(policy.Static(p), nil, nil), "clicks", "viewer"}, - {"unknown table", NewHub(policy.Static(p), reg, nil), "missing", "viewer"}, - {"role cannot read the table", NewHub(policy.Static(p), reg, nil), "clicks", "stranger"}, + {"no registry", NewHub(tenant.Default, staticPolicy(p), nil, nil), "clicks", "viewer"}, + {"unknown table", NewHub(tenant.Default, staticPolicy(p), reg, nil), "missing", "viewer"}, + {"role cannot read the table", NewHub(tenant.Default, staticPolicy(p), reg, nil), "clicks", "stranger"}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -1370,7 +1371,7 @@ func TestHub_SubscribeSchemaFrame_NothingToAnnounce(t *testing.T) { // the client with every digit — a float64 round trip would round it. func TestHub_DataFrame_PreservesRawCellBytes(t *testing.T) { t.Parallel() - hub := NewHub(nil, nil, nil) + hub := NewHub(tenant.Default, nil, nil, nil) const topic = "clicks" sub := NewSubscriber(nil, nil) hub.Add(topic, "public", sub) @@ -1390,7 +1391,7 @@ func TestHub_DataFrame_PreservesRawCellBytes(t *testing.T) { // ahead of it with nothing to zip against. Replay announces on its own. func TestHub_ReplayProjector_SchemaTrackingIsIndependentOfLive(t *testing.T) { t.Parallel() - hub := NewHub(nil, nil, nil) + hub := NewHub(tenant.Default, nil, nil, nil) sub := NewSubscriber(nil, nil) raw := rawEventCols(t, "clicks", "t1", []string{"page"}, map[string]any{"page": "/a"}) @@ -1414,7 +1415,7 @@ func TestHub_ReplayProjector_SchemaTrackingIsIndependentOfLive(t *testing.T) { // for good. func TestHub_SchemaFrame_DroppedAnnouncementDropsItsRow(t *testing.T) { t.Parallel() - hub := NewHub(nil, nil, nil) + hub := NewHub(tenant.Default, nil, nil, nil) const topic = "clicks" sub := newSubscriber(1, nil) // cap-1: room for the announcement, not the row hub.Add(topic, "public", sub) @@ -1447,7 +1448,7 @@ func TestHub_SubscribeSchemaFrame_ExcludesComputedColumns(t *testing.T) { {Name: "country", Type: "String"}, }}, }) - hub := NewHub(nil, reg, nil) + hub := NewHub(tenant.Default, nil, reg, nil) sub := NewSubscriber(nil, nil) f, ok := hub.SubscribeSchemaFrame("clicks", "public", sub) diff --git a/internal/stream/roweval_test.go b/internal/stream/roweval_test.go index 21600d58..05826492 100644 --- a/internal/stream/roweval_test.go +++ b/internal/stream/roweval_test.go @@ -7,6 +7,7 @@ import ( "github.com/stretchr/testify/require" "github.com/Wave-RF/WaveHouse/internal/policy" + "github.com/Wave-RF/WaveHouse/internal/tenant" ) // recordingEvaluator answers every row the same way and counts the calls, so a @@ -57,7 +58,7 @@ func TestHub_RowEvaluatorSeam_LiveBroadcast(t *testing.T) { t.Run(tt.name, func(t *testing.T) { t.Parallel() eval := &recordingEvaluator{visible: tt.visible} - hub := NewHub(policy.Static(filteredPolicy()), nil, nil) + hub := NewHub(tenant.Default, staticPolicy(filteredPolicy()), nil, nil) hub.RowEvaluator = eval sub := NewSubscriber(map[string]any{"tenant": "t1"}, nil) @@ -85,7 +86,7 @@ func TestHub_RowEvaluatorSeam_LiveBroadcast(t *testing.T) { func TestHub_RowEvaluatorSeam_Replay(t *testing.T) { t.Parallel() eval := &recordingEvaluator{visible: false} - hub := NewHub(policy.Static(filteredPolicy()), nil, nil) + hub := NewHub(tenant.Default, staticPolicy(filteredPolicy()), nil, nil) hub.RowEvaluator = eval project := hub.ReplayProjector("viewer", NewSubscriber(map[string]any{"tenant": "t1"}, nil)) @@ -101,7 +102,7 @@ func TestHub_RowEvaluatorSeam_Replay(t *testing.T) { // row-level security. A nil seam must never read as "everything is visible". func TestHub_DefaultRowEvaluator_WhenUnwired(t *testing.T) { t.Parallel() - hub := NewHub(policy.Static(filteredPolicy()), nil, nil) + hub := NewHub(tenant.Default, staticPolicy(filteredPolicy()), nil, nil) require.Nil(t, hub.RowEvaluator) assert.IsType(t, policyRowEvaluator{}, hub.rowEvaluator()) diff --git a/internal/stream/tenant_test.go b/internal/stream/tenant_test.go new file mode 100644 index 00000000..10c1898a --- /dev/null +++ b/internal/stream/tenant_test.go @@ -0,0 +1,11 @@ +package stream + +import ( + "github.com/Wave-RF/WaveHouse/internal/policy" + "github.com/Wave-RF/WaveHouse/internal/tenant" +) + +// staticPolicy is a PolicySource fixed to p, whatever the tenant. +func staticPolicy(p *policy.Policy) PolicySource { + return func(tenant.ID) *policy.Policy { return p } +} diff --git a/internal/tenant/tenant.go b/internal/tenant/tenant.go new file mode 100644 index 00000000..2a08c97c --- /dev/null +++ b/internal/tenant/tenant.go @@ -0,0 +1,50 @@ +// Package tenant defines the tenant identifier: the token that selects a +// settings folder, and with it the tables, policies, and pipes a request or +// an event belongs to (#583). It imports nothing from the rest of the +// repository, so every package can name a tenant without a cycle. +package tenant + +import ( + "errors" + "fmt" +) + +// ID is a validated tenant identifier. It is a string, never a number: ids +// issued upstream can be 19 digits long, which already round as a float64. +type ID string + +const ( + // Default is the reserved tenant every request without a tenant header + // resolves to, and the only tenant of a flat settings directory. + Default ID = "0" + + // Header carries the tenant id on a request. Absent means Default. + Header = "X-Tenant-ID" + + // MaxLen caps an id's length in bytes. + MaxLen = 64 +) + +// Parse validates s against the one grammar an id must satisfy to be safe +// both as a folder name and as a message-queue subject token: ASCII letters, +// digits, '_' and '-', at most MaxLen bytes. Dots, slashes, spaces, and +// wildcards are rejected because each means something to one of the two. +func Parse(s string) (ID, error) { + if s == "" { + return "", errors.New("tenant id is empty") + } + if len(s) > MaxLen { + return "", fmt.Errorf("tenant id is %d bytes, the limit is %d", len(s), MaxLen) + } + for i := 0; i < len(s); i++ { + c := s[i] + switch { + case c >= 'a' && c <= 'z', c >= 'A' && c <= 'Z', c >= '0' && c <= '9', c == '_', c == '-': + default: + return "", fmt.Errorf("tenant id has %q at byte %d: only letters, digits, '_' and '-' are allowed", c, i) + } + } + return ID(s), nil +} + +func (id ID) String() string { return string(id) } diff --git a/internal/tenant/tenant_test.go b/internal/tenant/tenant_test.go new file mode 100644 index 00000000..26341da7 --- /dev/null +++ b/internal/tenant/tenant_test.go @@ -0,0 +1,53 @@ +package tenant + +import ( + "strings" + "testing" +) + +func TestParse(t *testing.T) { + tests := []struct { + name string + in string + wantErr bool + }{ + {name: "default", in: "0"}, + {name: "letters digits underscore dash", in: "Acme_co-42"}, + {name: "19 digit id", in: "9223372036854775807"}, + {name: "at the length cap", in: strings.Repeat("a", MaxLen)}, + {name: "empty", in: "", wantErr: true}, + {name: "over the length cap", in: strings.Repeat("a", MaxLen+1), wantErr: true}, + {name: "dot", in: "a.b", wantErr: true}, + {name: "parent directory", in: "..", wantErr: true}, + {name: "slash", in: "a/b", wantErr: true}, + {name: "backslash", in: `a\b`, wantErr: true}, + {name: "space", in: "a b", wantErr: true}, + {name: "subject wildcard star", in: "*", wantErr: true}, + {name: "subject wildcard tail", in: ">", wantErr: true}, + {name: "non-ascii letter", in: "ténant", wantErr: true}, + {name: "newline", in: "a\n", wantErr: true}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got, err := Parse(tt.in) + if tt.wantErr { + if err == nil { + t.Fatalf("Parse(%q) = %q, want an error", tt.in, got) + } + return + } + if err != nil { + t.Fatalf("Parse(%q): %v", tt.in, err) + } + if got.String() != tt.in { + t.Errorf("Parse(%q) = %q", tt.in, got) + } + }) + } +} + +func TestDefaultSatisfiesTheGrammar(t *testing.T) { + if _, err := Parse(Default.String()); err != nil { + t.Errorf("Default %q fails its own grammar: %v", Default, err) + } +} diff --git a/internal/testutil/testutil.go b/internal/testutil/testutil.go index 63f9a4c5..f0e39695 100644 --- a/internal/testutil/testutil.go +++ b/internal/testutil/testutil.go @@ -16,6 +16,7 @@ import ( "github.com/stretchr/testify/require" "github.com/Wave-RF/WaveHouse/internal/discovery" + "github.com/Wave-RF/WaveHouse/internal/tenant" ) // NopLogger returns a *slog.Logger that discards all output. @@ -35,7 +36,7 @@ func NopLogger() *slog.Logger { // type string), not the caller's structs. func NewTestSchemaRegistry(t testing.TB, tables []*discovery.TableSchema) *discovery.SchemaRegistry { t.Helper() - reg := discovery.NewSchemaRegistry(&schemaConn{tables: tables}, func() string { return "test" }, func() time.Duration { return time.Hour }, NopLogger()) + reg := discovery.NewSchemaRegistry(&schemaConn{tables: tables}, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, NopLogger()) require.NoError(t, reg.Refresh(context.Background())) return reg } diff --git a/tests/integration/boot_resilience_test.go b/tests/integration/boot_resilience_test.go index 460dc314..b0ecf10d 100644 --- a/tests/integration/boot_resilience_test.go +++ b/tests/integration/boot_resilience_test.go @@ -17,6 +17,7 @@ import ( "github.com/Wave-RF/WaveHouse/internal/api" "github.com/Wave-RF/WaveHouse/internal/discovery" + "github.com/Wave-RF/WaveHouse/internal/tenant" "github.com/Wave-RF/WaveHouse/internal/testutil" ) @@ -67,7 +68,7 @@ func TestBootResilience_StickyHealthVsConditionalReady(t *testing.T) { require.NoError(t, err, "reopen driver against stopped CH") bootState := api.NewBootState(nil) - registry := discovery.NewSchemaRegistry(ch.conn, func() string { return testCHDatabase }, func() time.Duration { return time.Minute }, logger) + registry := discovery.NewSchemaRegistry(ch.conn, func() string { return testCHDatabase }, tenant.Default, func(tenant.ID) time.Duration { return time.Minute }, logger) // === Row 1: Boot, CH down === err = registry.Refresh(ctx) @@ -89,7 +90,7 @@ func TestBootResilience_StickyHealthVsConditionalReady(t *testing.T) { _ = ch.conn.Close() ch.conn, err = openDriver(ch.nativeAddr()) require.NoError(t, err, "reopen driver against restarted CH") - registry = discovery.NewSchemaRegistry(ch.conn, func() string { return testCHDatabase }, func() time.Duration { return time.Minute }, logger) + registry = discovery.NewSchemaRegistry(ch.conn, func() string { return testCHDatabase }, tenant.Default, func(tenant.ID) time.Duration { return time.Minute }, logger) h.CHConn = ch.conn require.NoError(t, waitForNativeReady(ctx, ch.conn, 30*time.Second), "CH native should be ready after restart") diff --git a/tests/integration/query_limits_test.go b/tests/integration/query_limits_test.go index f629644c..4d9b95de 100644 --- a/tests/integration/query_limits_test.go +++ b/tests/integration/query_limits_test.go @@ -17,6 +17,7 @@ import ( "github.com/Wave-RF/WaveHouse/internal/api" "github.com/Wave-RF/WaveHouse/internal/auth" "github.com/Wave-RF/WaveHouse/internal/policy" + "github.com/Wave-RF/WaveHouse/internal/settings" "github.com/Wave-RF/WaveHouse/internal/testutil" ) @@ -92,19 +93,22 @@ func TestStructuredQuery_ResourceCapsEnforcedServerSide(t *testing.T) { // actually executes against ClickHouse (no cross-case cache hit // masking enforcement). defaultMaxRows 0 falls back to the builder's // constant. singleflight's zero value is ready to use. - store := policy.Static(&policy.Policy{ + p := &policy.Policy{ AdminRole: "admin", Tables: map[string]policy.TablePolicy{ table: {"viewer": {Select: &tt.perms}}, }, - }) + } h := api.NewStructuredQueryHandler( - e.chConn, nil, e.registry, store, func() int { return 60 }, func() time.Duration { return 30 * time.Second }, nil, testutil.NopLogger(), + e.chConn, nil, e.registry, func(*settings.Store) *policy.Policy { return p }, func(*settings.Store) int { return 60 }, func() time.Duration { return 30 * time.Second }, nil, testutil.NopLogger(), ) req := httptest.NewRequest(http.MethodPost, "/v1/query?table="+table, strings.NewReader(`{"select_all":true}`)) req = req.WithContext(auth.WithRole(req.Context(), "viewer")) + // The handler is served without the router, so the test stands in + // for TenantMW; the fixed getters above never read the store. + req = req.WithContext(api.WithStore(req.Context(), &settings.Store{})) rec := httptest.NewRecorder() h.Handle(rec, req) From 60e4d6071f18e19b92923a199c398b707897266d Mon Sep 17 00:00:00 2001 From: taitelee Date: Fri, 18 Sep 2026 09:25:36 -0400 Subject: [PATCH 02/11] refactor(logging): log through the slog default logger, ops pipe reads take ?tenant= --- .github/labeler.yml | 5 + AGENTS.md | 6 +- CHANGELOG.md | 4 +- docs/src/content/docs/api.md | 6 +- docs/src/content/docs/architecture.md | 4 +- docs/src/content/docs/development.md | 2 +- internal/api/boot_chain_test.go | 5 +- internal/api/dlq.go | 7 +- internal/api/dlq_test.go | 17 +- internal/api/errors.go | 11 +- internal/api/errors_test.go | 47 +++--- internal/api/ingest.go | 65 ++++---- internal/api/ingest_seams_test.go | 14 +- internal/api/ingest_test.go | 186 +++++++++++----------- internal/api/main_test.go | 15 ++ internal/api/pipes.go | 37 +++-- internal/api/pipes_test.go | 38 +++-- internal/api/router.go | 7 +- internal/api/router_test.go | 39 ++--- internal/api/settings.go | 11 +- internal/api/settings_test.go | 5 +- internal/api/structured_query.go | 5 +- internal/api/structured_query_test.go | 4 +- internal/api/tenant.go | 63 +++++--- internal/api/tenant_test.go | 47 +++++- internal/app/wire.go | 33 ++-- internal/auth/auth.go | 77 ++++----- internal/auth/auth_test.go | 60 ++++--- internal/auth/main_test.go | 15 ++ internal/chconn/chconn.go | 6 +- internal/config/persistence.go | 14 +- internal/config/persistence_test.go | 43 +++-- internal/discovery/discovery.go | 12 +- internal/discovery/discovery_test.go | 57 ++----- internal/discovery/main_test.go | 15 ++ internal/discovery/timestamp.go | 5 +- internal/discovery/timestamp_test.go | 22 +-- internal/ingest/main_test.go | 15 ++ internal/ingest/sweeper.go | 8 +- internal/ingest/sweeper_test.go | 8 +- internal/ingest/worker.go | 34 ++-- internal/ingest/worker_test.go | 18 +-- internal/mq/embedded.go | 54 +++---- internal/mq/embedded_test.go | 14 +- internal/mq/main_test.go | 14 ++ internal/settings/main_test.go | 13 ++ internal/settings/store.go | 41 +++-- internal/settings/store_test.go | 8 +- internal/settings/watch.go | 9 +- internal/stream/hub_test.go | 2 +- internal/testutil/logtest/logtest.go | 59 +++++++ internal/testutil/logtest/logtest_test.go | 29 ++++ internal/testutil/testutil.go | 10 +- tests/integration/boot_resilience_test.go | 6 +- tests/integration/query_limits_test.go | 3 +- 55 files changed, 740 insertions(+), 614 deletions(-) create mode 100644 internal/api/main_test.go create mode 100644 internal/auth/main_test.go create mode 100644 internal/discovery/main_test.go create mode 100644 internal/ingest/main_test.go create mode 100644 internal/mq/main_test.go create mode 100644 internal/settings/main_test.go create mode 100644 internal/testutil/logtest/logtest.go create mode 100644 internal/testutil/logtest/logtest_test.go diff --git a/.github/labeler.yml b/.github/labeler.yml index 502b8c6c..724d59bf 100644 --- a/.github/labeler.yml +++ b/.github/labeler.yml @@ -13,6 +13,11 @@ - any-glob-to-any-file: - "internal/api/**" +"area/tenant": + - changed-files: + - any-glob-to-any-file: + - "internal/tenant/**" + "area/ingest": - changed-files: - any-glob-to-any-file: diff --git a/AGENTS.md b/AGENTS.md index 85a39dc5..c5396666 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -75,10 +75,10 @@ The invariant index — what must stay true. Full narrative and rationale live i ## Code Conventions - **Go 1.26**, strict formatting (`gofumpt`, enforced by CI) -- **Structured logging** with `log/slog` (JSON handler) +- **Structured logging** with `log/slog` (JSON handler), through the default logger: call `slog.InfoContext(ctx, …)` and its siblings (the context carries the trace ids the handler stamps) rather than taking a `*slog.Logger` parameter or field. `cmd/wavehouse` and `internal/app` install the default; tests silence or capture it with `internal/testutil/logtest` (a capturing test must not call `t.Parallel()`) - **Chi v5** for HTTP routing - **Error handling**: Return errors, don't panic. Wrap with `fmt.Errorf("context: %w", err)`. -- **No global state**: Dependencies are passed explicitly (constructor injection). +- **No global state**: Dependencies are passed explicitly (constructor injection). The `slog` default logger is the one sanctioned exception. - **Package naming**: Lowercase, single word (or abbreviated). `internal/` enforces module privacy. ## Craftsmanship @@ -442,7 +442,7 @@ internal/query/ → Structured query AST + SQL builder internal/settings/ → Settings directory (validate, adopted snapshot + reload, watcher, embedded seed) internal/stream/ → SSE fan-out (event Hub: project once per role, Subscriber outbound queue, Bucket fan-out, keepalive Heartbeater wheel) internal/tenant/ → Tenant id (type, grammar, reserved default, request header name) -internal/testutil/ → Shared test helpers (NopLogger, etc.) +internal/testutil/ → Shared test helpers (mocks, JWT + schema helpers; logtest/ captures or silences the default logger) tests/ → Integration & E2E tests tests/integration/ → Go integration tests (//go:build integration; ClickHouse testcontainer) tests/e2e/ → E2E test stack (scripts/orchestrator boots a ClickHouse testcontainer + the wavehouse-cov binary) diff --git a/CHANGELOG.md b/CHANGELOG.md index 33909991..79ef0776 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **Requests resolve to a tenant before authentication, and the tenant is threaded through every settings read** (`internal/tenant/` (new, + tests), `internal/settings/registry.go` (new, + tests), `internal/api/tenant.go` (new, + tests), `internal/api/{router,ingest,structured_query,pipes}.go`, `internal/ingest/{worker,sweeper}.go`, `internal/stream/hub.go`, `internal/discovery/discovery.go`, `internal/app/{app,wire}.go`): story 1 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)), with no behavior change for a deployment that sends no tenant header. `internal/tenant` defines the id — a validated string (letters, digits, `_`, `-`; at most 64 bytes, so it is safe as a folder name and as an MQ subject token), the reserved default `0`, and the `X-Tenant-ID` header name — and imports nothing from the rest of the repository. `api.TenantMW` runs ahead of the auth middleware on every `/v1` route outside `/v1/ops/*`: an absent or empty header is tenant `0`, a malformed id or a repeated header is a `400`, a well-formed id the new `settings.Registry` does not hold is a `404`, and the resolved `*settings.Store` rides the request context. The registry holds the one store `settings.Open` adopted, keyed `0`. Handlers read the store once and pass it down as an argument — the ingest, structured-query, and pipe getters (`PolicySource`, `DedupeSettings`, `bucketSecs`, `defaultMaxRows`, the pipes source) now take it as a parameter — and a tenant route reached without a resolved tenant answers `500` rather than fall back to one. The ingest worker, sweeper, stream hub, and schema registry are constructed with a `tenant.ID` (`tenant.Default` in `internal/app`) and their settings getters take it. The probes, `/version`, the metrics path, and `/v1/ops/*` stay tenant-exempt, with the ops tree behind the auth middleware and the admin gate exactly as before. `X-Tenant-ID` joins the CORS `Access-Control-Allow-Headers` list so a browser client can send it; the SDK needs no change (`options.headers`). +- **Requests resolve to a tenant before authentication, and the tenant is threaded through every settings read** (`internal/tenant/` (new, + tests), `internal/settings/registry.go` (new, + tests), `internal/api/tenant.go` (new, + tests), `internal/api/{router,ingest,structured_query,pipes}.go`, `internal/ingest/{worker,sweeper}.go`, `internal/stream/hub.go`, `internal/discovery/discovery.go`, `internal/app/{app,wire}.go`): story 1 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)), with no behavior change for a deployment that sends no tenant header. `internal/tenant` defines the id — a validated string (letters, digits, `_`, `-`; at most 64 bytes, so it is safe as a folder name and as an MQ subject token), the reserved default `0`, and the `X-Tenant-ID` header name — and imports nothing from the rest of the repository. `api.TenantMW` runs ahead of the auth middleware on every `/v1` route outside `/v1/ops/*`: an absent or empty header is tenant `0`, a malformed id or a repeated header is a `400`, a well-formed id the new `settings.Registry` does not hold is a `404`, and the resolved `*settings.Store` rides the request context. The registry holds the one store `settings.Open` adopted, keyed `0`. Handlers read the store once and pass it down as an argument — the ingest, structured-query, and pipe getters (`PolicySource`, `DedupeSettings`, `bucketSecs`, `defaultMaxRows`, the pipes source) now take it as a parameter — and a tenant route reached without a resolved tenant answers `500` rather than fall back to one. The ingest worker, sweeper, stream hub, and schema registry are constructed with a `tenant.ID` (`tenant.Default` in `internal/app`) and their settings getters take it. The probes, `/version`, the metrics path, and `/v1/ops/*` stay tenant-exempt, with the ops tree behind the auth middleware and the admin gate exactly as before; the admin pipe reads (`GET /v1/ops/pipes[/{name}]`) name their tenant with an optional `?tenant=` query parameter (absent means `0`, same `400`/`404` answers as the header). `X-Tenant-ID` joins the CORS `Access-Control-Allow-Headers` list so a browser client can send it; the SDK needs no change (`options.headers`). - **Schema discovery captures each table's DDL, its columns' ordinals and default expressions, and the server version** (`internal/discovery/discovery.go`, `internal/testutil/testutil.go`): `Column` gains `DefaultExpression` and `Position` (both from a widened `system.columns` select), `TableSchema` gains `DDL` from `system.tables.create_table_query`, and `SchemaRegistry` gains `ServerVersion()` from a `SELECT version()` probe next to the existing `SELECT timezone()`. Groundwork for the native type layer, captured on the same refresh as the columns so a stale version cannot outlive the schemas it describes. That is a publication guarantee, not a same-server one: `chconn.Manager` resolves the connection per call, so a reload changing `clickhouse.addr` mid-refresh can still pair a version from one server with schemas from another — narrow, and self-correcting on the next refresh. `DDL` is `json:"-"` and does **not** appear in `/v1/ops/schema`: that endpoint marshals `TableSchema` straight to the client, and an external-engine table (S3, MySQL, PostgreSQL, Kafka) renders its wiring there unconditionally — endpoint, bucket or host, database, username, S3 access key id. ClickHouse masks the password itself as `[HIDDEN]` from ~23.9 (verified on 26.7.3), so the exposure is the topology rather than the secret — except on an older server, or one with `display_secrets_in_show_and_select` enabled. `position` and `default_expression` are additive fields in the response. A table listed in `system.tables` with no `system.columns` rows is skipped rather than published column-less, and both new queries fail the refresh on error exactly as `timezone()` and `system.columns` do — callers keep the prior cache and retry. @@ -22,6 +22,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Changed +- **Logging goes through the `slog` default logger; no constructor takes a `*slog.Logger` anymore** (`internal/mq/embedded.go`, `internal/api/{ingest,pipes,structured_query,dlq,settings,errors,router}.go`, `internal/auth/auth.go`, `internal/discovery/{discovery,timestamp}.go`, `internal/ingest/{sweeper,worker}.go`, `internal/settings/{store,watch}.go`, `internal/chconn/chconn.go`, `internal/config/persistence.go`, `internal/app/wire.go`, `internal/testutil/logtest/` (new, + tests), `internal/testutil/testutil.go`): the general-notes refactor of [#583](https://github.com/Wave-RF/WaveHouse/issues/583) and the cleanup deferred from [#586](https://github.com/Wave-RF/WaveHouse/pull/586), which left `internal/mq` logging half through an injected logger and half through the default. The logger parameter or field is gone from `mq.NewEmbedded`, `api.NewIngestHandler` / `NewPipesHandler` / `NewStructuredQueryHandler` / `NewDLQHandler` / `NewSettingsHandler`, `api.RequireAdmin` and `api.Dependencies.Logger`, `auth.NewAuthenticator`, `discovery.NewSchemaRegistry`, `ingest.NewSweeper` and the ingest worker, `settings.Open`, `chconn.Open` (whose field was never read), and `config.WarnIfFreshDataDir` / `LogStorageInitError`. Call sites use the context-aware calls (`slog.ErrorContext(ctx, …)`) wherever a context is in scope, so the trace handler can stamp them. Two visible differences: the ingest worker's lines no longer carry `component=ingest_worker`, and the auth middleware's operator-key audit lines and the settings reload lines are no longer skippable by passing a nil logger (only tests did). Tests reach log output through the new `internal/testutil/logtest`: `Silence()` from a package's `TestMain`, and `Capture(t, level)` for a test that asserts on log lines — which therefore runs serially, since the default logger is process-wide. `testutil.NopLogger` is removed. + - **The process wiring moves out of `main.go` into `internal/app`** (`internal/app/` (new: `app.go`, `wire.go`, + tests), `cmd/wavehouse/main.go` (+ tests), `tests/integration/setup_test.go`, `.testcoverage.yml`, `.github/labeler.yml`): `app.New` builds every component from the boot config and the settings directory, `Run` drives the long-lived ones — ingest worker, sweeper, hub bridge, keepalive wheel, schema refresh, SIGHUP and the directory watcher, the API server and the Prometheus sidecar — under one `errgroup` until the signal context is cancelled or one of them fails, and `Close` releases what `New` opened in reverse order. Each component is wired in one place — what it opens, what it loops, what it releases — with the settings store handed to its wiring function whole, so the per-tenant registry ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)) lands there rather than in `main`. `main.go` shrinks to argv dispatch, the logger, `config.Load`, `CheckDataDir`, `app.New`, `app.Run`; `run(ctx)` takes the context `main` cancels on the first `SIGINT`/`SIGTERM`, so it is unit-tested end to end and the per-suite coverage exclude for it is gone. The integration suite boots through the same `app.New` against its testcontainer (the seed settings patched to the container, `default_role` set to the admin role) instead of a hand-built handler subset that had drifted from the binary. Behavior is unchanged except for the stop, which is now bounded end to end in three phases whose budgets add rather than multiply — `server.shutdown_timeout` for the drain, then fixed 5s and 3s for the release and the telemetry flush, so the worst case is the timeout plus 8s: `Run` drains the ingest worker and the API server's in-flight requests — the process could previously exit while the worker drain was still in flight — while open SSE streams are ended the moment the drain begins (a stream is a connection to close, not work to wait for; the client reconnects via `Last-Event-ID`) instead of holding the stop for the whole timeout; then `Close(ctx)` releases the stores under a context of its own, so a remote store's close can give up at the deadline rather than hang the exit, and finally flushes telemetry on a separate short budget so the flush that reports on the stop is never starved by a slow close. A settings reload caught mid-hook by the stop gives up with it. A second `SIGTERM`/`SIGINT` during the stop abandons it and exits non-zero, and `SIGHUP` is ignored once a stop has begun (it was briefly fatal: the reload loop's `signal.Stop` restored the default disposition at the start of the drain). The `mq.max_bytes_gb` reload hook bounds its JetStream calls to ten seconds, and when the DLQ resize fails it rolls the ingest stream back under a budget of its own instead of the one that just expired. `deployments/compose/standalone.yaml` sets `stop_grace_period` to cover all three phases, and the deployment docs gain a [Stopping](https://github.com/Wave-RF/WaveHouse/blob/main/docs/src/content/docs/deployment.md#stopping) section. Closes [#140](https://github.com/Wave-RF/WaveHouse/issues/140); story 0 of #583. - **Boot refuses an unbound `WH_*` environment variable and an unusable `data_dir`** (BREAKING; `internal/config/check.go` (new, + tests), `internal/config/{config,persistence}.go`, `cmd/wavehouse/main.go`, `docs/src/integrations/diagram-png.mjs`): the environment half of the strict YAML loader. `config.Load` now errors, naming every offender, on a `WH_*` variable that no `Config` field binds — the two variables read outside the struct, `WH_CONFIG` and `WH_LOG_LEVEL`, are exempt — `WH_DEDUPE_ENABLED=true` left in a compose file from before the settings-directory move, or a misspelling, was set, ignored, and believed. **An existing deployment that still exports a variable this release moved to the settings directory stops booting until it is unset**; the upgrade runbook in `deployment.md` gains that audit. Only the `WH_` prefix is checked, since the environment always carries unrelated names; the one outside source that shares it — Kubernetes service-link variables for a Service named `wh` or `wh-*` — is named in the error with the `enableServiceLinks: false` remediation, and the docs build's opt-out knob is renamed from `WH_SKIP_DIAGRAM_PNG` to `DOCS_SKIP_DIAGRAM_PNG` so an exported one no longer refuses a local boot. Right after `Load`, before ClickHouse or the settings directory are touched, `config.CheckDataDir` probes `data_dir` and refuses boot on any of: an empty or blank value (reachable through `WH_DATA_DIR=`), refused outright since the ancestor walk would otherwise fall back to the working directory and NATS and Pebble state would land under it; a path that exists and is not a directory; a dangling symlink at `data_dir` or any component above it (the walk to the nearest existing ancestor uses `Lstat`, so a failed mount is not skipped over as "does not exist" and passed in an unrelated directory); and a directory the process cannot write to — or, when it does not exist, an unwritable nearest ancestor — probed by creating and removing one temp file. So an unusable `data_dir` refuses boot before schema discovery rather than after it; a permission denial — on the probe, or on reaching the path at all through a parent without search permission — carries the UID-65532 remediation (a bind mount owned by root is the typical cause), and that hint string is now shared with `LogStorageInitError`. `EnvConfig` and `EnvLogLevel` join `EnvSettingsDir` as the exported names for the process-level variables. Boot is the validator for the non-hot-reloadable half — there is no dry-run subcommand, by decision on #530: boot config only takes effect through a restart, so the restart is where it is checked, and the docs say so. Closes #530. diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index ab54f255..5d3d638d 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -69,7 +69,7 @@ A tenant id is 1–64 characters of ASCII letters, digits, `_`, and `-`. It is a Both are decided before authentication, so they are returned whatever token the request carries. -The probes (`/livez`, `/readyz`, `/healthz`), `/version`, the Prometheus metrics path, and `/v1/ops/*` are tenant-exempt: they ignore the header entirely. +The probes (`/livez`, `/readyz`, `/healthz`), `/version`, the Prometheus metrics path, and `/v1/ops/*` are tenant-exempt: they ignore the header entirely. An ops route that reads a tenant's settings names it with a `?tenant=` query parameter instead ([`GET /v1/ops/pipes`](#get-v1opspipes--list-named-pipes)), with the same grammar and the same `400`/`404` answers. `X-Tenant-ID` is in the CORS `Access-Control-Allow-Headers` list, so a browser client can send it cross-origin. The SDK sends it through [`options.headers`](/sdk#custom-headers). @@ -781,9 +781,11 @@ The policy has no endpoints: it is the settings directory's [`policies.json`](/s Returns every adopted named query pipe — the settings directory's [`pipes.json`](/settings-directory#pipesjson). Pipes have no write endpoints: edit the file and reload. +The ops routes are [tenant-exempt](#tenant-selection), so this read and `GET /v1/ops/pipes/{name}` name their tenant with an optional `?tenant=` query parameter instead of the header. Absent or empty means tenant `0`; a malformed id or a repeated parameter is a `400`, and an unknown tenant a `404`, with the same bodies as the header. + #### `GET /v1/ops/pipes/{name}` — Get Named Pipe -Returns a specific named pipe definition: +Returns a specific named pipe definition (of the `?tenant=`, as above): ```json { diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 9c309bcb..04cd5c30 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -77,7 +77,7 @@ The API layer uses [Chi](https://github.com/go-chi/chi) for routing with Request - **router.go** — Route definitions. Public: `/livez`, `/readyz`, and the content-free `/v1/health` SDK ping (plus the permanent `/healthz` alias and the deprecated `/health`, `/ready` aliases). Policy-gated: `/v1/ingest?table={table}`, `/v1/query?table={table}` (structured), `/v1/pipes/{name}` (named pipes), `/v1/stream`. Admin-only (`RequireAdmin` — role == `policy.admin_role`, or a request bearing the operator key's operator bit, which passes even under a nil policy): `/v1/ops/schema/*`, `/v1/ops/dlq/stats`, `GET /v1/ops/pipes[/{name}]`, `/v1/ops/settings/reload`, `/v1/ops/query` (raw SQL — same gate as the rest of `/v1/ops/*`). - **auth middleware** — the JWT/JWKS authentication middleware is its own package, [`auth/`](#auth--authentication); the router runs it on every `/v1/*` route. -- **tenant.go** — `TenantMW` resolves the request's tenant ahead of the auth middleware on every `/v1` route outside `/v1/ops/*`: the [`X-Tenant-ID`](/api#tenant-selection) header (absent means `tenant.Default`), validated by `tenant.Parse` (`400`), looked up in the `settings.Registry` (`404` on a miss), and the resolved `*settings.Store` stored in the request context (`WithStore` / `StoreFromContext` — here rather than in `tenant/`, because `settings` names `tenant.ID`). A handler reads the store once and passes it down as an argument — the per-tenant getters it holds take it as a parameter (`(*settings.Store).Policy`, `.DedupeFor`, `.DefaultMaxRows`, … in production) — and nothing below a handler reads the context; a tenant route reached without a resolved store answers `500` rather than fall back to a tenant. The probes, `/version`, the metrics path, and `/v1/ops/*` are tenant-exempt; the ops pipe reads serve the default tenant. +- **tenant.go** — `TenantMW` resolves the request's tenant ahead of the auth middleware on every `/v1` route outside `/v1/ops/*`: the [`X-Tenant-ID`](/api#tenant-selection) header (absent means `tenant.Default`), validated by `tenant.Parse` (`400`), looked up in the `settings.Registry` (`404` on a miss), and the resolved `*settings.Store` stored in the request context (`WithStore` / `StoreFromContext` — here rather than in `tenant/`, because `settings` names `tenant.ID`). A handler reads the store once and passes it down as an argument — the per-tenant getters it holds take it as a parameter (`(*settings.Store).Policy`, `.DedupeFor`, `.DefaultMaxRows`, … in production) — and nothing below a handler reads the context; a tenant route reached without a resolved store answers `500` rather than fall back to a tenant. The probes, `/version`, the metrics path, and `/v1/ops/*` are tenant-exempt; an ops read that needs a tenant names it with `?tenant=` (`opsStore`, the pipe reads), resolved through the same `resolveTenant`. - **pipes.go** — Named query pipe handlers: admin listing (`GET /v1/ops/pipes[/{name}]`, read per request from its `pipes.Source`) and execution with parameter binding. `pipes.json` is the only write path. - **structured_query.go** — Handler for `POST /v1/query?table={table}`: validates query AST, enforces permissions, builds and executes SQL. - **ingest.go** — Accepts `POST /v1/ingest?table={table}` in three body shapes: one flat JSON object, a JSON array of them, or NDJSON. The **required** `Content-Type` chooses the format *family* — `application/json` versus the four NDJSON spellings — and within the JSON family the body's first non-whitespace byte picks array versus single object; the bytes never choose the family. Anything that is not exactly one readable media type is a `415`, decided before the body is read: the header is parsed per RFC 9110 §8.3, and because `Content-Type` is a singleton field, repeated header lines must all resolve to the same format and a value carrying a comma is refused unless the value as a whole parses as one media type — a comma inside a *quoted* parameter value is data, so `application/json; a=", application/x-ndjson; b="` is accepted. It then reads the whole (`MaxBytesReader`-capped) body into a pooled buffer and runs the per-format record readers over those bytes, so the `413` lands before any record is processed and peak memory per request is O(body) rather than O(record). Then it validates each record against the discovered schema, optional dedup, and publishes each row through `mq.Publisher` on `mq.Topic{Table, Scope}` (raw names — the subject it becomes is `internal/mq`'s; a full queue comes back as `mq.ErrQueueFull`, which is the `503` + `Retry-After`). When dedup is on, a row missing the configured `id_field` can't be deduped: it is logged at `WARN` and counted by `wavehouse_ingest_dedupe_missing_id_total` (labeled by `table`), then published un-deduped — or rejected when `dedupe.require_id` is set ([#219](https://github.com/Wave-RF/WaveHouse/issues/219)). @@ -104,7 +104,7 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ ### `auth/` — Authentication -- **auth.go** — `NewAuthenticator(cfg, policySource, logger)` owns the verifier; its `Middleware()` reads the current one per request, and `Reconfigure(cfg)` swaps the whole verifier — key source plus its pinned `alg` allowlist — atomically after a settings reload (`auth.jwks_url` / `auth.role_claim`; see [Settings Directory — Authentication](/settings-directory#authentication)). Verifies JWT tokens with HMAC **or** JWKS (never both), with the accepted `alg` pinned to the active verifier and checked before any key is consulted (rejects `alg: none` and cross-family confusion). Extracts the caller's role from a configurable dot-path claim (`auth.role_claim`, default `role`). Claims parse with `jwt.WithJSONNumber()`, so a numeric claim reaches the policy engine as its exact digits (`json.Number`, never a rounded float64) — part of the row-visibility guarantee (AGENTS.md invariant 12). It always runs and never rejects — a missing/invalid/expired token yields an empty role (resolved to `default_role` downstream), with the token error stashed in context so a denying gate can fail loud (`401`, not a bare `403`). Before the Bearer token it checks a non-JWT operator key (`auth.operator_key`): a constant-time match on the presented credential — an `Authorization: Operator ` header, or the `X-Operator-Key` alias — stamps the live admin role plus an operator bit (`auth.WithOperator`) that `RequireAdmin` honors even under a nil policy — a full-access break-glass credential, audit-logged at Info with no client IP (`store`/`logger` back this path). A presented-but-wrong operator key is logged at `WARN` and counted by `wavehouse_auth_operator_key_failures_total` — a probing signal on the most privileged credential — then falls through like any unauthenticated request (the middleware never rejects). +- **auth.go** — `NewAuthenticator(cfg, policySource)` owns the verifier; its `Middleware()` reads the current one per request, and `Reconfigure(cfg)` swaps the whole verifier — key source plus its pinned `alg` allowlist — atomically after a settings reload (`auth.jwks_url` / `auth.role_claim`; see [Settings Directory — Authentication](/settings-directory#authentication)). Verifies JWT tokens with HMAC **or** JWKS (never both), with the accepted `alg` pinned to the active verifier and checked before any key is consulted (rejects `alg: none` and cross-family confusion). Extracts the caller's role from a configurable dot-path claim (`auth.role_claim`, default `role`). Claims parse with `jwt.WithJSONNumber()`, so a numeric claim reaches the policy engine as its exact digits (`json.Number`, never a rounded float64) — part of the row-visibility guarantee (AGENTS.md invariant 12). It always runs and never rejects — a missing/invalid/expired token yields an empty role (resolved to `default_role` downstream), with the token error stashed in context so a denying gate can fail loud (`401`, not a bare `403`). Before the Bearer token it checks a non-JWT operator key (`auth.operator_key`): a constant-time match on the presented credential — an `Authorization: Operator ` header, or the `X-Operator-Key` alias — stamps the live admin role plus an operator bit (`auth.WithOperator`) that `RequireAdmin` honors even under a nil policy — a full-access break-glass credential, audit-logged at Info with no client IP (`store`/`logger` back this path). A presented-but-wrong operator key is logged at `WARN` and counted by `wavehouse_auth_operator_key_failures_total` — a probing signal on the most privileged credential — then falls through like any unauthenticated request (the middleware never rejects). - **context.go** — request-context accessors and their setters for the role, claims, and token error (`RoleFromContext`, `ClaimsFromContext`, `AuthErrorFromContext`, and the matching `With*` helpers). ### `cache/` — Query Cache diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index 143ad90f..68eaff68 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -347,7 +347,7 @@ Each test target writes `covdata` to `tmp/coverage//data/`, renders a tex - **Unit tests** live beside the code they test (e.g., `internal/discovery/discovery_test.go`). They use mocks or embedded NATS (in-process, no Docker needed). - **Integration tests** use the `//go:build integration` build tag. `TestMain` starts one ClickHouse testcontainer and boots the production wiring against it through `app.New` (embedded NATS, ingest worker, sweeper, hub, the API server on a random loopback port); tests reach it via `env(t)` and create their own tables. DLQ tests use `assert.Eventually` with a 30-second timeout for the 5-second ingest worker batch window. -Shared test utilities live in `internal/testutil/` (e.g., `testutil.NopLogger()` for silencing embedded NATS output). +Shared test utilities live in `internal/testutil/`. The packages log through `slog.Default()`, so tests reach log output through `internal/testutil/logtest`: `logtest.Silence()` in a package's `TestMain` discards it, and `logtest.Capture(t, level)` routes it to a buffer for a test that asserts on log lines — such a test must not call `t.Parallel()`, because the default logger is process-wide. ### Adding New Tests diff --git a/internal/api/boot_chain_test.go b/internal/api/boot_chain_test.go index 088557b4..213d7e5a 100644 --- a/internal/api/boot_chain_test.go +++ b/internal/api/boot_chain_test.go @@ -4,8 +4,6 @@ import ( "context" "errors" "fmt" - "io" - "log/slog" "net/http" "net/http/httptest" "strings" @@ -89,8 +87,7 @@ func TestBoot_Chain_DegradedThenRecovers(t *testing.T) { // comes up partway through the retry backoff. conn := &errsThenSuccessConn{errs: []error{connRefused, connRefused, dbMissing}} - logger := slog.New(slog.NewJSONHandler(io.Discard, nil)) - registry := discovery.NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, logger) + registry := discovery.NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }) // Phase 0 — synchronous boot Refresh fails. internal/app records the // diagnostic in BootState and proceeds with the retry loop in a diff --git a/internal/api/dlq.go b/internal/api/dlq.go index e5826b70..9de69ad5 100644 --- a/internal/api/dlq.go +++ b/internal/api/dlq.go @@ -12,11 +12,10 @@ import ( // DLQHandler exposes Dead Letter Queue statistics. type DLQHandler struct { Counts mq.DeadLetterStats - Logger *slog.Logger } -func NewDLQHandler(stats mq.DeadLetterStats, logger *slog.Logger) *DLQHandler { - return &DLQHandler{Counts: stats, Logger: logger} +func NewDLQHandler(stats mq.DeadLetterStats) *DLQHandler { + return &DLQHandler{Counts: stats} } // Stats returns per-table message counts on the dead-letter queue. @@ -25,7 +24,7 @@ func (h *DLQHandler) Stats(w http.ResponseWriter, r *http.Request) { counts, err := h.Counts.DeadLetterCounts(r.Context(), r.URL.Query().Get("table")) if err != nil { if !errors.Is(err, mq.ErrNoDeadLetterQueue) { - h.Logger.ErrorContext(r.Context(), "dlq stats failed", "error", err) + slog.ErrorContext(r.Context(), "dlq stats failed", "error", err) writeJSONError(w, http.StatusInternalServerError, "stream info failed") return } diff --git a/internal/api/dlq_test.go b/internal/api/dlq_test.go index a9bcd088..03f8acf9 100644 --- a/internal/api/dlq_test.go +++ b/internal/api/dlq_test.go @@ -4,7 +4,6 @@ import ( "context" "encoding/json" "errors" - "log/slog" "net/http" "net/http/httptest" "testing" @@ -26,7 +25,7 @@ func parkedMsg(table string) *mq.Message { func TestDLQStats_EmptyWhenNoStream(t *testing.T) { // The embedded MQ always has a dead-letter queue, so its absence comes // from a mock. - handler := NewDLQHandler(&testutil.MockDeadLetterStats{Err: mq.ErrNoDeadLetterQueue}, slog.Default()) + handler := NewDLQHandler(&testutil.MockDeadLetterStats{Err: mq.ErrNoDeadLetterQueue}) req := httptest.NewRequestWithContext(context.Background(), http.MethodGet, "/v1/ops/dlq/stats", nil) rec := httptest.NewRecorder() @@ -46,7 +45,7 @@ func TestDLQStats_EmptyWhenNoStream(t *testing.T) { func TestDLQStats_ReturnsCorrectCounts(t *testing.T) { dir := t.TempDir() - emb, err := mq.NewEmbedded(dir, 1024*1024, testutil.NopLogger()) + emb, err := mq.NewEmbedded(dir, 1024*1024) require.NoError(t, err) defer func() { _ = emb.Close() }() @@ -60,7 +59,7 @@ func TestDLQStats_ReturnsCorrectCounts(t *testing.T) { require.NoError(t, emb.DeadLetter(ctx, parkedMsg("users"))) } - handler := NewDLQHandler(emb, slog.Default()) + handler := NewDLQHandler(emb) req := httptest.NewRequestWithContext(context.Background(), http.MethodGet, "/v1/ops/dlq/stats", nil) rec := httptest.NewRecorder() @@ -80,7 +79,7 @@ func TestDLQStats_ReturnsCorrectCounts(t *testing.T) { func TestDLQStats_SingleTable(t *testing.T) { dir := t.TempDir() - emb, err := mq.NewEmbedded(dir, 1024*1024, testutil.NopLogger()) + emb, err := mq.NewEmbedded(dir, 1024*1024) require.NoError(t, err) defer func() { _ = emb.Close() }() @@ -88,7 +87,7 @@ func TestDLQStats_SingleTable(t *testing.T) { require.NoError(t, emb.DeadLetter(ctx, parkedMsg("orders"))) - handler := NewDLQHandler(emb, slog.Default()) + handler := NewDLQHandler(emb) req := httptest.NewRequestWithContext(context.Background(), http.MethodGet, "/v1/ops/dlq/stats", nil) rec := httptest.NewRecorder() @@ -105,7 +104,7 @@ func TestDLQStats_SingleTable(t *testing.T) { } func TestDLQStats_BrokerFailureIsAnError(t *testing.T) { - handler := NewDLQHandler(&testutil.MockDeadLetterStats{Err: errors.New("broker unavailable")}, testutil.NopLogger()) + handler := NewDLQHandler(&testutil.MockDeadLetterStats{Err: errors.New("broker unavailable")}) req := httptest.NewRequestWithContext(context.Background(), http.MethodGet, "/v1/ops/dlq/stats", nil) rec := httptest.NewRecorder() @@ -116,7 +115,7 @@ func TestDLQStats_BrokerFailureIsAnError(t *testing.T) { } func TestDLQStats_PassesTheTableFilter(t *testing.T) { - emb, err := mq.NewEmbedded(t.TempDir(), 1024*1024, testutil.NopLogger()) + emb, err := mq.NewEmbedded(t.TempDir(), 1024*1024) require.NoError(t, err) defer func() { _ = emb.Close() }() @@ -124,7 +123,7 @@ func TestDLQStats_PassesTheTableFilter(t *testing.T) { require.NoError(t, emb.DeadLetter(ctx, parkedMsg("default.orders"))) require.NoError(t, emb.DeadLetter(ctx, parkedMsg("users"))) - handler := NewDLQHandler(emb, slog.Default()) + handler := NewDLQHandler(emb) req := httptest.NewRequestWithContext(ctx, http.MethodGet, "/v1/ops/dlq/stats?table=default.orders", nil) rec := httptest.NewRecorder() diff --git a/internal/api/errors.go b/internal/api/errors.go index 876b397b..ab71601a 100644 --- a/internal/api/errors.go +++ b/internal/api/errors.go @@ -41,10 +41,7 @@ func writeJSONError(w http.ResponseWriter, status int, message string) { // "gate" (admin / policy / pipe) so a denial is attributable to the check that // raised it without parsing the route pattern, and the policy paths add the // table + action they evaluated. -// logger is the calling gate's injected logger (each handler holds one; main -// wires it, tests pass their own) — the denial WARN goes there, not to a -// package global. -func writeAuthzDenied(w http.ResponseWriter, r *http.Request, logger *slog.Logger, role string, allowedRoles []string, attrs ...slog.Attr) { +func writeAuthzDenied(w http.ResponseWriter, r *http.Request, role string, allowedRoles []string, attrs ...slog.Attr) { authErr := auth.AuthErrorFromContext(r.Context()) // reason tracks the response: a present-but-invalid token fails loud (401) @@ -59,7 +56,7 @@ func writeAuthzDenied(w http.ResponseWriter, r *http.Request, logger *slog.Logge reason = "no role and no default_role configured" } - logAuthzDenied(logger, r, reason, role, allowedRoles, status, attrs...) + logAuthzDenied(r, reason, role, allowedRoles, status, attrs...) if authErr != nil { writeJSONError(w, http.StatusUnauthorized, authErr.Error()) @@ -82,7 +79,7 @@ func writeAuthzDenied(w http.ResponseWriter, r *http.Request, logger *slog.Logge // slog escapes control characters in string values, so the request-derived // fields (route, method, role) carry no log-injection risk despite originating // in an *http.Request scope. -func logAuthzDenied(logger *slog.Logger, r *http.Request, reason, resolvedRole string, allowedRoles []string, status int, attrs ...slog.Attr) { +func logAuthzDenied(r *http.Request, reason, resolvedRole string, allowedRoles []string, status int, attrs ...slog.Attr) { // Prefer the matched route template (e.g. /v1/pipes/{name}) over the raw // path: it keeps the field low-cardinality and avoids logging concrete path // params. Falls back to the path when there's no chi route context (a gate @@ -103,7 +100,7 @@ func logAuthzDenied(logger *slog.Logger, r *http.Request, reason, resolvedRole s slog.Int("status", status), } - logger.LogAttrs(r.Context(), slog.LevelWarn, "authorization denied", append(fields, attrs...)...) + slog.LogAttrs(r.Context(), slog.LevelWarn, "authorization denied", append(fields, attrs...)...) } // forbiddenForRole returns the 403 message body for a policy/allowlist denial. diff --git a/internal/api/errors_test.go b/internal/api/errors_test.go index f1e23f24..ae35469a 100644 --- a/internal/api/errors_test.go +++ b/internal/api/errors_test.go @@ -1,7 +1,6 @@ package api import ( - "bytes" "context" "encoding/json" "errors" @@ -16,6 +15,7 @@ import ( "github.com/Wave-RF/WaveHouse/internal/stream" "github.com/Wave-RF/WaveHouse/internal/tenant" "github.com/Wave-RF/WaveHouse/internal/testutil" + "github.com/Wave-RF/WaveHouse/internal/testutil/logtest" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) @@ -44,14 +44,12 @@ func TestWriteJSONError_EscapesSpecialCharacters(t *testing.T) { assert.Equal(t, `oops "quoted" \n`, body["error"]) } -// warnBufLogger returns a WARN-level JSON logger that writes to the returned -// buffer. It's injected into the gate/handler under test (the way -// internal/policy/store_test.go injects one into NewStore), so reading a -// denial's structured WARN needs no process-global slog.SetDefault swap — which -// is why, unlike the old default-logger capture, these tests can run in parallel. -func warnBufLogger() (*slog.Logger, *bytes.Buffer) { - var buf bytes.Buffer - return slog.New(slog.NewJSONHandler(&buf, &slog.HandlerOptions{Level: slog.LevelWarn})), &buf +// captureWarns routes the default logger's WARN+ records to the returned +// buffer. The default logger is process-wide, so the tests that call it run +// serially (see logtest.Capture). +func captureWarns(t *testing.T) *logtest.Buffer { + t.Helper() + return logtest.Capture(t, slog.LevelWarn) } // TestRequireAdmin_DenialLogsStructuredWarn pins the structured WARN emitted on @@ -59,9 +57,8 @@ func warnBufLogger() (*slog.Logger, *bytes.Buffer) { // allowlist" reason, and gate=admin (so the denial is attributable to the admin // check, which the route pattern alone can't convey). func TestRequireAdmin_DenialLogsStructuredWarn(t *testing.T) { - t.Parallel() - logger, buf := warnBufLogger() - handler := RequireAdmin(policy.Static(&policy.Policy{}), logger)(http.HandlerFunc(func(http.ResponseWriter, *http.Request) { + buf := captureWarns(t) + handler := RequireAdmin(policy.Static(&policy.Policy{}))(http.HandlerFunc(func(http.ResponseWriter, *http.Request) { t.Fatal("handler must not run on a denied request") })) @@ -87,10 +84,9 @@ func TestRequireAdmin_DenialLogsStructuredWarn(t *testing.T) { // while role_resolved is the default — the signal that says "the public default // role can't reach admin", not "the client sent the wrong role". func TestRequireAdmin_EmptyRoleDenialLogsResolvedRole(t *testing.T) { - t.Parallel() - logger, buf := warnBufLogger() + buf := captureWarns(t) store := policy.Static(&policy.Policy{DefaultRole: "viewer"}) - handler := RequireAdmin(store, logger)(http.HandlerFunc(func(http.ResponseWriter, *http.Request) { + handler := RequireAdmin(store)(http.HandlerFunc(func(http.ResponseWriter, *http.Request) { t.Fatal("handler must not run on a denied request") })) @@ -109,9 +105,8 @@ func TestRequireAdmin_EmptyRoleDenialLogsResolvedRole(t *testing.T) { // 401, distinguishing it from an ordinary roleless 403. The admin gate logs no // explicit allowlist, so roles_allowed is null. func TestRequireAdmin_InvalidTokenDenialLogsFailLoudReason(t *testing.T) { - t.Parallel() - logger, buf := warnBufLogger() - handler := RequireAdmin(nil, logger)(http.HandlerFunc(func(http.ResponseWriter, *http.Request) { + buf := captureWarns(t) + handler := RequireAdmin(nil)(http.HandlerFunc(func(http.ResponseWriter, *http.Request) { t.Fatal("handler must not run on a denied request") })) @@ -132,12 +127,11 @@ func TestRequireAdmin_InvalidTokenDenialLogsFailLoudReason(t *testing.T) { // caller through — and which pipe (the route pattern is /v1/pipes/{name}, so the // concrete name isn't in the route field). func TestPipesHandler_Execute_DenialLogsAllowedRoles(t *testing.T) { - t.Parallel() - logger, buf := warnBufLogger() + buf := captureWarns(t) store := staticPipes( &pipes.NamedQuery{Name: "report", SQL: "SELECT * FROM clicks", AllowedRoles: []string{"analyst", "viewer"}}, ) - h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout, logger) + h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout) r := pipesRequest(t, http.MethodPost, "/v1/pipes/report/execute", "report", nil) r = r.WithContext(auth.WithRole(r.Context(), "guest")) @@ -160,9 +154,8 @@ func TestPipesHandler_Execute_DenialLogsAllowedRoles(t *testing.T) { // denial is distinguishable from an admin-gate or pipe-allowlist denial — the // /v1/ingest route pattern alone doesn't say which check failed, or on what. func TestIngest_DenialLogsPolicyGate(t *testing.T) { - t.Parallel() - logger, buf := warnBufLogger() - h := NewIngestHandler(testRegistry(t), &testutil.MockPublisher{}, logger) + buf := captureWarns(t) + h := NewIngestHandler(testRegistry(t), &testutil.MockPublisher{}) h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": {"viewer": {Select: &policy.SelectPermissions{}}}, // no insert for viewer @@ -189,19 +182,17 @@ func TestIngest_DenialLogsPolicyGate(t *testing.T) { // it denies before sub-route matching and the template is /v1/ops/*; the // gate=admin attribute tells the operator which check denied it. func TestAuthzDenied_LogsChiRoutePattern(t *testing.T) { - t.Parallel() - logger, buf := warnBufLogger() + buf := captureWarns(t) reg := testutil.NewTestSchemaRegistry(t, nil) router := NewRouter(Dependencies{ Tenants: testTenants(), - Ingest: NewIngestHandler(reg, &testutil.MockPublisher{}, logger), + Ingest: NewIngestHandler(reg, &testutil.MockPublisher{}), Query: &QueryHandler{}, SSE: NewStreamHandler(stream.NewHub(tenant.Default, nil, nil, nil), nil), Health: &HealthHandler{}, Schema: NewSchemaHandler(reg), AuthMW: func(next http.Handler) http.Handler { return next }, PolicySource: policy.Static(&policy.Policy{}), - Logger: logger, }) ctx := auth.WithRole(context.Background(), "viewer") diff --git a/internal/api/ingest.go b/internal/api/ingest.go index 56f64b0e..bc0f2014 100644 --- a/internal/api/ingest.go +++ b/internal/api/ingest.go @@ -47,7 +47,6 @@ type IngestHandler struct { DedupeSettings func(store *settings.Store, table string) (enabled bool, idField string, requireID bool) Publisher mq.Publisher PolicySource PolicySource - logger *slog.Logger // Validator and Checker are the per-record seams a native type layer will // take over (see ingest_seams.go). Both are optional: nil means the default @@ -62,8 +61,8 @@ type IngestHandler struct { maxRequestBytes int64 } -func NewIngestHandler(registry *discovery.SchemaRegistry, pub mq.Publisher, logger *slog.Logger) *IngestHandler { - return &IngestHandler{Registry: registry, Publisher: pub, logger: logger} +func NewIngestHandler(registry *discovery.SchemaRegistry, pub mq.Publisher) *IngestHandler { + return &IngestHandler{Registry: registry, Publisher: pub} } var dedupeMissingIDCounter, _ = otel.Meter("wavehouse-ingest").Int64Counter( @@ -152,14 +151,14 @@ func (h *IngestHandler) Handle(w http.ResponseWriter, r *http.Request) { defer span.End() // Add a standard log to prove we are inside the span logic - h.logger.DebugContext(ctx, "debug: span started for ingest", "table", table) + slog.DebugContext(ctx, "debug: span started for ingest", "table", table) r = r.WithContext(ctx) // TODO: what should the order of these be to maximize speed + limit risk of data leakage or DoS/resource exhaustion? if table == "" { - h.logger.ErrorContext(ctx, "missing table parameter in request") + slog.ErrorContext(ctx, "missing table parameter in request") writeJSONError(w, http.StatusBadRequest, "missing table") return } @@ -167,7 +166,7 @@ func (h *IngestHandler) Handle(w http.ResponseWriter, r *http.Request) { // TODO: prevent table-enumeration... schema := h.Registry.Get(table) if schema == nil { - h.logger.WarnContext(ctx, "unknown table requested", "table", table) + slog.WarnContext(ctx, "unknown table requested", "table", table) writeJSONError(w, http.StatusNotFound, "unknown table: "+table) return } @@ -185,7 +184,7 @@ func (h *IngestHandler) Handle(w http.ResponseWriter, r *http.Request) { claims, _ := auth.ClaimsFromContext(ctx) perms = policy.Evaluate(p, role, table, "insert", claims) if !perms.Allowed { - writeAuthzDenied(w, r, h.logger, role, nil, + writeAuthzDenied(w, r, role, nil, slog.String("gate", "policy"), slog.String("table", table), slog.String("action", "insert"), @@ -221,7 +220,7 @@ func (h *IngestHandler) Handle(w http.ResponseWriter, r *http.Request) { // caller's own doing, and a proxy that starts duplicating the header // would otherwise produce a wall of client-side 415s whose // server-side record named only one of the declarations involved. - h.logger.WarnContext(ctx, "conflicting ingest content-type declarations", + slog.WarnContext(ctx, "conflicting ingest content-type declarations", "content_types", decls, "table", table) } else { // Logs the same bounded set the response shows, like the conflicting @@ -229,7 +228,7 @@ func (h *IngestHandler) Handle(w http.ResponseWriter, r *http.Request) { // disagree with what the caller was told — for // ["", "text/csv"] the client sees `Content-Type "", "text/csv"` // while Header.Get would have logged only content_type="". - h.logger.WarnContext(ctx, "ingest content-type not declared or not supported", + slog.WarnContext(ctx, "ingest content-type not declared or not supported", "content_types", decls, "table", table) } writeJSONError(w, http.StatusUnsupportedMediaType, contentTypeMessage(decls, conflicting)) @@ -254,7 +253,7 @@ func (h *IngestHandler) Handle(w http.ResponseWriter, r *http.Request) { if writeMaxBytesError(w, err, reqCap) { return } - h.logger.WarnContext(ctx, "ingest body read failed", "error", err, "table", table) + slog.WarnContext(ctx, "ingest body read failed", "error", err, "table", table) writeJSONError(w, http.StatusBadRequest, "invalid request body") return } @@ -265,7 +264,7 @@ func (h *IngestHandler) Handle(w http.ResponseWriter, r *http.Request) { // only error left is an empty body. rr, batch, err := newRecordReader(format, body.Bytes()) if err != nil { - h.logger.ErrorContext(ctx, "empty ingest body", "table", table, "format", format.String()) + slog.ErrorContext(ctx, "empty ingest body", "table", table, "format", format.String()) writeJSONError(w, http.StatusBadRequest, emptyBodyMessage(format)) return } @@ -301,7 +300,7 @@ func (h *IngestHandler) handleSingle( if writeMaxBytesError(w, err, reqCap) { return } - h.logger.ErrorContext(ctx, "invalid json payload", "error", err, "table", table) + slog.ErrorContext(ctx, "invalid json payload", "error", err, "table", table) writeJSONError(w, http.StatusBadRequest, "invalid json") return } @@ -321,7 +320,7 @@ func (h *IngestHandler) handleSingle( return } - h.logger.InfoContext(ctx, "event successfully ingested", "table", table) + slog.InfoContext(ctx, "event successfully ingested", "table", table) w.Header().Set("Content-Type", "application/json") _ = json.NewEncoder(w).Encode(map[string]bool{"ok": true}) } @@ -370,7 +369,7 @@ func (h *IngestHandler) handleBatch( // NDJSON line) — the reader can't resume, so fail the request rather // than report a misleading partial summary. Not a body read error: // the readers run over an in-memory slice now. - h.logger.WarnContext(ctx, "ingest read error", "error", err, "table", table) + slog.WarnContext(ctx, "ingest read error", "error", err, "table", table) writeJSONError(w, http.StatusBadRequest, "invalid json: "+err.Error()) return } @@ -398,7 +397,7 @@ func (h *IngestHandler) handleBatch( appendResult(&result, recordResult{Index: idx, Ok: true}) } - h.logger.InfoContext(ctx, "batch ingested", "table", table, + slog.InfoContext(ctx, "batch ingested", "table", table, "total", result.Total, "succeeded", result.Succeeded, "failed", result.Failed, "duplicates", result.Duplicates) w.Header().Set("Content-Type", "application/json") @@ -477,11 +476,11 @@ func (h *IngestHandler) policyCheckGuard( schemaCol, known := schema.Lookup(col) switch { case !known: - h.logger.ErrorContext(ctx, "policy check references a column the table does not have", + slog.ErrorContext(ctx, "policy check references a column the table does not have", "column", col, "table", table, "role", role) reasons = append(reasons, fmt.Sprintf("%q, which table %q does not have", col, table)) case !schemaCol.IsInsertable(): - h.logger.ErrorContext(ctx, "policy check references a column no record may write", + slog.ErrorContext(ctx, "policy check references a column no record may write", "column", col, "table", table, "role", role, "default_kind", schemaCol.DefaultKind) reasons = append(reasons, fmt.Sprintf("%q of table %q, which is %s and cannot be inserted", col, table, strings.ToLower(schemaCol.DefaultKind))) @@ -492,7 +491,7 @@ func (h *IngestHandler) policyCheckGuard( // Accepting a check that provably does nothing is worse than refusing // it. An operator wanting this should check the DEFAULT column derived // from the ephemeral one, which is stored and therefore enforceable. - h.logger.ErrorContext(ctx, "policy check references an ephemeral column, which is never stored", + slog.ErrorContext(ctx, "policy check references an ephemeral column, which is never stored", "column", col, "table", table, "role", role) reasons = append(reasons, fmt.Sprintf("%q of table %q, which is ephemeral and is never stored", col, table)) @@ -531,7 +530,7 @@ func (h *IngestHandler) processRecord( checkGuard *recordReject, ) (duplicate bool, reject *recordReject, abort *requestAbort) { if err := h.validator().Validate(schema, data); err != nil { - h.logger.WarnContext(ctx, "schema validation failed", "error", err, "table", table) + slog.WarnContext(ctx, "schema validation failed", "error", err, "table", table) return false, &recordReject{Status: http.StatusBadRequest, Message: err.Error()}, nil } @@ -539,7 +538,7 @@ func (h *IngestHandler) processRecord( if perms != nil { for col := range data { if !perms.IsColumnAllowed(col, true) { - h.logger.WarnContext(ctx, "column insertion forbidden", "column", col, "role", role) + slog.WarnContext(ctx, "column insertion forbidden", "column", col, "role", role) return false, &recordReject{ Status: http.StatusForbidden, Message: fmt.Sprintf("column %q not allowed for insert", col), @@ -562,7 +561,7 @@ func (h *IngestHandler) processRecord( // so this is true for every record or none. As a reject, a 10k-record // batch would emit 10k ERROR lines and report 10k independent // permission failures for one mis-wired grant. - h.logger.ErrorContext(ctx, "insert checks consulted on a grant resolved for another operation", + slog.ErrorContext(ctx, "insert checks consulted on a grant resolved for another operation", "table", table, "role", role) return false, nil, &requestAbort{ Status: http.StatusForbidden, @@ -587,7 +586,7 @@ func (h *IngestHandler) processRecord( if set, isSet := requiredVal.([]any); isSet { actual, ok := data[col] if !ok || !h.checker().InSet(actual, set) { - h.logger.WarnContext(ctx, "check clause failed", "column", col, "allowed", set, "actual", actual, "present", ok) + slog.WarnContext(ctx, "check clause failed", "column", col, "allowed", set, "actual", actual, "present", ok) return false, &recordReject{ Status: http.StatusForbidden, Message: fmt.Sprintf("check failed for column %q", col), @@ -607,7 +606,7 @@ func (h *IngestHandler) processRecord( // claim-derived value arrives as a plain string and never gains a // reading the token's own JSON type didn't give it. if !h.checker().Matches(actual, requiredVal) { - h.logger.WarnContext(ctx, "check clause failed", "column", col, "expected", requiredVal, "actual", actual) + slog.WarnContext(ctx, "check clause failed", "column", col, "expected", requiredVal, "actual", actual) return false, &recordReject{ Status: http.StatusForbidden, Message: fmt.Sprintf("check failed for column %q", col), @@ -642,13 +641,13 @@ func (h *IngestHandler) processRecord( if !ok { dedupeMissingIDCounter.Add(ctx, 1, metric.WithAttributes(attribute.String("table", table))) if requireID { - h.logger.WarnContext(ctx, "dedupe id_field missing; rejecting", "id_field", idField, "table", table) + slog.WarnContext(ctx, "dedupe id_field missing; rejecting", "id_field", idField, "table", table) return false, &recordReject{ Status: http.StatusBadRequest, Message: fmt.Sprintf("missing dedupe id field %q", idField), }, nil } - h.logger.WarnContext(ctx, "dedupe id_field missing; publishing without idempotency", "id_field", idField, "table", table) + slog.WarnContext(ctx, "dedupe id_field missing; publishing without idempotency", "id_field", idField, "table", table) } else { eventID := fmt.Sprint(idVal) dup, err := h.Dedup.CheckAndMark(ctx, eventID) @@ -662,12 +661,12 @@ func (h *IngestHandler) processRecord( // is the store and settings out of step), so the line is // Debug rather than a WARN per record. dedupeDisabledCounter.Add(ctx, 1, metric.WithAttributes(attribute.String("table", table))) - h.logger.DebugContext(ctx, "dedupe switched off mid-reload; publishing without idempotency", "event_id", eventID, "table", table) + slog.DebugContext(ctx, "dedupe switched off mid-reload; publishing without idempotency", "event_id", eventID, "table", table) case err != nil: - h.logger.ErrorContext(ctx, "dedupe check failed", "error", err, "event_id", eventID) + slog.ErrorContext(ctx, "dedupe check failed", "error", err, "event_id", eventID) return false, nil, &requestAbort{Status: http.StatusInternalServerError, Message: "dedupe failed"} case dup: - h.logger.InfoContext(ctx, "duplicate event skipped", "event_id", eventID) + slog.InfoContext(ctx, "duplicate event skipped", "event_id", eventID) return true, nil, nil } } @@ -681,7 +680,7 @@ func (h *IngestHandler) processRecord( cols := schema.InsertableColumns() row, err := ingest.EncodeCompactRow(cols, data) if err != nil { - h.logger.ErrorContext(ctx, "failed to encode compact row", "error", err, "table", table) + slog.ErrorContext(ctx, "failed to encode compact row", "error", err, "table", table) return false, nil, &requestAbort{Status: http.StatusInternalServerError, Message: "marshal failed"} } @@ -696,17 +695,17 @@ func (h *IngestHandler) processRecord( payload, err := json.Marshal(evt) if err != nil { - h.logger.ErrorContext(ctx, "failed to marshal event message", "error", err) + slog.ErrorContext(ctx, "failed to marshal event message", "error", err) return false, nil, &requestAbort{Status: http.StatusInternalServerError, Message: "marshal failed"} } - h.logger.DebugContext(ctx, "publishing event to the ingest queue", "table", table, "scope", scope) + slog.DebugContext(ctx, "publishing event to the ingest queue", "table", table, "scope", scope) if err := h.Publisher.Publish(ctx, mq.Topic{Table: table, Scope: scope}, payload); err != nil { if errors.Is(err, mq.ErrQueueFull) { - h.logger.WarnContext(ctx, "ingest queue is full", "table", table, "scope", scope) + slog.WarnContext(ctx, "ingest queue is full", "table", table, "scope", scope) return false, nil, &requestAbort{Status: http.StatusServiceUnavailable, Message: "service unavailable", RetryAfter: "30"} } - h.logger.ErrorContext(ctx, "failed to publish to the ingest queue", "error", err, "table", table, "scope", scope) + slog.ErrorContext(ctx, "failed to publish to the ingest queue", "error", err, "table", table, "scope", scope) return false, nil, &requestAbort{Status: http.StatusInternalServerError, Message: "publish failed"} } diff --git a/internal/api/ingest_seams_test.go b/internal/api/ingest_seams_test.go index f365335a..dc7de285 100644 --- a/internal/api/ingest_seams_test.go +++ b/internal/api/ingest_seams_test.go @@ -48,7 +48,7 @@ func TestIngest_RecordValidatorSeam_IsUsed(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} v := &recordingValidator{validateErr: errors.New("seam says no")} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.Validator = v w := httptest.NewRecorder() @@ -66,7 +66,7 @@ func TestIngest_RecordValidatorSeam_IsUsed(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} v := &recordingValidator{canonicalizeAs: "/rewritten"} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.Validator = v w := httptest.NewRecorder() @@ -85,7 +85,7 @@ func TestIngest_RecordValidatorSeam_IsUsed(t *testing.T) { func TestIngest_DefaultValidator_WhenUnwired(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) require.Nil(t, h.Validator) assert.IsType(t, discoveryValidator{}, h.validator()) @@ -129,7 +129,7 @@ func TestIngest_InsertCheckerSeam_IsUsed(t *testing.T) { t.Run(tt.name, func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.PolicySource = staticPolicy(p) h.Checker = alwaysChecker{matches: tt.matches} @@ -168,7 +168,7 @@ func TestIngest_InsertCheckerSeam_InSet_IsUsed(t *testing.T) { t.Run(tt.name, func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.PolicySource = checkInStore() h.Checker = alwaysChecker{inSet: tt.inSet} @@ -198,7 +198,7 @@ func TestIngest_DefaultChecker_WhenUnwired(t *testing.T) { }}}, }} pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.PolicySource = staticPolicy(p) require.Nil(t, h.Checker) assert.IsType(t, canonicalChecker{}, h.checker()) @@ -224,7 +224,7 @@ func TestIngest_SeamOrdering_ChecksSitBetweenValidateAndCanonicalize(t *testing. }} pub := &testutil.MockPublisher{} v := &recordingValidator{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.PolicySource = staticPolicy(p) h.Validator = v diff --git a/internal/api/ingest_test.go b/internal/api/ingest_test.go index 8cf57a78..64cb277c 100644 --- a/internal/api/ingest_test.go +++ b/internal/api/ingest_test.go @@ -23,6 +23,7 @@ import ( "github.com/Wave-RF/WaveHouse/internal/policy" "github.com/Wave-RF/WaveHouse/internal/settings" "github.com/Wave-RF/WaveHouse/internal/testutil" + "github.com/Wave-RF/WaveHouse/internal/testutil/logtest" "github.com/golang-jwt/jwt/v5" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" @@ -57,7 +58,7 @@ func ingestRequest(t *testing.T, table string, body any) *http.Request { func TestIngest_ValidPayload(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := ingestRequest(t, "clicks", map[string]any{"page": "/home", "count": 1}) w := httptest.NewRecorder() @@ -107,7 +108,7 @@ func TestIngest_MissingTable(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := httptest.NewRequestWithContext( context.Background(), @@ -130,7 +131,7 @@ func TestIngest_MissingTable(t *testing.T) { func TestIngest_UnknownTable(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := ingestRequest(t, "nonexistent", map[string]any{"x": 1}) w := httptest.NewRecorder() @@ -144,7 +145,7 @@ func TestIngest_UnknownTable(t *testing.T) { func TestIngest_InvalidJSON(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) r := rawIngestRequest(t, "clicks", "application/json", "not json") @@ -158,7 +159,7 @@ func TestIngest_InvalidJSON(t *testing.T) { func TestIngest_SchemaValidation_UnknownField(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := ingestRequest(t, "clicks", map[string]any{"page": "/home", "nonexistent_field": 42}) w := httptest.NewRecorder() @@ -172,7 +173,7 @@ func TestIngest_Dedup_FirstTime(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} dedup := testutil.NewMockDeduplicator() - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.Dedup = dedup h.DedupeSettings = func(*settings.Store, string) (bool, string, bool) { return true, "event_id", false } @@ -188,7 +189,7 @@ func TestIngest_Dedup_Duplicate(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} dedup := testutil.NewMockDeduplicator() - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.Dedup = dedup h.DedupeSettings = func(*settings.Store, string) (bool, string, bool) { return true, "event_id", false } @@ -214,7 +215,7 @@ func TestIngest_Dedup_Duplicate(t *testing.T) { func TestIngest_PublishError_503(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{Err: fmt.Errorf("%w: maximum bytes exceeded", mq.ErrQueueFull)} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := ingestRequest(t, "clicks", map[string]any{"page": "/home"}) w := httptest.NewRecorder() @@ -228,7 +229,7 @@ func TestIngest_PublishError_503(t *testing.T) { func TestIngest_PublishError_500(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{Err: errors.New("some other error")} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := ingestRequest(t, "clicks", map[string]any{"page": "/home"}) w := httptest.NewRecorder() @@ -242,7 +243,7 @@ func TestIngest_PublishError_500(t *testing.T) { func TestIngest_Policy_Forbidden(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": { @@ -267,7 +268,7 @@ func TestIngest_Policy_Forbidden(t *testing.T) { func TestIngest_Policy_ColumnDenied(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": { @@ -291,7 +292,7 @@ func TestIngest_Policy_ColumnDenied(t *testing.T) { func TestIngest_Policy_CheckClause_Mismatch(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) orgTemplate := "{{ jwt.org_id }}" h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ @@ -319,7 +320,7 @@ func TestIngest_Policy_CheckClause_Mismatch(t *testing.T) { func TestIngest_Policy_CheckClause_Match(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) orgTemplate := "{{ jwt.org_id }}" h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ @@ -353,7 +354,7 @@ func TestIngest_Policy_CheckClause_Match(t *testing.T) { func TestIngest_Policy_CheckClause_NumericSpellingMatch(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) countTemplate := "{{ jwt.max_count }}" h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ @@ -397,7 +398,7 @@ func TestIngest_Policy_CheckClause_StaticNumericSpelling(t *testing.T) { t.Run(tt.name, func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) staticCount := "1.0" h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ @@ -442,7 +443,7 @@ func TestIngest_Policy_CheckClause_StringClaimStrictEquality(t *testing.T) { t.Run(tt.name, func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) orgTemplate := "{{ jwt.org_id }}" h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ @@ -475,7 +476,7 @@ func TestIngest_Policy_CheckClause_StringClaimStrictEquality(t *testing.T) { func TestIngest_Policy_CheckClause_NullValue_FailsClosed(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) orgTemplate := "{{ jwt.org_id }}" h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ @@ -504,7 +505,7 @@ func TestIngest_Policy_CheckClause_NullValue_FailsClosed(t *testing.T) { func TestIngest_Policy_CheckClause_AutoInject(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) orgTemplate := "{{ jwt.org_id }}" h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ @@ -550,7 +551,7 @@ func checkInStore() PolicySource { func TestIngest_Policy_CheckIn_InSet(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.PolicySource = checkInStore() // org_id is one of the token's allowed orgs — should pass. @@ -569,7 +570,7 @@ func TestIngest_Policy_CheckIn_InSet(t *testing.T) { func TestIngest_Policy_CheckIn_NotInSet(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.PolicySource = checkInStore() // org_id is NOT one of the token's allowed orgs — forging another tenant's row. @@ -594,7 +595,7 @@ func TestIngest_Policy_CheckIn_NotInSet(t *testing.T) { func TestIngest_Policy_CheckIn_NullValue_FailsClosed(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.PolicySource = checkInStore() req := ingestRequest(t, "clicks", map[string]any{"page": "/home", "org_id": nil}) @@ -613,7 +614,7 @@ func TestIngest_Policy_CheckIn_NullValue_FailsClosed(t *testing.T) { func TestIngest_Policy_CheckIn_Absent_FailsClosed(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.PolicySource = checkInStore() // org_id omitted — unlike _eq there's no single value to auto-inject, so the @@ -640,7 +641,7 @@ func TestIngest_Policy_CheckIn_Absent_FailsClosed(t *testing.T) { func TestIngest_Policy_CheckIn_AbsentClaim_FailsClosed(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.PolicySource = checkInStore() // The `orgs` claim is absent entirely, so the _in set resolves to a typed-nil @@ -664,7 +665,7 @@ func TestIngest_Dedup_MissingIDField(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} dedup := testutil.NewMockDeduplicator() - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.Dedup = dedup h.DedupeSettings = func(*settings.Store, string) (bool, string, bool) { return true, "event_id", false } @@ -683,7 +684,7 @@ func TestIngest_Dedup_MissingIDField(t *testing.T) { func TestIngest_Dedup_RequireID_Rejects(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.Dedup = testutil.NewMockDeduplicator() h.DedupeSettings = func(*settings.Store, string) (bool, string, bool) { return true, "event_id", true } @@ -705,7 +706,7 @@ func TestIngest_Dedup_RequireID_Rejects(t *testing.T) { func TestIngest_NDJSON_RequireID_Rejects(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.Dedup = testutil.NewMockDeduplicator() h.DedupeSettings = func(*settings.Store, string) (bool, string, bool) { return true, "event_id", true } @@ -732,7 +733,7 @@ func TestIngest_NDJSON_RequireID_Rejects(t *testing.T) { func TestIngest_Policy_DenyColumns(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": { @@ -755,7 +756,7 @@ func TestIngest_Policy_DenyColumns(t *testing.T) { func TestIngest_AdminRole_NoPolicy(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": {}, @@ -819,7 +820,7 @@ func resultAt(t *testing.T, resp batchResult, index int) recordResult { func TestIngest_NDJSON_AllValid(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := ndjsonRequest(t, "clicks", jsonLine(t, map[string]any{"page": "/a", "count": 1}), @@ -847,7 +848,7 @@ func TestIngest_NDJSON_AllValid(t *testing.T) { func TestIngest_NDJSON_PartialFailure_Validation(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := ndjsonRequest(t, "clicks", jsonLine(t, map[string]any{"page": "/a"}), @@ -873,7 +874,7 @@ func TestIngest_NDJSON_PartialFailure_Validation(t *testing.T) { func TestIngest_NDJSON_MalformedLine(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := ndjsonRequest(t, "clicks", jsonLine(t, map[string]any{"page": "/a"}), @@ -898,7 +899,7 @@ func TestIngest_NDJSON_MalformedLine(t *testing.T) { func TestIngest_NDJSON_BlankLinesSkipped(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) // Leading, interior, and whitespace-only lines are all skipped; only real // records are counted. @@ -935,7 +936,7 @@ func TestIngest_NDJSON_EmptyBody(t *testing.T) { t.Run(tt.name, func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := ndjsonRequest(t, "clicks", tt.lines...) w := httptest.NewRecorder() @@ -953,7 +954,7 @@ func TestIngest_NDJSON_Dedup(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} dedup := testutil.NewMockDeduplicator() - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.Dedup = dedup h.DedupeSettings = func(*settings.Store, string) (bool, string, bool) { return true, "event_id", false } @@ -982,7 +983,7 @@ func TestIngest_NDJSON_Backpressure_503(t *testing.T) { // Publisher rejects every publish with the backpressure sentinel; the first // valid record aborts the whole batch with 503 + Retry-After. pub := &testutil.MockPublisher{Err: fmt.Errorf("%w: maximum bytes exceeded", mq.ErrQueueFull)} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := ndjsonRequest(t, "clicks", jsonLine(t, map[string]any{"page": "/a"}), @@ -999,7 +1000,7 @@ func TestIngest_NDJSON_Backpressure_503(t *testing.T) { func TestIngest_NDJSON_PublishError_500(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{Err: errors.New("some other error")} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := ndjsonRequest(t, "clicks", jsonLine(t, map[string]any{"page": "/a"})) w := httptest.NewRecorder() @@ -1013,7 +1014,7 @@ func TestIngest_NDJSON_PublishError_500(t *testing.T) { func TestIngest_NDJSON_Policy_ColumnDenied_PerLine(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": { @@ -1046,7 +1047,7 @@ func TestIngest_NDJSON_Policy_ColumnDenied_PerLine(t *testing.T) { func TestIngest_NDJSON_Policy_TableForbidden(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": { @@ -1073,7 +1074,7 @@ func TestIngest_NDJSON_Policy_TableForbidden(t *testing.T) { func TestIngest_NDJSON_Policy_CheckClause_PerLineAndAutoInject(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) orgTemplate := "{{ jwt.org_id }}" h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ @@ -1113,7 +1114,7 @@ func TestIngest_NDJSON_Policy_CheckClause_PerLineAndAutoInject(t *testing.T) { func TestIngest_NDJSON_ContentTypeWithCharset(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := ndjsonRequest(t, "clicks", jsonLine(t, map[string]any{"page": "/a"})) req.Header.Set("Content-Type", "application/x-ndjson; charset=utf-8") @@ -1130,7 +1131,7 @@ func TestIngest_NDJSON_ContentTypeWithCharset(t *testing.T) { func TestIngest_NDJSON_ErrorsTruncated(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) const total = maxReportedResults + 50 lines := make([]string, total) @@ -1314,7 +1315,7 @@ func TestIngest_UndeclaredOrUnsupportedContentType_415(t *testing.T) { t.Run(tt.name, func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) w := httptest.NewRecorder() h.Handle(w, withTenant(rawIngestRequest(t, "clicks", tt.ct, `{"page":"/a"}`))) @@ -1364,7 +1365,7 @@ func TestIngest_ContentTypeRefusalBeatsEmptyBody(t *testing.T) { t.Run(name, func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) w := httptest.NewRecorder() h.Handle(w, withTenant(rawIngestRequest(t, "clicks", ct, ""))) @@ -1383,7 +1384,7 @@ func TestIngest_ContentTypeRefusalBeatsEmptyBody(t *testing.T) { func TestIngest_DeclaredNDJSON_ArrayBodyIsNotReframed(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) w := httptest.NewRecorder() h.Handle(w, withTenant(rawIngestRequest(t, "clicks", "application/x-ndjson", `[{"page":"/a"},{"page":"/b"}]`))) @@ -1430,7 +1431,7 @@ func TestIngest_DuplicateContentTypeHeaders(t *testing.T) { t.Run("disagreeing declarations are refused", func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := rawIngestRequest(t, "clicks", "application/json", ndjson) req.Header.Add("Content-Type", "application/x-ndjson") @@ -1455,7 +1456,7 @@ func TestIngest_DuplicateContentTypeHeaders(t *testing.T) { t.Run("a supported and an unsupported declaration are refused", func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := rawIngestRequest(t, "clicks", "application/json", ndjson) req.Header.Add("Content-Type", "text/csv") @@ -1474,7 +1475,7 @@ func TestIngest_DuplicateContentTypeHeaders(t *testing.T) { t.Run("different spellings of the same format are accepted", func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := rawIngestRequest(t, "clicks", "application/x-ndjson", ndjson) req.Header.Add("Content-Type", "application/ndjson; charset=utf-8") @@ -1504,7 +1505,7 @@ func TestIngest_DuplicateContentTypeHeaders(t *testing.T) { t.Run(name, func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) w := httptest.NewRecorder() h.Handle(w, withTenant(rawIngestRequest(t, "clicks", ct, ndjson))) @@ -1549,7 +1550,7 @@ func TestIngest_DuplicateContentTypeHeaders(t *testing.T) { t.Run(name, func(t *testing.T) { t.Parallel() wJ := httptest.NewRecorder() - NewIngestHandler(testRegistry(t), &testutil.MockPublisher{}, testutil.NopLogger()). + NewIngestHandler(testRegistry(t), &testutil.MockPublisher{}). Handle(wJ, withTenant(rawIngestRequest(t, "clicks", tc.joined, `{"page":"/a"}`))) assert.Equal(t, tc.wJoined, wJ.Code, "joined") @@ -1558,7 +1559,7 @@ func TestIngest_DuplicateContentTypeHeaders(t *testing.T) { req.Header.Add("Content-Type", v) } wR := httptest.NewRecorder() - NewIngestHandler(testRegistry(t), &testutil.MockPublisher{}, testutil.NopLogger()). + NewIngestHandler(testRegistry(t), &testutil.MockPublisher{}). Handle(wR, withTenant(req)) assert.Equal(t, tc.wRepeat, wR.Code, "repeated") }) @@ -1568,7 +1569,7 @@ func TestIngest_DuplicateContentTypeHeaders(t *testing.T) { t.Run("a quoted comma does not split a declaration", func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) w := httptest.NewRecorder() h.Handle(w, withTenant(rawIngestRequest(t, "clicks", `application/json; profile="a,b"`, `{"page":"/a"}`))) @@ -1579,7 +1580,7 @@ func TestIngest_DuplicateContentTypeHeaders(t *testing.T) { t.Run("a third line that disagrees is refused", func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := rawIngestRequest(t, "clicks", "application/json", ndjson) req.Header.Add("Content-Type", "application/json") req.Header.Add("Content-Type", "application/x-ndjson") @@ -1601,7 +1602,7 @@ func TestIngest_DuplicateContentTypeHeaders(t *testing.T) { t.Parallel() for _, first := range []bool{false, true} { pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := rawIngestRequest(t, "clicks", "", ndjson) if first { req.Header.Add("Content-Type", empty) @@ -1627,7 +1628,7 @@ func TestIngest_DuplicateContentTypeHeaders(t *testing.T) { t.Run("two unsupported lines name both", func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := rawIngestRequest(t, "clicks", "text/csv", ndjson) req.Header.Add("Content-Type", "text/plain") @@ -1646,7 +1647,7 @@ func TestIngest_DuplicateContentTypeHeaders(t *testing.T) { t.Run("an identical declaration repeated is not ambiguous", func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := rawIngestRequest(t, "clicks", "application/x-ndjson", ndjson) req.Header.Add("Content-Type", "application/x-ndjson") @@ -1661,7 +1662,7 @@ func TestIngest_DuplicateContentTypeHeaders(t *testing.T) { func TestIngest_JSONArray_AllValid(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) // A JSON array declared as application/json is read as a batch — the body's // first byte picks arity within the family ingestRequest declares. @@ -1686,7 +1687,7 @@ func TestIngest_JSONArray_AllValid(t *testing.T) { func TestIngest_JSONArray_SingleElement(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) // A one-element array is still a batch (returns the results envelope, not // the single-object {"ok":true}). @@ -1706,7 +1707,7 @@ func TestIngest_JSONArray_SingleElement(t *testing.T) { func TestIngest_JSONArray_PartialValidationFailure(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := ingestRequest(t, "clicks", []map[string]any{ {"page": "/a"}, @@ -1731,7 +1732,7 @@ func TestIngest_JSONArray_PartialValidationFailure(t *testing.T) { func TestIngest_JSONArray_ScalarElements(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) // Non-object elements (number, string, nested array) are wrong-typed: the // decoder stays in sync, so each is a per-record error and the objects @@ -1762,7 +1763,7 @@ func TestIngest_JSONArray_ScalarElements(t *testing.T) { func TestIngest_JSONArray_SyntaxError_Fatal(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) // A structural syntax error desyncs the decoder — the whole request fails // (400), unlike a per-element type error. The leading good element may have @@ -1796,7 +1797,7 @@ func TestIngest_JSONArray_Truncated_Fatal(t *testing.T) { t.Run(tt.name, func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := rawIngestRequest(t, "clicks", "application/json", tt.body) w := httptest.NewRecorder() @@ -1812,7 +1813,7 @@ func TestIngest_JSONArray_Truncated_Fatal(t *testing.T) { func TestIngest_JSONArray_Empty(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) // An explicit empty array is a valid, record-less batch → 200 with no rows. req := rawIngestRequest(t, "clicks", "application/json", `[]`) @@ -1829,7 +1830,7 @@ func TestIngest_JSONArray_Empty(t *testing.T) { func TestIngest_SingleObject_PrettyPrinted(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) // A multi-line (pretty-printed) single object must not be mistaken for // NDJSON — it's one record on the single-object path. @@ -1847,7 +1848,7 @@ func TestIngest_SingleObject_PrettyPrinted(t *testing.T) { func TestIngest_DeclaredJSON_ConcatenatedObjects_FirstOnly(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) // Two concatenated objects declared as application/json take the // single-object path and ingest only the first (matching the historical @@ -1880,7 +1881,7 @@ func TestIngest_LeadingWhitespace_Sniff(t *testing.T) { t.Run(tt.name, func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := rawIngestRequest(t, "clicks", "application/json", tt.body) w := httptest.NewRecorder() @@ -1918,7 +1919,7 @@ func TestIngest_EmptyBody(t *testing.T) { t.Run(tt.name, func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := rawIngestRequest(t, "clicks", tt.contentType, tt.body) w := httptest.NewRecorder() @@ -1949,7 +1950,7 @@ func TestIngest_BodyReadFailure_400(t *testing.T) { t.Run(ct, func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := httptest.NewRequestWithContext(context.Background(), http.MethodPost, "/v1/ingest?table=clicks", iotest.ErrReader(errors.New("connection reset by peer"))) @@ -2001,7 +2002,7 @@ func TestIngest_BodyCap_413(t *testing.T) { t.Run(tt.name, func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.maxRequestBytes = tt.cap // below the body req := rawIngestRequest(t, "clicks", tt.ct, tt.body) @@ -2035,7 +2036,7 @@ func TestIngest_ContentTypeResolvesBeforeTheBodyIsRead(t *testing.T) { t.Run("a body that cannot be read at all", func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := httptest.NewRequestWithContext(context.Background(), http.MethodPost, "/v1/ingest?table=clicks", iotest.ErrReader(errors.New("connection reset by peer"))) @@ -2053,7 +2054,7 @@ func TestIngest_ContentTypeResolvesBeforeTheBodyIsRead(t *testing.T) { t.Run("a body over the cap", func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.maxRequestBytes = 50 // below the body req := rawIngestRequest(t, "clicks", "text/csv", @@ -2117,7 +2118,7 @@ func publishedRow(t *testing.T, payload []byte) map[string]any { func TestIngest_TimestampsCanonicalized(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(tsRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(tsRegistry(t), pub) req := ingestRequest(t, "events", map[string]any{ "name": "e", @@ -2145,7 +2146,7 @@ func TestIngest_TimestampsCanonicalized(t *testing.T) { func TestIngest_AutoInjectedLiteralTimestampCanonicalized(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(tsRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(tsRegistry(t), pub) staticTS := "2026-06-21 04:00:00" h.PolicySource = staticPolicy(&policy.Policy{ Tables: map[string]policy.TablePolicy{ @@ -2177,7 +2178,7 @@ func TestIngest_AutoInjectedLiteralTimestampCanonicalized(t *testing.T) { func TestIngest_TimestampGarbage_PassesThrough(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(tsRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(tsRegistry(t), pub) req := ingestRequest(t, "events", map[string]any{"name": "e", "ts": "banana"}) w := httptest.NewRecorder() @@ -2192,7 +2193,7 @@ func TestIngest_TimestampGarbage_PassesThrough(t *testing.T) { func TestIngest_Batch_MixedTimestampSpellings(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(tsRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(tsRegistry(t), pub) req := ingestRequest(t, "events", []map[string]any{ {"name": "a", "ts": "2026-06-21T04:00:00Z"}, @@ -2244,7 +2245,7 @@ func TestIngest_Dedup_DisabledBySettings(t *testing.T) { pub := &testutil.MockPublisher{} dedup := testutil.NewMockDeduplicator() dedup.Err = errors.New("must not be called while disabled") - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.Dedup = dedup h.DedupeSettings = func(*settings.Store, string) (bool, string, bool) { return false, "event_id", true } @@ -2264,7 +2265,7 @@ func TestIngest_Dedup_DisabledMidReload(t *testing.T) { pub := &testutil.MockPublisher{} dedup := testutil.NewMockDeduplicator() dedup.Err = dedupe.ErrDisabled - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.Dedup = dedup h.DedupeSettings = func(*settings.Store, string) (bool, string, bool) { return true, "event_id", true } @@ -2297,7 +2298,7 @@ func TestProcessRecord_UnresolvedInsertSideAborts(t *testing.T) { }, } reg := testutil.NewTestSchemaRegistry(t, []*discovery.TableSchema{schema}) - h := NewIngestHandler(reg, &testutil.MockPublisher{}, testutil.NopLogger()) + h := NewIngestHandler(reg, &testutil.MockPublisher{}) // A grant resolved for SELECT, reaching the insert path. selectResolved := policy.Evaluate(&policy.Policy{ @@ -2355,7 +2356,7 @@ func TestIngest_ContentTypeEchoIsBounded(t *testing.T) { t.Run(name, func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) w := httptest.NewRecorder() h.Handle(w, withTenant(build(t))) @@ -2383,7 +2384,7 @@ func TestIngest_ContentTypeEchoIsBounded(t *testing.T) { req.Header.Add("Content-Type", fmt.Sprintf("application/%04d", i)+strings.Repeat("\xff", 112)) } w := httptest.NewRecorder() - NewIngestHandler(testRegistry(t), &testutil.MockPublisher{}, testutil.NopLogger()).Handle(w, withTenant(req)) + NewIngestHandler(testRegistry(t), &testutil.MockPublisher{}).Handle(w, withTenant(req)) require.Equal(t, http.StatusUnsupportedMediaType, w.Code) return w.Body.Len() } @@ -2407,7 +2408,7 @@ func TestIngest_ContentTypeEchoIsBounded(t *testing.T) { func TestIngest_ConflictMessageNamesTheDisagreement(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := rawIngestRequest(t, "clicks", "", "{\"page\":\"/a\"}\n{\"page\":\"/b\"}") for range 4 { req.Header.Add("Content-Type", "application/json") @@ -2437,7 +2438,7 @@ func TestIngest_ConflictMessageNamesTheDisagreement(t *testing.T) { func TestIngest_ConflictMessageNamesADifferentSpelling(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) req := rawIngestRequest(t, "clicks", "", `{"page":"/a"}`) for _, ct := range []string{ "application/json", @@ -2464,12 +2465,10 @@ func TestIngest_ConflictMessageNamesADifferentSpelling(t *testing.T) { // This drifted invisibly once already: the response passed the pinned echo while // the log passed -1, so the operator debugging a header-duplicating proxy — who // never sees the client's 415 body — got the version with the disagreeing -// declaration buried. Nothing covered log CONTENT, because NopLogger discards. +// declaration buried. Nothing covered log CONTENT, because the tests discarded it. func TestIngest_ConflictLogNamesTheDisagreement(t *testing.T) { - t.Parallel() - var buf bytes.Buffer - h := NewIngestHandler(testRegistry(t), &testutil.MockPublisher{}, - slog.New(slog.NewJSONHandler(&buf, nil))) + buf := logtest.Capture(t, slog.LevelInfo) + h := NewIngestHandler(testRegistry(t), &testutil.MockPublisher{}) req := rawIngestRequest(t, "clicks", "", `{"page":"/a"}`) for _, ct := range []string{ @@ -2497,7 +2496,7 @@ func TestIngest_ConflictLogNamesTheDisagreement(t *testing.T) { func TestIngest_CheckColumnNotInSchema_Rejected(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.PolicySource = staticPolicy(checkColumnPolicy(t, "tenant_id", "acme")) w := httptest.NewRecorder() @@ -2530,7 +2529,7 @@ func TestIngest_CheckOnComputedColumn_Rejected(t *testing.T) { }}}, }} pub := &testutil.MockPublisher{} - h := NewIngestHandler(computedRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(computedRegistry(t), pub) h.PolicySource = staticPolicy(p) w := httptest.NewRecorder() @@ -2553,7 +2552,7 @@ func TestIngest_CheckOnComputedColumn_Rejected(t *testing.T) { func TestIngest_CheckColumnInSchema_StillInjects(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.PolicySource = staticPolicy(checkColumnPolicy(t, "org_id", "org-42")) w := httptest.NewRecorder() @@ -2576,10 +2575,9 @@ func TestIngest_CheckColumnInSchema_StillInjects(t *testing.T) { // inline one: TestIngest_CheckColumnNotInSchema_BatchRejectsPerRecord passes // either way, because the per-record REJECT is deliberately kept. func TestIngest_CheckGuardLogsOncePerRequest(t *testing.T) { - t.Parallel() - var buf bytes.Buffer + buf := logtest.Capture(t, slog.LevelInfo) pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, slog.New(slog.NewJSONHandler(&buf, nil))) + h := NewIngestHandler(testRegistry(t), pub) h.PolicySource = staticPolicy(checkColumnPolicy(t, "tenant_id", "acme")) const n = 25 @@ -2611,7 +2609,7 @@ func TestIngest_CheckGuardLogsOncePerRequest(t *testing.T) { func TestIngest_CheckColumnNotInSchema_BatchRejectsPerRecord(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub) h.PolicySource = staticPolicy(checkColumnPolicy(t, "tenant_id", "acme")) req := rawIngestRequest(t, "clicks", "application/json", @@ -2670,7 +2668,7 @@ func computedRegistry(t testing.TB) *discovery.SchemaRegistry { func TestIngest_CheckOnEphemeralColumn_Rejected(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(computedRegistry(t), pub, testutil.NopLogger()) + h := NewIngestHandler(computedRegistry(t), pub) h.PolicySource = staticPolicy(checkColumnPolicy(t, "raw", "anything")) w := httptest.NewRecorder() diff --git a/internal/api/main_test.go b/internal/api/main_test.go new file mode 100644 index 00000000..d091ce5e --- /dev/null +++ b/internal/api/main_test.go @@ -0,0 +1,15 @@ +package api + +import ( + "testing" + + "github.com/Wave-RF/WaveHouse/internal/testutil/logtest" +) + +// TestMain silences the default logger the package logs through, so a +// failing test's output is not buried; tests that assert on log output +// capture it (logtest.Capture). +func TestMain(m *testing.M) { + logtest.Silence() + m.Run() +} diff --git a/internal/api/pipes.go b/internal/api/pipes.go index 494b6647..1117e722 100644 --- a/internal/api/pipes.go +++ b/internal/api/pipes.go @@ -24,17 +24,16 @@ type PipesHandler struct { // Source yields a tenant's pipes (the store itself in production). Source func(*settings.Store) pipes.Source PolicySource PolicySource // resolves empty role to default_role; may be nil - // OpsStore is the store the admin reads (List, Get) serve: /v1/ops is - // tenant-exempt, so they carry no request tenant and read the default one. - OpsStore *settings.Store - CHConn driver.Conn - Cache cache.Cache - sf singleflight.Group + // Tenants resolves the ?tenant= of the admin reads (List, Get): the ops + // tree is tenant-exempt, so they name their tenant rather than carry one. + Tenants *settings.Registry + CHConn driver.Conn + Cache cache.Cache + sf singleflight.Group // queryTimeout bounds each pipe execution, read per request // (chconn.Manager.QueryTimeout in production) so a settings reload // applies without a restart. queryTimeout func() time.Duration - logger *slog.Logger // maxRequestBytes optionally overrides the default inbound request body // cap (maxControlBodyBytes) for the body-decoding path (Execute). @@ -44,24 +43,32 @@ type PipesHandler struct { maxRequestBytes int64 } -func NewPipesHandler(source func(*settings.Store) pipes.Source, policySource PolicySource, conn driver.Conn, c cache.Cache, queryTimeout func() time.Duration, logger *slog.Logger) *PipesHandler { - return &PipesHandler{Source: source, PolicySource: policySource, CHConn: conn, Cache: c, queryTimeout: queryTimeout, logger: logger} +func NewPipesHandler(source func(*settings.Store) pipes.Source, policySource PolicySource, conn driver.Conn, c cache.Cache, queryTimeout func() time.Duration) *PipesHandler { + return &PipesHandler{Source: source, PolicySource: policySource, CHConn: conn, Cache: c, queryTimeout: queryTimeout} } -// List returns all named queries (admin endpoint). -func (h *PipesHandler) List(w http.ResponseWriter, _ *http.Request) { +// List returns all named queries of the ?tenant= (admin endpoint). +func (h *PipesHandler) List(w http.ResponseWriter, r *http.Request) { + store, ok := opsStore(w, r, h.Tenants) + if !ok { + return + } w.Header().Set("Content-Type", "application/json") - q := h.Source(h.OpsStore).Pipes() + q := h.Source(store).Pipes() if q == nil { q = []*pipes.NamedQuery{} } _ = json.NewEncoder(w).Encode(q) } -// Get returns a specific named query (admin endpoint). +// Get returns a specific named query of the ?tenant= (admin endpoint). func (h *PipesHandler) Get(w http.ResponseWriter, r *http.Request) { + store, ok := opsStore(w, r, h.Tenants) + if !ok { + return + } name := chi.URLParam(r, "name") - q := h.Source(h.OpsStore).Pipe(name) + q := h.Source(store).Pipe(name) if q == nil { writeJSONError(w, http.StatusNotFound, "pipe not found") return @@ -98,7 +105,7 @@ func (h *PipesHandler) Execute(w http.ResponseWriter, r *http.Request) { } role := policy.ResolveRole(p, auth.RoleFromContext(r.Context())) if !policy.RoleAllowed(p, role, q.AllowedRoles) { - writeAuthzDenied(w, r, h.logger, role, q.AllowedRoles, + writeAuthzDenied(w, r, role, q.AllowedRoles, slog.String("gate", "pipe"), slog.String("pipe", q.Name), ) diff --git a/internal/api/pipes_test.go b/internal/api/pipes_test.go index 4621f513..a19a8c19 100644 --- a/internal/api/pipes_test.go +++ b/internal/api/pipes_test.go @@ -47,7 +47,8 @@ func TestPipesHandler_List(t *testing.T) { &pipes.NamedQuery{Name: "top_pages", SQL: "SELECT page, count(*) FROM clicks GROUP BY page"}, &pipes.NamedQuery{Name: "recent", SQL: "SELECT * FROM clicks ORDER BY ts DESC LIMIT 10"}, ) - h := NewPipesHandler(store, nil, nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, nil, nil, nil, noTimeout) + h.Tenants = testTenants() w := httptest.NewRecorder() r := httptest.NewRequestWithContext(context.Background(), http.MethodGet, "/v1/ops/pipes", nil) @@ -64,7 +65,8 @@ func TestPipesHandler_Get_Found(t *testing.T) { store := staticPipes( &pipes.NamedQuery{Name: "top_pages", SQL: "SELECT page FROM clicks"}, ) - h := NewPipesHandler(store, nil, nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, nil, nil, nil, noTimeout) + h.Tenants = testTenants() w := httptest.NewRecorder() r := pipesRequest(t, http.MethodGet, "/v1/ops/pipes/top_pages", "top_pages", nil) @@ -79,7 +81,8 @@ func TestPipesHandler_Get_Found(t *testing.T) { func TestPipesHandler_Get_NotFound(t *testing.T) { t.Parallel() store := staticPipes() - h := NewPipesHandler(store, nil, nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, nil, nil, nil, noTimeout) + h.Tenants = testTenants() w := httptest.NewRecorder() r := pipesRequest(t, http.MethodGet, "/v1/ops/pipes/nope", "nope", nil) @@ -93,7 +96,8 @@ func TestPipesHandler_Get_NotFound(t *testing.T) { func TestPipesHandler_List_Empty(t *testing.T) { t.Parallel() store := staticPipes() - h := NewPipesHandler(store, nil, nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, nil, nil, nil, noTimeout) + h.Tenants = testTenants() w := httptest.NewRecorder() r := pipesRequest(t, http.MethodGet, "/v1/ops/pipes", "", nil) @@ -109,7 +113,7 @@ func TestPipesHandler_List_Empty(t *testing.T) { func TestPipesHandler_Execute_NotFound(t *testing.T) { t.Parallel() store := staticPipes() - h := NewPipesHandler(store, nil, nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, nil, nil, nil, noTimeout) w := httptest.NewRecorder() r := pipesRequest(t, http.MethodPost, "/v1/pipes/nope/execute", "nope", nil) @@ -133,7 +137,7 @@ func TestPipesHandler_Execute_RoleAuthorization(t *testing.T) { // A real (non-nil) policy so the default admin role ("admin") is defined // and bypasses the allowlist, per the matrix. With a nil policy nobody is // admin (total lockout) — covered separately in internal/policy tests. - h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout) w := httptest.NewRecorder() r := pipesRequest(t, http.MethodPost, "/v1/pipes/report/execute", "report", nil) @@ -160,7 +164,7 @@ func TestPipesHandler_Execute_RestrictedPipe_EmptyRoleDenied(t *testing.T) { AllowedRoles: []string{"admin"}, }, ) - h := NewPipesHandler(store, nil, nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, nil, nil, nil, noTimeout) w := httptest.NewRecorder() // No ContextKeyRole set, which simulates no token or a JWT without the role claim. @@ -181,7 +185,7 @@ func TestPipesHandler_Execute_DefaultRoleGrantsAccess(t *testing.T) { store := staticPipes( &pipes.NamedQuery{Name: "report", SQL: "SELECT * FROM clicks", AllowedRoles: []string{"viewer"}}, ) - h := NewPipesHandler(store, nil, nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, nil, nil, nil, noTimeout) h.PolicySource = staticPolicy(&policy.Policy{DefaultRole: "viewer"}) w := httptest.NewRecorder() @@ -202,7 +206,7 @@ func TestPipesHandler_Execute_DefaultRoleNotInAllowedRolesDenied(t *testing.T) { store := staticPipes( &pipes.NamedQuery{Name: "admin_report", SQL: "SELECT * FROM clicks", AllowedRoles: []string{"admin"}}, ) - h := NewPipesHandler(store, nil, nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, nil, nil, nil, noTimeout) h.PolicySource = staticPolicy(&policy.Policy{DefaultRole: "viewer"}) w := httptest.NewRecorder() @@ -226,7 +230,7 @@ func TestPipesHandler_Execute_MissingParam(t *testing.T) { }, }, ) - h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout) w := httptest.NewRecorder() // No query params or body — missing "page". @@ -251,7 +255,7 @@ func TestPipesHandler_Execute_ParamsFromQuery(t *testing.T) { }, }, ) - h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout) w := httptest.NewRecorder() r := httptest.NewRequestWithContext(context.Background(), http.MethodGet, "/v1/pipes/by_page/execute?page=/home", nil) @@ -280,7 +284,7 @@ func TestPipesHandler_Execute_RequestBodyCap(t *testing.T) { Parameters: []pipes.ParamDef{{Name: "page", Type: "string", Required: true}}, }, ) - h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout) h.maxRequestBytes = 64 w := httptest.NewRecorder() @@ -307,7 +311,7 @@ func TestPipesHandler_Execute_PostBodyParams(t *testing.T) { }, }, ) - h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout) w := httptest.NewRecorder() body := map[string]any{"page": "/about"} @@ -330,7 +334,7 @@ func TestPipesHandler_Execute_NoAllowedRoles_NonAdminDenied(t *testing.T) { store := staticPipes( &pipes.NamedQuery{Name: "open", SQL: "SELECT * FROM clicks"}, // no AllowedRoles ) - h := NewPipesHandler(store, nil, nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, nil, nil, nil, noTimeout) w := httptest.NewRecorder() r := pipesRequest(t, http.MethodPost, "/v1/pipes/open/execute", "open", nil) @@ -356,7 +360,7 @@ func TestPipesHandler_Execute_ArrayParamBinds(t *testing.T) { Parameters: []pipes.ParamDef{{Name: "ids", Type: "array", Required: true}}, }, ) - h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout) w := httptest.NewRecorder() body := map[string]any{"ids": []any{"a", "b"}} @@ -377,7 +381,7 @@ func TestPipesHandler_Execute_ObjectParamRejected(t *testing.T) { store := staticPipes( &pipes.NamedQuery{Name: "by_col", SQL: "SELECT * FROM clicks WHERE col = {{p}}"}, ) - h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout) w := httptest.NewRecorder() body := map[string]any{"p": map[string]any{"k": "v"}} @@ -398,7 +402,7 @@ func TestPipesHandler_Execute_NoAllowedRoles_AdminAllowed(t *testing.T) { store := staticPipes( &pipes.NamedQuery{Name: "open", SQL: "SELECT * FROM clicks"}, ) - h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout, testutil.NopLogger()) + h := NewPipesHandler(store, staticPolicy(&policy.Policy{}), nil, nil, noTimeout) w := httptest.NewRecorder() r := pipesRequest(t, http.MethodPost, "/v1/pipes/open/execute", "open", nil) diff --git a/internal/api/router.go b/internal/api/router.go index 6288135e..84b2e033 100644 --- a/internal/api/router.go +++ b/internal/api/router.go @@ -45,7 +45,6 @@ type Dependencies struct { // production). An empty or nil list — including a nil func — denies every // browser origin; ["*"] is the only allow-all spelling. CORSOrigins func() []string - Logger *slog.Logger // MetricsHandler, if non-nil, is mounted at MetricsPath as an unauthenticated // endpoint (Prometheus convention). Wired by internal/app from the OTel Prometheus // exporter when observability.metrics.prometheus.enabled is true AND port is 0. @@ -168,7 +167,7 @@ func NewRouter(deps Dependencies) http.Handler { // default), read live from the policy store so changes apply // without a restart. r.Use(deps.AuthMW) - r.Use(RequireAdmin(deps.PolicySource, deps.Logger)) + r.Use(RequireAdmin(deps.PolicySource)) // Schema discovery. r.Get("/schema", deps.Schema.Get) @@ -257,7 +256,7 @@ func jsonRecoverer(next http.Handler) http.Handler { // resolves to an empty (non-admin) role and is denied here. Denials go through // writeAuthzDenied, so a present-but-invalid token fails loud (401 + token // reason) rather than as a bare 403. -func RequireAdmin(store policy.Source, logger *slog.Logger) func(http.Handler) http.Handler { +func RequireAdmin(store policy.Source) func(http.Handler) http.Handler { return func(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { var p *policy.Policy @@ -273,7 +272,7 @@ func RequireAdmin(store policy.Source, logger *slog.Logger) func(http.Handler) h next.ServeHTTP(w, r) return } - writeAuthzDenied(w, r, logger, role, nil, slog.String("gate", "admin")) + writeAuthzDenied(w, r, role, nil, slog.String("gate", "admin")) }) } } diff --git a/internal/api/router_test.go b/internal/api/router_test.go index e664025b..9cb6e056 100644 --- a/internal/api/router_test.go +++ b/internal/api/router_test.go @@ -21,7 +21,7 @@ import ( func TestRequireAdmin_AdminAllowed(t *testing.T) { t.Parallel() - handler := RequireAdmin(policy.Static(&policy.Policy{}), testutil.NopLogger())(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + handler := RequireAdmin(policy.Static(&policy.Policy{}))(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusOK) })) ctx := auth.WithRole(context.Background(), "admin") @@ -33,7 +33,7 @@ func TestRequireAdmin_AdminAllowed(t *testing.T) { func TestRequireAdmin_NonAdminForbidden(t *testing.T) { t.Parallel() - handler := RequireAdmin(policy.Static(&policy.Policy{}), testutil.NopLogger())(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + handler := RequireAdmin(policy.Static(&policy.Policy{}))(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { t.Fatal("handler should not be called") })) ctx := auth.WithRole(context.Background(), "viewer") @@ -49,7 +49,7 @@ func TestRequireAdmin_NonAdminForbidden(t *testing.T) { // admin route — fail closed with 403. func TestRequireAdmin_NoRoleForbidden(t *testing.T) { t.Parallel() - handler := RequireAdmin(nil, testutil.NopLogger())(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + handler := RequireAdmin(nil)(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { t.Fatal("handler should not be called - a roleless request must not reach an admin route") })) req := httptest.NewRequestWithContext(context.Background(), http.MethodGet, "/", nil) @@ -64,7 +64,7 @@ func TestRequireAdmin_NoRoleForbidden(t *testing.T) { func TestRequireAdmin_CustomAdminRole(t *testing.T) { t.Parallel() store := policy.Static(&policy.Policy{AdminRole: "superuser"}) - handler := RequireAdmin(store, testutil.NopLogger())(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + handler := RequireAdmin(store)(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusOK) })) for role, want := range map[string]int{"superuser": http.StatusOK, "admin": http.StatusForbidden} { @@ -81,7 +81,7 @@ func TestRequireAdmin_CustomAdminRole(t *testing.T) { // (401) rather than a bare 403. func TestRequireAdmin_InvalidTokenFailsLoud(t *testing.T) { t.Parallel() - handler := RequireAdmin(nil, testutil.NopLogger())(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + handler := RequireAdmin(nil)(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { t.Fatal("handler should not be called") })) ctx := auth.WithAuthError(context.Background(), errors.New("token expired")) @@ -97,7 +97,7 @@ func TestRequireAdmin_InvalidTokenFailsLoud(t *testing.T) { // non-admin one. func TestRequireAdmin_OperatorBypass(t *testing.T) { t.Parallel() - handler := RequireAdmin(policy.Static(&policy.Policy{}), testutil.NopLogger())(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + handler := RequireAdmin(policy.Static(&policy.Policy{}))(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusOK) })) ctx := auth.WithOperator(auth.WithRole(context.Background(), "viewer")) @@ -112,7 +112,7 @@ func TestRequireAdmin_OperatorBypass(t *testing.T) { // operator can trigger a settings reload while locked out. func TestRequireAdmin_OperatorBypassesNilPolicy(t *testing.T) { t.Parallel() - handler := RequireAdmin(nil, testutil.NopLogger())(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + handler := RequireAdmin(nil)(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusOK) })) ctx := auth.WithOperator(context.Background()) @@ -324,23 +324,22 @@ func TestNewRouter_RoutesRegistered(t *testing.T) { pub := &testutil.MockPublisher{} hub := stream.NewHub(tenant.Default, nil, nil, nil) - emb, err := mq.NewEmbedded(t.TempDir(), 1024*1024, testutil.NopLogger()) + emb, err := mq.NewEmbedded(t.TempDir(), 1024*1024) require.NoError(t, err) t.Cleanup(func() { _ = emb.Close() }) deps := Dependencies{ Tenants: testTenants(), - Ingest: NewIngestHandler(reg, pub, testutil.NopLogger()), + Ingest: NewIngestHandler(reg, pub), Query: &QueryHandler{}, SSE: NewStreamHandler(hub, nil), Health: &HealthHandler{}, Version: NewVersionHandler("test", "test", "test"), Schema: NewSchemaHandler(reg), - DLQ: NewDLQHandler(emb, testutil.NopLogger()), - Pipes: NewPipesHandler(staticPipes(), staticPolicy(&policy.Policy{}), nil, nil, nil, testutil.NopLogger()), + DLQ: NewDLQHandler(emb), + Pipes: &PipesHandler{Source: staticPipes(), PolicySource: staticPolicy(&policy.Policy{}), Tenants: testTenants()}, AuthMW: func(next http.Handler) http.Handler { return next }, PolicySource: policy.Static(&policy.Policy{}), - Logger: testutil.NopLogger(), } router := NewRouter(deps) @@ -420,7 +419,6 @@ func TestNewRouter_CORSOnStream(t *testing.T) { Health: &HealthHandler{}, AuthMW: func(next http.Handler) http.Handler { return next }, CORSOrigins: func() []string { return []string{"https://app.example.com"} }, - Logger: testutil.NopLogger(), }) // A fetch-based EventSource resuming cross-origin sends both Authorization @@ -490,14 +488,13 @@ func TestNewRouter_RawSQLAdminGate(t *testing.T) { router := NewRouter(Dependencies{ Tenants: testTenants(), - Ingest: NewIngestHandler(reg, pub, testutil.NopLogger()), + Ingest: NewIngestHandler(reg, pub), Query: &QueryHandler{}, SSE: NewStreamHandler(hub, nil), Health: &HealthHandler{}, Schema: NewSchemaHandler(reg), AuthMW: func(next http.Handler) http.Handler { return next }, PolicySource: policy.Static(&policy.Policy{}), - Logger: testutil.NopLogger(), }) post := func(role string) *httptest.ResponseRecorder { @@ -552,14 +549,13 @@ func TestNewRouter_OptionalDepsNil(t *testing.T) { deps := Dependencies{ Tenants: testTenants(), - Ingest: NewIngestHandler(reg, pub, testutil.NopLogger()), + Ingest: NewIngestHandler(reg, pub), Query: &QueryHandler{}, SSE: NewStreamHandler(hub, nil), Health: &HealthHandler{}, Schema: NewSchemaHandler(reg), AuthMW: func(next http.Handler) http.Handler { return next }, PolicySource: policy.Static(&policy.Policy{}), - Logger: testutil.NopLogger(), } // Should not panic. @@ -631,13 +627,12 @@ func TestNewRouter_NotFoundEmitsJSON(t *testing.T) { hub := stream.NewHub(tenant.Default, nil, nil, nil) deps := Dependencies{ Tenants: testTenants(), - Ingest: NewIngestHandler(reg, pub, testutil.NopLogger()), + Ingest: NewIngestHandler(reg, pub), Query: &QueryHandler{}, SSE: NewStreamHandler(hub, nil), Health: &HealthHandler{}, Schema: NewSchemaHandler(reg), AuthMW: func(next http.Handler) http.Handler { return next }, - Logger: testutil.NopLogger(), } router := NewRouter(deps) @@ -657,13 +652,12 @@ func TestNewRouter_MethodNotAllowedEmitsJSON(t *testing.T) { hub := stream.NewHub(tenant.Default, nil, nil, nil) deps := Dependencies{ Tenants: testTenants(), - Ingest: NewIngestHandler(reg, pub, testutil.NopLogger()), + Ingest: NewIngestHandler(reg, pub), Query: &QueryHandler{}, SSE: NewStreamHandler(hub, nil), Health: &HealthHandler{}, Schema: NewSchemaHandler(reg), AuthMW: func(next http.Handler) http.Handler { return next }, - Logger: testutil.NopLogger(), } router := NewRouter(deps) @@ -758,14 +752,13 @@ func TestNewRouter_SchemaAdminOnly(t *testing.T) { router := NewRouter(Dependencies{ Tenants: testTenants(), - Ingest: NewIngestHandler(reg, pub, testutil.NopLogger()), + Ingest: NewIngestHandler(reg, pub), Query: &QueryHandler{}, SSE: NewStreamHandler(hub, nil), Health: &HealthHandler{}, Schema: NewSchemaHandler(reg), AuthMW: func(next http.Handler) http.Handler { return next }, PolicySource: policy.Static(&policy.Policy{}), - Logger: testutil.NopLogger(), }) get := func(path, role string) *httptest.ResponseRecorder { diff --git a/internal/api/settings.go b/internal/api/settings.go index 3848a2f9..a25b92f8 100644 --- a/internal/api/settings.go +++ b/internal/api/settings.go @@ -13,12 +13,11 @@ import ( // none there is nothing to reload, so the route is simply absent (the same // pattern as the DLQ and policy handlers). type SettingsHandler struct { - Store *settings.Store - logger *slog.Logger + Store *settings.Store } -func NewSettingsHandler(store *settings.Store, logger *slog.Logger) *SettingsHandler { - return &SettingsHandler{Store: store, logger: logger} +func NewSettingsHandler(store *settings.Store) *SettingsHandler { + return &SettingsHandler{Store: store} } // reloadResponse is the POST /v1/ops/settings/reload body: whether the @@ -34,7 +33,7 @@ type reloadResponse struct { // serialized reload path SIGHUP and the directory watcher run. 200 when the // directory was adopted (warnings included in the body), 422 when validation // rejected it and the previous settings remain in effect. -func (h *SettingsHandler) Reload(w http.ResponseWriter, _ *http.Request) { +func (h *SettingsHandler) Reload(w http.ResponseWriter, r *http.Request) { findings, adopted := h.Store.Reload("api") if findings == nil { findings = []settings.Finding{} @@ -46,6 +45,6 @@ func (h *SettingsHandler) Reload(w http.ResponseWriter, _ *http.Request) { w.Header().Set("Content-Type", "application/json") w.WriteHeader(status) if err := json.NewEncoder(w).Encode(reloadResponse{Adopted: adopted, Findings: findings}); err != nil { - h.logger.Error("settings reload response encode", "error", err) + slog.ErrorContext(r.Context(), "settings reload response encode", "error", err) } } diff --git a/internal/api/settings_test.go b/internal/api/settings_test.go index 245bc7ae..2f3a3499 100644 --- a/internal/api/settings_test.go +++ b/internal/api/settings_test.go @@ -10,7 +10,6 @@ import ( "testing" "github.com/Wave-RF/WaveHouse/internal/settings" - "github.com/Wave-RF/WaveHouse/internal/testutil" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) @@ -41,9 +40,9 @@ func writeSettingsFixture(t *testing.T, configJSON string) string { // case adopted survives), so neither the parent nor the subtests are parallel. func TestSettingsReload(t *testing.T) { dir := writeSettingsFixture(t, fullConfig(100)) - store, _ := settings.Open(dir, testutil.NopLogger()) + store, _ := settings.Open(dir) require.NotNil(t, store) - h := NewSettingsHandler(store, testutil.NopLogger()) + h := NewSettingsHandler(store) post := func() (*httptest.ResponseRecorder, reloadResponse) { rec := httptest.NewRecorder() diff --git a/internal/api/structured_query.go b/internal/api/structured_query.go index 1486b9c9..525b91cf 100644 --- a/internal/api/structured_query.go +++ b/internal/api/structured_query.go @@ -40,7 +40,6 @@ type StructuredQueryHandler struct { // constant. bucketSecs func(*settings.Store) int defaultMaxRows func(*settings.Store) int - logger *slog.Logger // maxRequestBytes optionally overrides the default inbound request body // cap (maxControlBodyBytes). When 0, the default applies. Exists so @@ -58,7 +57,6 @@ func NewStructuredQueryHandler( bucketSecs func(*settings.Store) int, queryTimeout func() time.Duration, defaultMaxRows func(*settings.Store) int, - logger *slog.Logger, ) *StructuredQueryHandler { return &StructuredQueryHandler{ CHConn: conn, @@ -68,7 +66,6 @@ func NewStructuredQueryHandler( bucketSecs: bucketSecs, queryTimeout: queryTimeout, defaultMaxRows: defaultMaxRows, - logger: logger, } } @@ -115,7 +112,7 @@ func (h *StructuredQueryHandler) Handle(w http.ResponseWriter, r *http.Request) claims, _ := auth.ClaimsFromContext(r.Context()) perms := policy.Evaluate(p, role, table, "select", claims) if !perms.Allowed { - writeAuthzDenied(w, r, h.logger, role, nil, + writeAuthzDenied(w, r, role, nil, slog.String("gate", "policy"), slog.String("table", table), slog.String("action", "select"), diff --git a/internal/api/structured_query_test.go b/internal/api/structured_query_test.go index 223eda1e..6f1769e0 100644 --- a/internal/api/structured_query_test.go +++ b/internal/api/structured_query_test.go @@ -41,7 +41,7 @@ func newStructuredQueryHandler(t testing.TB) *StructuredQueryHandler { }, }, }) - return NewStructuredQueryHandler(nil, nil, reg, nil, func(*settings.Store) int { return 60 }, func() time.Duration { return 5 * time.Second }, nil, testutil.NopLogger()) + return NewStructuredQueryHandler(nil, nil, reg, nil, func(*settings.Store) int { return 60 }, func() time.Duration { return 5 * time.Second }, nil) } func TestStructuredQuery_MissingTable(t *testing.T) { @@ -290,7 +290,7 @@ func newCapturingHandler(t *testing.T, conn driver.Conn, p *policy.Policy) *Stru }, }, }) - return NewStructuredQueryHandler(conn, nil, reg, staticPolicy(p), func(*settings.Store) int { return 60 }, func() time.Duration { return 5 * time.Second }, nil, testutil.NopLogger()) + return NewStructuredQueryHandler(conn, nil, reg, staticPolicy(p), func(*settings.Store) int { return 60 }, func() time.Duration { return 5 * time.Second }, nil) } func viewerRequest(t *testing.T, sq query.StructuredQuery) *http.Request { diff --git a/internal/api/tenant.go b/internal/api/tenant.go index e608ec44..cbc017da 100644 --- a/internal/api/tenant.go +++ b/internal/api/tenant.go @@ -46,34 +46,55 @@ func requestStore(w http.ResponseWriter, r *http.Request) (*settings.Store, bool return store, ok } -// TenantMW resolves the request's tenant before authentication runs: the -// tenant.Header value, tenant.Default when absent. A malformed id is a 400 -// and a well-formed id the registry does not hold is a 404. A repeated -// header is refused rather than picked from, so a value a proxy sets can -// never be shadowed by one the client sent. +// resolveTenant turns the tenant values a request carries — one header line +// or one query value, none meaning tenant.Default — into that tenant's store. +// A malformed id is a 400 and a well-formed id the registry does not hold is +// a 404. A repeated value is refused rather than picked from, so a value a +// proxy sets can never be shadowed by one the client sent. where names the +// source in the error body. +func resolveTenant(w http.ResponseWriter, tenants *settings.Registry, where string, values []string) (*settings.Store, bool) { + id := tenant.Default + if len(values) > 1 { + writeJSONError(w, http.StatusBadRequest, "invalid "+where+": sent more than once") + return nil, false + } + if len(values) == 1 && values[0] != "" { + parsed, err := tenant.Parse(values[0]) + if err != nil { + writeJSONError(w, http.StatusBadRequest, "invalid "+where+": "+err.Error()) + return nil, false + } + id = parsed + } + store, ok := tenants.For(id) + if !ok { + writeJSONError(w, http.StatusNotFound, "unknown tenant: "+id.String()) + return nil, false + } + return store, true +} + +// TenantMW resolves the request's tenant from the tenant.Header before +// authentication runs and stores it in the request context. func TenantMW(tenants *settings.Registry) func(http.Handler) http.Handler { return func(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - id := tenant.Default - values := r.Header.Values(tenant.Header) - if len(values) > 1 { - writeJSONError(w, http.StatusBadRequest, "invalid "+tenant.Header+": sent more than once") - return - } - if len(values) == 1 && values[0] != "" { - parsed, err := tenant.Parse(values[0]) - if err != nil { - writeJSONError(w, http.StatusBadRequest, "invalid "+tenant.Header+": "+err.Error()) - return - } - id = parsed - } - store, ok := tenants.For(id) + store, ok := resolveTenant(w, tenants, tenant.Header, r.Header.Values(tenant.Header)) if !ok { - writeJSONError(w, http.StatusNotFound, "unknown tenant: "+id.String()) return } next.ServeHTTP(w, r.WithContext(WithStore(r.Context(), store))) }) } } + +// opsTenantParam is the query parameter an ops route takes its tenant from. +// The ops tree is tenant-exempt — it runs no TenantMW and ignores the header +// — so an admin names the tenant explicitly, and none means tenant.Default. +const opsTenantParam = "tenant" + +// opsStore resolves the tenant an ops route addresses from its +// opsTenantParam, with the same answers as TenantMW. +func opsStore(w http.ResponseWriter, r *http.Request, tenants *settings.Registry) (*settings.Store, bool) { + return resolveTenant(w, tenants, "?"+opsTenantParam, r.URL.Query()[opsTenantParam]) +} diff --git a/internal/api/tenant_test.go b/internal/api/tenant_test.go index f34c84c8..e6f7efc9 100644 --- a/internal/api/tenant_test.go +++ b/internal/api/tenant_test.go @@ -9,6 +9,7 @@ import ( "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" + "github.com/Wave-RF/WaveHouse/internal/pipes" "github.com/Wave-RF/WaveHouse/internal/policy" "github.com/Wave-RF/WaveHouse/internal/settings" "github.com/Wave-RF/WaveHouse/internal/stream" @@ -73,9 +74,9 @@ func TestTenantRouteHandlers_NoResolvedTenantIs500(t *testing.T) { t.Parallel() reg := testRegistry(t) handlers := map[string]http.HandlerFunc{ - "ingest": NewIngestHandler(reg, &testutil.MockPublisher{}, testutil.NopLogger()).Handle, + "ingest": NewIngestHandler(reg, &testutil.MockPublisher{}).Handle, "structured query": newStructuredQueryHandler(t).Handle, - "pipe execute": NewPipesHandler(staticPipes(), nil, nil, nil, noTimeout, testutil.NopLogger()).Execute, + "pipe execute": NewPipesHandler(staticPipes(), nil, nil, nil, noTimeout).Execute, } for name, handle := range handlers { t.Run(name, func(t *testing.T) { @@ -94,7 +95,7 @@ func tenantProbeRouter(t *testing.T, sawStore *[]bool) http.Handler { reg := testRegistry(t) return NewRouter(Dependencies{ Tenants: testTenants(), - Ingest: NewIngestHandler(reg, &testutil.MockPublisher{}, testutil.NopLogger()), + Ingest: NewIngestHandler(reg, &testutil.MockPublisher{}), Query: &QueryHandler{}, SSE: NewStreamHandler(stream.NewHub(tenant.Default, nil, nil, nil), nil), Health: &HealthHandler{}, @@ -108,7 +109,6 @@ func tenantProbeRouter(t *testing.T, sawStore *[]bool) http.Handler { }) }, PolicySource: policy.Static(&policy.Policy{}), - Logger: testutil.NopLogger(), }) } @@ -162,3 +162,42 @@ func TestNewRouter_TenantExemptRoutes(t *testing.T) { }) } } + +// The ops tree ignores the tenant header, so the admin pipe reads name their +// tenant with ?tenant= instead — with TenantMW's answers. +func TestPipesHandler_AdminReads_TenantParam(t *testing.T) { + t.Parallel() + tests := []struct { + name string + query string + wantStatus int + wantBody string + }{ + {name: "absent resolves to the default tenant", wantStatus: http.StatusOK}, + {name: "empty resolves to the default tenant", query: "?tenant=", wantStatus: http.StatusOK}, + {name: "explicit default tenant", query: "?tenant=0", wantStatus: http.StatusOK}, + {name: "unknown tenant", query: "?tenant=acme", wantStatus: http.StatusNotFound, wantBody: "unknown tenant: acme"}, + {name: "malformed tenant", query: "?tenant=a.b", wantStatus: http.StatusBadRequest, wantBody: "invalid ?tenant"}, + {name: "repeated parameter", query: "?tenant=0&tenant=0", wantStatus: http.StatusBadRequest, wantBody: "sent more than once"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + h := NewPipesHandler(staticPipes(&pipes.NamedQuery{Name: "top_pages", SQL: "SELECT 1"}), nil, nil, nil, noTimeout) + h.Tenants = testTenants() + + w := httptest.NewRecorder() + h.List(w, httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/v1/ops/pipes"+tt.query, nil)) + require.Equal(t, tt.wantStatus, w.Code, "List body: %s", w.Body.String()) + + w = httptest.NewRecorder() + h.Get(w, pipesRequest(t, http.MethodGet, "/v1/ops/pipes/top_pages"+tt.query, "top_pages", nil)) + require.Equal(t, tt.wantStatus, w.Code, "Get body: %s", w.Body.String()) + if tt.wantBody != "" { + assert.Contains(t, w.Body.String(), tt.wantBody) + } else { + assert.Contains(t, w.Body.String(), `"top_pages"`) + } + }) + } +} diff --git a/internal/app/wire.go b/internal/app/wire.go index 81401975..569f169c 100644 --- a/internal/app/wire.go +++ b/internal/app/wire.go @@ -54,7 +54,7 @@ func withoutContext(release func() error) func(context.Context) error { // are read per request off the adopted snapshot, so a reload applies to the // next request with no hook. func (a *App) wireSettings() error { - store, _ := settings.Open(a.cfg.Settings.Dir, slog.Default()) + store, _ := settings.Open(a.cfg.Settings.Dir) if store == nil { return fmt.Errorf("settings directory %s invalid, refusing to start — findings above; `wavehouse validate` reproduces them, `wavehouse bootstrap` writes a starter directory", a.cfg.Settings.Dir) } @@ -154,7 +154,7 @@ func (a *App) wireClickHouse() error { QueryTimeout: c.QueryTimeout, } } - ch, err := chconn.Open(params(), slog.Default()) + ch, err := chconn.Open(params()) if err != nil { return fmt.Errorf("clickhouse open: %w", err) } @@ -181,7 +181,7 @@ func (a *App) wireDiscovery(ctx context.Context) { a.bootState = api.NewBootState(nil) // Both sources are read per refresh, so a settings reload retunes the // cadence and a ClickHouse reconfigure moves the database without a restart. - registry := discovery.NewSchemaRegistry(a.ch, a.ch.Database, tenant.Default, perTenant(a.tenants, (*settings.Store).SchemaRefreshInterval), slog.Default()) + registry := discovery.NewSchemaRegistry(a.ch, a.ch.Database, tenant.Default, perTenant(a.tenants, (*settings.Store).SchemaRefreshInterval)) a.registry = registry bootErr := registry.Refresh(ctx) if bootErr != nil { @@ -224,10 +224,10 @@ func (a *App) wireDedupe() error { reconcile := func() (bool, error) { enabled := a.store.DedupeEnabled() if enabled && !dedup.Open() { - config.WarnIfFreshDataDir(slog.Default(), "pebble", dir) + config.WarnIfFreshDataDir("pebble", dir) } if err := dedup.Apply(enabled); err != nil { - config.LogStorageInitError(slog.Default(), "dedupe", dir, err) + config.LogStorageInitError("dedupe", dir, err) return enabled, err } return enabled, nil @@ -250,11 +250,11 @@ func (a *App) wireDedupe() error { // them consistent (see mq.Broker.SetMaxBytes). func (a *App) wireMQ() error { dir := filepath.Join(a.cfg.DataDir, "nats") - config.WarnIfFreshDataDir(slog.Default(), "nats", dir) + config.WarnIfFreshDataDir("nats", dir) var broker mq.Broker broker, err := mq.NewEmbedded(dir, a.store.MQMaxBytes()) if err != nil { - config.LogStorageInitError(slog.Default(), "mq", dir, err) + config.LogStorageInitError("mq", dir, err) return fmt.Errorf("mq open: %w", err) } a.mq = broker @@ -303,7 +303,7 @@ func (a *App) wireCache() error { // written to ClickHouse and older than the SSE gap window // (stream.gap_window_minutes, re-read every sweep). Runs every minute. func (a *App) wireSweeper() { - sweeper := ingest.NewSweeper(a.mq, tenant.Default, perTenant(a.tenants, (*settings.Store).GapWindow), slog.Default()) + sweeper := ingest.NewSweeper(a.mq, tenant.Default, perTenant(a.tenants, (*settings.Store).GapWindow)) a.add(component{name: "sweeper", run: func(ctx context.Context) error { sweeper.Start(ctx) return nil @@ -418,7 +418,7 @@ func (a *App) wireAuth() (func(http.Handler) http.Handler, error) { OperatorKey: operatorKey, } } - authn, err := auth.NewAuthenticator(authConfig(), a.policies, slog.Default()) + authn, err := auth.NewAuthenticator(authConfig(), a.policies) if err != nil { return nil, fmt.Errorf("auth middleware init: %w", err) } @@ -477,9 +477,7 @@ func (a *App) wireReloadTriggers() { // prometheus.port set — the metrics sidecar. Same-port Prometheus mounts on // the API router instead. func (a *App) wireHTTP(authMW func(http.Handler) http.Handler) { - logger := slog.Default() - - ingestHandler := api.NewIngestHandler(a.registry, a.mq, logger) + ingestHandler := api.NewIngestHandler(a.registry, a.mq) ingestHandler.PolicySource = (*settings.Store).Policy ingestHandler.Dedup = a.dedup ingestHandler.DedupeSettings = (*settings.Store).DedupeFor @@ -495,8 +493,8 @@ func (a *App) wireHTTP(authMW func(http.Handler) http.Handler) { closing := make(chan struct{}) streamHandler.Closing = closing - pipesHandler := api.NewPipesHandler(func(s *settings.Store) pipes.Source { return s }, (*settings.Store).Policy, a.ch, a.cache, a.ch.QueryTimeout, logger) - pipesHandler.OpsStore = a.store + pipesHandler := api.NewPipesHandler(func(s *settings.Store) pipes.Source { return s }, (*settings.Store).Policy, a.ch, a.cache, a.ch.QueryTimeout) + pipesHandler.Tenants = a.tenants deps := api.Dependencies{ Ingest: ingestHandler, @@ -508,16 +506,15 @@ func (a *App) wireHTTP(authMW func(http.Handler) http.Handler) { Health: healthHandler, Version: api.NewVersionHandler(a.build.Version, a.build.GitCommit, a.build.BuildTime), Schema: api.NewSchemaHandler(a.registry), - DLQ: api.NewDLQHandler(a.mq, logger), + DLQ: api.NewDLQHandler(a.mq), Pipes: pipesHandler, - StructuredQuery: api.NewStructuredQueryHandler(a.ch, a.cache, a.registry, (*settings.Store).Policy, (*settings.Store).TimestampBucketSeconds, a.ch.QueryTimeout, (*settings.Store).DefaultMaxRows, logger), + StructuredQuery: api.NewStructuredQueryHandler(a.ch, a.cache, a.registry, (*settings.Store).Policy, (*settings.Store).TimestampBucketSeconds, a.ch.QueryTimeout, (*settings.Store).DefaultMaxRows), AuthMW: authMW, Tenants: a.tenants, PolicySource: a.policies, - Logger: logger, CORSOrigins: a.store.CORSOrigins, - Settings: api.NewSettingsHandler(a.store, logger), + Settings: api.NewSettingsHandler(a.store), } prom := a.cfg.Prometheus diff --git a/internal/auth/auth.go b/internal/auth/auth.go index 4ece479a..2a275bf5 100644 --- a/internal/auth/auth.go +++ b/internal/auth/auth.go @@ -66,8 +66,8 @@ func (v *verifier) keyFunc(t *jwt.Token) (any, error) { // refresh (or the refresh an unknown key id triggers) succeeds, no JWT // validates and requests fall to the policy default_role — the same // fail-closed posture as an unreachable ClickHouse, fixed by the next -// reload. Refresh failures are logged through logger when non-nil. -func newVerifier(cfg Config, requireFetch bool, logger *slog.Logger) (*verifier, error) { +// reload. Refresh failures are logged. +func newVerifier(cfg Config, requireFetch bool) (*verifier, error) { v := &verifier{secret: cfg.JWTSecret, roleClaim: cfg.RoleClaim, url: cfg.JWKSURL} if v.roleClaim == "" { v.roleClaim = "role" @@ -76,11 +76,9 @@ func newVerifier(cfg Config, requireFetch bool, logger *slog.Logger) (*verifier, ctx, cancel := context.WithCancel(context.Background()) noErrorReturnFirstHTTPReq := !requireFetch override := keyfunc.Override{NoErrorReturnFirstHTTPReq: &noErrorReturnFirstHTTPReq} - if logger != nil { - override.RefreshErrorHandlerFunc = func(u string) func(context.Context, error) { - return func(_ context.Context, err error) { - logger.Warn("jwks refresh failed; no token validates until it succeeds", "url", u, "error", err) - } + override.RefreshErrorHandlerFunc = func(u string) func(context.Context, error) { + return func(ctx context.Context, err error) { + slog.WarnContext(ctx, "jwks refresh failed; no token validates until it succeeds", "url", u, "error", err) } } jwks, err := keyfunc.NewDefaultOverrideCtx(ctx, []string{cfg.JWKSURL}, override) @@ -109,19 +107,18 @@ func newVerifier(cfg Config, requireFetch bool, logger *slog.Logger) (*verifier, type Authenticator struct { operatorKey string store policy.Source - logger *slog.Logger mu sync.Mutex // serializes Reconfigure cur atomic.Pointer[verifier] } // NewAuthenticator builds the boot-time verifier from cfg. An unreachable // JWKS endpoint is an error so boot fails loudly rather than degraded. -func NewAuthenticator(cfg Config, store policy.Source, logger *slog.Logger) (*Authenticator, error) { - v, err := newVerifier(cfg, true, logger) +func NewAuthenticator(cfg Config, store policy.Source) (*Authenticator, error) { + v, err := newVerifier(cfg, true) if err != nil { return nil, err } - a := &Authenticator{operatorKey: cfg.OperatorKey, store: store, logger: logger} + a := &Authenticator{operatorKey: cfg.OperatorKey, store: store} a.cur.Store(v) return a, nil } @@ -145,11 +142,9 @@ func (a *Authenticator) Reconfigure(cfg Config) { // The URL was validated as absolute http(s) and the first fetch is not // required, so construction cannot fail; a nil verifier would fail closed // anyway (every token rejected), matching the fetch-pending state. - v, err := newVerifier(cfg, false, a.logger) + v, err := newVerifier(cfg, false) if err != nil { - if a.logger != nil { - a.logger.Error("auth reconfigure: build verifier", "error", err) - } + slog.Error("auth reconfigure: build verifier", "error", err) return } a.cur.Store(v) @@ -161,7 +156,7 @@ func (a *Authenticator) Reconfigure(cfg Config) { // Middleware returns the http middleware bound to this Authenticator; it // reads the current verifier on every request. func (a *Authenticator) Middleware() func(http.Handler) http.Handler { - return middleware(a.cur.Load, a.operatorKey, a.store, a.logger) + return middleware(a.cur.Load, a.operatorKey, a.store) } var ( @@ -205,13 +200,13 @@ var operatorKeyFailures, _ = otel.Meter("wavehouse-auth").Int64Counter( // When cfg.OperatorKey is set, a non-JWT operator path is checked before the // Bearer token (see operatorKey below): a constant-time match on the presented // credential authorizes a full-access platform operator independent of the JWT verifier. -// store and logger back that path — the live admin role is read from store per -// request, and operator authentications are logged at info (audit). A presented +// store backs that path — the live admin role is read from store per +// request — and operator authentications are logged at info (audit). A presented // credential that does not match is logged at warn and counted by // wavehouse_auth_operator_key_failures_total (a probing signal), then falls -// through like any unauthenticated request. Both store and logger may be nil -// when no operator key is configured. -func middleware(current func() *verifier, operatorKeyCfg string, store policy.Source, logger *slog.Logger) func(http.Handler) http.Handler { +// through like any unauthenticated request. store may be nil when no operator +// key is configured. +func middleware(current func() *verifier, operatorKeyCfg string, store policy.Source) func(http.Handler) http.Handler { return func(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // One verifier per request: the swap is atomic, so a reload lands @@ -245,22 +240,20 @@ func middleware(current func() *verifier, operatorKeyCfg string, store policy.So match := presented != "" && subtle.ConstantTimeCompare([]byte(presented), []byte(operatorKeyCfg)) == 1 if match { - if logger != nil { - // Audit at Info (not Debug): the operator key is the most - // privileged credential in the system — full data-plane + - // admin, honored even when the policy is wiped — so its use - // must be visible in production logs (Info+), mirroring the - // WARN emitted on an authz denial. Correlation fields - // (request_id, and eventually the trusted-proxy client IP) are - // deliberately NOT stamped per-call-site — they belong in the - // global TraceHandler (internal/observability) so every log line - // gets them uniformly; tracked in #333. When OTel is enabled this - // line already carries trace_id/span_id from that handler. - logger.LogAttrs(r.Context(), slog.LevelInfo, "operator key authenticated request", - slog.String("path", r.URL.Path), - slog.String("method", r.Method), - ) - } + // Audit at Info (not Debug): the operator key is the most + // privileged credential in the system — full data-plane + + // admin, honored even when the policy is wiped — so its use + // must be visible in production logs (Info+), mirroring the + // WARN emitted on an authz denial. Correlation fields + // (request_id, and eventually the trusted-proxy client IP) are + // deliberately NOT stamped per-call-site — they belong in the + // global TraceHandler (internal/observability) so every log line + // gets them uniformly; tracked in #333. When OTel is enabled this + // line already carries trace_id/span_id from that handler. + slog.LogAttrs(r.Context(), slog.LevelInfo, "operator key authenticated request", + slog.String("path", r.URL.Path), + slog.String("method", r.Method), + ) var p *policy.Policy if store != nil { p = store() @@ -280,12 +273,10 @@ func middleware(current func() *verifier, operatorKeyCfg string, store policy.So // sends a wrong operator key by accident. Same correlation-field // deferral (request_id / client IP → #333) as the audit line above. operatorKeyFailures.Add(r.Context(), 1) - if logger != nil { - logger.LogAttrs(r.Context(), slog.LevelWarn, "operator key authentication failed", - slog.String("path", r.URL.Path), - slog.String("method", r.Method), - ) - } + slog.LogAttrs(r.Context(), slog.LevelWarn, "operator key authentication failed", + slog.String("path", r.URL.Path), + slog.String("method", r.Method), + ) } } diff --git a/internal/auth/auth_test.go b/internal/auth/auth_test.go index 95988ca8..3197754b 100644 --- a/internal/auth/auth_test.go +++ b/internal/auth/auth_test.go @@ -1,7 +1,6 @@ package auth import ( - "bytes" "context" "crypto/ed25519" "crypto/rand" @@ -15,6 +14,7 @@ import ( "github.com/Wave-RF/WaveHouse/internal/policy" "github.com/Wave-RF/WaveHouse/internal/testutil" + "github.com/Wave-RF/WaveHouse/internal/testutil/logtest" "github.com/golang-jwt/jwt/v5" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" @@ -38,19 +38,19 @@ type captured struct { // run drives cfg's middleware over a request decorated by setup, returning what // the downstream handler observed. The middleware never rejects — it always // reaches the handler — so the interesting output is the captured context, not -// the status code. It uses no policy store or logger (the operator-key path is +// the status code. It uses no policy store (the operator-key path is // exercised by runOp). func run(t *testing.T, cfg Config, setup func(*http.Request)) captured { t.Helper() - return runOp(t, cfg, nil, nil, setup) + return runOp(t, cfg, nil, setup) } -// runOp is run with an explicit policy store and logger, so the operator-key +// runOp is run with an explicit policy store, so the operator-key // path (which reads the live admin role from the store) can be exercised. -func runOp(t *testing.T, cfg Config, store policy.Source, logger *slog.Logger, setup func(*http.Request)) captured { +func runOp(t *testing.T, cfg Config, store policy.Source, setup func(*http.Request)) captured { t.Helper() var c captured - a, err := NewAuthenticator(cfg, store, logger) + a, err := NewAuthenticator(cfg, store) require.NoError(t, err) h := a.Middleware()(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { c.called = true @@ -318,7 +318,7 @@ func TestMiddleware_JWKSUnreachableAtBoot_FailsLoud(t *testing.T) { })) defer srv.Close() - _, err := NewAuthenticator(Config{JWKSURL: srv.URL}, nil, nil) + _, err := NewAuthenticator(Config{JWKSURL: srv.URL}, nil) require.Error(t, err, "an unreachable/erroring JWKS at boot must fail loudly") } @@ -330,7 +330,7 @@ func TestMiddleware_JWKSReachableAtBoot_OK(t *testing.T) { })) defer srv.Close() - a, err := NewAuthenticator(Config{JWKSURL: srv.URL}, nil, nil) + a, err := NewAuthenticator(Config{JWKSURL: srv.URL}, nil) require.NoError(t, err, "a reachable JWKS endpoint must construct successfully") require.NotNil(t, a.Middleware()) } @@ -374,7 +374,6 @@ func TestMiddleware_OperatorKey(t *testing.T) { name string cfg Config store policy.Source - logger *slog.Logger setup func(*http.Request) wantOp bool wantRole string @@ -384,7 +383,6 @@ func TestMiddleware_OperatorKey(t *testing.T) { name: "match sets operator bit and stamps the live admin role", cfg: Config{JWTSecret: testutil.TestJWTSecret, RoleClaim: "role", OperatorKey: testOperatorKey}, store: adminStore(), - logger: testutil.NopLogger(), // exercise the audit-log (logger != nil) branch setup: operatorHeader(testOperatorKey), wantOp: true, wantRole: "admin", @@ -489,7 +487,7 @@ func TestMiddleware_OperatorKey(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { t.Parallel() - c := runOp(t, tt.cfg, tt.store, tt.logger, tt.setup) + c := runOp(t, tt.cfg, tt.store, tt.setup) assert.Equal(t, tt.wantOp, c.isOperator, "operator bit") assert.Equal(t, tt.wantRole, c.role, "role") assert.Equal(t, tt.wantClaims, c.hasClaims, "claims present") @@ -507,7 +505,7 @@ func TestMiddleware_OperatorKey_StripsQueryToken(t *testing.T) { t.Parallel() store := policy.Static(&policy.Policy{AdminRole: "admin"}) query := testutil.MakeJWT(t, map[string]any{"role": "viewer"}) - c := runOp(t, Config{OperatorKey: testOperatorKey}, store, nil, func(r *http.Request) { + c := runOp(t, Config{OperatorKey: testOperatorKey}, store, func(r *http.Request) { r.URL.RawQuery = "table=clicks&token=" + query r.Header.Set("X-Operator-Key", testOperatorKey) }) @@ -517,12 +515,13 @@ func TestMiddleware_OperatorKey_StripsQueryToken(t *testing.T) { assert.Equal(t, "clicks", c.otherQueryParam) } -// infoBufLogger returns a logger writing JSON records at Info+ to buf, so a test -// can assert both that the failed-operator-attempt WARN fires and that ordinary -// traffic does not emit it. Mirrors warnBufLogger in internal/api/errors_test.go. -func infoBufLogger() (*slog.Logger, *bytes.Buffer) { - var buf bytes.Buffer - return slog.New(slog.NewJSONHandler(&buf, &slog.HandlerOptions{Level: slog.LevelInfo})), &buf +// captureInfo routes the default logger's Info+ records to the returned +// buffer, so a test can assert both that the failed-operator-attempt WARN +// fires and that ordinary traffic does not emit it. The default logger is +// process-wide, so the tests that call it run serially (see logtest.Capture). +func captureInfo(t *testing.T) *logtest.Buffer { + t.Helper() + return logtest.Capture(t, slog.LevelInfo) } // A presented-but-wrong operator credential is recorded at WARN (so operators @@ -535,14 +534,12 @@ func infoBufLogger() (*slog.Logger, *bytes.Buffer) { // the failed-attempt cases in TestMiddleware_OperatorKey already exercise the // Add call (proving it's safe under the default no-op meter). func TestMiddleware_OperatorKey_FailedAttemptLogged(t *testing.T) { - t.Parallel() cfg := Config{OperatorKey: testOperatorKey} store := policy.Static(&policy.Policy{AdminRole: "admin"}) t.Run("wrong key via X-Operator-Key logs WARN and falls through", func(t *testing.T) { - t.Parallel() - logger, buf := infoBufLogger() - c := runOp(t, cfg, store, logger, operatorHeader("wrong-key")) + buf := captureInfo(t) + c := runOp(t, cfg, store, operatorHeader("wrong-key")) assert.False(t, c.isOperator, "a wrong key never sets the operator bit") assert.Empty(t, c.role, "wrong key + no JWT → roleless fall-through") assert.NoError(t, c.authErr) @@ -554,9 +551,8 @@ func TestMiddleware_OperatorKey_FailedAttemptLogged(t *testing.T) { }) t.Run("wrong key via Authorization Operator scheme logs WARN", func(t *testing.T) { - t.Parallel() - logger, buf := infoBufLogger() - c := runOp(t, cfg, store, logger, func(r *http.Request) { + buf := captureInfo(t) + c := runOp(t, cfg, store, func(r *http.Request) { r.Header.Set("Authorization", "Operator wrong-key") }) assert.False(t, c.isOperator) @@ -564,18 +560,16 @@ func TestMiddleware_OperatorKey_FailedAttemptLogged(t *testing.T) { }) t.Run("absent operator credential does not emit the failed-attempt WARN", func(t *testing.T) { - t.Parallel() - logger, buf := infoBufLogger() - c := runOp(t, cfg, store, logger, nil) // no operator header at all + buf := captureInfo(t) + c := runOp(t, cfg, store, nil) // no operator header at all assert.False(t, c.isOperator) assert.NotContains(t, buf.String(), "operator key authentication failed", "an absent operator credential is an ordinary request, not a failed attempt") }) t.Run("successful operator auth logs the INFO audit, not the failure WARN", func(t *testing.T) { - t.Parallel() - logger, buf := infoBufLogger() - c := runOp(t, cfg, store, logger, operatorHeader(testOperatorKey)) + buf := captureInfo(t) + c := runOp(t, cfg, store, operatorHeader(testOperatorKey)) assert.True(t, c.isOperator) out := buf.String() assert.Contains(t, out, `"level":"INFO"`) @@ -634,7 +628,7 @@ func TestExtractClaim(t *testing.T) { // changes role_claim is visible to the next request with no rebuild. func TestAuthenticator_ReconfigureSwapsRoleClaim(t *testing.T) { t.Parallel() - a, err := NewAuthenticator(Config{JWTSecret: testutil.TestJWTSecret, RoleClaim: "role"}, nil, nil) + a, err := NewAuthenticator(Config{JWTSecret: testutil.TestJWTSecret, RoleClaim: "role"}, nil) require.NoError(t, err) var got string h := a.Middleware()(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) { got = RoleFromContext(r.Context()) })) @@ -665,7 +659,7 @@ func TestAuthenticator_ReconfigureAppliesUnreachableJWKS(t *testing.T) { })) defer srv.Close() - a, err := NewAuthenticator(Config{JWTSecret: testutil.TestJWTSecret}, nil, nil) + a, err := NewAuthenticator(Config{JWTSecret: testutil.TestJWTSecret}, nil) require.NoError(t, err) var got string diff --git a/internal/auth/main_test.go b/internal/auth/main_test.go new file mode 100644 index 00000000..9ee030b4 --- /dev/null +++ b/internal/auth/main_test.go @@ -0,0 +1,15 @@ +package auth + +import ( + "testing" + + "github.com/Wave-RF/WaveHouse/internal/testutil/logtest" +) + +// TestMain silences the default logger the package logs through, so a +// failing test's output is not buried; tests that assert on log output +// capture it (logtest.Capture). +func TestMain(m *testing.M) { + logtest.Silence() + m.Run() +} diff --git a/internal/chconn/chconn.go b/internal/chconn/chconn.go index d63216ae..d9399388 100644 --- a/internal/chconn/chconn.go +++ b/internal/chconn/chconn.go @@ -11,7 +11,6 @@ import ( "errors" "fmt" "io" - "log/slog" "net" "strconv" "sync" @@ -62,7 +61,6 @@ type state struct { // Manager is a driver.Conn whose backing connection is swapped by Reconfigure. type Manager struct { - logger *slog.Logger // dial is the connection factory; tests substitute it. dial func(Params) (driver.Conn, error) // grace is how long a replaced connection stays open for in-flight @@ -78,8 +76,8 @@ var _ driver.Conn = (*Manager)(nil) // Open builds the boot-time connection. Like clickhouse.Open it does not // dial — boot tolerates an unreachable ClickHouse (schema discovery degrades // and retries) — so only a malformed option errors here. -func Open(p Params, logger *slog.Logger) (*Manager, error) { - m := &Manager{logger: logger, dial: dial, grace: p.QueryTimeout} +func Open(p Params) (*Manager, error) { + m := &Manager{dial: dial, grace: p.QueryTimeout} conn, err := m.dial(p) if err != nil { return nil, err diff --git a/internal/config/persistence.go b/internal/config/persistence.go index c1640165..19b75a38 100644 --- a/internal/config/persistence.go +++ b/internal/config/persistence.go @@ -21,12 +21,12 @@ const permissionHint = "if running in a container with a host bind mount, the ho // // `kind` is a short label (e.g. "mq", "dedupe"). The caller still decides // whether to exit; this only logs. -func LogStorageInitError(logger *slog.Logger, kind, path string, err error) { +func LogStorageInitError(kind, path string, err error) { fields := []any{"error", err, "path", path} if errors.Is(err, os.ErrPermission) { fields = append(fields, "hint", permissionHint) } - logger.Error(kind+" init failed", fields...) + slog.Error(kind+" init failed", fields...) } // WarnIfFreshDataDir logs a startup `WARN` if dir doesn't already exist or is @@ -39,7 +39,7 @@ func LogStorageInitError(logger *slog.Logger, kind, path string, err error) { // recreated. // // `kind` is a short label for the log message (e.g. "nats", "pebble"). -func WarnIfFreshDataDir(logger *slog.Logger, kind, dir string) { +func WarnIfFreshDataDir(kind, dir string) { if dir == "" { return } @@ -47,20 +47,20 @@ func WarnIfFreshDataDir(logger *slog.Logger, kind, dir string) { entries, err := os.ReadDir(dir) switch { case errors.Is(err, fs.ErrNotExist): - logger.Warn( + slog.Warn( "data directory does not exist — starting with no prior state. If this is a redeploy, your persistent volume is not actually persisting; verify your mount.", "kind", kind, "path", dir, ) case err != nil: - logger.Warn("could not read data directory", "kind", kind, "path", dir, "error", err) + slog.Warn("could not read data directory", "kind", kind, "path", dir, "error", err) case len(entries) == 0: - logger.Warn( + slog.Warn( "data directory is empty — starting with no prior state. If this is a redeploy, your persistent volume is not actually persisting; verify your mount.", "kind", kind, "path", dir, ) default: - logger.Info("data directory found with prior state", "kind", kind, "path", dir) + slog.Info("data directory found with prior state", "kind", kind, "path", dir) } } diff --git a/internal/config/persistence_test.go b/internal/config/persistence_test.go index 320b8832..bbb2eac6 100644 --- a/internal/config/persistence_test.go +++ b/internal/config/persistence_test.go @@ -1,7 +1,6 @@ package config import ( - "bytes" "encoding/json" "errors" "io/fs" @@ -11,18 +10,20 @@ import ( "strings" "testing" + "github.com/Wave-RF/WaveHouse/internal/testutil/logtest" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) -func captureLog(t *testing.T) (*slog.Logger, *bytes.Buffer) { +// captureLog routes the default logger to the returned buffer. The default +// logger is process-wide, so the tests that call it run serially (see +// logtest.Capture). +func captureLog(t *testing.T) *logtest.Buffer { t.Helper() - var buf bytes.Buffer - h := slog.NewJSONHandler(&buf, &slog.HandlerOptions{Level: slog.LevelDebug}) - return slog.New(h), &buf + return logtest.Capture(t, slog.LevelDebug) } -func records(t *testing.T, buf *bytes.Buffer) []map[string]any { +func records(t *testing.T, buf *logtest.Buffer) []map[string]any { t.Helper() var out []map[string]any for line := range strings.SplitSeq(strings.TrimRight(buf.String(), "\n"), "\n") { @@ -37,11 +38,10 @@ func records(t *testing.T, buf *bytes.Buffer) []map[string]any { } func TestWarnIfFreshDataDir_Missing(t *testing.T) { - t.Parallel() - logger, buf := captureLog(t) + buf := captureLog(t) missing := filepath.Join(t.TempDir(), "does-not-exist") - WarnIfFreshDataDir(logger, "nats", missing) + WarnIfFreshDataDir("nats", missing) recs := records(t, buf) require.Len(t, recs, 1) @@ -52,11 +52,10 @@ func TestWarnIfFreshDataDir_Missing(t *testing.T) { } func TestWarnIfFreshDataDir_Empty(t *testing.T) { - t.Parallel() - logger, buf := captureLog(t) + buf := captureLog(t) // t.TempDir() returns a fresh empty dir. - WarnIfFreshDataDir(logger, "pebble", t.TempDir()) + WarnIfFreshDataDir("pebble", t.TempDir()) recs := records(t, buf) require.Len(t, recs, 1) @@ -65,13 +64,12 @@ func TestWarnIfFreshDataDir_Empty(t *testing.T) { } func TestWarnIfFreshDataDir_Populated(t *testing.T) { - t.Parallel() - logger, buf := captureLog(t) + buf := captureLog(t) dir := t.TempDir() require.NoError(t, os.WriteFile(filepath.Join(dir, "marker"), []byte("x"), 0o600)) - WarnIfFreshDataDir(logger, "nats", dir) + WarnIfFreshDataDir("nats", dir) recs := records(t, buf) require.Len(t, recs, 1) @@ -80,23 +78,21 @@ func TestWarnIfFreshDataDir_Populated(t *testing.T) { } func TestWarnIfFreshDataDir_EmptyDirArgIsNoop(t *testing.T) { - t.Parallel() - logger, buf := captureLog(t) + buf := captureLog(t) - WarnIfFreshDataDir(logger, "nats", "") + WarnIfFreshDataDir("nats", "") assert.Empty(t, buf.String()) } func TestLogStorageInitError_AddsHintOnPermissionDenied(t *testing.T) { - t.Parallel() - logger, buf := captureLog(t) + buf := captureLog(t) // fs.ErrPermission wraps to os.ErrPermission via errors.Is — this is // the canonical "permission denied" signal across the stdlib filesystem // surface, so any wrapped EACCES/EPERM bubbling up from NATS or Pebble // will satisfy the same check. - LogStorageInitError(logger, "mq", "/app/data/nats", fs.ErrPermission) + LogStorageInitError("mq", "/app/data/nats", fs.ErrPermission) recs := records(t, buf) require.Len(t, recs, 1) @@ -107,10 +103,9 @@ func TestLogStorageInitError_AddsHintOnPermissionDenied(t *testing.T) { } func TestLogStorageInitError_NoHintOnGenericError(t *testing.T) { - t.Parallel() - logger, buf := captureLog(t) + buf := captureLog(t) - LogStorageInitError(logger, "mq", "/app/data/nats", errors.New("disk full")) + LogStorageInitError("mq", "/app/data/nats", errors.New("disk full")) recs := records(t, buf) require.Len(t, recs, 1) diff --git a/internal/discovery/discovery.go b/internal/discovery/discovery.go index 6ae58ad7..02ede3d8 100644 --- a/internal/discovery/discovery.go +++ b/internal/discovery/discovery.go @@ -184,7 +184,6 @@ type SchemaRegistry struct { // tick, so a settings reload retunes the cadence without restarting the // loop (settings.Store.SchemaRefreshInterval in production). refreshInterval func(tenant.ID) time.Duration - logger *slog.Logger mu sync.RWMutex tables map[string]*TableSchema // serverVersion is the ClickHouse version string from the last successful @@ -194,13 +193,12 @@ type SchemaRegistry struct { // NewSchemaRegistry creates the registry of tenant id, which discovers // schemas from system.columns. -func NewSchemaRegistry(conn driver.Conn, database func() string, id tenant.ID, refreshInterval func(tenant.ID) time.Duration, logger *slog.Logger) *SchemaRegistry { +func NewSchemaRegistry(conn driver.Conn, database func() string, id tenant.ID, refreshInterval func(tenant.ID) time.Duration) *SchemaRegistry { return &SchemaRegistry{ conn: conn, database: database, tenant: id, refreshInterval: refreshInterval, - logger: logger, tables: make(map[string]*TableSchema), } } @@ -242,7 +240,7 @@ func (sr *SchemaRegistry) Refresh(ctx context.Context) error { } else { // Unresolvable — warn, not fatal, and no UTC fallback (that could move // instants). A nil server zone means zone-less values pass through. - sr.logger.Warn("cannot resolve server timezone; zone-less timestamps will pass through un-canonicalized", + slog.WarnContext(ctx, "cannot resolve server timezone; zone-less timestamps will pass through un-canonicalized", "timezone", tzName, "error", err) } @@ -301,7 +299,7 @@ func (sr *SchemaRegistry) Refresh(ctx context.Context) error { } for _, ts := range tables { - resolveTimestampSpecs(ts, serverTZ, sr.logger) + resolveTimestampSpecs(ctx, ts, serverTZ) ts.cacheInsertable() } @@ -309,7 +307,7 @@ func (sr *SchemaRegistry) Refresh(ctx context.Context) error { sr.tables = tables sr.serverVersion = serverVersion sr.mu.Unlock() - sr.logger.Info("schema registry refreshed", "tables", len(tables), "server_tz", tzName, "server_version", serverVersion) + slog.InfoContext(ctx, "schema registry refreshed", "tables", len(tables), "server_tz", tzName, "server_version", serverVersion) return nil } @@ -443,7 +441,7 @@ func (sr *SchemaRegistry) StartAutoRefresh(ctx context.Context) { return case <-ticker.C: if err := sr.Refresh(ctx); err != nil { - sr.logger.Error("schema auto-refresh failed", "error", err) + slog.ErrorContext(ctx, "schema auto-refresh failed", "error", err) } if next := sr.refreshInterval(sr.tenant); next != interval { interval = next diff --git a/internal/discovery/discovery_test.go b/internal/discovery/discovery_test.go index 63859d18..26cb363d 100644 --- a/internal/discovery/discovery_test.go +++ b/internal/discovery/discovery_test.go @@ -1,14 +1,11 @@ package discovery import ( - "bytes" "context" "encoding/json" "errors" - "io" "log/slog" "strings" - "sync" "sync/atomic" "testing" "time" @@ -18,6 +15,7 @@ import ( "github.com/stretchr/testify/require" "github.com/Wave-RF/WaveHouse/internal/tenant" + "github.com/Wave-RF/WaveHouse/internal/testutil/logtest" ) func TestTableSchema_ColumnNames(t *testing.T) { @@ -52,12 +50,10 @@ func TestTableSchema_ColumnNames(t *testing.T) { func TestNewSchemaRegistry_ConstructorDefaults(t *testing.T) { t.Parallel() - logger := slog.New(slog.NewTextHandler(io.Discard, nil)) - sr := NewSchemaRegistry(nil, func() string { return "wavehouse" }, tenant.Default, func(tenant.ID) time.Duration { return 30 * time.Second }, logger) + sr := NewSchemaRegistry(nil, func() string { return "wavehouse" }, tenant.Default, func(tenant.ID) time.Duration { return 30 * time.Second }) require.NotNil(t, sr) assert.Equal(t, "wavehouse", sr.database()) assert.Equal(t, 30*time.Second, sr.refreshInterval(tenant.Default)) - assert.Same(t, logger, sr.logger) assert.NotNil(t, sr.tables) assert.Empty(t, sr.List()) assert.Nil(t, sr.Get("anything")) @@ -82,7 +78,7 @@ func TestRefresh_PopulatesAndLookups(t *testing.T) { {"ghost", "CREATE TABLE test.ghost (`x` String) ENGINE = MergeTree"}, }, } - sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, discardLogger()) + sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }) require.NoError(t, sr.Refresh(context.Background())) clicks := sr.Get("clicks") @@ -115,7 +111,7 @@ func TestRefresh_DDLIsNotSerialized(t *testing.T) { // topology is not. The field is withheld for the topology. tables: [][2]string{{"clicks", "CREATE TABLE test.clicks (`id` String) ENGINE = S3('https://acme-private.s3.amazonaws.com/events.csv', 'AKIAEXAMPLEKEY', '[HIDDEN]', 'CSV')"}}, } - sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, discardLogger()) + sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }) require.NoError(t, sr.Refresh(context.Background())) encoded, err := json.Marshal(sr.Get("clicks")) @@ -135,7 +131,7 @@ func TestRefresh_ServerVersionQueryFails(t *testing.T) { version: "25.3.2.2", columns: []fakeColumn{{table: "clicks", name: "id", chType: "String", position: 1}}, } - sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, discardLogger()) + sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }) require.NoError(t, sr.Refresh(context.Background())) require.Equal(t, "25.3.2.2", sr.ServerVersion()) @@ -156,7 +152,7 @@ func TestRefresh_TablesQueryFails(t *testing.T) { columns: []fakeColumn{{table: "clicks", name: "id", chType: "String", position: 1}}, tables: [][2]string{{"clicks", "CREATE TABLE test.clicks (id String) ENGINE = MergeTree"}}, } - sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, discardLogger()) + sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }) require.NoError(t, sr.Refresh(context.Background())) require.NotNil(t, sr.Get("clicks")) require.NotEmpty(t, sr.Get("clicks").DDL) @@ -358,8 +354,7 @@ func (*fakeTableRows) Err() error { return nil } func newFakeRegistry(t *testing.T, errs []error) (*SchemaRegistry, *fakeConn) { t.Helper() conn := &fakeConn{errsThenSuccess: errs} - logger := slog.New(slog.NewJSONHandler(io.Discard, nil)) - return NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, logger), conn + return NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }), conn } // TestRefresh_UnresolvableServerTimezone_NotFatal: an unresolvable server zone @@ -367,7 +362,7 @@ func newFakeRegistry(t *testing.T, errs []error) (*SchemaRegistry, *fakeConn) { func TestRefresh_UnresolvableServerTimezone_NotFatal(t *testing.T) { t.Parallel() conn := &fakeConn{tz: "Not/AZone"} - sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, discardLogger()) + sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }) require.NoError(t, sr.Refresh(context.Background())) } @@ -381,7 +376,7 @@ func TestRefresh_RowsIterationError_Fails(t *testing.T) { columns: []fakeColumn{{table: "events", name: "id", chType: "String", position: 1}}, iterErr: errors.New("network drop mid-stream"), } - sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, discardLogger()) + sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }) err := sr.Refresh(context.Background()) require.ErrorContains(t, err, "network drop mid-stream") require.Nil(t, sr.Get("events"), "truncated scan must not be published") @@ -580,7 +575,7 @@ func TestClampBackoff(t *testing.T) { func TestStartAutoRefresh_ExitsOnContextCancel(t *testing.T) { t.Parallel() // Long interval so the ticker never fires before cancel. - sr := NewSchemaRegistry(nil, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, discardLogger()) + sr := NewSchemaRegistry(nil, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }) ctx, cancel := context.WithCancel(context.Background()) @@ -599,28 +594,6 @@ func TestStartAutoRefresh_ExitsOnContextCancel(t *testing.T) { } } -// concurrentBuffer wraps bytes.Buffer with a mutex so the test goroutine can -// read the buffer while slog's JSON handler writes to it from another. slog -// serialises its own writes via an internal mutex, but the test's reads sit -// outside that mutex — without external synchronisation, `go test -race` -// reports the buf access as a data race. -type concurrentBuffer struct { - mu sync.Mutex - b bytes.Buffer -} - -func (c *concurrentBuffer) Write(p []byte) (int, error) { - c.mu.Lock() - defer c.mu.Unlock() - return c.b.Write(p) -} - -func (c *concurrentBuffer) String() string { - c.mu.Lock() - defer c.mu.Unlock() - return c.b.String() -} - // TestStartAutoRefresh_LogsAndContinuesOnError covers the error branch in // StartAutoRefresh's ticker loop: a failed Refresh logs an ERROR line and // the loop keeps going. Operators rely on this so transient ClickHouse @@ -631,16 +604,14 @@ func (c *concurrentBuffer) String() string { // branch in milliseconds, then asserts the log line is present via // assert.Eventually (no time.Sleep-based scheduling assumption). func TestStartAutoRefresh_LogsAndContinuesOnError(t *testing.T) { - t.Parallel() errs := make([]error, 50) for i := range errs { errs[i] = errors.New("transient") } conn := &fakeConn{errsThenSuccess: errs} - var buf concurrentBuffer - logger := slog.New(slog.NewJSONHandler(&buf, &slog.HandlerOptions{Level: slog.LevelDebug})) - sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return 5 * time.Millisecond }, logger) + buf := logtest.Capture(t, slog.LevelDebug) + sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return 5 * time.Millisecond }) ctx, cancel := context.WithCancel(context.Background()) done := make(chan struct{}) @@ -694,7 +665,7 @@ func TestRefresh_DatabaseSnapshottedForWholeRefresh(t *testing.T) { } return "old" } - sr := NewSchemaRegistry(conn, db, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, discardLogger()) + sr := NewSchemaRegistry(conn, db, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }) require.NoError(t, sr.Refresh(context.Background())) require.Len(t, seen, 2, "both scans should be parameterised by a database") @@ -767,7 +738,7 @@ func TestRefresh_CapturesDefaultKind(t *testing.T) { {table: "t", name: "mat", chType: "String", defaultKind: "MATERIALIZED", defaultExpr: "concat('m', id)", position: 2}, {table: "t", name: "page", chType: "String", defaultKind: "DEFAULT", defaultExpr: "'/'", position: 3}, }} - sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, discardLogger()) + sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }) require.NoError(t, sr.Refresh(context.Background())) ts := sr.Get("t") diff --git a/internal/discovery/main_test.go b/internal/discovery/main_test.go new file mode 100644 index 00000000..036a7063 --- /dev/null +++ b/internal/discovery/main_test.go @@ -0,0 +1,15 @@ +package discovery + +import ( + "testing" + + "github.com/Wave-RF/WaveHouse/internal/testutil/logtest" +) + +// TestMain silences the default logger the package logs through, so a +// failing test's output is not buried; tests that assert on log output +// capture it (logtest.Capture). +func TestMain(m *testing.M) { + logtest.Silence() + m.Run() +} diff --git a/internal/discovery/timestamp.go b/internal/discovery/timestamp.go index 072096c4..dc0d8f81 100644 --- a/internal/discovery/timestamp.go +++ b/internal/discovery/timestamp.go @@ -1,6 +1,7 @@ package discovery import ( + "context" "encoding/json" "fmt" "log/slog" @@ -95,7 +96,7 @@ type timestampSpec struct { // type strings and loads no zones. An unresolvable zone (no embedded tzdata — // resolution needs the runtime's zone database) keeps a nil spec — warned, not // fatal: those values pass through un-canonicalized. -func resolveTimestampSpecs(ts *TableSchema, serverTZ *time.Location, logger *slog.Logger) { +func resolveTimestampSpecs(ctx context.Context, ts *TableSchema, serverTZ *time.Location) { for i := range ts.Columns { col := &ts.Columns[i] if !isTimestampType(col.Type) { @@ -103,7 +104,7 @@ func resolveTimestampSpecs(ts *TableSchema, serverTZ *time.Location, logger *slo } spec, err := resolveTimestampSpec(col.Type, serverTZ) if err != nil { - logger.Warn("cannot resolve timestamp column spec; its ingest values will pass through un-canonicalized", + slog.WarnContext(ctx, "cannot resolve timestamp column spec; its ingest values will pass through un-canonicalized", "table", ts.Name, "column", col.Name, "type", col.Type, "error", err) continue } diff --git a/internal/discovery/timestamp_test.go b/internal/discovery/timestamp_test.go index 7a5573bf..4f96e49c 100644 --- a/internal/discovery/timestamp_test.go +++ b/internal/discovery/timestamp_test.go @@ -3,8 +3,6 @@ package discovery import ( "context" "encoding/json" - "io" - "log/slog" "testing" "time" @@ -96,7 +94,7 @@ func TestCanonicalizeTimestamps(t *testing.T) { t.Parallel() // The production path: specs resolved once at schema-build time. schema := tsSchema(tt.colType) - resolveTimestampSpecs(schema, tt.serverTZ, discardLogger()) + resolveTimestampSpecs(t.Context(), schema, tt.serverTZ) data := map[string]any{"ts": tt.value} CanonicalizeTimestamps(schema, data) assert.Equal(t, tt.want, data["ts"]) @@ -133,7 +131,7 @@ func TestColumnTimeParser(t *testing.T) { t.Run(tt.name, func(t *testing.T) { t.Parallel() schema := tsSchema(tt.colType) - resolveTimestampSpecs(schema, nil, discardLogger()) + resolveTimestampSpecs(t.Context(), schema, nil) parse := schema.Columns[0].TimeParser() require.NotNil(t, parse) got, ok := parse(tt.value) @@ -158,7 +156,7 @@ func TestColumnTimeParser_NilOrZoneLimited(t *testing.T) { {Name: "d", Type: "Date"}, {Name: "ts", Type: "DateTime"}, }} - resolveTimestampSpecs(schema, nil, discardLogger()) + resolveTimestampSpecs(t.Context(), schema, nil) assert.Nil(t, schema.Columns[0].TimeParser(), "String column: no parser") assert.Nil(t, schema.Columns[1].TimeParser(), "Date column: excluded from timestamp handling") assert.Nil(t, tsSchema("DateTime").Columns[0].TimeParser(), "hand-built literal without spec resolution: no parser") @@ -189,7 +187,7 @@ func TestCanonicalizeTimestamps_AbsentColumn(t *testing.T) { {Name: "ts", Type: "DateTime", HasDefault: true}, {Name: "page", Type: "String"}, }} - resolveTimestampSpecs(schema, time.UTC, discardLogger()) + resolveTimestampSpecs(t.Context(), schema, time.UTC) data := map[string]any{"page": "/home"} CanonicalizeTimestamps(schema, data) assert.Equal(t, map[string]any{"page": "/home"}, data) @@ -255,7 +253,7 @@ func TestCanonicalizeTimestamps_Unparseable_PassThrough(t *testing.T) { t.Run(tt.name, func(t *testing.T) { t.Parallel() schema := tsSchema(tt.colType) - resolveTimestampSpecs(schema, time.UTC, discardLogger()) + resolveTimestampSpecs(t.Context(), schema, time.UTC) data := map[string]any{"ts": tt.value} CanonicalizeTimestamps(schema, data) assert.Equal(t, tt.value, data["ts"]) @@ -277,7 +275,7 @@ func TestResolveTimestampSpecs(t *testing.T) { {Name: "broken", Type: "DateTime('Not/AZone')"}, {Name: "etc_utc", Type: "DateTime('Etc/UTC')"}, }} - resolveTimestampSpecs(schema, nyc, discardLogger()) + resolveTimestampSpecs(t.Context(), schema, nyc) require.NotNil(t, schema.Columns[0].tsSpec) assert.Equal(t, nyc, schema.Columns[0].tsSpec.loc, "zone-less column takes the server zone") @@ -303,15 +301,9 @@ func TestResolveTimestampSpecs(t *testing.T) { func TestRefresh_PrecomputesSpecs(t *testing.T) { t.Parallel() conn := &fakeConn{columns: []fakeColumn{{table: "t", name: "ts", chType: "DateTime", position: 1}}} - reg := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, discardLogger()) + reg := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }) require.NoError(t, reg.Refresh(context.Background())) col := reg.Get("t").Columns[0] require.NotNil(t, col.tsSpec) assert.Equal(t, time.UTC, col.tsSpec.loc) } - -// discardLogger mirrors the registries' test logger: spec-resolution warnings are -// asserted via behavior (nil specs), not log output. -func discardLogger() *slog.Logger { - return slog.New(slog.NewTextHandler(io.Discard, nil)) -} diff --git a/internal/ingest/main_test.go b/internal/ingest/main_test.go new file mode 100644 index 00000000..bb042eac --- /dev/null +++ b/internal/ingest/main_test.go @@ -0,0 +1,15 @@ +package ingest + +import ( + "testing" + + "github.com/Wave-RF/WaveHouse/internal/testutil/logtest" +) + +// TestMain silences the default logger the package logs through, so a +// failing test's output is not buried; tests that assert on log output +// capture it (logtest.Capture). +func TestMain(m *testing.M) { + logtest.Silence() + m.Run() +} diff --git a/internal/ingest/sweeper.go b/internal/ingest/sweeper.go index 1e0f037b..3b87ca53 100644 --- a/internal/ingest/sweeper.go +++ b/internal/ingest/sweeper.go @@ -26,18 +26,16 @@ type Sweeper struct { // (settings.Store.GapWindow in production) so a reload of // stream.gap_window_minutes applies from the next sweep without a restart. gapWindow func(tenant.ID) time.Duration - logger *slog.Logger } // NewSweeper creates the Active Sweeper of tenant id. gapWindow is resolved // per sweep. // TODO: (future) need leader election or shared lock to only run one instance of the sweeper in clustered mode -func NewSweeper(purger mq.Purger, id tenant.ID, gapWindow func(tenant.ID) time.Duration, logger *slog.Logger) *Sweeper { +func NewSweeper(purger mq.Purger, id tenant.ID, gapWindow func(tenant.ID) time.Duration) *Sweeper { return &Sweeper{ purger: purger, tenant: id, gapWindow: gapWindow, - logger: logger, } } @@ -60,9 +58,9 @@ func (s *Sweeper) sweep(ctx context.Context) { if err != nil { if errors.Is(err, mq.ErrConsumerNotFound) { // Consumer may not exist yet if no messages have been ingested. - s.logger.Warn("sweeper: buffer consumer not found (may not exist yet)", "error", err) + slog.WarnContext(ctx, "sweeper: buffer consumer not found (may not exist yet)", "error", err) return } - s.logger.Error("sweeper: purge", "error", err) + slog.ErrorContext(ctx, "sweeper: purge", "error", err) } } diff --git a/internal/ingest/sweeper_test.go b/internal/ingest/sweeper_test.go index ebb76c94..2371d6eb 100644 --- a/internal/ingest/sweeper_test.go +++ b/internal/ingest/sweeper_test.go @@ -20,7 +20,7 @@ func TestSweep_AsksForTheBufferConsumerAndTheGapWindow(t *testing.T) { t.Parallel() gapWindow := 5 * time.Minute purger := &testutil.MockPurger{Purged: true} - s := NewSweeper(purger, tenant.Default, func(tenant.ID) time.Duration { return gapWindow }, testutil.NopLogger()) + s := NewSweeper(purger, tenant.Default, func(tenant.ID) time.Duration { return gapWindow }) before := time.Now() s.sweep(context.Background()) @@ -37,7 +37,7 @@ func TestSweep_RereadsTheGapWindowEverySweep(t *testing.T) { t.Parallel() gapWindow := time.Minute purger := &testutil.MockPurger{} - s := NewSweeper(purger, tenant.Default, func(tenant.ID) time.Duration { return gapWindow }, testutil.NopLogger()) + s := NewSweeper(purger, tenant.Default, func(tenant.ID) time.Duration { return gapWindow }) s.sweep(context.Background()) gapWindow = time.Hour // a settings reload @@ -51,7 +51,7 @@ func TestSweep_ErrorsDoNotPanic(t *testing.T) { t.Parallel() for _, err := range []error{mq.ErrConsumerNotFound, errors.New("broker unavailable")} { purger := &testutil.MockPurger{Err: err} - s := NewSweeper(purger, tenant.Default, func(tenant.ID) time.Duration { return time.Minute }, testutil.NopLogger()) + s := NewSweeper(purger, tenant.Default, func(tenant.ID) time.Duration { return time.Minute }) s.sweep(context.Background()) assert.Len(t, purger.Calls, 1) } @@ -63,7 +63,7 @@ func TestSweep_ErrorsDoNotPanic(t *testing.T) { func TestStart_ContextCancellation(t *testing.T) { t.Parallel() - s := NewSweeper(&testutil.MockPurger{}, tenant.Default, func(tenant.ID) time.Duration { return 5 * time.Minute }, testutil.NopLogger()) + s := NewSweeper(&testutil.MockPurger{}, tenant.Default, func(tenant.ID) time.Duration { return 5 * time.Minute }) ctx, cancel := context.WithCancel(context.Background()) cancel() // Cancel immediately. diff --git a/internal/ingest/worker.go b/internal/ingest/worker.go index bc039d3b..c97f9fcc 100644 --- a/internal/ingest/worker.go +++ b/internal/ingest/worker.go @@ -58,7 +58,6 @@ type IngestWorker struct { failed chan error httpClient *http.Client cache cache.Cache - logger *slog.Logger // target resolves the ClickHouse HTTP wiring per insert // (chconn.Manager.Target in production) so a settings reload that // re-points ClickHouse applies to the next flush. @@ -178,7 +177,6 @@ func StartIngestWorker( Timeout: 30 * time.Second, }, cache: cache, - logger: slog.Default().With("component", "ingest_worker"), target: target, maxBatch: defaultMaxBatch, maxWait: defaultMaxWait, @@ -239,7 +237,7 @@ func (w *IngestWorker) dispatchLoop(ctx context.Context, cons mq.Consumer) { } }, pullMaxMessages) if err != nil { - w.logger.Error("failed to start consumer", "error", err) + slog.ErrorContext(ctx, "failed to start consumer", "error", err) w.failed <- fmt.Errorf("ingest worker: start consumer: %w", err) return } @@ -278,7 +276,7 @@ func (w *IngestWorker) dispatchLoop(ctx context.Context, cons mq.Consumer) { // what is already in hand — those rows are delivered and the // publish path is not what broke — then fail loud. Messages still // in msgChan are unacked and redelivered to the next consumer. - w.logger.Error("ingest consumer delivery ended; ingestion has stopped", "error", err) + slog.ErrorContext(ctx, "ingest consumer delivery ended; ingestion has stopped", "error", err) shutdown() w.failed <- fmt.Errorf("ingest worker: %w", err) return @@ -463,19 +461,19 @@ func (w *IngestWorker) parseMsg(ctx context.Context, m *mq.Message) (parsedMsg, var envelope EventMessage if err := json.Unmarshal(m.Data, &envelope); err != nil { - w.logger.ErrorContext(ctx, "failed to parse event envelope", "error", err) + slog.ErrorContext(ctx, "failed to parse event envelope", "error", err) w.rejectPoison(ctx, m, "", "malformed", err.Error()) return parsedMsg{}, false } if envelope.Format != FormatJSONCompactEachRow { - w.logger.ErrorContext(ctx, "event envelope declares an unknown row format", + slog.ErrorContext(ctx, "event envelope declares an unknown row format", "format", envelope.Format, "table", envelope.TableName) w.rejectPoison(ctx, m, envelope.TableName, "unknown_format", fmt.Sprintf("unknown row format %q (a pre-v2 envelope carries none); drain the ingest queue before upgrading", envelope.Format)) return parsedMsg{}, false } if len(envelope.Columns) == 0 || len(envelope.Row) == 0 { - w.logger.ErrorContext(ctx, "event envelope carries no columns or no row", + slog.ErrorContext(ctx, "event envelope carries no columns or no row", "table", envelope.TableName, "columns", len(envelope.Columns)) w.rejectPoison(ctx, m, envelope.TableName, "unpairable", "envelope carries no columns or no row, so its values cannot be mapped to columns") @@ -489,14 +487,14 @@ func (w *IngestWorker) parseMsg(ctx context.Context, m *mq.Message) (parsedMsg, // same check; this is the ingest half of the contract AGENTS.md states. var cells []json.RawMessage if dup, ok := firstDuplicate(envelope.Columns); ok { - w.logger.ErrorContext(ctx, "unreadable envelope: a column name repeats", + slog.ErrorContext(ctx, "unreadable envelope: a column name repeats", "table", envelope.TableName, "column", dup) w.rejectPoison(ctx, m, envelope.TableName, "unpairable", fmt.Sprintf("column %q appears more than once, so its values cannot be mapped to columns", dup)) return parsedMsg{}, false } if err := json.Unmarshal(envelope.Row, &cells); err != nil || len(cells) != len(envelope.Columns) { - w.logger.ErrorContext(ctx, "event envelope row does not pair with its columns", + slog.ErrorContext(ctx, "event envelope row does not pair with its columns", "table", envelope.TableName, "columns", len(envelope.Columns), "error", err) w.rejectPoison(ctx, m, envelope.TableName, "unpairable", fmt.Sprintf("row does not pair with its %d column(s), so its values cannot be mapped to columns", len(envelope.Columns))) @@ -568,7 +566,7 @@ func (w *IngestWorker) flushGroup(ctx context.Context, tableName string, group [ return } - w.logger.WarnContext(ctx, "bulk insert failed, falling back to 1-by-1 isolation", "table", tableName, "error", err) + slog.WarnContext(ctx, "bulk insert failed, falling back to 1-by-1 isolation", "table", tableName, "error", err) // ISOLATE & DLQ: re-insert one row at a time so a single poison row can't // sink the whole batch. @@ -577,10 +575,10 @@ func (w *IngestWorker) flushGroup(ctx context.Context, tableName string, group [ singleErr := w.insertToClickHouse(ctx, tableName, cols, []parsedMsg{pm}) if singleErr != nil { if w.dlqEnabled != nil && !w.dlqEnabled(w.tenant, tableName) { - w.logger.ErrorContext(ctx, "isolated bad row, DLQ disabled for table — left unacked, NATS will redeliver it until it inserts or dlq is enabled", "table", tableName, "error", singleErr) + slog.ErrorContext(ctx, "isolated bad row, DLQ disabled for table — left unacked, NATS will redeliver it until it inserts or dlq is enabled", "table", tableName, "error", singleErr) continue } - w.logger.ErrorContext(ctx, "isolated bad row, sending to DLQ", "table", tableName, "error", singleErr) + slog.ErrorContext(ctx, "isolated bad row, sending to DLQ", "table", tableName, "error", singleErr) w.sendToDLQ(ctx, tableName, pm, singleErr.Error()) } else { w.handleSuccess(ctx, tableName, []parsedMsg{pm}) @@ -682,7 +680,7 @@ func (w *IngestWorker) handleSuccess(ctx context.Context, tableName string, msgs invCtx := trace.ContextWithSpanContext(context.WithoutCancel(ctx), trace.SpanContextFromContext(ctx)) _, err := w.cache.Invalidate(invCtx, namespaces) if err != nil { - w.logger.ErrorContext(invCtx, "failed to invalidate cache after insert - your cache is holding stale data now!", "table", tableName, "error", err) + slog.ErrorContext(invCtx, "failed to invalidate cache after insert - your cache is holding stale data now!", "table", tableName, "error", err) } } @@ -695,7 +693,7 @@ func (w *IngestWorker) handleSuccess(ctx context.Context, tableName string, msgs for _, pm := range msgs { acks.Go(func() { if err := pm.msg.DoubleAck(context.WithoutCancel(ctx)); err != nil { - w.logger.ErrorContext(context.WithoutCancel(ctx), "double ack failed for processed message", "error", err, "table", tableName) + slog.ErrorContext(context.WithoutCancel(ctx), "double ack failed for processed message", "error", err, "table", tableName) } }) } @@ -725,7 +723,7 @@ func (w *IngestWorker) rejectPoison(ctx context.Context, m *mq.Message, tableNam }) return } - w.logger.ErrorContext(ctx, "unreadable envelope dropped — the DLQ is disabled for this table, and a message that can never insert must not redeliver forever", + slog.ErrorContext(ctx, "unreadable envelope dropped — the DLQ is disabled for this table, and a message that can never insert must not redeliver forever", "table", tableName, "reason", reason, "detail", detail) // Counted only once the ack lands, for the same reason the parked path waits // on parkOnDLQ's verdict: a failed ack leaves the message in the stream to be @@ -734,7 +732,7 @@ func (w *IngestWorker) rejectPoison(ctx context.Context, m *mq.Message, tableNam // that about a row still sitting in the queue, once per redelivery. w.ackWg.Go(func() { if err := m.DoubleAck(ctx); err != nil { - w.logger.ErrorContext(ctx, "ack of a dropped unreadable envelope failed, so it stays in the stream and will be refused again", + slog.ErrorContext(ctx, "ack of a dropped unreadable envelope failed, so it stays in the stream and will be refused again", "table", tableName, "reason", reason, "error", err) return } @@ -765,7 +763,7 @@ func (w *IngestWorker) parkOnDLQ(ctx context.Context, msg *mq.Message, tableName mq.WithHeader("X-DLQ-Timestamp", time.Now().UTC().Format(time.RFC3339)), ) if pubErr != nil { - w.logger.ErrorContext(ctx, "DLQ publish failed, this data will continue retrying insertion indefinitely until the DLQ recovers", "table", tableName, "topic", msg.TopicKey(), "error", pubErr) + slog.ErrorContext(ctx, "DLQ publish failed, this data will continue retrying insertion indefinitely until the DLQ recovers", "table", tableName, "topic", msg.TopicKey(), "error", pubErr) return false } @@ -775,7 +773,7 @@ func (w *IngestWorker) parkOnDLQ(ctx context.Context, msg *mq.Message, tableName // parking again on every retry. The duplicate copy is the residual cost: // a publish is not idempotent, so it cannot be taken back here. if err := msg.DoubleAck(ctx); err != nil { - w.logger.ErrorContext(ctx, "parked on the DLQ but the ack failed, so the envelope stays in the stream and will be parked again on redelivery", + slog.ErrorContext(ctx, "parked on the DLQ but the ack failed, so the envelope stays in the stream and will be parked again on redelivery", "table", tableName, "topic", msg.TopicKey(), "error", err) return false } diff --git a/internal/ingest/worker_test.go b/internal/ingest/worker_test.go index c34f5b2e..5b68914d 100644 --- a/internal/ingest/worker_test.go +++ b/internal/ingest/worker_test.go @@ -35,7 +35,7 @@ import ( ) // Shared mocks come from internal/testutil: MockMessage, MockPublisher, -// MockRoundTripper, MockCache, NopLogger. +// MockRoundTripper, MockCache. // newTestWorker builds an IngestWorker wired to in-process mocks. wait() blocks // until all background ack goroutines kicked off by handleSuccess finish. @@ -47,7 +47,6 @@ func newTestWorker(rt http.RoundTripper) (*IngestWorker, *testutil.MockPublisher failed: make(chan error, 1), httpClient: &http.Client{Transport: rt}, cache: cache, - logger: testutil.NopLogger(), target: func() chconn.Target { return chconn.Target{URL: "http://test-clickhouse:8123", Username: "test_user", Password: "test_pass", Database: "test_db"} }, @@ -121,7 +120,7 @@ func TestStartIngestWorker_Validation(t *testing.T) { { name: "nil cache", setup: func(t *testing.T) (Queue, cache.Cache) { - emb, err := mq.NewEmbedded(t.TempDir(), 1024*1024, testutil.NopLogger()) + emb, err := mq.NewEmbedded(t.TempDir(), 1024*1024) require.NoError(t, err) t.Cleanup(func() { _ = emb.Close() }) return emb, nil @@ -153,7 +152,7 @@ func TestStartIngestWorker_EndToEnd(t *testing.T) { t.Parallel() // ── Embedded MQ ── - emb, err := mq.NewEmbedded(t.TempDir(), 4*1024*1024, testutil.NopLogger()) + emb, err := mq.NewEmbedded(t.TempDir(), 4*1024*1024) require.NoError(t, err) t.Cleanup(func() { _ = emb.Close() }) @@ -196,7 +195,6 @@ func TestStartIngestWorker_EndToEnd(t *testing.T) { dlq: emb, httpClient: &http.Client{Timeout: 30 * time.Second}, cache: cache, - logger: testutil.NopLogger(), target: func() chconn.Target { return chconn.Target{URL: fmt.Sprintf("http://%s:%s", host, port), Username: "u", Password: "p", Database: "db"} }, @@ -247,7 +245,7 @@ func TestStartIngestWorker_EndToEnd(t *testing.T) { func TestStartIngestWorker_StopFunc_RespectsShutdownDeadline(t *testing.T) { t.Parallel() - emb, err := mq.NewEmbedded(t.TempDir(), 1024*1024, testutil.NopLogger()) + emb, err := mq.NewEmbedded(t.TempDir(), 1024*1024) require.NoError(t, err) t.Cleanup(func() { _ = emb.Close() }) @@ -298,7 +296,7 @@ func TestStartIngestWorker_StopFunc_RespectsShutdownDeadline(t *testing.T) { func TestStartIngestWorker_StopFunc_CleanShutdown(t *testing.T) { t.Parallel() - emb, err := mq.NewEmbedded(t.TempDir(), 1024*1024, testutil.NopLogger()) + emb, err := mq.NewEmbedded(t.TempDir(), 1024*1024) require.NoError(t, err) t.Cleanup(func() { _ = emb.Close() }) @@ -1052,7 +1050,7 @@ func TestDispatchLoop_PerTableBatching_NoCrossTableContamination(t *testing.T) { batchB = maxBatch ) - emb, err := mq.NewEmbedded(t.TempDir(), 8*1024*1024, testutil.NopLogger()) + emb, err := mq.NewEmbedded(t.TempDir(), 8*1024*1024) require.NoError(t, err) t.Cleanup(func() { _ = emb.Close() }) @@ -1088,7 +1086,6 @@ func TestDispatchLoop_PerTableBatching_NoCrossTableContamination(t *testing.T) { dlq: emb, httpClient: &http.Client{Timeout: 30 * time.Second}, cache: &testutil.MockCache{}, - logger: testutil.NopLogger(), target: func() chconn.Target { return chconn.Target{URL: fmt.Sprintf("http://%s:%s", host, port), Username: "u", Password: "p", Database: "db"} }, @@ -1146,7 +1143,7 @@ func TestDispatchLoop_PartialBatchWaitsForOwnTrigger(t *testing.T) { total = 4 // 3 → one full batch on the size trigger; 1 leftover ) - emb, err := mq.NewEmbedded(t.TempDir(), 8*1024*1024, testutil.NopLogger()) + emb, err := mq.NewEmbedded(t.TempDir(), 8*1024*1024) require.NoError(t, err) t.Cleanup(func() { _ = emb.Close() }) @@ -1184,7 +1181,6 @@ func TestDispatchLoop_PartialBatchWaitsForOwnTrigger(t *testing.T) { dlq: emb, httpClient: &http.Client{Timeout: 30 * time.Second}, cache: &testutil.MockCache{}, - logger: testutil.NopLogger(), target: func() chconn.Target { return chconn.Target{URL: fmt.Sprintf("http://%s:%s", host, port), Username: "u", Password: "p", Database: "db"} }, diff --git a/internal/mq/embedded.go b/internal/mq/embedded.go index bd814be0..8f536dcb 100644 --- a/internal/mq/embedded.go +++ b/internal/mq/embedded.go @@ -16,31 +16,32 @@ import ( "github.com/nats-io/nats.go/jetstream" ) -// slogNATSLogger adapts slog to the natsserver.Logger interface. -type slogNATSLogger struct{ l *slog.Logger } +// slogNATSLogger adapts the default slog logger to the natsserver.Logger +// interface. +type slogNATSLogger struct{} -func (s *slogNATSLogger) Noticef(format string, v ...any) { - s.l.Info(fmt.Sprintf(format, v...), "component", "nats") +func (slogNATSLogger) Noticef(format string, v ...any) { + slog.Info(fmt.Sprintf(format, v...), "component", "nats") } -func (s *slogNATSLogger) Warnf(format string, v ...any) { - s.l.Warn(fmt.Sprintf(format, v...), "component", "nats") +func (slogNATSLogger) Warnf(format string, v ...any) { + slog.Warn(fmt.Sprintf(format, v...), "component", "nats") } -func (s *slogNATSLogger) Fatalf(format string, v ...any) { - s.l.Error(fmt.Sprintf(format, v...), "component", "nats") +func (slogNATSLogger) Fatalf(format string, v ...any) { + slog.Error(fmt.Sprintf(format, v...), "component", "nats") } -func (s *slogNATSLogger) Errorf(format string, v ...any) { - s.l.Error(fmt.Sprintf(format, v...), "component", "nats") +func (slogNATSLogger) Errorf(format string, v ...any) { + slog.Error(fmt.Sprintf(format, v...), "component", "nats") } -func (s *slogNATSLogger) Debugf(format string, v ...any) { - s.l.Debug(fmt.Sprintf(format, v...), "component", "nats") +func (slogNATSLogger) Debugf(format string, v ...any) { + slog.Debug(fmt.Sprintf(format, v...), "component", "nats") } -func (s *slogNATSLogger) Tracef(format string, v ...any) { - s.l.Debug(fmt.Sprintf(format, v...), "component", "nats") +func (slogNATSLogger) Tracef(format string, v ...any) { + slog.Debug(fmt.Sprintf(format, v...), "component", "nats") } // EmbeddedNATS runs an in-process NATS server with JetStream. @@ -48,7 +49,6 @@ type EmbeddedNATS struct { server *natsserver.Server conn *nats.Conn js jetstream.JetStream - logger *slog.Logger limitMu sync.Mutex maxBytes int64 // the ingest stream cap both streams were last reconciled to @@ -79,16 +79,10 @@ const ( // stream at a tenth of it. The DLQ stream is always present — an empty // limits-policy stream costs nothing, and whether a poison row lands on it is // the ingest worker's decision at the moment of the failure. -// An optional *slog.Logger can be passed to control server log output; -// if omitted, slog.Default() is used. The stream names are fixed (see -// subject.go) — the embedded server is private to this process, so there's -// no namespacing to do. -func NewEmbedded(storeDir string, maxBytes int64, logger ...*slog.Logger) (*EmbeddedNATS, error) { - l := slog.Default() - if len(logger) > 0 && logger[0] != nil { - l = logger[0] - } - +// The server logs through slog's default logger. The stream names are fixed +// (see subject.go) — the embedded server is private to this process, so +// there's no namespacing to do. +func NewEmbedded(storeDir string, maxBytes int64) (*EmbeddedNATS, error) { opts := &natsserver.Options{ DontListen: true, JetStream: true, @@ -105,7 +99,7 @@ func NewEmbedded(storeDir string, maxBytes int64, logger ...*slog.Logger) (*Embe if err != nil { return nil, fmt.Errorf("new nats server: %w", err) } - ns.SetLogger(&slogNATSLogger{l: l}, false, false) + ns.SetLogger(slogNATSLogger{}, false, false) ns.Start() if !ns.ReadyForConnections(5 * time.Second) { @@ -136,7 +130,7 @@ func NewEmbedded(storeDir string, maxBytes int64, logger ...*slog.Logger) (*Embe return nil, fmt.Errorf("create dlq stream: %w", err) } - return &EmbeddedNATS{server: ns, conn: nc, js: js, logger: l, maxBytes: maxBytes}, nil + return &EmbeddedNATS{server: ns, conn: nc, js: js, maxBytes: maxBytes}, nil } // ingestStreamConfig is the WAVEHOUSE stream. LimitsPolicy: standard @@ -339,7 +333,7 @@ func (c *jsConsumer) Consume(handler func(msg *Message), prefetch int) (func(), opts := []jetstream.PullConsumeOpt{ jetstream.ConsumeErrHandler(func(_ jetstream.ConsumeContext, err error) { lastErr.Store(&err) - slog.Default().Warn("mq: consumer reported an error", "component", "nats", "error", err) + slog.Warn("mq: consumer reported an error", "component", "nats", "error", err) }), } if prefetch > 0 { @@ -448,13 +442,13 @@ func (e *EmbeddedNATS) PurgeAcked(ctx context.Context, consumer string, olderTha // this package speaks. switch { case report.purged: - e.logger.Info("sweeper: purged", + slog.InfoContext(ctx, "sweeper: purged", "purged_below_seq", report.target, "ack_floor", report.ackFloor, "gap_seq", report.gapSeq, ) case report.gapSeq == 0: - e.logger.Debug("sweeper: all messages within gap window, skipping purge") + slog.DebugContext(ctx, "sweeper: all messages within gap window, skipping purge") } return report.purged, nil } diff --git a/internal/mq/embedded_test.go b/internal/mq/embedded_test.go index e9b0b9db..94783252 100644 --- a/internal/mq/embedded_test.go +++ b/internal/mq/embedded_test.go @@ -2,8 +2,6 @@ package mq import ( "context" - "io" - "log/slog" "sync" "testing" "time" @@ -14,12 +12,11 @@ import ( "github.com/stretchr/testify/require" ) -// newTestEmbedded spins up an EmbeddedNATS with a silent logger and a -// temporary store directory that is cleaned up by the test framework. +// newTestEmbedded spins up an EmbeddedNATS with a temporary store directory +// that is cleaned up by the test framework. func newTestEmbedded(t *testing.T) *EmbeddedNATS { t.Helper() - logger := slog.New(slog.NewTextHandler(io.Discard, nil)) - e, err := NewEmbedded(t.TempDir(), 64<<20, logger) + e, err := NewEmbedded(t.TempDir(), 64<<20) require.NoError(t, err) t.Cleanup(func() { _ = e.Close() }) return e @@ -293,7 +290,7 @@ func TestEmbeddedNATS_SubscribeCancellation(t *testing.T) { func TestSlogNATSLogger_Levels(t *testing.T) { t.Parallel() - l := &slogNATSLogger{l: slog.New(slog.NewTextHandler(io.Discard, nil))} + l := slogNATSLogger{} l.Noticef("notice %d", 1) l.Warnf("warn %s", "w") l.Errorf("err %v", "e") @@ -511,8 +508,7 @@ func TestEmbeddedNATS_DeadLetter_IsAPrefixSwap(t *testing.T) { } func TestEmbeddedNATS_Publish_QueueFull(t *testing.T) { - logger := slog.New(slog.NewTextHandler(io.Discard, nil)) - e, err := NewEmbedded(t.TempDir(), 4<<10, logger) + e, err := NewEmbedded(t.TempDir(), 4<<10) require.NoError(t, err) t.Cleanup(func() { _ = e.Close() }) ctx, cancel := context.WithTimeout(t.Context(), 10*time.Second) diff --git a/internal/mq/main_test.go b/internal/mq/main_test.go new file mode 100644 index 00000000..f759977d --- /dev/null +++ b/internal/mq/main_test.go @@ -0,0 +1,14 @@ +package mq + +import ( + "testing" + + "github.com/Wave-RF/WaveHouse/internal/testutil/logtest" +) + +// TestMain silences the default logger, which the embedded server logs +// through. +func TestMain(m *testing.M) { + logtest.Silence() + m.Run() +} diff --git a/internal/settings/main_test.go b/internal/settings/main_test.go new file mode 100644 index 00000000..c67c7228 --- /dev/null +++ b/internal/settings/main_test.go @@ -0,0 +1,13 @@ +package settings + +import ( + "testing" + + "github.com/Wave-RF/WaveHouse/internal/testutil/logtest" +) + +// TestMain silences the default logger, which every reload reports through. +func TestMain(m *testing.M) { + logtest.Silence() + m.Run() +} diff --git a/internal/settings/store.go b/internal/settings/store.go index a17c46b0..2010dbd2 100644 --- a/internal/settings/store.go +++ b/internal/settings/store.go @@ -25,8 +25,7 @@ import ( // Validate, so the snapshot is exactly what the files said when they were // adopted. Defaults live in the seed directory (Seed / WriteSeed). type Store struct { - dir string - logger *slog.Logger + dir string // mu serializes Reload: concurrent triggers queue rather than racing // validate-then-swap sequences (a stale document must not overwrite a newer one). @@ -42,8 +41,8 @@ type Store struct { // Open validates dir and returns a Store holding its document. A rejected // directory returns a nil Store with the findings — the caller (boot) // refuses to start; it must never run without adopted settings. -func Open(dir string, logger *slog.Logger) (*Store, []Finding) { - s := &Store{dir: dir, logger: logger} +func Open(dir string) (*Store, []Finding) { + s := &Store{dir: dir} findings, adopted := s.Reload("boot") if !adopted { return nil, findings @@ -71,26 +70,24 @@ func (s *Store) Reload(trigger string) ([]Finding, bool) { fn() } } - if s.logger != nil { - var errs, warns int - for _, f := range findings { - if f.Severity == SeverityError { - errs++ - s.logger.Error("settings finding", "trigger", trigger, "finding", f.String()) - } else { - warns++ - s.logger.Warn("settings finding", "trigger", trigger, "finding", f.String()) - } - } - switch { - case adopted: - s.logger.Info("settings adopted", "trigger", trigger, "dir", s.dir, "warnings", warns) - case s.snap.Load() == nil: - s.logger.Error("settings rejected", "trigger", trigger, "dir", s.dir, "errors", errs, "warnings", warns) - default: - s.logger.Error("settings rejected — keeping previous settings", "trigger", trigger, "dir", s.dir, "errors", errs, "warnings", warns) + var errs, warns int + for _, f := range findings { + if f.Severity == SeverityError { + errs++ + slog.Error("settings finding", "trigger", trigger, "finding", f.String()) + } else { + warns++ + slog.Warn("settings finding", "trigger", trigger, "finding", f.String()) } } + switch { + case adopted: + slog.Info("settings adopted", "trigger", trigger, "dir", s.dir, "warnings", warns) + case s.snap.Load() == nil: + slog.Error("settings rejected", "trigger", trigger, "dir", s.dir, "errors", errs, "warnings", warns) + default: + slog.Error("settings rejected — keeping previous settings", "trigger", trigger, "dir", s.dir, "errors", errs, "warnings", warns) + } return findings, adopted } diff --git a/internal/settings/store_test.go b/internal/settings/store_test.go index 078cb02d..f0ed9e1e 100644 --- a/internal/settings/store_test.go +++ b/internal/settings/store_test.go @@ -18,7 +18,7 @@ func newLoadedStore(t *testing.T, overrides map[string]string) *Store { for name, content := range overrides { files[name] = content } - s, findings := Open(writeDir(t, files), nil) + s, findings := Open(writeDir(t, files)) require.NotNil(t, s, "findings: %s", findingStrings(findings)) return s } @@ -88,11 +88,11 @@ func TestStore_OpenRejectsInvalid(t *testing.T) { t.Parallel() files := validFiles() files[FileConfig] = `{}` // every key missing - s, findings := Open(writeDir(t, files), nil) + s, findings := Open(writeDir(t, files)) assert.Nil(t, s) assert.True(t, HasErrors(findings)) - s, findings = Open(filepath.Join(t.TempDir(), "nope"), nil) + s, findings = Open(filepath.Join(t.TempDir(), "nope")) assert.Nil(t, s) assert.True(t, HasErrors(findings)) } @@ -122,7 +122,7 @@ func TestStore_SeedIsValid(t *testing.T) { t.Parallel() dir := filepath.Join(t.TempDir(), "settings") require.NoError(t, WriteSeed(dir)) - s, findings := Open(dir, nil) + s, findings := Open(dir) require.NotNil(t, s, "findings: %s", findingStrings(findings)) assert.False(t, HasErrors(findings)) // The one expected finding: an empty policies.json is fail-closed and diff --git a/internal/settings/watch.go b/internal/settings/watch.go index 1d8f6325..6a1f3f7b 100644 --- a/internal/settings/watch.go +++ b/internal/settings/watch.go @@ -3,6 +3,7 @@ package settings import ( "context" "fmt" + "log/slog" "path/filepath" "time" @@ -50,8 +51,8 @@ func (s *Store) Watch(ctx context.Context) error { // Best effort: a parent that can't be watched (e.g. "/" permissions) // costs only the recreate case, not the watcher. if parent := filepath.Dir(dir); parent != dir { - if err := w.Add(parent); err != nil && s.logger != nil { - s.logger.Warn("settings watcher: parent directory not watched; a deleted-and-recreated settings directory won't reload until SIGHUP or POST /v1/ops/settings/reload", "parent", parent, "error", err) + if err := w.Add(parent); err != nil { + slog.WarnContext(ctx, "settings watcher: parent directory not watched; a deleted-and-recreated settings directory won't reload until SIGHUP or POST /v1/ops/settings/reload", "parent", parent, "error", err) } } // Catch up on the gap between Open's read at boot and the watch existing: @@ -89,9 +90,7 @@ func (s *Store) Watch(ctx context.Context) error { if !ok { return nil } - if s.logger != nil { - s.logger.Error("settings watcher error", "dir", s.dir, "error", werr) - } + slog.ErrorContext(ctx, "settings watcher error", "dir", s.dir, "error", werr) case <-timer.C: // Re-arm the directory watch before reloading: after a remove or // rename fsnotify has dropped it, and Add is a no-op while it diff --git a/internal/stream/hub_test.go b/internal/stream/hub_test.go index acd70224..ef16df7c 100644 --- a/internal/stream/hub_test.go +++ b/internal/stream/hub_test.go @@ -36,7 +36,7 @@ import ( // nested-object claims decode unchanged, so literal maps stay faithful there. func jwtClaims(t *testing.T, claims map[string]any) map[string]any { t.Helper() - authn, err := auth.NewAuthenticator(auth.Config{JWTSecret: testutil.TestJWTSecret}, nil, nil) + authn, err := auth.NewAuthenticator(auth.Config{JWTSecret: testutil.TestJWTSecret}, nil) require.NoError(t, err) var got map[string]any h := authn.Middleware()(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) { diff --git a/internal/testutil/logtest/logtest.go b/internal/testutil/logtest/logtest.go new file mode 100644 index 00000000..59ae43cc --- /dev/null +++ b/internal/testutil/logtest/logtest.go @@ -0,0 +1,59 @@ +// Package logtest points slog's default logger somewhere a test can use. The +// packages log through slog.Default() rather than an injected logger, so a +// test reaches their output by swapping the default. It imports nothing from +// the repository, so every package's tests can use it. +package logtest + +import ( + "bytes" + "io" + "log/slog" + "sync" + "testing" +) + +// Silence discards everything logged through the default logger. Call it from +// a package's TestMain so a failing test's output is not buried in log lines. +func Silence() { + slog.SetDefault(slog.New(slog.NewTextHandler(io.Discard, nil))) +} + +// Buffer is a concurrency-safe log sink: the code under test may log from +// its own goroutines while the test reads. +type Buffer struct { + mu sync.Mutex + b bytes.Buffer +} + +func (b *Buffer) Write(p []byte) (int, error) { + b.mu.Lock() + defer b.mu.Unlock() + return b.b.Write(p) +} + +func (b *Buffer) String() string { + b.mu.Lock() + defer b.mu.Unlock() + return b.b.String() +} + +// Bytes returns a copy of what has been logged so far. +func (b *Buffer) Bytes() []byte { + b.mu.Lock() + defer b.mu.Unlock() + return bytes.Clone(b.b.Bytes()) +} + +// Capture routes the default logger to the returned Buffer as JSON records at +// level and above, until the test ends. The default logger is process-wide, +// so a test that captures must not call t.Parallel: serial tests never +// overlap the package's parallel ones, which is what keeps another test's +// lines out of the buffer. +func Capture(t testing.TB, level slog.Level) *Buffer { + t.Helper() + buf := &Buffer{} + prev := slog.Default() + slog.SetDefault(slog.New(slog.NewJSONHandler(buf, &slog.HandlerOptions{Level: level}))) + t.Cleanup(func() { slog.SetDefault(prev) }) + return buf +} diff --git a/internal/testutil/logtest/logtest_test.go b/internal/testutil/logtest/logtest_test.go new file mode 100644 index 00000000..2069c2e3 --- /dev/null +++ b/internal/testutil/logtest/logtest_test.go @@ -0,0 +1,29 @@ +package logtest + +import ( + "log/slog" + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestCapture_RoutesAndRestoresTheDefaultLogger(t *testing.T) { + prev := slog.Default() + t.Run("captured", func(t *testing.T) { + buf := Capture(t, slog.LevelWarn) + slog.Info("below the level") + slog.Warn("kept", "k", "v") + assert.NotContains(t, buf.String(), "below the level") + assert.Contains(t, buf.String(), `"msg":"kept"`) + assert.Contains(t, string(buf.Bytes()), `"k":"v"`) + }) + assert.Same(t, prev, slog.Default(), "the previous default is restored when the capturing test ends") +} + +func TestSilence(t *testing.T) { + prev := slog.Default() + t.Cleanup(func() { slog.SetDefault(prev) }) + Silence() + assert.NotSame(t, prev, slog.Default()) + slog.Error("discarded") +} diff --git a/internal/testutil/testutil.go b/internal/testutil/testutil.go index f0e39695..80e141de 100644 --- a/internal/testutil/testutil.go +++ b/internal/testutil/testutil.go @@ -4,8 +4,6 @@ import ( "context" "encoding/json" "fmt" - "io" - "log/slog" "net/http/httptest" "strings" "testing" @@ -19,12 +17,6 @@ import ( "github.com/Wave-RF/WaveHouse/internal/tenant" ) -// NopLogger returns a *slog.Logger that discards all output. -// Use in tests to suppress noisy log output from embedded NATS, etc. -func NopLogger() *slog.Logger { - return slog.New(slog.NewTextHandler(io.Discard, nil)) -} - // NewTestSchemaRegistry creates a SchemaRegistry pre-loaded with the given // table schemas, without a real ClickHouse: a mock connection serves the // schemas as system.columns rows (UTC as the server zone) and the registry is @@ -36,7 +28,7 @@ func NopLogger() *slog.Logger { // type string), not the caller's structs. func NewTestSchemaRegistry(t testing.TB, tables []*discovery.TableSchema) *discovery.SchemaRegistry { t.Helper() - reg := discovery.NewSchemaRegistry(&schemaConn{tables: tables}, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }, NopLogger()) + reg := discovery.NewSchemaRegistry(&schemaConn{tables: tables}, func() string { return "test" }, tenant.Default, func(tenant.ID) time.Duration { return time.Hour }) require.NoError(t, reg.Refresh(context.Background())) return reg } diff --git a/tests/integration/boot_resilience_test.go b/tests/integration/boot_resilience_test.go index b0ecf10d..e511953f 100644 --- a/tests/integration/boot_resilience_test.go +++ b/tests/integration/boot_resilience_test.go @@ -18,7 +18,6 @@ import ( "github.com/Wave-RF/WaveHouse/internal/api" "github.com/Wave-RF/WaveHouse/internal/discovery" "github.com/Wave-RF/WaveHouse/internal/tenant" - "github.com/Wave-RF/WaveHouse/internal/testutil" ) // TestBootResilience_StickyHealthVsConditionalReady exercises the full @@ -41,7 +40,6 @@ import ( // package, which is exactly the assumption this test needs to violate. func TestBootResilience_StickyHealthVsConditionalReady(t *testing.T) { ctx := context.Background() - logger := testutil.NopLogger() ch, err := startClickHouse(ctx) require.NoError(t, err, "starting initial CH container") @@ -68,7 +66,7 @@ func TestBootResilience_StickyHealthVsConditionalReady(t *testing.T) { require.NoError(t, err, "reopen driver against stopped CH") bootState := api.NewBootState(nil) - registry := discovery.NewSchemaRegistry(ch.conn, func() string { return testCHDatabase }, tenant.Default, func(tenant.ID) time.Duration { return time.Minute }, logger) + registry := discovery.NewSchemaRegistry(ch.conn, func() string { return testCHDatabase }, tenant.Default, func(tenant.ID) time.Duration { return time.Minute }) // === Row 1: Boot, CH down === err = registry.Refresh(ctx) @@ -90,7 +88,7 @@ func TestBootResilience_StickyHealthVsConditionalReady(t *testing.T) { _ = ch.conn.Close() ch.conn, err = openDriver(ch.nativeAddr()) require.NoError(t, err, "reopen driver against restarted CH") - registry = discovery.NewSchemaRegistry(ch.conn, func() string { return testCHDatabase }, tenant.Default, func(tenant.ID) time.Duration { return time.Minute }, logger) + registry = discovery.NewSchemaRegistry(ch.conn, func() string { return testCHDatabase }, tenant.Default, func(tenant.ID) time.Duration { return time.Minute }) h.CHConn = ch.conn require.NoError(t, waitForNativeReady(ctx, ch.conn, 30*time.Second), "CH native should be ready after restart") diff --git a/tests/integration/query_limits_test.go b/tests/integration/query_limits_test.go index 4d9b95de..caa2a0ed 100644 --- a/tests/integration/query_limits_test.go +++ b/tests/integration/query_limits_test.go @@ -18,7 +18,6 @@ import ( "github.com/Wave-RF/WaveHouse/internal/auth" "github.com/Wave-RF/WaveHouse/internal/policy" "github.com/Wave-RF/WaveHouse/internal/settings" - "github.com/Wave-RF/WaveHouse/internal/testutil" ) // TestStructuredQuery_ResourceCapsEnforcedServerSide is the executable proof @@ -100,7 +99,7 @@ func TestStructuredQuery_ResourceCapsEnforcedServerSide(t *testing.T) { }, } h := api.NewStructuredQueryHandler( - e.chConn, nil, e.registry, func(*settings.Store) *policy.Policy { return p }, func(*settings.Store) int { return 60 }, func() time.Duration { return 30 * time.Second }, nil, testutil.NopLogger(), + e.chConn, nil, e.registry, func(*settings.Store) *policy.Policy { return p }, func(*settings.Store) int { return 60 }, func() time.Duration { return 30 * time.Second }, nil, ) req := httptest.NewRequest(http.MethodPost, From a9d76a7f966e759d82bb6479ed2e39e2b2328be3 Mon Sep 17 00:00:00 2001 From: taitelee Date: Fri, 18 Sep 2026 10:08:54 -0400 Subject: [PATCH 03/11] docs(tenant): address round-one review findings --- docs/src/content/docs/api.md | 8 ++++---- docs/src/content/docs/architecture.md | 2 +- docs/src/content/docs/development.md | 1 + docs/src/content/docs/reverse-proxy.mdx | 1 + internal/api/structured_query.go | 6 +++--- internal/app/wire.go | 16 ++++++++-------- 6 files changed, 18 insertions(+), 16 deletions(-) diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index 5d3d638d..48ed5b21 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -58,7 +58,7 @@ Every `/v1` route outside `/v1/ops/*` resolves a tenant before it authenticates X-Tenant-ID: 0 ``` -A request without the header, or with an empty one, resolves to tenant `0`, the default tenant, whose settings are the [settings directory](/settings-directory). Setting the header on every request is the client's or the fronting proxy's job; WaveHouse never derives it from the token. +A request without the header, or with an empty one, resolves to tenant `0`, the default tenant, whose settings are the [settings directory](/settings-directory). A settings directory defines that one tenant, so any other id is unknown. Setting the header on every request is the client's or the fronting proxy's job; WaveHouse never derives it from the token. A tenant id is 1–64 characters of ASCII letters, digits, `_`, and `-`. It is a string, not a number, so a long numeric id keeps every digit. @@ -173,7 +173,7 @@ Status code: `503 Service Unavailable` ### `GET /v1/health` — Liveness ping (public, content-free) -Returns **`200 OK` with an empty body** once the gateway is past boot, or **`503 Service Unavailable`** (also empty) while boot-time schema discovery is still failing. No authentication required and no response body — the caller only branches on the status code, so there's nothing to JSON-encode or cache per request. +Returns **`200 OK` with an empty body** once the gateway is past boot, or **`503 Service Unavailable`** (also empty) while boot-time schema discovery is still failing. Like every other `/v1` route it [resolves a tenant](#tenant-selection) first, so a bad `X-Tenant-ID` answers `400`/`404` before the probe runs. No authentication required and no response body — the caller only branches on the status code, so there's nothing to JSON-encode or cache per request. This is what the SDK's `wh.sys.health()` calls, and the endpoint to use when choosing among multiple servers in a distributed setup. It mirrors `/livez` under the hood but is intentionally a `/v1` API route rather than a Kubernetes probe path: an operator may filter the bare probe paths (`/livez`, `/readyz`, `/healthz`) out at the reverse proxy since they're internal probes, so the SDK relies on `/v1/health`, which is documented public API surface meant to stay reachable. It does **not** ping ClickHouse — readiness-based load balancing is the proxy/LB's job (via `/readyz`), not the client's. @@ -781,11 +781,11 @@ The policy has no endpoints: it is the settings directory's [`policies.json`](/s Returns every adopted named query pipe — the settings directory's [`pipes.json`](/settings-directory#pipesjson). Pipes have no write endpoints: edit the file and reload. -The ops routes are [tenant-exempt](#tenant-selection), so this read and `GET /v1/ops/pipes/{name}` name their tenant with an optional `?tenant=` query parameter instead of the header. Absent or empty means tenant `0`; a malformed id or a repeated parameter is a `400`, and an unknown tenant a `404`, with the same bodies as the header. +The ops routes are [tenant-exempt](#tenant-selection), so this read and `GET /v1/ops/pipes/{name}` name their tenant with an optional `?tenant=` query parameter instead of the header. Absent or empty means tenant `0`; a malformed id or a repeated parameter is a `400` (`{"error": "invalid ?tenant: …"}`), and an unknown tenant a `404` with the same body as the header. #### `GET /v1/ops/pipes/{name}` — Get Named Pipe -Returns a specific named pipe definition (of the `?tenant=`, as above): +Returns a specific named pipe definition from the tenant named by `?tenant=`, as above: ```json { diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 04cd5c30..5a6cd367 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -104,7 +104,7 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ ### `auth/` — Authentication -- **auth.go** — `NewAuthenticator(cfg, policySource)` owns the verifier; its `Middleware()` reads the current one per request, and `Reconfigure(cfg)` swaps the whole verifier — key source plus its pinned `alg` allowlist — atomically after a settings reload (`auth.jwks_url` / `auth.role_claim`; see [Settings Directory — Authentication](/settings-directory#authentication)). Verifies JWT tokens with HMAC **or** JWKS (never both), with the accepted `alg` pinned to the active verifier and checked before any key is consulted (rejects `alg: none` and cross-family confusion). Extracts the caller's role from a configurable dot-path claim (`auth.role_claim`, default `role`). Claims parse with `jwt.WithJSONNumber()`, so a numeric claim reaches the policy engine as its exact digits (`json.Number`, never a rounded float64) — part of the row-visibility guarantee (AGENTS.md invariant 12). It always runs and never rejects — a missing/invalid/expired token yields an empty role (resolved to `default_role` downstream), with the token error stashed in context so a denying gate can fail loud (`401`, not a bare `403`). Before the Bearer token it checks a non-JWT operator key (`auth.operator_key`): a constant-time match on the presented credential — an `Authorization: Operator ` header, or the `X-Operator-Key` alias — stamps the live admin role plus an operator bit (`auth.WithOperator`) that `RequireAdmin` honors even under a nil policy — a full-access break-glass credential, audit-logged at Info with no client IP (`store`/`logger` back this path). A presented-but-wrong operator key is logged at `WARN` and counted by `wavehouse_auth_operator_key_failures_total` — a probing signal on the most privileged credential — then falls through like any unauthenticated request (the middleware never rejects). +- **auth.go** — `NewAuthenticator(cfg, policySource)` owns the verifier; its `Middleware()` reads the current one per request, and `Reconfigure(cfg)` swaps the whole verifier — key source plus its pinned `alg` allowlist — atomically after a settings reload (`auth.jwks_url` / `auth.role_claim`; see [Settings Directory — Authentication](/settings-directory#authentication)). Verifies JWT tokens with HMAC **or** JWKS (never both), with the accepted `alg` pinned to the active verifier and checked before any key is consulted (rejects `alg: none` and cross-family confusion). Extracts the caller's role from a configurable dot-path claim (`auth.role_claim`, default `role`). Claims parse with `jwt.WithJSONNumber()`, so a numeric claim reaches the policy engine as its exact digits (`json.Number`, never a rounded float64) — part of the row-visibility guarantee (AGENTS.md invariant 12). It always runs and never rejects — a missing/invalid/expired token yields an empty role (resolved to `default_role` downstream), with the token error stashed in context so a denying gate can fail loud (`401`, not a bare `403`). Before the Bearer token it checks a non-JWT operator key (`auth.operator_key`): a constant-time match on the presented credential — an `Authorization: Operator ` header, or the `X-Operator-Key` alias — stamps the live admin role plus an operator bit (`auth.WithOperator`) that `RequireAdmin` honors even under a nil policy — a full-access break-glass credential, audit-logged at Info with no client IP (`store` backs this path; the audit line goes through the default logger). A presented-but-wrong operator key is logged at `WARN` and counted by `wavehouse_auth_operator_key_failures_total` — a probing signal on the most privileged credential — then falls through like any unauthenticated request (the middleware never rejects). - **context.go** — request-context accessors and their setters for the role, claims, and token error (`RoleFromContext`, `ClaimsFromContext`, `AuthErrorFromContext`, and the matching `With*` helpers). ### `cache/` — Query Cache diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index 68eaff68..758a6c93 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -468,6 +468,7 @@ WaveHouse/ │ ├── query/ # Structured query AST + SQL builder │ ├── settings/ # Settings directory: validate, adopted snapshot, reload │ ├── stream/ # SSE fan-out: Hub, Subscriber queue, Bucket, keepalive wheel +│ ├── tenant/ # Tenant id: type, grammar, reserved default, request header name │ └── testutil/ # Shared test helpers and mocks ├── tests/ # Integration & E2E tests │ ├── integration/ # Go integration tests (//go:build integration) diff --git a/docs/src/content/docs/reverse-proxy.mdx b/docs/src/content/docs/reverse-proxy.mdx index 8705956b..187d29ce 100644 --- a/docs/src/content/docs/reverse-proxy.mdx +++ b/docs/src/content/docs/reverse-proxy.mdx @@ -194,6 +194,7 @@ These limit the *whole* request regardless of traffic, so no keepalive extends t - **`Authorization`** — forward verbatim. WaveHouse validates a `Bearer` JWT (resolving the role from it) or an `Operator ` [operator credential](/access-control#operator-key). Most proxies forward `Authorization` unchanged, so it's the transport to prefer for the operator key — the exception to check for is an auth-terminating layer that consumes or rewrites the header (e.g. a gateway doing its own auth). - **`X-Operator-Key`** (optional) — only relevant if you present the [operator key](/access-control#operator-key) via this alias header instead of `Authorization: Operator `. Custom request headers are forwarded by default, but confirm your proxy doesn't strip it — and note nginx silently drops header names containing **underscores** (this one uses hyphens, so it's fine as named). The `Authorization` form needs none of this. +- **`X-Tenant-ID`** (optional) — selects the [tenant](/api#tenant-selection); absent means tenant `0`. It is resolved before authentication and never derived from the token, so if the proxy owns tenant selection it must **replace or strip** a client-supplied value, not add to it: two `X-Tenant-ID` lines are refused with `400`, the same append-vs-replace hazard as `Content-Type` below. The name is hyphenated, so nginx's underscore rule does not apply. - **`X-Forwarded-For` / `X-Forwarded-Proto` / `Host`** — set these for your own logs and any upstream that reads them. WaveHouse does not currently derive a client IP from `X-Forwarded-For` (see the caution below); forwarding it is good hygiene and is what the trusted-proxy client-IP work ([#333](https://github.com/Wave-RF/WaveHouse/issues/333)) will consume. :::caution[Don't expose `:8080` directly] diff --git a/internal/api/structured_query.go b/internal/api/structured_query.go index 525b91cf..61c0da18 100644 --- a/internal/api/structured_query.go +++ b/internal/api/structured_query.go @@ -35,9 +35,9 @@ type StructuredQueryHandler struct { // ((*settings.Store).TimestampBucketSeconds in production) and // defaultMaxRows its current fallback result LIMIT // ((*settings.Store).DefaultMaxRows) — funcs, not ints, so a settings - // reload takes effect on the next query without a restart. A nil bucketSecs means no bucketing; a nil - // defaultMaxRows or a non-positive return means the builder's compiled - // constant. + // reload takes effect on the next query without a restart. A nil + // bucketSecs means no bucketing; a nil defaultMaxRows or a non-positive + // return means the builder's compiled constant. bucketSecs func(*settings.Store) int defaultMaxRows func(*settings.Store) int diff --git a/internal/app/wire.go b/internal/app/wire.go index 569f169c..0371a6b7 100644 --- a/internal/app/wire.go +++ b/internal/app/wire.go @@ -67,14 +67,6 @@ func (a *App) wireSettings() error { return nil } -// wireObservability initializes the OTel pipeline whenever either OTLP push -// or Prometheus exposition is wanted — Prometheus-only operation -// (Alloy/scrape, no collector) is a first-class mode, and the OTel SDK -// MeterProvider is the shared substrate. Endpoint, TLS, and auth headers -// come from the standard OTEL_EXPORTER_OTLP_* env vars, read by the SDK. A -// malformed header is logged and skipped by the SDK (fail-soft); -// InitProvider's own error is likewise non-fatal — logged, stdout-only from -// there on. // perTenant adapts a store accessor to the tenant-keyed getter the async // paths take: they hold a tenant id (tenant.Default today, the MQ subject's // from #583 story 5), not a request's resolved store. Only tenant.Default @@ -91,6 +83,14 @@ func perTenant[T any](tenants *settings.Registry, get func(*settings.Store) T) f } } +// wireObservability initializes the OTel pipeline whenever either OTLP push +// or Prometheus exposition is wanted — Prometheus-only operation +// (Alloy/scrape, no collector) is a first-class mode, and the OTel SDK +// MeterProvider is the shared substrate. Endpoint, TLS, and auth headers +// come from the standard OTEL_EXPORTER_OTLP_* env vars, read by the SDK. A +// malformed header is logged and skipped by the SDK (fail-soft); +// InitProvider's own error is likewise non-fatal — logged, stdout-only from +// there on. func (a *App) wireObservability(ctx context.Context) { cfg := a.cfg if !cfg.OTel.Enabled && !cfg.Prometheus.Enabled { From e7f0c16309258ca088dc2e0f77499db489c83c46 Mon Sep 17 00:00:00 2001 From: taitelee Date: Fri, 18 Sep 2026 10:25:53 -0400 Subject: [PATCH 04/11] fix(tenant): reject a malformed ops query, re-read policy per replayed row --- AGENTS.md | 2 +- CHANGELOG.md | 4 +++- docs/src/content/docs/api.md | 2 +- docs/src/content/docs/architecture.md | 11 ++++++---- docs/src/content/docs/sdk/index.mdx | 2 +- docs/src/content/docs/settings-directory.mdx | 2 +- internal/api/tenant.go | 13 ++++++++++-- internal/api/tenant_test.go | 8 ++++++- internal/stream/hub.go | 4 +++- internal/stream/hub_test.go | 22 ++++++++++++++++++++ 10 files changed, 57 insertions(+), 13 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index c5396666..b27a54e3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -29,7 +29,7 @@ One binary: Eighteen internal packages under `internal/` (plus `internal/testutil/` for shared test helpers): - **`api/`** — Chi HTTP router, JWT/JWKS middleware (from `auth/`), ingest/query/structured-query/SSE/schema/DLQ/pipes handlers -- **`app/`** — the process wiring: `New` builds every component from the boot config and the settings directory (each one wired in one place — what it opens, what it loops, what it releases — with the settings store handed to its wiring function whole, the injection point the per-tenant registry of #583 lands on), `Run` drives the long-lived ones under one `errgroup` until the context is cancelled or one fails, `Close` releases them in reverse order. `cmd/wavehouse` and `tests/integration` both boot through it +- **`app/`** — the process wiring: `New` builds every component from the boot config and the settings directory (each one wired in one place — what it opens, what it loops, what it releases — with the settings store handed to its wiring function whole, the injection point of the per-tenant registry of #583: store-keyed getters for the handlers, `perTenant` for the async paths), `Run` drives the long-lived ones under one `errgroup` until the context is cancelled or one fails, `Close` releases them in reverse order. `cmd/wavehouse` and `tests/integration` both boot through it - **`auth/`** — JWT auth middleware: HMAC **or** JWKS verification with `alg` pinned to the active verifier, role extraction from a configurable claim path; always runs, never rejects (bad token → empty role + stashed reason) - **`cache/`** — `Cache` interface → `LocalCache` (Ristretto) + `SharedCache` (TBD) + `TieredCache` (singleflight) - **`chconn/`** — `Manager`, the one ClickHouse `driver.Conn` every consumer holds; `Reconfigure` swaps the connection behind it after a settings reload changes the wiring (never dials; the old connection closes after a `query_timeout` grace) diff --git a/CHANGELOG.md b/CHANGELOG.md index 79ef0776..4e2a57a0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **Requests resolve to a tenant before authentication, and the tenant is threaded through every settings read** (`internal/tenant/` (new, + tests), `internal/settings/registry.go` (new, + tests), `internal/api/tenant.go` (new, + tests), `internal/api/{router,ingest,structured_query,pipes}.go`, `internal/ingest/{worker,sweeper}.go`, `internal/stream/hub.go`, `internal/discovery/discovery.go`, `internal/app/{app,wire}.go`): story 1 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)), with no behavior change for a deployment that sends no tenant header. `internal/tenant` defines the id — a validated string (letters, digits, `_`, `-`; at most 64 bytes, so it is safe as a folder name and as an MQ subject token), the reserved default `0`, and the `X-Tenant-ID` header name — and imports nothing from the rest of the repository. `api.TenantMW` runs ahead of the auth middleware on every `/v1` route outside `/v1/ops/*`: an absent or empty header is tenant `0`, a malformed id or a repeated header is a `400`, a well-formed id the new `settings.Registry` does not hold is a `404`, and the resolved `*settings.Store` rides the request context. The registry holds the one store `settings.Open` adopted, keyed `0`. Handlers read the store once and pass it down as an argument — the ingest, structured-query, and pipe getters (`PolicySource`, `DedupeSettings`, `bucketSecs`, `defaultMaxRows`, the pipes source) now take it as a parameter — and a tenant route reached without a resolved tenant answers `500` rather than fall back to one. The ingest worker, sweeper, stream hub, and schema registry are constructed with a `tenant.ID` (`tenant.Default` in `internal/app`) and their settings getters take it. The probes, `/version`, the metrics path, and `/v1/ops/*` stay tenant-exempt, with the ops tree behind the auth middleware and the admin gate exactly as before; the admin pipe reads (`GET /v1/ops/pipes[/{name}]`) name their tenant with an optional `?tenant=` query parameter (absent means `0`, same `400`/`404` answers as the header). `X-Tenant-ID` joins the CORS `Access-Control-Allow-Headers` list so a browser client can send it; the SDK needs no change (`options.headers`). +- **Requests resolve to a tenant before authentication, and the tenant is threaded through every settings read** (`internal/tenant/` (new, + tests), `internal/settings/registry.go` (new, + tests), `internal/api/tenant.go` (new, + tests), `internal/api/{router,ingest,structured_query,pipes}.go`, `internal/ingest/{worker,sweeper}.go`, `internal/stream/hub.go`, `internal/discovery/discovery.go`, `internal/app/{app,wire}.go`): story 1 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)), with no behavior change for a deployment that sends no tenant header. `internal/tenant` defines the id — a validated string (letters, digits, `_`, `-`; at most 64 bytes, so it is safe as a folder name and as an MQ subject token), the reserved default `0`, and the `X-Tenant-ID` header name — and imports nothing from the rest of the repository. `api.TenantMW` runs ahead of the auth middleware on every `/v1` route outside `/v1/ops/*`: an absent or empty header is tenant `0`, a malformed id or a repeated header is a `400`, a well-formed id the new `settings.Registry` does not hold is a `404`, and the resolved `*settings.Store` rides the request context. The registry holds the one store `settings.Open` adopted, keyed `0`. Handlers read the store once and pass it down as an argument — the ingest, structured-query, and pipe getters (`PolicySource`, `DedupeSettings`, `bucketSecs`, `defaultMaxRows`, the pipes source) now take it as a parameter — and a tenant route reached without a resolved tenant answers `500` rather than fall back to one. The ingest worker, sweeper, stream hub, and schema registry are constructed with a `tenant.ID` (`tenant.Default` in `internal/app`) and their settings getters take it. The probes, `/version`, the metrics path, and `/v1/ops/*` stay tenant-exempt, with the ops tree behind the auth middleware and the admin gate exactly as before; the admin pipe reads (`GET /v1/ops/pipes[/{name}]`) name their tenant with an optional `?tenant=` query parameter (absent means `0`, same `400`/`404` answers as the header; a query string that does not parse is a `400` too, rather than silently reading as the default tenant). `X-Tenant-ID` joins the CORS `Access-Control-Allow-Headers` list so a browser client can send it; the SDK needs no change (`options.headers`). - **Schema discovery captures each table's DDL, its columns' ordinals and default expressions, and the server version** (`internal/discovery/discovery.go`, `internal/testutil/testutil.go`): `Column` gains `DefaultExpression` and `Position` (both from a widened `system.columns` select), `TableSchema` gains `DDL` from `system.tables.create_table_query`, and `SchemaRegistry` gains `ServerVersion()` from a `SELECT version()` probe next to the existing `SELECT timezone()`. Groundwork for the native type layer, captured on the same refresh as the columns so a stale version cannot outlive the schemas it describes. That is a publication guarantee, not a same-server one: `chconn.Manager` resolves the connection per call, so a reload changing `clickhouse.addr` mid-refresh can still pair a version from one server with schemas from another — narrow, and self-correcting on the next refresh. `DDL` is `json:"-"` and does **not** appear in `/v1/ops/schema`: that endpoint marshals `TableSchema` straight to the client, and an external-engine table (S3, MySQL, PostgreSQL, Kafka) renders its wiring there unconditionally — endpoint, bucket or host, database, username, S3 access key id. ClickHouse masks the password itself as `[HIDDEN]` from ~23.9 (verified on 26.7.3), so the exposure is the topology rather than the secret — except on an older server, or one with `display_secrets_in_show_and_select` enabled. `position` and `default_expression` are additive fields in the response. A table listed in `system.tables` with no `system.columns` rows is skipped rather than published column-less, and both new queries fail the refresh on error exactly as `timezone()` and `system.columns` do — callers keep the prior cache and retry. @@ -62,6 +62,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Fixed +- **SSE gap-fill re-reads the policy per replayed row** (`internal/stream/hub.go`): `ReplayProjector` captured the policy once when the replay began, so a policy adopted mid-fill — a revoked grant, say — applied only after the fill ended. It now reads it per event, as `Broadcast` does on the live path. + - **`classify-paths.sh` no longer reads a `grep` failure as "no match"** (`scripts/classify-paths.sh`, `scripts/classify-paths.test.sh`): both decisions were `if printf … | grep -qE …; then A; else B; fi`. `grep` exits `0` on match, `1` on no match and **`2` on error** (can't fork/exec, read error, bad pattern), and the `else` branch collapsed `1` and `2` into the same answer — `set -euo pipefail` does not help, since `set -e` is suppressed for a command used as an `if` condition. Observed twice in local `make ci` runs whose static checks run at `-j 14`: a different single case failed each time (`mixed-docs-go` answering `docs=false`, then `dep-bump-go` answering `code=false`) while every other case passed, which is the signature of a transient `grep` failure rather than a pattern bug. The test caught it only because it asserts expected values; **the production path has no such check** — CI's `changes` job gates the docs pipeline on this answer, so a `docs=false` produced by an errored `grep` silently skips the docs build and still reports success. The two greps now go through a `matches` helper that aborts with a diagnostic on any exit above 1, and the test suite stubs `grep` onto `PATH` to prove the abort fires (that case fails against the previous script). A second instance of the same class, found reviewing the first fix: the helper piped its input into `grep -q`, which exits at the first match — so once the file list outgrew the pipe buffer (a few thousand paths) the upstream `printf` died of SIGPIPE, `pipefail` reported 141, and the new error arm aborted on an ordinary large change set. Reproduced at 5,000 paths. It now reads from a here-string instead, and the suite pins that case. `scripts/ci/classify-changes.sh` also stopped reading the classifier through process substitution, which discarded its exit status: a classifier that aborted left `code`/`docs` empty, every `needs.changes.outputs.code == 'true'` job skipped, and the `CI` aggregator reported green having run nothing. It now captures the status, and fails closed — running everything — on a failed *or* partial classification, matching the rule already used for an empty file list. Also here, unrelated and one line: `biome.json` declared `$schema` 2.4.15 while the lockfile pins the 2.5.8 CLI, so `biome check --error-on-warnings` failed on the config itself for any change touching TypeScript. Bumped to match; it changes no lint rule. - **The role-first split makes two of #541's rule rejections structural, and adds the resolver-side half of a third** (`internal/policy/policy.go`): [#541](https://github.com/Wave-RF/WaveHouse/pull/541) rejects `filter` under an `insert` grant and `check` under a `select` one at validation time. With `select` and `insert` as separate types those fields do not exist on the wrong side at all, so the strict decode refuses them as unknown keys and the runtime checks are gone — the same document is still refused, one layer earlier. #541's operator-less `filter`/`check` rejection is unchanged and keeps its message; what is added here is the matching deny in `evaluateSelect`/`evaluateInsert` — for the operator-less shape and, separately, for a check using an operator the resolver does not honor (`_neq`/`_gt`/`_lt`, or the ambiguous `_eq`+`_in`), which main's `Evaluate` resolved to no clause at all and authorized the insert with the rule silently gone; `Evaluate` does not re-validate the policy it is handed and `policy.Static` is a Validate-free `policy.Source`, so without the operator-less deny a policy reaching the resolvers unvalidated still resolved an operator-less `filter` entry to no predicate and answered `RowVisible` true for every row. `evaluateInsert` also gained the bind-unsafe check-column deny — the insert-side mirror of the `filter`-side guard main already had. diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index 48ed5b21..bcfd7a9e 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -781,7 +781,7 @@ The policy has no endpoints: it is the settings directory's [`policies.json`](/s Returns every adopted named query pipe — the settings directory's [`pipes.json`](/settings-directory#pipesjson). Pipes have no write endpoints: edit the file and reload. -The ops routes are [tenant-exempt](#tenant-selection), so this read and `GET /v1/ops/pipes/{name}` name their tenant with an optional `?tenant=` query parameter instead of the header. Absent or empty means tenant `0`; a malformed id or a repeated parameter is a `400` (`{"error": "invalid ?tenant: …"}`), and an unknown tenant a `404` with the same body as the header. +The ops routes are [tenant-exempt](#tenant-selection), so this read and `GET /v1/ops/pipes/{name}` name their tenant with an optional `?tenant=` query parameter instead of the header. Absent or empty means tenant `0`; a malformed id, a repeated parameter, or a query string that does not parse is a `400` (`{"error": "invalid ?tenant: …"}`), and an unknown tenant a `404` with the same body as the header. #### `GET /v1/ops/pipes/{name}` — Get Named Pipe diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 5a6cd367..7164e0f1 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -90,7 +90,7 @@ The API layer uses [Chi](https://github.com/go-chi/chi) for routing with Request ### `app/` — Process wiring - **app.go** — `New(ctx, Options)` builds every component from the boot config (`Options.Config`) and the settings directory it names, in dependency order: settings store, observability, ClickHouse connection, schema discovery, dedupe store, embedded NATS (ingest + DLQ streams), cache, sweeper, streaming (hub, MQ→hub bridge, keepalive wheel), ingest worker, auth, reload triggers, HTTP. Each is one `component` value — what it opens, what it loops, what it releases — so a failure part-way releases what was already opened and returns the error. `Run(ctx)` drives every loop under one `errgroup` until `ctx` is canceled (a clean stop: every loop drains, the API server and the ingest worker within `server.shutdown_timeout`; open SSE streams are ended as the drain begins rather than waited on) or a component fails, which stops the rest and returns that error. `Close(ctx)` releases what `New` opened, newest first, under the caller's release budget (`ReleaseTimeout`, 5s), a real bound: a remote implementation's close gives up at the deadline itself, and a close that ignores the context (the local stores) is abandoned at it, with the components below it left unreleased rather than overlapping it, both named in the error — and then flushes telemetry under its own 3s budget, so the flush that reports on the stop is never handed a deadline a slow close already spent. The SIGHUP registration is released last of all. `Handler`, `Registry`, and `MQ` expose the pieces a harness needs; `Options.Listener` lets one serve the API on its own listener instead of `server.port`. -- **wire.go** — one `wire*` function per component, each handed the settings store whole and deriving the per-call getters the internal packages take (`DLQFor`, `DedupeFor`, `GapWindow`, …) and registering its `AfterAdopt` hook there where it has one. Those wiring functions are where the per-tenant registry of [#583](https://github.com/Wave-RF/WaveHouse/issues/583) lands, not `main`. The reload triggers (SIGHUP, the directory watcher) only start in `Run`, after `New` has registered every hook, so the watcher's first reload already drives all of them. The `mq.max_bytes_gb` hook only hands the adopted budget to `mq.Broker.SetMaxBytes` under the App's stop context; how it is split across the streams, the time bounds, and the rollback are `internal/mq`'s. +- **wire.go** — one `wire*` function per component, each handed the settings store whole and deriving the per-call getters the internal packages take (`DLQFor`, `DedupeFor`, `GapWindow`, …) and registering its `AfterAdopt` hook there where it has one. Those wiring functions are where the per-tenant registry of [#583](https://github.com/Wave-RF/WaveHouse/issues/583) is injected, not `main`: `wireSettings` builds the `settings.Registry`, the HTTP handlers get store-keyed getters (method expressions such as `(*settings.Store).Policy`), and `perTenant` adapts a store accessor into the `func(tenant.ID) T` getter the async packages take. The reload triggers (SIGHUP, the directory watcher) only start in `Run`, after `New` has registered every hook, so the watcher's first reload already drives all of them. The `mq.max_bytes_gb` hook only hands the adopted budget to `mq.Broker.SetMaxBytes` under the App's stop context; how it is split across the streams, the time bounds, and the rollback are `internal/mq`'s. ### `stream/` — SSE keepalive & fan-out @@ -164,7 +164,7 @@ The package's design invariants — stdout always 100%, WARN+ERROR always export - **rowfilter.go** — the in-memory row-visibility twin of the SQL `WHERE`: `HasRowFilter`, `RowVisible` (evaluates the resolved predicates against a decoded event, per subscriber), and `ColumnSpec` — the per-column comparison contract (`ColumnKind` `Numeric`/`Text`/`Time`/`Opaque`, plus each kind's parameters: the caller-supplied instant parser for `Time`, the `NumericSpec` storage model for `Numeric`) whose zero value is the fail-closed floor: numeric columns compare in the column's **storage domain** (operands rendered by canonical.go, compared by numeric.go — next two bullets), `String` bytewise, `DateTime`/`DateTime64` chronologically (both operands through the ingest grammar; either side unreadable ⇒ withheld), and everything else (including any column with no usable schema) admits byte-equality only, failing `!=`/`>`/`<` closed. Both `HasRowFilter` and `RowVisible` fail closed on a denied or unresolved grant: `HasRowFilter` is the gate in front of `RowVisible`, so it must answer *true* there or the whole-bucket fast path skips the check entirely. - **canonical.go** — the one rendering layer for comparison operands: every value a `filter` or `check` compares — a JWT claim (`CanonicalScalar`), a policy-authored literal (`CanonicalNumericLiteral`), an ingested payload value (`numericCanonical`) — converges on one exact canonical decimal form (positional, digit-bounded, never a float64 round-trip), so what a read filter binds and what the stream compares can't drift; `scalarString` is the deliberate exception, the raw byte rendering that `Text`/`Opaque` equality compares. - **numeric.go** — compares canonical forms the way the column that stores them would: `compareCanonicalDecimals` orders by exact digit-string arithmetic, and `NumericSpec` first narrows both operands the way ClickHouse narrows the stored value and the bound constant — `Float32`/`Float64` width rounding, `Decimal` scale truncation, integers exact at any width, with an operand outside the column's width or a `Decimal`'s precision budget refused rather than modeled; the `tests/integration` differential oracle holds in-range verdicts equal to a live ClickHouse's and asserts the never-admit-where-SQL-hides direction for the refused out-of-range operands. -- **source.go** — `Source`, a `func() *Policy` every consumer (the auth middleware, ingest, structured query, pipes, the stream hub, the `/v1/ops` gate) reads per call, so a settings reload applies to the very next request. In production it is `settings.Store.Policy`; `Static(p)` fixes one for tests. A `nil` result is a deliberate lockout. +- **source.go** — `Source`, a `func() *Policy` the auth middleware and the `/v1/ops` gate read per call, so a settings reload applies to the very next request; in production it is the default tenant's `settings.Store.Policy`, and `Static(p)` fixes one for tests. The tenant-aware surfaces take a keyed variant that resolves to the same `Store.Policy`: `api.PolicySource` (`func(*settings.Store) *policy.Policy`) for ingest, structured query and pipes, and `stream.PolicySource` (`func(tenant.ID) *policy.Policy`) for the hub. A `nil` result is a deliberate lockout. ### `pipes/` — Named Query Pipes @@ -182,11 +182,10 @@ The hot-reloadable half of configuration: a directory of four JSON files (`confi - **validate.go** — `Validate(dir)` reads, decodes, and checks the directory in one pass (strict JSON — unknown fields and duplicate keys are errors; per-file shape rules; cross-file role references) and returns every `Finding` at once. Shared by `wavehouse validate`, boot, and every reload. - **finding.go** — `Finding` / `Severity`: errors make the directory invalid, warnings don't block adoption. The JSON shape is part of the ops API (`POST /v1/ops/settings/reload` returns them). - **store.go** — `Store` owns the adopted snapshot. `Open` validates and adopts at boot; `Reload` re-validates and swaps the document atomically when there are no errors (a rejected reload keeps the previous snapshot). Consumers read typed accessors per call (`ClickHouse()`, `Auth()`, `DedupeFor(table)`, `DLQFor(table)`, `Keepalive()`, …) rather than holding values, and `AfterAdopt` registers hooks (dedupe store open/close, keepalive-wheel rebuild) that run after each successful reload. +- **registry.go** — `Registry` maps a tenant id to its `Store` (`For(id)`). It holds the one store `Open` adopted, under `tenant.Default`; reload and the watcher stay on the `Store`. - **watch.go** — fsnotify on the *directory* (not the files, so atomic-writer replaces and Kubernetes ConfigMap symlink swaps aren't lost), debounced into one reload; reloads once as soon as the watch exists so an edit between the boot read and the watch is never missed. `SIGHUP` and the reload endpoint funnel through the same serialized `Reload`. - **seed.go** / **seed/** — The `go:embed`ded starter directory with every key at its default. The binary carries no compiled defaults: `wavehouse bootstrap [dir]` writes this seed, and the compose stack and e2e fixture ship copies of it. -- **registry.go** — `Registry` maps a tenant id to its `Store` (`For(id)`). It holds the one store `Open` adopted, under `tenant.Default`; reload and the watcher stay on the `Store`. - ### `tenant/` — Tenant Identifier - **tenant.go** — `ID`, a validated string (never a number: a 19-digit id already rounds as a float64), and `Parse`, the one grammar that makes an id safe both as a folder name and as a message-queue subject token: ASCII letters, digits, `_`, `-`, at most `MaxLen` (64) bytes. `Default` (`"0"`) is the tenant a request without the header resolves to; `Header` is `X-Tenant-ID`. The package imports nothing from the rest of the repository, so any package can name a tenant. HTTP handlers receive the tenant as its resolved `*settings.Store`; the asynchronous paths — ingest worker, sweeper, stream hub, schema registry — are constructed with a `tenant.ID` and their settings getters take it as a parameter, which `internal/app` resolves through the registry. @@ -205,6 +204,8 @@ The hot-reloadable half of configuration: a directory of four JSON files (`confi ```text wrap=false Client POST /v1/ingest?table={table} + → Tenant resolution: X-Tenant-ID → settings.Registry → the request's *settings.Store + (absent = tenant 0; 400 malformed / 404 unknown, before auth) → JWT auth middleware (always runs; token optional) → Look up table schema from SchemaRegistry → Policy check: role allowed to insert into this table (before the body is parsed) @@ -297,6 +298,8 @@ The proxy-pattern wins are: zero classification logic on the WaveHouse side (no ```text Client GET /v1/stream + → Tenant resolution: X-Tenant-ID → settings.Registry → the request's *settings.Store + (absent = tenant 0; 400 malformed / 404 unknown, before auth) → JWT auth middleware (always runs; token optional) → Announce the caller's projected column list as an `event: schema` frame (no `id:`, so it never moves Last-Event-ID) BEFORE registering, so a client diff --git a/docs/src/content/docs/sdk/index.mdx b/docs/src/content/docs/sdk/index.mdx index 4779f204..4a234ff2 100644 --- a/docs/src/content/docs/sdk/index.mdx +++ b/docs/src/content/docs/sdk/index.mdx @@ -335,7 +335,7 @@ The token is never placed in the URL. The server still accepts a `?token=` query :::caution[Streaming asks more of a custom `fetch`] `.stream()` and `.liveQuery()` read the response as it arrives, so an `options.fetch` used with them must return a response with a live streaming body. A response handed back with an absent or already-consumed body fails fast with `SSE_NO_STREAM_BODY`. What the SDK cannot rescue is a wrapper that *awaits* the body before returning — `await res.text()`, or the `res.clone().text()` a logging wrapper reaches for — because on a stream that never ends it never resolves and your function never returns. Both satisfy every REST call, which is what makes the trap easy to walk into. -`options.headers` is also subject to CORS in a browser: a header outside the safelist joins the preflight, which the origin has to allow. WaveHouse advertises a fixed set, so custom headers reach it cross-origin only when a proxy in front terminates the preflight — the deployment they exist for. Server-side callers never preflight. +`options.headers` is also subject to CORS in a browser: a header outside the safelist joins the preflight, which the origin has to allow. WaveHouse advertises a fixed set (see [below](#custom-headers)), so a header outside it reaches WaveHouse cross-origin only when a proxy in front terminates the preflight — the deployment such headers exist for. Server-side callers never preflight. ::: #### Serving under a path prefix diff --git a/docs/src/content/docs/settings-directory.mdx b/docs/src/content/docs/settings-directory.mdx index 6103bdc1..4ca04d40 100644 --- a/docs/src/content/docs/settings-directory.mdx +++ b/docs/src/content/docs/settings-directory.mdx @@ -189,7 +189,7 @@ The `WAVEHOUSE_DLQ` stream always exists (an empty stream costs nothing) and the ## Message Queue -- `mq.max_bytes_gb` (seed default `50`) — disk budget for the embedded JetStream `WAVEHOUSE` stream that buffers ingested events until the worker writes them to ClickHouse; the `WAVEHOUSE_DLQ` stream gets a tenth of it. The stream runs `DiscardNew`, so when it's full new publishes are rejected and `POST /v1/ingest` returns `503` — [backpressure by construction](/ingest-pipeline#backpressure-and-durability-knobs). A reload updates both streams' limits in place without touching what's buffered: growing takes effect immediately; shrinking below what's currently on disk makes the stream refuse new publishes until the worker drains it back under the limit — nothing already accepted is dropped. If NATS rejects the update, the reload does not take effect and the failure is logged. Size it from [Durability & Storage](/durability). +- `mq.max_bytes_gb` (seed default `50`) — disk budget for the embedded JetStream `WAVEHOUSE` stream that buffers ingested events until the worker writes them to ClickHouse; the `WAVEHOUSE_DLQ` stream gets a tenth of it. The stream runs `DiscardNew`, so when it's full new publishes are rejected and `POST /v1/ingest` returns `503` — [backpressure by construction](/ingest-pipeline#backpressure-and-durability-knobs). A reload updates both streams' limits in place without touching what's buffered: growing takes effect immediately; shrinking below what's currently on disk makes the stream refuse new publishes until the worker drains it back under the limit — nothing already accepted is dropped. If NATS rejects the update, the rest of the reload is still adopted; the streams stay on the previously applied budget, the failure is logged, and the next reload retries it. Size it from [Durability & Storage](/durability). ## Streaming diff --git a/internal/api/tenant.go b/internal/api/tenant.go index cbc017da..32a81c91 100644 --- a/internal/api/tenant.go +++ b/internal/api/tenant.go @@ -4,6 +4,7 @@ import ( "context" "log/slog" "net/http" + "net/url" "github.com/Wave-RF/WaveHouse/internal/policy" "github.com/Wave-RF/WaveHouse/internal/settings" @@ -94,7 +95,15 @@ func TenantMW(tenants *settings.Registry) func(http.Handler) http.Handler { const opsTenantParam = "tenant" // opsStore resolves the tenant an ops route addresses from its -// opsTenantParam, with the same answers as TenantMW. +// opsTenantParam, with the same answers as TenantMW. The query string is +// parsed strictly: url.Values silently drops a malformed pair, which would +// turn "?tenant=acme;x=1" into the default tenant rather than a 400. func opsStore(w http.ResponseWriter, r *http.Request, tenants *settings.Registry) (*settings.Store, bool) { - return resolveTenant(w, tenants, "?"+opsTenantParam, r.URL.Query()[opsTenantParam]) + where := "?" + opsTenantParam + values, err := url.ParseQuery(r.URL.RawQuery) + if err != nil { + writeJSONError(w, http.StatusBadRequest, "invalid "+where+": malformed query string") + return nil, false + } + return resolveTenant(w, tenants, where, values[opsTenantParam]) } diff --git a/internal/api/tenant_test.go b/internal/api/tenant_test.go index e6f7efc9..4035e02d 100644 --- a/internal/api/tenant_test.go +++ b/internal/api/tenant_test.go @@ -55,8 +55,8 @@ func TestTenantMW(t *testing.T) { return } assert.Nil(t, resolved, "a refused request must not reach the handler") + testutil.AssertJSONErrorResponse(t, w) assert.Contains(t, w.Body.String(), tt.wantBody) - assert.Equal(t, "application/json", w.Header().Get("Content-Type")) }) } } @@ -179,6 +179,8 @@ func TestPipesHandler_AdminReads_TenantParam(t *testing.T) { {name: "unknown tenant", query: "?tenant=acme", wantStatus: http.StatusNotFound, wantBody: "unknown tenant: acme"}, {name: "malformed tenant", query: "?tenant=a.b", wantStatus: http.StatusBadRequest, wantBody: "invalid ?tenant"}, {name: "repeated parameter", query: "?tenant=0&tenant=0", wantStatus: http.StatusBadRequest, wantBody: "sent more than once"}, + // url.Values would drop the malformed pair and default the tenant. + {name: "malformed query string", query: "?tenant=acme;x=1", wantStatus: http.StatusBadRequest, wantBody: "malformed query string"}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -189,11 +191,15 @@ func TestPipesHandler_AdminReads_TenantParam(t *testing.T) { w := httptest.NewRecorder() h.List(w, httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/v1/ops/pipes"+tt.query, nil)) require.Equal(t, tt.wantStatus, w.Code, "List body: %s", w.Body.String()) + if tt.wantBody != "" { + testutil.AssertJSONErrorResponse(t, w) + } w = httptest.NewRecorder() h.Get(w, pipesRequest(t, http.MethodGet, "/v1/ops/pipes/top_pages"+tt.query, "top_pages", nil)) require.Equal(t, tt.wantStatus, w.Code, "Get body: %s", w.Body.String()) if tt.wantBody != "" { + testutil.AssertJSONErrorResponse(t, w) assert.Contains(t, w.Body.String(), tt.wantBody) } else { assert.Contains(t, w.Body.String(), `"top_pages"`) diff --git a/internal/stream/hub.go b/internal/stream/hub.go index 997b6648..66c47aa1 100644 --- a/internal/stream/hub.go +++ b/internal/stream/hub.go @@ -451,7 +451,6 @@ func (h *Hub) snapshotPolicy() (p *policy.Policy, filter bool) { // The closure is for a single goroutine — each connection makes its own. The live // path uses Broadcast. func (h *Hub) ReplayProjector(role string, sub *Subscriber) func(raw []byte) []Frame { - p, filter := h.snapshotPolicy() var colSpecs map[string]policy.ColumnSpec specsFor := "" // table name colSpecs was resolved for ("" ⇒ not yet resolved) // Schema-drift state is LOCAL to this gap-fill, not the connection's shared @@ -476,6 +475,9 @@ func (h *Hub) ReplayProjector(role string, sub *Subscriber) func(raw []byte) []F // availability; a reconnect resynchronizes. lastSig := "" return func(raw []byte) []Frame { + // Read per event, like Broadcast, so a policy adopted mid-gap-fill + // applies to the next replayed row rather than after the fill ends. + p, filter := h.snapshotPolicy() ev := newEventView(raw) plan, ok := planForRole(p, filter, role, ev, KindReplay) if !ok { diff --git a/internal/stream/hub_test.go b/internal/stream/hub_test.go index ef16df7c..8fa977f7 100644 --- a/internal/stream/hub_test.go +++ b/internal/stream/hub_test.go @@ -10,6 +10,7 @@ import ( "slices" "strings" "sync" + "sync/atomic" "testing" "time" @@ -862,6 +863,27 @@ func TestHub_ReplayProjector(t *testing.T) { } } +// A policy adopted while a gap-fill is in flight applies to the next replayed +// row, as it does on the live path: the projector reads the policy per event +// rather than once at construction. +func TestHub_ReplayProjector_ReadsPolicyPerEvent(t *testing.T) { + t.Parallel() + granted := &policy.Policy{ + Tables: map[string]policy.TablePolicy{ + "clicks": {"viewer": {Select: &policy.SelectPermissions{AllowColumns: []string{"page"}}}}, + }, + } + var current atomic.Pointer[policy.Policy] + current.Store(granted) + hub := NewHub(tenant.Default, func(tenant.ID) *policy.Policy { return current.Load() }, nil, nil) + raw := rawEvent(t, "clicks", "2026-06-26T00:00:00Z", map[string]any{"page": "/home"}) + + project := hub.ReplayProjector("viewer", NewSubscriber(nil, nil)) + require.Len(t, project(raw), 2, "granted before the reload") + current.Store(&policy.Policy{}) // the reload revokes the grant mid-fill + assert.Empty(t, project(raw), "the next replayed row sees the revocation") +} + // TestHub_ReplayProjector_RowFilter exercises the row-filter branch of replay: the // #319 fix applies row-level security on the per-connection replay path too, so a // gap-fill event is projected only when the connection's claims satisfy the filter. From b5183dd77836b7eba5e506b9ff61e6034cf9ac97 Mon Sep 17 00:00:00 2001 From: taitelee Date: Fri, 18 Sep 2026 10:41:01 -0400 Subject: [PATCH 05/11] docs(stream): gap-fill re-reads the policy per replayed row --- docs/src/content/docs/access-control.mdx | 2 +- docs/src/content/docs/api.md | 2 +- docs/src/content/docs/architecture.md | 2 +- docs/src/content/docs/sdk/reference.md | 4 ++-- docs/src/content/docs/sdk/streaming.md | 4 ++-- internal/stream/hub.go | 10 +++++----- 6 files changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/src/content/docs/access-control.mdx b/docs/src/content/docs/access-control.mdx index 1962a11f..6befb392 100644 --- a/docs/src/content/docs/access-control.mdx +++ b/docs/src/content/docs/access-control.mdx @@ -360,7 +360,7 @@ A few more edges worth knowing when you write a policy — the stream evaluates - **A non-scalar event value** (array/object/null) under a filtered column withholds the row. - **Insert-time numeric narrowing is simulated, not skipped.** The insert narrows a payload carrying more precision than the column's declared type — a `Decimal` **truncates** at its scale (`1.005`, `1.006` and `1.009` all store as `1.00` in a `Decimal(10, 2)`), a `Float32` **rounds** to its nearest representable value (`16777217` stores as `16777216`) — and ClickHouse applies the same narrowing to a bound filter constant at compare time. The stream narrows **both operands** identically before comparing, so its verdict matches the query path's on narrowing columns under every operator: a `_gt: "1.004"` filter on a `Decimal(10, 2)` column withholds a `1.005` payload exactly as the query path hides the stored `1.00`. (An earlier revision of this feature compared the raw payload and could deliver such an event; that fail-open is closed, and an integration test holds every in-range numeric stream verdict equal to a live ClickHouse's — for the out-of-range operands the range gate refuses, it asserts the half that matters: the stream never admits a row ClickHouse hides.) What remains payload-vs-stored: an event whose insert later **fails outright** (an out-of-range value, a batch error, the DLQ) was already streamed to whichever subscribers the filter admitted, and its row never becomes queryable. -One more boundary is temporal: a subscriber's claims (and role) are captured when the SSE connection is established and are never re-read, while the policy itself is re-read on every live event. A gap-fill replay is the one exception: it runs under the single policy snapshot taken when the connection opened, so a policy change landing mid-replay applies from the first live event after it. Tightening a policy therefore applies from the next live event, but a token that expires — or claims revoked at the identity provider — keeps its open stream until the client disconnects, so treat connection lifetime as the revocation window for stream row-scoping. +One more boundary is temporal: a subscriber's claims (and role) are captured when the SSE connection is established and are never re-read, while the policy itself is re-read on every live event. The policy itself is re-read on every event, live and replayed alike, so a policy adopted mid-gap-fill applies to the next replayed row. Tightening a policy therefore applies from the next event either way, but a token that expires — or claims revoked at the identity provider — keeps its open stream until the client disconnects, so treat connection lifetime as the revocation window for stream row-scoping. Each row withheld **from a subscriber** by row-level security is counted in `wavehouse_sse_rows_withheld_total` (labeled by table and role; a row withheld from three subscribers counts three times) — check it before concluding a quiet stream simply has no matching rows. diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index bcfd7a9e..a729bf4f 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -639,7 +639,7 @@ Each SSE connection is bound to a single `?table=`; to consume multiple tables, Values of top-level `DateTime`/`DateTime64` columns inside `row` arrive in the canonical RFC 3339 UTC form (ingest rewrites them before publishing — see [timestamp canonicalization](#timestamp-canonicalization)), so a live event and a `/v1/query` read of the same row agree on the instant in zone-explicit form — a zone-less spelling no longer parses as local time in a browser ([#372](https://github.com/Wave-RF/WaveHouse/issues/372)). The two renderings are byte-identical regardless of the declared time zone or a `Nullable` wrapper — a column declared with a non-UTC zone also streams as `Z`, and `/v1/query` normalizes it (nullable or not) to UTC before rendering. Canonicalization is fail-open at ingest, so a value outside the accepted input forms streams in whatever spelling the producer sent — and for exactly those events the byte-identity above does not hold: a spelling ClickHouse accepts anyway is stored and still queries back canonical, while one it too rejects lands in the DLQ and never becomes queryable at all. -**Note:** When access control policies are active, streamed events are filtered per the caller's role: tables without `select` permission are skipped, denied columns are removed from each event, and the role's [row-level `filter`](/access-control#row-level-security) is evaluated per subscriber against the caller's JWT claims — supplied by the connection's token (the `Authorization` header, or the `?token=` fallback above), with replayed gap-fill events filtered the same way. For a filter constant the query path's SQL also accepts ([the enforcement caution](/access-control#where-each-rule-is-enforced) gives per-type guidance), a connection is never delivered a row the query path would hide for that role — every comparison the stream can't prove fails closed and withholds the row instead. Numeric comparisons run in the column's storage domain — both operands narrowed the way ClickHouse narrows the stored value and the bound constant — so columns that narrow on insert (`Float32`/`Float64` width, a `Decimal`'s scale) agree with the query path too; the residual payload-vs-stored case is an event whose insert later fails into the DLQ, which the caution documents. The connection's claims are captured once, when the stream is established — a policy change applies from the next live event (an in-flight gap-fill finishes under the policy snapshot taken when the stream opened), but an expired token or changed claims take effect only when the client reconnects. +**Note:** When access control policies are active, streamed events are filtered per the caller's role: tables without `select` permission are skipped, denied columns are removed from each event, and the role's [row-level `filter`](/access-control#row-level-security) is evaluated per subscriber against the caller's JWT claims — supplied by the connection's token (the `Authorization` header, or the `?token=` fallback above), with replayed gap-fill events filtered the same way. For a filter constant the query path's SQL also accepts ([the enforcement caution](/access-control#where-each-rule-is-enforced) gives per-type guidance), a connection is never delivered a row the query path would hide for that role — every comparison the stream can't prove fails closed and withholds the row instead. Numeric comparisons run in the column's storage domain — both operands narrowed the way ClickHouse narrows the stored value and the bound constant — so columns that narrow on insert (`Float32`/`Float64` width, a `Decimal`'s scale) agree with the query path too; the residual payload-vs-stored case is an event whose insert later fails into the DLQ, which the caution documents. The connection's claims are captured once, when the stream is established — a policy change applies from the next event, replayed or live (a gap-fill re-reads the policy per event too), but an expired token or changed claims take effect only when the client reconnects. **CORS:** `/v1/stream` honors the `cors.allowed_origins` allowlist (settings directory) like every endpoint. Note that a **header-authenticated stream preflights before it connects** — `Authorization` is not CORS-safelisted — where a bare `EventSource` never preflighted at all: its request is not a `fetch()`, so Fetch's unsafe-request flag is never set and `Last-Event-ID` rides on the plain `GET`. Both headers are allow-listed, so an allowed origin connects *and* resumes cross-origin. diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 7164e0f1..a48d376c 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -96,7 +96,7 @@ The API layer uses [Chi](https://github.com/go-chi/chi) for routing with Request The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https://github.com/Wave-RF/WaveHouse/issues/294)) lives next to the keepalive primitives it shares. One abstraction per file. -- **hub.go** — `Hub`, the event fan-out. Subscribers register under `(topic, role)`; `Broadcast` decodes each event once, applies each subscribed role's column policy once, builds one SSE frame per role, and fans it to every member of that role's `Bucket` — prepending a per-connection `event: schema` frame wherever that connection's announced column list has drifted, and withholding the row if the announcement cannot be queued — collapsing the prior per-subscriber `unmarshal → evaluate → filter → marshal` into one pass per distinct `(role, table)` output shape (the [#294](https://github.com/Wave-RF/WaveHouse/issues/294) lever; the measured ceiling was ~2 270 deliveries/s from re-projecting per subscriber). That schema-before-row guarantee is the LIVE path's: `ReplayProjector` tracks drift in its own state and the two are not reconciled ([#543](https://github.com/Wave-RF/WaveHouse/issues/543)). The column projection is claims-independent, so it is shared across a role's whole bucket; the role's row-level `filter` predicate is not — it is resolved against each subscriber's JWT claims, so for a role that carries a filter `Broadcast` keeps the shared column projection but delivers it only to the subscribers whose claims admit each row (`ResolvedPermissions.RowVisible`, evaluated against the full event via the type-aware comparison seeded from the schema registry — `policy.ColumnSpec`: numeric columns compare numerically, `String` bytewise, `DateTime`/`DateTime64` as instants through the same parser ingest canonicalization uses (`discovery.Column.TimeParser` — one grammar, so filter constants and canonicalized payloads can't disagree on the instant), everything else admits byte-equality only and fails ordering/`!=` closed, so a missing schema can never downgrade the comparison to a leak). Each row withheld this way increments `wavehouse_sse_rows_withheld_total`. This is the [#319](https://github.com/Wave-RF/WaveHouse/issues/319) fix that closes the query/stream row-level-security drift; roles without a filter keep the pure once-per-role fast path. `ReplayProjector` shares the same projection and per-connection row check for the handler's gap-fill, holding one policy snapshot per gap-fill and caching the per-table column-kind lookup across the replay loop. +- **hub.go** — `Hub`, the event fan-out. Subscribers register under `(topic, role)`; `Broadcast` decodes each event once, applies each subscribed role's column policy once, builds one SSE frame per role, and fans it to every member of that role's `Bucket` — prepending a per-connection `event: schema` frame wherever that connection's announced column list has drifted, and withholding the row if the announcement cannot be queued — collapsing the prior per-subscriber `unmarshal → evaluate → filter → marshal` into one pass per distinct `(role, table)` output shape (the [#294](https://github.com/Wave-RF/WaveHouse/issues/294) lever; the measured ceiling was ~2 270 deliveries/s from re-projecting per subscriber). That schema-before-row guarantee is the LIVE path's: `ReplayProjector` tracks drift in its own state and the two are not reconciled ([#543](https://github.com/Wave-RF/WaveHouse/issues/543)). The column projection is claims-independent, so it is shared across a role's whole bucket; the role's row-level `filter` predicate is not — it is resolved against each subscriber's JWT claims, so for a role that carries a filter `Broadcast` keeps the shared column projection but delivers it only to the subscribers whose claims admit each row (`ResolvedPermissions.RowVisible`, evaluated against the full event via the type-aware comparison seeded from the schema registry — `policy.ColumnSpec`: numeric columns compare numerically, `String` bytewise, `DateTime`/`DateTime64` as instants through the same parser ingest canonicalization uses (`discovery.Column.TimeParser` — one grammar, so filter constants and canonicalized payloads can't disagree on the instant), everything else admits byte-equality only and fails ordering/`!=` closed, so a missing schema can never downgrade the comparison to a leak). Each row withheld this way increments `wavehouse_sse_rows_withheld_total`. This is the [#319](https://github.com/Wave-RF/WaveHouse/issues/319) fix that closes the query/stream row-level-security drift; roles without a filter keep the pure once-per-role fast path. `ReplayProjector` shares the same projection and per-connection row check for the handler's gap-fill, reading the policy per replayed event as `Broadcast` does, and caching the per-table column-kind lookup across the replay loop. - **subscriber.go** — `Subscriber`, the per-connection handle. It carries the connection's JWT claims, fixed at construction (`NewSubscriber(claims, metrics)`, no setter) — the claims the `Hub` resolves a role's row-level `filter` against, and immutability is what makes the fan-out's unsynchronized claims read race-free structurally. It owns a single ready-to-write outbound queue of `Frame`s (each tagged with its `kind`, so the handler labels the write where it happens): producers — the keepalive wheel and the event `Hub` — fan frames in with `Send` (non-blocking; a full queue drops, and `Send` itself counts the drop by frame kind, so no producer can forget to), and the handler drains `Frames()` to the client verbatim. The queue is sized for buffering live events (cap 64, up from the keepalive-only cap 1; #152 will make it a knob), and an `Evicted()` channel is the seam the slow-consumer follow-up closes to disconnect a wedged consumer. - **bucket.go** — `Bucket`, the reusable fan-out primitive: a concurrency-safe set of subscribers. `Push` fans one `Frame` to every member fire-and-forget — the keepalive wheel's ring is its only caller now that both `Hub` paths iterate `Snapshot`, since the schema announcement is per connection even where the projection is shared per role; `Snapshot` exposes the members so the event `Hub` can evaluate row visibility per subscriber before sending (drop counting lives in `Send` itself). The `Hub` holds one `Bucket` per `(topic, role)` so a projected frame is built once and sent to every member instead of re-projected per subscriber. - **heartbeat.go** — The keepalive wheel (`Heartbeater`). A single process-wide ticker fans a minimal `:` comment across the ring of `Bucket`s, waking ~1/N of live streams per tick so the writes don't synchronize. The effective per-connection keepalive period is `stream.keepalive_interval` in the settings directory (the wheel ticks every `keepalive_interval ÷ keepalive_buckets`, so one rotation spans the interval; a reload calls `Reconfigure`, which rebuilds the ring in place with every live subscriber carried over); the owning handler goroutine does the actual write, so the shared ticker never touches a `ResponseWriter` directly. diff --git a/docs/src/content/docs/sdk/reference.md b/docs/src/content/docs/sdk/reference.md index d5f3f3a2..24bc2447 100644 --- a/docs/src/content/docs/sdk/reference.md +++ b/docs/src/content/docs/sdk/reference.md @@ -30,7 +30,7 @@ The SDK **never throws** for anything the server returns — all API errors come | 400 | `HTTP_400` | No | Bad request (validation, missing fields) | | 401 | `HTTP_401` | No | On REST, a present-but-invalid or expired JWT that a gate then denied. **WaveHouse itself** never returns `401` for a *missing* token — that resolves to `default_role`, and a denial is `403`. On a stream it is always from something in front, since `/v1/stream` is ungated | | 403 | `HTTP_403` | No | Insufficient permissions | -| 404 | `HTTP_404` | No | Table or pipe not found | +| 404 | `HTTP_404` | No | Table, pipe, or tenant not found | | 500 | `HTTP_500` | Yes | Server error (retried per `maxRetries`) | | 503 | `HTTP_503` | Yes | Service unavailable (auto-retries with `Retry-After`) | | 0 | `NETWORK_ERROR` | Yes | Network failure (retried with exponential backoff) | @@ -56,7 +56,7 @@ On REST, `ABORTED` is the one error raised *by* a backoff rather than by an atte On a stream, a retryable failure is re-dialed on a jittered exponential backoff (capped at 30s, and reset only once a connection has held for a few seconds — so a server that accepts and instantly closes still backs off), with the `status` callback moving `reconnecting` → `live`. -Rejected requests surface the real status and message rather than an opaque connection failure — in a browser going cross-origin, though, only when the rejection passes CORS and the gateway answered whatever preflight the request triggers — `Authorization`, configured `headers`, or `Last-Event-ID` once the stream resumes; a rejected preflight or a response without `Access-Control-Allow-Origin` reaches you as a retryable network error instead — indistinguishable from a drop, and retried. Any `4xx` ends the stream, since repeating the request won't usually talk whatever rejected it round — the exception being a `429` or `408` from a fronting rate limiter, which is transient even though the stream still ends, so catch it and open a new one after a delay ([#469](https://github.com/Wave-RF/WaveHouse/issues/469)). Note that **WaveHouse never rejects a stream for authentication**: `/v1/stream` is ungated, so an expired or missing token resolves to `default_role` and you get a `200` with a filtered view, not a `401`. The one 4xx it raises itself is `400` for a missing or empty `table`, and only on the stream route: a `404` or `405` means the request never reached that route, most often a `baseURL` path prefix your proxy didn't strip. Any other 4xx comes from something in front — an auth gateway, a proxy. That silent-downgrade behavior is exactly why `auth` is re-read on every connection attempt, and [#239](https://github.com/Wave-RF/WaveHouse/issues/239) tracks enforcing expiry server-side. `SSE_CONNECT_ERROR` and `SSE_NO_STREAM_BODY` are configuration faults, so fix the cause and start a new stream. +Rejected requests surface the real status and message rather than an opaque connection failure — in a browser going cross-origin, though, only when the rejection passes CORS and the gateway answered whatever preflight the request triggers — `Authorization`, configured `headers`, or `Last-Event-ID` once the stream resumes; a rejected preflight or a response without `Access-Control-Allow-Origin` reaches you as a retryable network error instead — indistinguishable from a drop, and retried. Any `4xx` ends the stream, since repeating the request won't usually talk whatever rejected it round — the exception being a `429` or `408` from a fronting rate limiter, which is transient even though the stream still ends, so catch it and open a new one after a delay ([#469](https://github.com/Wave-RF/WaveHouse/issues/469)). Note that **WaveHouse never rejects a stream for authentication**: `/v1/stream` is ungated, so an expired or missing token resolves to `default_role` and you get a `200` with a filtered view, not a `401`. The 4xx it raises itself are `400` for a missing or empty `table` and, when you send `X-Tenant-ID`, the `400`/`404` of [tenant resolution](/api#tenant-selection); any other `404` or `405` means the request never reached that route, most often a `baseURL` path prefix your proxy didn't strip. Any other 4xx comes from something in front — an auth gateway, a proxy. That silent-downgrade behavior is exactly why `auth` is re-read on every connection attempt, and [#239](https://github.com/Wave-RF/WaveHouse/issues/239) tracks enforcing expiry server-side. `SSE_CONNECT_ERROR` and `SSE_NO_STREAM_BODY` are configuration faults, so fix the cause and start a new stream. `SSE_PARSE_ERROR` is the one code that isn't a connection outcome: it's reported and *skipped*, and the connection keeps reading — one bad frame shouldn't cost you the stream. For an ordinary bad frame its `retryable: true` is therefore vestigial — nothing is re-dialed. Two exceptions, one to each half of that reported-and-skipped rule. A frame the SDK can't turn into a row is skipped but never *reported* — `console.warn` and dropped, with no `error` callback: `data` that isn't valid JSON, a row arriving before any `event: schema` frame, a `row` that is valid JSON but not an array, or a row whose length disagrees with the announced column list. Every one of those is **bounded** — three per cause per connection, then one "further occurrences suppressed" line, with the malformed-schema frame below counted as its own cause. So against a server that never announces a schema you get a handful of lines rather than one per row, and a quiet console is **not** evidence the stream is healthy. A malformed schema frame is the one to watch, because it discards the list rather than keeping a stale one — so every row after it is dropped until the next announcement or a reconnect. The parser's 16 MiB buffer cap is reported but not *skipped*: an overflow terminates the parser, so the transport stops reading and reconnects rather than feeding it again. diff --git a/docs/src/content/docs/sdk/streaming.md b/docs/src/content/docs/sdk/streaming.md index eadfcb6d..89e64eb8 100644 --- a/docs/src/content/docs/sdk/streaming.md +++ b/docs/src/content/docs/sdk/streaming.md @@ -127,13 +127,13 @@ The SSE reader and writer changed in the same release. A **new SDK against an ol `@wavehouse/sdk` publishes to npm independently of the server, so pinning the SDK in a frontend while the backend upgrades on its own schedule (or the reverse) is the normal deployment shape. ::: -A `4xx` is terminal and surfaces through `error` with the real status code rather than an opaque connection failure — in a browser going cross-origin, only when the rejection passes CORS and the gateway answered whatever preflight the request triggers (`Authorization`, configured `headers`, or `Last-Event-ID` once the stream resumes); otherwise it arrives as a retryable network error instead — indistinguishable from a drop, and retried. It won't be an *authentication* rejection from WaveHouse, which leaves `/v1/stream` ungated and answers an expired token with a filtered view rather than a `401`; the only 4xx it raises itself is `400` for a missing or empty table name, and only on the stream route — a `404` or `405` means the request never reached it, usually a `baseURL` path prefix the proxy didn't strip. Anything else means something in front of it (an auth gateway, a proxy) turned the request away — and note the exception to "retrying wouldn't help": a `429` or `408` from a rate limiter *is* transient, but the stream still ends, so catch it and open a new one after a delay ([#469](https://github.com/Wave-RF/WaveHouse/issues/469)). See [Error Handling](/sdk/reference#error-handling) for every code a stream can report and which ones re-dial. +A `4xx` is terminal and surfaces through `error` with the real status code rather than an opaque connection failure — in a browser going cross-origin, only when the rejection passes CORS and the gateway answered whatever preflight the request triggers (`Authorization`, configured `headers`, or `Last-Event-ID` once the stream resumes); otherwise it arrives as a retryable network error instead — indistinguishable from a drop, and retried. It won't be an *authentication* rejection from WaveHouse, which leaves `/v1/stream` ungated and answers an expired token with a filtered view rather than a `401`; the 4xx it raises itself are `400` for a missing or empty table name and, when you send `X-Tenant-ID`, the `400`/`404` of [tenant resolution](/api#tenant-selection) — any other `404` or `405` means the request never reached the route, usually a `baseURL` path prefix the proxy didn't strip. Anything else means something in front of it (an auth gateway, a proxy) turned the request away — and note the exception to "retrying wouldn't help": a `429` or `408` from a rate limiter *is* transient, but the stream still ends, so catch it and open a new one after a delay ([#469](https://github.com/Wave-RF/WaveHouse/issues/469)). See [Error Handling](/sdk/reference#error-handling) for every code a stream can report and which ones re-dial. Streams go through `options.fetch`, `options.headers`, and `options.fetchOptions` like every other request — which is what lets a stream reach a header-gated origin. A custom `fetch` is asked more of on this path; see [Supplying your own fetch](/sdk#supplying-your-own-fetch). ### Server-Side Policy Filtering -Access-control policy applies on the server before anything reaches the client: tables the connection's role can't `select` are skipped, denied columns are stripped from each event, and the role's row-level `filter` is evaluated per subscriber against the connection's JWT claims. A stream on a row-policied table therefore delivers only the rows the policy admits for that connection — and, where the server's in-memory comparison can't prove a match, fewer; see [Access control — where each rule is enforced](/access-control#where-each-rule-is-enforced). Claims are captured when the connection opens: a policy change applies from the next live event (an in-flight gap-fill replay finishes under the policy snapshot taken at connect), while token expiry or claim changes take effect on reconnect. +Access-control policy applies on the server before anything reaches the client: tables the connection's role can't `select` are skipped, denied columns are stripped from each event, and the role's row-level `filter` is evaluated per subscriber against the connection's JWT claims. A stream on a row-policied table therefore delivers only the rows the policy admits for that connection — and, where the server's in-memory comparison can't prove a match, fewer; see [Access control — where each rule is enforced](/access-control#where-each-rule-is-enforced). Claims are captured when the connection opens: a policy change applies from the next event, replayed or live (a gap-fill re-reads the policy per event too), while token expiry or claim changes take effect on reconnect. ### Client-Side Stream Filtering diff --git a/internal/stream/hub.go b/internal/stream/hub.go index 66c47aa1..440ec038 100644 --- a/internal/stream/hub.go +++ b/internal/stream/hub.go @@ -443,11 +443,11 @@ func (h *Hub) snapshotPolicy() (p *policy.Policy, filter bool) { // replay shares the Hub's policy store and schema registry with the live fan-out — // the handler can't accidentally project replay against a different (or nil) // policy. Replay is already per-connection, so row-level security evaluates against -// this connection's claims directly; the returned closure holds one policy snapshot -// for the whole gap-fill (matching Broadcast's one-snapshot-per-event — a reload -// landing mid-replay applies from the first live event) and caches the per-table -// column-kind lookup across the replay loop, so a large Last-Event-ID gap-fill -// doesn't pay a store read-lock plus a registry lookup and map build per event. +// this connection's claims directly; the returned closure reads the policy per +// replayed event (matching Broadcast, so a reload landing mid-replay applies to +// the next replayed row) and caches only the per-table column-kind lookup across +// the replay loop, so a large Last-Event-ID gap-fill doesn't pay a registry +// lookup and map build per event. // The closure is for a single goroutine — each connection makes its own. The live // path uses Broadcast. func (h *Hub) ReplayProjector(role string, sub *Subscriber) func(raw []byte) []Frame { From f53507ff96eb4ed51195b4af01e32a73db363edb Mon Sep 17 00:00:00 2001 From: taitelee Date: Fri, 18 Sep 2026 10:52:18 -0400 Subject: [PATCH 06/11] fix(app): log a settings-registry miss on the async paths --- docs/src/content/docs/access-control.mdx | 2 +- docs/src/content/docs/development.md | 2 +- internal/app/wire.go | 11 ++++++++--- 3 files changed, 10 insertions(+), 5 deletions(-) diff --git a/docs/src/content/docs/access-control.mdx b/docs/src/content/docs/access-control.mdx index 6befb392..5dba919c 100644 --- a/docs/src/content/docs/access-control.mdx +++ b/docs/src/content/docs/access-control.mdx @@ -360,7 +360,7 @@ A few more edges worth knowing when you write a policy — the stream evaluates - **A non-scalar event value** (array/object/null) under a filtered column withholds the row. - **Insert-time numeric narrowing is simulated, not skipped.** The insert narrows a payload carrying more precision than the column's declared type — a `Decimal` **truncates** at its scale (`1.005`, `1.006` and `1.009` all store as `1.00` in a `Decimal(10, 2)`), a `Float32` **rounds** to its nearest representable value (`16777217` stores as `16777216`) — and ClickHouse applies the same narrowing to a bound filter constant at compare time. The stream narrows **both operands** identically before comparing, so its verdict matches the query path's on narrowing columns under every operator: a `_gt: "1.004"` filter on a `Decimal(10, 2)` column withholds a `1.005` payload exactly as the query path hides the stored `1.00`. (An earlier revision of this feature compared the raw payload and could deliver such an event; that fail-open is closed, and an integration test holds every in-range numeric stream verdict equal to a live ClickHouse's — for the out-of-range operands the range gate refuses, it asserts the half that matters: the stream never admits a row ClickHouse hides.) What remains payload-vs-stored: an event whose insert later **fails outright** (an out-of-range value, a batch error, the DLQ) was already streamed to whichever subscribers the filter admitted, and its row never becomes queryable. -One more boundary is temporal: a subscriber's claims (and role) are captured when the SSE connection is established and are never re-read, while the policy itself is re-read on every live event. The policy itself is re-read on every event, live and replayed alike, so a policy adopted mid-gap-fill applies to the next replayed row. Tightening a policy therefore applies from the next event either way, but a token that expires — or claims revoked at the identity provider — keeps its open stream until the client disconnects, so treat connection lifetime as the revocation window for stream row-scoping. +One more boundary is temporal: a subscriber's claims (and role) are captured when the SSE connection is established and are never re-read, while the policy itself is re-read on every event, live and replayed alike — so a policy adopted mid-gap-fill applies to the next replayed row. Tightening a policy therefore applies from the next event either way, but a token that expires — or claims revoked at the identity provider — keeps its open stream until the client disconnects, so treat connection lifetime as the revocation window for stream row-scoping. Each row withheld **from a subscriber** by row-level security is counted in `wavehouse_sse_rows_withheld_total` (labeled by table and role; a row withheld from three subscribers counts three times) — check it before concluding a quiet stream simply has no matching rows. diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index 758a6c93..3437fc39 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -495,7 +495,7 @@ WaveHouse/ - **Strict Go formatting**: Use `gofumpt` (a stricter superset of `gofmt`, enforced by CI). Run `make fmt` to format. - **Interface-first design**: Core behaviors (`Cache`, `Deduplicator`, `Publisher`, `Subscriber`) are defined as interfaces so implementations can be swapped behind a stable contract. - **Package boundaries**: The `internal/` directory ensures packages are private to this module. -- **Error handling**: Return errors to callers. Use `slog` for structured logging. +- **Error handling**: Return errors to callers. Use `slog` for structured logging, through the default logger (`slog.InfoContext(ctx, …)` and its siblings) — constructors don't take a `*slog.Logger`; tests silence or capture it with `internal/testutil/logtest`. - **Schema-driven**: ClickHouse is the schema source of truth. WaveHouse discovers and validates against real table schemas. ## Makefile Targets diff --git a/internal/app/wire.go b/internal/app/wire.go index 0371a6b7..98cff2ba 100644 --- a/internal/app/wire.go +++ b/internal/app/wire.go @@ -70,12 +70,13 @@ func (a *App) wireSettings() error { // perTenant adapts a store accessor to the tenant-keyed getter the async // paths take: they hold a tenant id (tenant.Default today, the MQ subject's // from #583 story 5), not a request's resolved store. Only tenant.Default -// exists, so a miss is a wiring bug and reads as T's zero value; what a -// removed tenant means to each async path is story 3's to decide. +// exists, so a miss is a wiring bug: logged, and read as T's zero value — +// what a removed tenant means to each async path is story 3's to decide. func perTenant[T any](tenants *settings.Registry, get func(*settings.Store) T) func(tenant.ID) T { return func(id tenant.ID) T { store, ok := tenants.For(id) if !ok { + slog.Error("no settings store for tenant; reading the zero value", "tenant", id) var zero T return zero } @@ -356,7 +357,11 @@ func (a *App) wireIngestWorker() { a.add(component{name: "ingest worker", run: func(ctx context.Context) error { dlqEnabled := func(id tenant.ID, table string) bool { store, ok := a.tenants.For(id) - return ok && store.DLQFor(table) + if !ok { + slog.Error("no settings store for tenant; treating its DLQ as off", "tenant", id, "table", table) + return false + } + return store.DLQFor(table) } stop, failed, err := ingest.StartIngestWorker(ctx, a.mq, a.cache, a.ch.Target, tenant.Default, dlqEnabled) if err != nil { From 9aa495eefbfb57d443136198f1bd628d1750b9bc Mon Sep 17 00:00:00 2001 From: taitelee Date: Fri, 18 Sep 2026 12:07:40 -0400 Subject: [PATCH 07/11] docs(settings): state the split-limit case of a failed stream resize --- AGENTS.md | 2 +- docs/src/content/docs/architecture.md | 2 +- docs/src/content/docs/settings-directory.mdx | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index b27a54e3..2a3c307a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -43,7 +43,7 @@ Eighteen internal packages under `internal/` (plus `internal/testutil/` for shar - **`pipes/`** — Named query pipes: `NamedQuery` type + `BindParams` + `Source` (read per request; `settings.Store` in production, `Static(q...)` in tests) - **`policy/`** — Hasura-style access control, **role-first**: `TablePolicy` is `map[string]RolePermissions`, and a role's grant splits by operation into `SelectPermissions` (columns, row `filter`, aggregations, the `max_*` limits) and `InsertPermissions` (columns, `check`) — so a field only one side honors does not exist on the other. `Evaluate()` resolves ONE operation and leaves the other side **nil** (`Select *ResolvedSelect` / `Insert *ResolvedInsert`), which every accessor fails closed on — nil is "not resolved", distinct from an empty side, which is "unrestricted" (what the admin return builds). Claim templating (`{{ jwt.claim.path }}`) resolves during that call. Policies come from `Source`, a `func() *Policy` read per call (`settings.Store.Policy` in production, `Static(p)` in tests) - **`query/`** — Structured query AST types + SQL builder with schema validation, structural policy predicate/limit emission, timestamp bucketing -- **`settings/`** — the settings directory: `Validate` (strict JSON, per-file rules, cross-file role references), `Store` (the adopted snapshot + serialized `Reload`, typed accessors read per call, `AfterAdopt` hooks), `Registry` (tenant id → `Store`; holds the one store under `tenant.Default`), the fsnotify `Watch`, and the `go:embed`ded seed `wavehouse bootstrap` writes +- **`settings/`** — the settings directory: `Validate` (strict JSON, per-file rules, cross-file role references), `Store` (the adopted snapshot + serialized `Reload`, typed accessors read per call, `AfterAdopt` hooks), `Registry` (tenant id → `Store`; holds the one store under `tenant.Default`), the fsnotify `Watch`, and the embedded (`go:embed`) seed `wavehouse bootstrap` writes - **`stream/`** — SSE fan-out: rows travel POSITIONALLY, so each connection is told its projected column list in an `event: schema` frame before its first row and again on drift — **not** guaranteed after a gap-fill across a column change, which can leave a connection reading live rows against a stale list until it reconnects ([#543](https://github.com/Wave-RF/WaveHouse/issues/543)) — (tracked per connection; replay tracks its own). The event `Hub` (registers subscribers by `(topic, role)`; `Broadcast` projects + serializes each event once per role, the #294 delivery hot path — a role carrying a row-level `filter` keeps the shared projection but delivers per subscriber, each subscriber's claims evaluated against the row, #319), `Subscriber` (per-connection outbound `Frame` queue, `Send`/`Frames`; claims fixed at construction, immutable), the `Bucket` fan-out set (`subscriberSet`, one per `(topic, role)`), the `Heartbeater` keepalive wheel, and `Metrics` (the `wavehouse_sse_*` stream instruments) - **`tenant/`** — the tenant identifier ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)): `ID` (a validated string), `Parse` (letters, digits, `_`, `-`; ≤ 64 bytes — safe as a folder name and as an MQ subject token), `Default` (`"0"`), and `Header` (`X-Tenant-ID`). Imports nothing from the rest of the repo. `api.TenantMW` resolves the header against `settings.Registry` before auth on every `/v1` route outside `/v1/ops/*` (`400` malformed, `404` unknown) and puts the resolved `*settings.Store` in the request context; handlers read it once (`api.StoreFromContext`) and pass it down as an argument, and nothing below a handler reads context. The async paths (ingest worker, sweeper, stream hub, schema registry) are constructed with a `tenant.ID` and their getters take it diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index a48d376c..dff15a1d 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -184,7 +184,7 @@ The hot-reloadable half of configuration: a directory of four JSON files (`confi - **store.go** — `Store` owns the adopted snapshot. `Open` validates and adopts at boot; `Reload` re-validates and swaps the document atomically when there are no errors (a rejected reload keeps the previous snapshot). Consumers read typed accessors per call (`ClickHouse()`, `Auth()`, `DedupeFor(table)`, `DLQFor(table)`, `Keepalive()`, …) rather than holding values, and `AfterAdopt` registers hooks (dedupe store open/close, keepalive-wheel rebuild) that run after each successful reload. - **registry.go** — `Registry` maps a tenant id to its `Store` (`For(id)`). It holds the one store `Open` adopted, under `tenant.Default`; reload and the watcher stay on the `Store`. - **watch.go** — fsnotify on the *directory* (not the files, so atomic-writer replaces and Kubernetes ConfigMap symlink swaps aren't lost), debounced into one reload; reloads once as soon as the watch exists so an edit between the boot read and the watch is never missed. `SIGHUP` and the reload endpoint funnel through the same serialized `Reload`. -- **seed.go** / **seed/** — The `go:embed`ded starter directory with every key at its default. The binary carries no compiled defaults: `wavehouse bootstrap [dir]` writes this seed, and the compose stack and e2e fixture ship copies of it. +- **seed.go** / **seed/** — The embedded (`go:embed`) starter directory with every key at its default. The binary carries no compiled defaults: `wavehouse bootstrap [dir]` writes this seed, and the compose stack and e2e fixture ship copies of it. ### `tenant/` — Tenant Identifier diff --git a/docs/src/content/docs/settings-directory.mdx b/docs/src/content/docs/settings-directory.mdx index 4ca04d40..3ab3c498 100644 --- a/docs/src/content/docs/settings-directory.mdx +++ b/docs/src/content/docs/settings-directory.mdx @@ -189,7 +189,7 @@ The `WAVEHOUSE_DLQ` stream always exists (an empty stream costs nothing) and the ## Message Queue -- `mq.max_bytes_gb` (seed default `50`) — disk budget for the embedded JetStream `WAVEHOUSE` stream that buffers ingested events until the worker writes them to ClickHouse; the `WAVEHOUSE_DLQ` stream gets a tenth of it. The stream runs `DiscardNew`, so when it's full new publishes are rejected and `POST /v1/ingest` returns `503` — [backpressure by construction](/ingest-pipeline#backpressure-and-durability-knobs). A reload updates both streams' limits in place without touching what's buffered: growing takes effect immediately; shrinking below what's currently on disk makes the stream refuse new publishes until the worker drains it back under the limit — nothing already accepted is dropped. If NATS rejects the update, the rest of the reload is still adopted; the streams stay on the previously applied budget, the failure is logged, and the next reload retries it. Size it from [Durability & Storage](/durability). +- `mq.max_bytes_gb` (seed default `50`) — disk budget for the embedded JetStream `WAVEHOUSE` stream that buffers ingested events until the worker writes them to ClickHouse; the `WAVEHOUSE_DLQ` stream gets a tenth of it. The stream runs `DiscardNew`, so when it's full new publishes are rejected and `POST /v1/ingest` returns `503` — [backpressure by construction](/ingest-pipeline#backpressure-and-durability-knobs). A reload updates both streams' limits in place without touching what's buffered: growing takes effect immediately; shrinking below what's currently on disk makes the stream refuse new publishes until the worker drains it back under the limit — nothing already accepted is dropped. If NATS rejects the update, the rest of the reload is still adopted, the failure is logged, and the next reload retries it. The two streams are resized as a pair: a failed DLQ resize undoes the ingest one so both stay on the previous budget, but if that undo fails too the ingest stream keeps the new limit and the DLQ the previous one until a later reload succeeds — the log line says which happened. Size it from [Durability & Storage](/durability). ## Streaming From af2ab5217401d13c503bfe048a349a52c7935cf2 Mon Sep 17 00:00:00 2001 From: taitelee Date: Sat, 19 Sep 2026 14:03:01 -0400 Subject: [PATCH 08/11] fix(tenant): vary tenant routes on X-Tenant-ID, scope the ?tenant= doc to the pipe reads --- CHANGELOG.md | 2 +- docs/src/content/docs/api.md | 4 ++-- internal/api/tenant.go | 6 +++++- internal/api/tenant_test.go | 2 ++ 4 files changed, 10 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4e2a57a0..6707a531 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **Requests resolve to a tenant before authentication, and the tenant is threaded through every settings read** (`internal/tenant/` (new, + tests), `internal/settings/registry.go` (new, + tests), `internal/api/tenant.go` (new, + tests), `internal/api/{router,ingest,structured_query,pipes}.go`, `internal/ingest/{worker,sweeper}.go`, `internal/stream/hub.go`, `internal/discovery/discovery.go`, `internal/app/{app,wire}.go`): story 1 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)), with no behavior change for a deployment that sends no tenant header. `internal/tenant` defines the id — a validated string (letters, digits, `_`, `-`; at most 64 bytes, so it is safe as a folder name and as an MQ subject token), the reserved default `0`, and the `X-Tenant-ID` header name — and imports nothing from the rest of the repository. `api.TenantMW` runs ahead of the auth middleware on every `/v1` route outside `/v1/ops/*`: an absent or empty header is tenant `0`, a malformed id or a repeated header is a `400`, a well-formed id the new `settings.Registry` does not hold is a `404`, and the resolved `*settings.Store` rides the request context. The registry holds the one store `settings.Open` adopted, keyed `0`. Handlers read the store once and pass it down as an argument — the ingest, structured-query, and pipe getters (`PolicySource`, `DedupeSettings`, `bucketSecs`, `defaultMaxRows`, the pipes source) now take it as a parameter — and a tenant route reached without a resolved tenant answers `500` rather than fall back to one. The ingest worker, sweeper, stream hub, and schema registry are constructed with a `tenant.ID` (`tenant.Default` in `internal/app`) and their settings getters take it. The probes, `/version`, the metrics path, and `/v1/ops/*` stay tenant-exempt, with the ops tree behind the auth middleware and the admin gate exactly as before; the admin pipe reads (`GET /v1/ops/pipes[/{name}]`) name their tenant with an optional `?tenant=` query parameter (absent means `0`, same `400`/`404` answers as the header; a query string that does not parse is a `400` too, rather than silently reading as the default tenant). `X-Tenant-ID` joins the CORS `Access-Control-Allow-Headers` list so a browser client can send it; the SDK needs no change (`options.headers`). +- **Requests resolve to a tenant before authentication, and the tenant is threaded through every settings read** (`internal/tenant/` (new, + tests), `internal/settings/registry.go` (new, + tests), `internal/api/tenant.go` (new, + tests), `internal/api/{router,ingest,structured_query,pipes}.go`, `internal/ingest/{worker,sweeper}.go`, `internal/stream/hub.go`, `internal/discovery/discovery.go`, `internal/app/{app,wire}.go`): story 1 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)), with no behavior change for a deployment that sends no tenant header. `internal/tenant` defines the id — a validated string (letters, digits, `_`, `-`; at most 64 bytes, so it is safe as a folder name and as an MQ subject token), the reserved default `0`, and the `X-Tenant-ID` header name — and imports nothing from the rest of the repository. `api.TenantMW` runs ahead of the auth middleware on every `/v1` route outside `/v1/ops/*`: an absent or empty header is tenant `0`, a malformed id or a repeated header is a `400`, a well-formed id the new `settings.Registry` does not hold is a `404`, and the resolved `*settings.Store` rides the request context. Every answer from the middleware, the `400` and `404` included, carries `Vary: X-Tenant-ID` so a shared cache can't replay one tenant's response to another. The registry holds the one store `settings.Open` adopted, keyed `0`. Handlers read the store once and pass it down as an argument — the ingest, structured-query, and pipe getters (`PolicySource`, `DedupeSettings`, `bucketSecs`, `defaultMaxRows`, the pipes source) now take it as a parameter — and a tenant route reached without a resolved tenant answers `500` rather than fall back to one. The ingest worker, sweeper, stream hub, and schema registry are constructed with a `tenant.ID` (`tenant.Default` in `internal/app`) and their settings getters take it. The probes, `/version`, the metrics path, and `/v1/ops/*` stay tenant-exempt, with the ops tree behind the auth middleware and the admin gate exactly as before; the admin pipe reads (`GET /v1/ops/pipes[/{name}]`) name their tenant with an optional `?tenant=` query parameter (absent means `0`, same `400`/`404` answers as the header; a query string that does not parse is a `400` too, rather than silently reading as the default tenant). `X-Tenant-ID` joins the CORS `Access-Control-Allow-Headers` list so a browser client can send it; the SDK needs no change (`options.headers`). - **Schema discovery captures each table's DDL, its columns' ordinals and default expressions, and the server version** (`internal/discovery/discovery.go`, `internal/testutil/testutil.go`): `Column` gains `DefaultExpression` and `Position` (both from a widened `system.columns` select), `TableSchema` gains `DDL` from `system.tables.create_table_query`, and `SchemaRegistry` gains `ServerVersion()` from a `SELECT version()` probe next to the existing `SELECT timezone()`. Groundwork for the native type layer, captured on the same refresh as the columns so a stale version cannot outlive the schemas it describes. That is a publication guarantee, not a same-server one: `chconn.Manager` resolves the connection per call, so a reload changing `clickhouse.addr` mid-refresh can still pair a version from one server with schemas from another — narrow, and self-correcting on the next refresh. `DDL` is `json:"-"` and does **not** appear in `/v1/ops/schema`: that endpoint marshals `TableSchema` straight to the client, and an external-engine table (S3, MySQL, PostgreSQL, Kafka) renders its wiring there unconditionally — endpoint, bucket or host, database, username, S3 access key id. ClickHouse masks the password itself as `[HIDDEN]` from ~23.9 (verified on 26.7.3), so the exposure is the topology rather than the secret — except on an older server, or one with `display_secrets_in_show_and_select` enabled. `position` and `default_expression` are additive fields in the response. A table listed in `system.tables` with no `system.columns` rows is skipped rather than published column-less, and both new queries fail the refresh on error exactly as `timezone()` and `system.columns` do — callers keep the prior cache and retry. diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index a729bf4f..b4460aa1 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -67,9 +67,9 @@ A tenant id is 1–64 characters of ASCII letters, digits, `_`, and `-`. It is a | `400` | `{"error": "invalid X-Tenant-ID: …"}` | The id breaks the grammar above, or the header was sent more than once | | `404` | `{"error": "unknown tenant: "}` | The id is well formed but no such tenant exists | -Both are decided before authentication, so they are returned whatever token the request carries. +Both are decided before authentication, so they are returned whatever token the request carries. Every response from a tenant route, these two included, carries `Vary: X-Tenant-ID`, so a shared cache keys on the header and never replays one tenant's response to another. -The probes (`/livez`, `/readyz`, `/healthz`), `/version`, the Prometheus metrics path, and `/v1/ops/*` are tenant-exempt: they ignore the header entirely. An ops route that reads a tenant's settings names it with a `?tenant=` query parameter instead ([`GET /v1/ops/pipes`](#get-v1opspipes--list-named-pipes)), with the same grammar and the same `400`/`404` answers. +The probes (`/livez`, `/readyz`, `/healthz`), `/version`, the Prometheus metrics path, and `/v1/ops/*` are tenant-exempt: they ignore the header entirely. The two admin pipe reads ([`GET /v1/ops/pipes`](#get-v1opspipes--list-named-pipes) and `GET /v1/ops/pipes/{name}`) name their tenant with an optional `?tenant=` query parameter instead, with the same grammar and the same `400`/`404` answers; no other ops route takes a tenant. `X-Tenant-ID` is in the CORS `Access-Control-Allow-Headers` list, so a browser client can send it cross-origin. The SDK sends it through [`options.headers`](/sdk#custom-headers). diff --git a/internal/api/tenant.go b/internal/api/tenant.go index 32a81c91..82678e0e 100644 --- a/internal/api/tenant.go +++ b/internal/api/tenant.go @@ -76,10 +76,14 @@ func resolveTenant(w http.ResponseWriter, tenants *settings.Registry, where stri } // TenantMW resolves the request's tenant from the tenant.Header before -// authentication runs and stores it in the request context. +// authentication runs and stores it in the request context. Every answer, +// the 400 and 404 included, carries Vary: X-Tenant-ID so a shared cache +// cannot replay one tenant's response to another — added, not set, so the +// CORS Vary: Origin survives. func TenantMW(tenants *settings.Registry) func(http.Handler) http.Handler { return func(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Add("Vary", tenant.Header) store, ok := resolveTenant(w, tenants, tenant.Header, r.Header.Values(tenant.Header)) if !ok { return diff --git a/internal/api/tenant_test.go b/internal/api/tenant_test.go index 4035e02d..95097ec0 100644 --- a/internal/api/tenant_test.go +++ b/internal/api/tenant_test.go @@ -50,6 +50,8 @@ func TestTenantMW(t *testing.T) { h.ServeHTTP(w, req) require.Equal(t, tt.wantStatus, w.Code, "body: %s", w.Body.String()) + assert.Equal(t, []string{tenant.Header}, w.Header().Values("Vary"), + "every answer varies on the tenant header, a refusal included") if tt.wantStatus == http.StatusOK { assert.Same(t, testStore, resolved, "the resolved store rides the request context") return From f1bbc279c27a63c19d7dfaa7eeaacb5fee605bf9 Mon Sep 17 00:00:00 2001 From: taitelee Date: Sat, 19 Sep 2026 14:18:03 -0400 Subject: [PATCH 09/11] test(api): pin the composed Vary header on a tenant route --- internal/api/router_test.go | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/internal/api/router_test.go b/internal/api/router_test.go index 9cb6e056..398ddb53 100644 --- a/internal/api/router_test.go +++ b/internal/api/router_test.go @@ -469,6 +469,30 @@ func TestNewRouter_CORSOnStream(t *testing.T) { }) } +// TestNewRouter_VaryOriginAndTenant pins the composed Vary header on a tenant +// route: corsMiddleware sets Vary: Origin at the root and TenantMW adds +// Vary: X-Tenant-ID inside /v1, so both survive only in that registration +// order. A reorder would drop one silently — a shared cache could then replay +// one tenant's response to another, or one origin's CORS answer to another. +func TestNewRouter_VaryOriginAndTenant(t *testing.T) { + t.Parallel() + + router := NewRouter(Dependencies{ + Tenants: testTenants(), + Health: &HealthHandler{}, + AuthMW: func(next http.Handler) http.Handler { return next }, + CORSOrigins: func() []string { return []string{"https://app.example.com"} }, + }) + + req := httptest.NewRequestWithContext(context.Background(), http.MethodGet, "/v1/health", nil) + req.Header.Set("Origin", "https://app.example.com") + rec := httptest.NewRecorder() + router.ServeHTTP(rec, req) + + assert.Equal(t, http.StatusOK, rec.Code) + assert.ElementsMatch(t, []string{"Origin", tenant.Header}, rec.Header().Values("Vary")) +} + // TestNewRouter_RawSQLAdminGate pins the contract for POST /v1/ops/query: // // admin role → reaches handler From d0913406d35fe6c85420d1da7ab226ce92f8ee22 Mon Sep 17 00:00:00 2001 From: taitelee Date: Mon, 21 Sep 2026 09:44:27 -0400 Subject: [PATCH 10/11] fix(tenant): defer ops ?tenant= to story 2, fail safe on a registry miss, move tenant docs to deployment --- CHANGELOG.md | 2 +- docs/src/content/docs/access-control.mdx | 4 ++ docs/src/content/docs/api.md | 29 +-------- docs/src/content/docs/architecture.md | 4 +- docs/src/content/docs/deployment.md | 31 ++++++++++ docs/src/content/docs/reverse-proxy.mdx | 4 +- docs/src/content/docs/sdk/index.mdx | 2 +- docs/src/content/docs/sdk/reference.md | 2 +- docs/src/content/docs/sdk/streaming.md | 2 +- internal/api/pipes.go | 30 ++++----- internal/api/pipes_test.go | 8 +-- internal/api/router_test.go | 2 +- internal/api/tenant.go | 78 ++++++++---------------- internal/api/tenant_test.go | 46 -------------- internal/app/app_test.go | 12 ++++ internal/app/wire.go | 27 +++++--- internal/discovery/discovery.go | 15 ++++- internal/discovery/discovery_test.go | 46 ++++++++++++++ 18 files changed, 175 insertions(+), 169 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6707a531..977c02d0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **Requests resolve to a tenant before authentication, and the tenant is threaded through every settings read** (`internal/tenant/` (new, + tests), `internal/settings/registry.go` (new, + tests), `internal/api/tenant.go` (new, + tests), `internal/api/{router,ingest,structured_query,pipes}.go`, `internal/ingest/{worker,sweeper}.go`, `internal/stream/hub.go`, `internal/discovery/discovery.go`, `internal/app/{app,wire}.go`): story 1 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)), with no behavior change for a deployment that sends no tenant header. `internal/tenant` defines the id — a validated string (letters, digits, `_`, `-`; at most 64 bytes, so it is safe as a folder name and as an MQ subject token), the reserved default `0`, and the `X-Tenant-ID` header name — and imports nothing from the rest of the repository. `api.TenantMW` runs ahead of the auth middleware on every `/v1` route outside `/v1/ops/*`: an absent or empty header is tenant `0`, a malformed id or a repeated header is a `400`, a well-formed id the new `settings.Registry` does not hold is a `404`, and the resolved `*settings.Store` rides the request context. Every answer from the middleware, the `400` and `404` included, carries `Vary: X-Tenant-ID` so a shared cache can't replay one tenant's response to another. The registry holds the one store `settings.Open` adopted, keyed `0`. Handlers read the store once and pass it down as an argument — the ingest, structured-query, and pipe getters (`PolicySource`, `DedupeSettings`, `bucketSecs`, `defaultMaxRows`, the pipes source) now take it as a parameter — and a tenant route reached without a resolved tenant answers `500` rather than fall back to one. The ingest worker, sweeper, stream hub, and schema registry are constructed with a `tenant.ID` (`tenant.Default` in `internal/app`) and their settings getters take it. The probes, `/version`, the metrics path, and `/v1/ops/*` stay tenant-exempt, with the ops tree behind the auth middleware and the admin gate exactly as before; the admin pipe reads (`GET /v1/ops/pipes[/{name}]`) name their tenant with an optional `?tenant=` query parameter (absent means `0`, same `400`/`404` answers as the header; a query string that does not parse is a `400` too, rather than silently reading as the default tenant). `X-Tenant-ID` joins the CORS `Access-Control-Allow-Headers` list so a browser client can send it; the SDK needs no change (`options.headers`). +- **Requests resolve to a tenant before authentication, and the tenant is threaded through every settings read** (`internal/tenant/` (new, + tests), `internal/settings/registry.go` (new, + tests), `internal/api/tenant.go` (new, + tests), `internal/api/{router,ingest,structured_query,pipes}.go`, `internal/ingest/{worker,sweeper}.go`, `internal/stream/hub.go`, `internal/discovery/discovery.go`, `internal/app/{app,wire}.go`): story 1 of the multi-tenant epic ([#583](https://github.com/Wave-RF/WaveHouse/issues/583)), with no behavior change for a deployment that sends no tenant header. `internal/tenant` defines the id — a validated string (letters, digits, `_`, `-`; at most 64 bytes, so it is safe as a folder name and as an MQ subject token), the reserved default `0`, and the `X-Tenant-ID` header name — and imports nothing from the rest of the repository. `api.TenantMW` runs ahead of the auth middleware on every `/v1` route outside `/v1/ops/*`: an absent or empty header is tenant `0`, a malformed id or a repeated header is a `400`, a well-formed id the new `settings.Registry` does not hold is a `404`, and the resolved `*settings.Store` rides the request context. Every answer from the middleware, the `400` and `404` included, carries `Vary: X-Tenant-ID` so a shared cache can't replay one tenant's response to another. The registry holds the one store `settings.Open` adopted, keyed `0`. Handlers read the store once and pass it down as an argument — the ingest, structured-query, and pipe getters (`PolicySource`, `DedupeSettings`, `bucketSecs`, `defaultMaxRows`, the pipes source) now take it as a parameter — and a tenant route reached without a resolved tenant answers `500` rather than fall back to one. The ingest worker, sweeper, stream hub, and schema registry are constructed with a `tenant.ID` (`tenant.Default` in `internal/app`) and their settings getters take it; a tenant the registry does not hold is logged and read as the getter's zero value, with two fail-safes — the worker's DLQ switch reads as on (an unreadable message is parked, never dropped) and the schema auto-refresh keeps its cadence rather than hand `time.NewTicker` a zero interval. The probes, `/version`, the metrics path, and `/v1/ops/*` stay tenant-exempt, with the ops tree behind the auth middleware and the admin gate exactly as before; the admin pipe reads (`GET /v1/ops/pipes[/{name}]`) serve the default tenant. `X-Tenant-ID` joins the CORS `Access-Control-Allow-Headers` list so a browser client can send it; the SDK needs no change (`options.headers`). - **Schema discovery captures each table's DDL, its columns' ordinals and default expressions, and the server version** (`internal/discovery/discovery.go`, `internal/testutil/testutil.go`): `Column` gains `DefaultExpression` and `Position` (both from a widened `system.columns` select), `TableSchema` gains `DDL` from `system.tables.create_table_query`, and `SchemaRegistry` gains `ServerVersion()` from a `SELECT version()` probe next to the existing `SELECT timezone()`. Groundwork for the native type layer, captured on the same refresh as the columns so a stale version cannot outlive the schemas it describes. That is a publication guarantee, not a same-server one: `chconn.Manager` resolves the connection per call, so a reload changing `clickhouse.addr` mid-refresh can still pair a version from one server with schemas from another — narrow, and self-correcting on the next refresh. `DDL` is `json:"-"` and does **not** appear in `/v1/ops/schema`: that endpoint marshals `TableSchema` straight to the client, and an external-engine table (S3, MySQL, PostgreSQL, Kafka) renders its wiring there unconditionally — endpoint, bucket or host, database, username, S3 access key id. ClickHouse masks the password itself as `[HIDDEN]` from ~23.9 (verified on 26.7.3), so the exposure is the topology rather than the secret — except on an older server, or one with `display_secrets_in_show_and_select` enabled. `position` and `default_expression` are additive fields in the response. A table listed in `system.tables` with no `system.columns` rows is skipped rather than published column-less, and both new queries fail the refresh on error exactly as `timezone()` and `system.columns` do — callers keep the prior cache and retry. diff --git a/docs/src/content/docs/access-control.mdx b/docs/src/content/docs/access-control.mdx index 5dba919c..b54d271a 100644 --- a/docs/src/content/docs/access-control.mdx +++ b/docs/src/content/docs/access-control.mdx @@ -212,6 +212,10 @@ On a structured query (`POST /v1/query?table={table}`) the allowlist is a **hard This produces `WHERE (tenant_id = ?)` with the caller's `app_metadata.tenant_id` claim bound as the parameter — so a `viewer` only ever sees rows for their own tenant, and the value comes from the signed token, not from anything the client sends. +:::note[Two meanings of "tenant"] +On this page a *tenant* is a row-scoping value carried in the signed token. The [`X-Tenant-ID` header](/deployment#multi-tenant-deployments) is a different axis: it selects which settings directory — and so which `policies.json` — serves the request. It is client-supplied, resolved before authentication, and isolates no rows. A settings directory is one such tenant (`0`), so most deployments never send it. +::: + Supported comparison operators: | Operator | SQL | Meaning | diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index b4460aa1..27ed88ae 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -50,29 +50,6 @@ WaveHouse extracts the role from a configurable JWT claim path (`auth.role_claim Policies support Hasura-style row-level and column-level permissions with JWT claim templating (e.g., `{{ jwt.app_metadata.tenant_id }}`). -## Tenant Selection - -Every `/v1` route outside `/v1/ops/*` resolves a tenant before it authenticates the request. The tenant comes from the `X-Tenant-ID` request header: - -```text -X-Tenant-ID: 0 -``` - -A request without the header, or with an empty one, resolves to tenant `0`, the default tenant, whose settings are the [settings directory](/settings-directory). A settings directory defines that one tenant, so any other id is unknown. Setting the header on every request is the client's or the fronting proxy's job; WaveHouse never derives it from the token. - -A tenant id is 1–64 characters of ASCII letters, digits, `_`, and `-`. It is a string, not a number, so a long numeric id keeps every digit. - -| Status | Body | When | -| ------ | ---- | ---- | -| `400` | `{"error": "invalid X-Tenant-ID: …"}` | The id breaks the grammar above, or the header was sent more than once | -| `404` | `{"error": "unknown tenant: "}` | The id is well formed but no such tenant exists | - -Both are decided before authentication, so they are returned whatever token the request carries. Every response from a tenant route, these two included, carries `Vary: X-Tenant-ID`, so a shared cache keys on the header and never replays one tenant's response to another. - -The probes (`/livez`, `/readyz`, `/healthz`), `/version`, the Prometheus metrics path, and `/v1/ops/*` are tenant-exempt: they ignore the header entirely. The two admin pipe reads ([`GET /v1/ops/pipes`](#get-v1opspipes--list-named-pipes) and `GET /v1/ops/pipes/{name}`) name their tenant with an optional `?tenant=` query parameter instead, with the same grammar and the same `400`/`404` answers; no other ops route takes a tenant. - -`X-Tenant-ID` is in the CORS `Access-Control-Allow-Headers` list, so a browser client can send it cross-origin. The SDK sends it through [`options.headers`](/sdk#custom-headers). - ## Response Format ### Error Responses @@ -173,7 +150,7 @@ Status code: `503 Service Unavailable` ### `GET /v1/health` — Liveness ping (public, content-free) -Returns **`200 OK` with an empty body** once the gateway is past boot, or **`503 Service Unavailable`** (also empty) while boot-time schema discovery is still failing. Like every other `/v1` route it [resolves a tenant](#tenant-selection) first, so a bad `X-Tenant-ID` answers `400`/`404` before the probe runs. No authentication required and no response body — the caller only branches on the status code, so there's nothing to JSON-encode or cache per request. +Returns **`200 OK` with an empty body** once the gateway is past boot, or **`503 Service Unavailable`** (also empty) while boot-time schema discovery is still failing. Like every other `/v1` route it [resolves a tenant](/deployment#multi-tenant-deployments) first, so a bad `X-Tenant-ID` answers `400`/`404` before the probe runs. No authentication required and no response body — the caller only branches on the status code, so there's nothing to JSON-encode or cache per request. This is what the SDK's `wh.sys.health()` calls, and the endpoint to use when choosing among multiple servers in a distributed setup. It mirrors `/livez` under the hood but is intentionally a `/v1` API route rather than a Kubernetes probe path: an operator may filter the bare probe paths (`/livez`, `/readyz`, `/healthz`) out at the reverse proxy since they're internal probes, so the SDK relies on `/v1/health`, which is documented public API surface meant to stay reachable. It does **not** ping ClickHouse — readiness-based load balancing is the proxy/LB's job (via `/readyz`), not the client's. @@ -781,11 +758,9 @@ The policy has no endpoints: it is the settings directory's [`policies.json`](/s Returns every adopted named query pipe — the settings directory's [`pipes.json`](/settings-directory#pipesjson). Pipes have no write endpoints: edit the file and reload. -The ops routes are [tenant-exempt](#tenant-selection), so this read and `GET /v1/ops/pipes/{name}` name their tenant with an optional `?tenant=` query parameter instead of the header. Absent or empty means tenant `0`; a malformed id, a repeated parameter, or a query string that does not parse is a `400` (`{"error": "invalid ?tenant: …"}`), and an unknown tenant a `404` with the same body as the header. - #### `GET /v1/ops/pipes/{name}` — Get Named Pipe -Returns a specific named pipe definition from the tenant named by `?tenant=`, as above: +Returns a specific named pipe definition: ```json { diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index dff15a1d..f9f4c4c2 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -77,7 +77,7 @@ The API layer uses [Chi](https://github.com/go-chi/chi) for routing with Request - **router.go** — Route definitions. Public: `/livez`, `/readyz`, and the content-free `/v1/health` SDK ping (plus the permanent `/healthz` alias and the deprecated `/health`, `/ready` aliases). Policy-gated: `/v1/ingest?table={table}`, `/v1/query?table={table}` (structured), `/v1/pipes/{name}` (named pipes), `/v1/stream`. Admin-only (`RequireAdmin` — role == `policy.admin_role`, or a request bearing the operator key's operator bit, which passes even under a nil policy): `/v1/ops/schema/*`, `/v1/ops/dlq/stats`, `GET /v1/ops/pipes[/{name}]`, `/v1/ops/settings/reload`, `/v1/ops/query` (raw SQL — same gate as the rest of `/v1/ops/*`). - **auth middleware** — the JWT/JWKS authentication middleware is its own package, [`auth/`](#auth--authentication); the router runs it on every `/v1/*` route. -- **tenant.go** — `TenantMW` resolves the request's tenant ahead of the auth middleware on every `/v1` route outside `/v1/ops/*`: the [`X-Tenant-ID`](/api#tenant-selection) header (absent means `tenant.Default`), validated by `tenant.Parse` (`400`), looked up in the `settings.Registry` (`404` on a miss), and the resolved `*settings.Store` stored in the request context (`WithStore` / `StoreFromContext` — here rather than in `tenant/`, because `settings` names `tenant.ID`). A handler reads the store once and passes it down as an argument — the per-tenant getters it holds take it as a parameter (`(*settings.Store).Policy`, `.DedupeFor`, `.DefaultMaxRows`, … in production) — and nothing below a handler reads the context; a tenant route reached without a resolved store answers `500` rather than fall back to a tenant. The probes, `/version`, the metrics path, and `/v1/ops/*` are tenant-exempt; an ops read that needs a tenant names it with `?tenant=` (`opsStore`, the pipe reads), resolved through the same `resolveTenant`. +- **tenant.go** — `TenantMW` resolves the request's tenant ahead of the auth middleware on every `/v1` route outside `/v1/ops/*`: the [`X-Tenant-ID`](/deployment#multi-tenant-deployments) header (absent means `tenant.Default`), validated by `tenant.Parse` (`400`), looked up in the `settings.Registry` (`404` on a miss), and the resolved `*settings.Store` stored in the request context (`WithStore` / `StoreFromContext` — here rather than in `tenant/`, because `settings` names `tenant.ID`). A handler reads the store once and passes it down as an argument — the per-tenant getters it holds take it as a parameter (`(*settings.Store).Policy`, `.DedupeFor`, `.DefaultMaxRows`, … in production) — and nothing below a handler reads the context; a tenant route reached without a resolved store answers `500` rather than fall back to a tenant. The probes, `/version`, the metrics path, and `/v1/ops/*` are tenant-exempt; the admin pipe reads serve the default tenant's store. - **pipes.go** — Named query pipe handlers: admin listing (`GET /v1/ops/pipes[/{name}]`, read per request from its `pipes.Source`) and execution with parameter binding. `pipes.json` is the only write path. - **structured_query.go** — Handler for `POST /v1/query?table={table}`: validates query AST, enforces permissions, builds and executes SQL. - **ingest.go** — Accepts `POST /v1/ingest?table={table}` in three body shapes: one flat JSON object, a JSON array of them, or NDJSON. The **required** `Content-Type` chooses the format *family* — `application/json` versus the four NDJSON spellings — and within the JSON family the body's first non-whitespace byte picks array versus single object; the bytes never choose the family. Anything that is not exactly one readable media type is a `415`, decided before the body is read: the header is parsed per RFC 9110 §8.3, and because `Content-Type` is a singleton field, repeated header lines must all resolve to the same format and a value carrying a comma is refused unless the value as a whole parses as one media type — a comma inside a *quoted* parameter value is data, so `application/json; a=", application/x-ndjson; b="` is accepted. It then reads the whole (`MaxBytesReader`-capped) body into a pooled buffer and runs the per-format record readers over those bytes, so the `413` lands before any record is processed and peak memory per request is O(body) rather than O(record). Then it validates each record against the discovered schema, optional dedup, and publishes each row through `mq.Publisher` on `mq.Topic{Table, Scope}` (raw names — the subject it becomes is `internal/mq`'s; a full queue comes back as `mq.ErrQueueFull`, which is the `503` + `Retry-After`). When dedup is on, a row missing the configured `id_field` can't be deduped: it is logged at `WARN` and counted by `wavehouse_ingest_dedupe_missing_id_total` (labeled by `table`), then published un-deduped — or rejected when `dedupe.require_id` is set ([#219](https://github.com/Wave-RF/WaveHouse/issues/219)). @@ -90,7 +90,7 @@ The API layer uses [Chi](https://github.com/go-chi/chi) for routing with Request ### `app/` — Process wiring - **app.go** — `New(ctx, Options)` builds every component from the boot config (`Options.Config`) and the settings directory it names, in dependency order: settings store, observability, ClickHouse connection, schema discovery, dedupe store, embedded NATS (ingest + DLQ streams), cache, sweeper, streaming (hub, MQ→hub bridge, keepalive wheel), ingest worker, auth, reload triggers, HTTP. Each is one `component` value — what it opens, what it loops, what it releases — so a failure part-way releases what was already opened and returns the error. `Run(ctx)` drives every loop under one `errgroup` until `ctx` is canceled (a clean stop: every loop drains, the API server and the ingest worker within `server.shutdown_timeout`; open SSE streams are ended as the drain begins rather than waited on) or a component fails, which stops the rest and returns that error. `Close(ctx)` releases what `New` opened, newest first, under the caller's release budget (`ReleaseTimeout`, 5s), a real bound: a remote implementation's close gives up at the deadline itself, and a close that ignores the context (the local stores) is abandoned at it, with the components below it left unreleased rather than overlapping it, both named in the error — and then flushes telemetry under its own 3s budget, so the flush that reports on the stop is never handed a deadline a slow close already spent. The SIGHUP registration is released last of all. `Handler`, `Registry`, and `MQ` expose the pieces a harness needs; `Options.Listener` lets one serve the API on its own listener instead of `server.port`. -- **wire.go** — one `wire*` function per component, each handed the settings store whole and deriving the per-call getters the internal packages take (`DLQFor`, `DedupeFor`, `GapWindow`, …) and registering its `AfterAdopt` hook there where it has one. Those wiring functions are where the per-tenant registry of [#583](https://github.com/Wave-RF/WaveHouse/issues/583) is injected, not `main`: `wireSettings` builds the `settings.Registry`, the HTTP handlers get store-keyed getters (method expressions such as `(*settings.Store).Policy`), and `perTenant` adapts a store accessor into the `func(tenant.ID) T` getter the async packages take. The reload triggers (SIGHUP, the directory watcher) only start in `Run`, after `New` has registered every hook, so the watcher's first reload already drives all of them. The `mq.max_bytes_gb` hook only hands the adopted budget to `mq.Broker.SetMaxBytes` under the App's stop context; how it is split across the streams, the time bounds, and the rollback are `internal/mq`'s. +- **wire.go** — one `wire*` function per component, each handed the settings store whole and deriving the per-call getters the internal packages take (`DLQFor`, `DedupeFor`, `GapWindow`, …) and registering its `AfterAdopt` hook there where it has one. Those wiring functions are where the per-tenant registry of [#583](https://github.com/Wave-RF/WaveHouse/issues/583) is injected, not `main`: `wireSettings` builds the `settings.Registry`, the HTTP handlers get store-keyed getters (method expressions such as `(*settings.Store).Policy`), and `perTenant` adapts a store accessor into the `func(tenant.ID) T` getter the async packages take — a tenant the registry does not hold is logged and read as the zero value, except in `dlqFor`, the ingest worker's DLQ switch, where it reads as on so a message the worker cannot read is parked rather than dropped. The reload triggers (SIGHUP, the directory watcher) only start in `Run`, after `New` has registered every hook, so the watcher's first reload already drives all of them. The `mq.max_bytes_gb` hook only hands the adopted budget to `mq.Broker.SetMaxBytes` under the App's stop context; how it is split across the streams, the time bounds, and the rollback are `internal/mq`'s. ### `stream/` — SSE keepalive & fan-out diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index db6c15cc..2ac19c79 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -328,6 +328,37 @@ Size the orchestrator's kill grace at `server.shutdown_timeout` plus 8s: at the WaveHouse serves plain HTTP on `:8080` and does **not** terminate TLS, manage certificates, or rate-limit — put a reverse proxy, CDN, or tunnel (nginx, Caddy, Cloudflare Tunnel) in front for any internet-facing deployment. A few behaviors only matter behind a proxy: TLS termination, the request-body size limits, Server-Sent Events buffering (WaveHouse now sends keepalive comments so quiet streams survive proxy idle timeouts, [#226](https://github.com/Wave-RF/WaveHouse/issues/226)), header/auth forwarding, and which health paths to expose. See **[Behind a reverse proxy](/reverse-proxy)** for the full guide and example nginx/Caddy/Cloudflare configs. +## Multi-tenant deployments + +Most deployments serve one tenant and can skip this section: send no `X-Tenant-ID` header and none of it applies, with one exception — [a proxy that already sends the header](#upgrading-behind-a-proxy-that-already-sends-x-tenant-id). + +A *tenant* here is a [settings directory](/settings-directory): the `X-Tenant-ID` request header selects whose `roles.json`, `policies.json`, `pipes.json`, and `config.json` serve the request. The header is client-supplied and resolved before authentication, so it is **not** a row-isolation boundary — it picks which `policies.json` applies, and scoping a caller to their own rows stays that policy's job, from a value in the signed token ([row-level security](/access-control#row-level-security)). Tenant selection and row scoping are different axes. + +Every `/v1` route outside `/v1/ops/*` resolves the tenant before it authenticates the request: + +```text +X-Tenant-ID: 0 +``` + +A request without the header, or with an empty one, resolves to tenant `0`, the default tenant, whose settings are the settings directory. A settings directory defines that one tenant, so any other id is unknown. Setting the header on every request is the client's or the fronting proxy's job; WaveHouse never derives it from the token. + +A tenant id is 1–64 characters of ASCII letters, digits, `_`, and `-`. It is a string, not a number, so a long numeric id keeps every digit. + +| Status | Body | When | +| ------ | ---- | ---- | +| `400` | `{"error": "invalid X-Tenant-ID: …"}` | The id breaks the grammar above, or the header was sent more than once | +| `404` | `{"error": "unknown tenant: "}` | The id is well formed but no such tenant exists | + +Both are decided before authentication, so they are returned whatever token the request carries. Every response that passes through tenant resolution — a route's own answer and these two alike — carries `Vary: X-Tenant-ID`, so a shared cache that stores one keys it on the header. A router-level `405` and the CORS preflight `204` are answered before tenant resolution and carry no such `Vary`; neither depends on the tenant. `Vary` covers the tenant and nothing else: a response also depends on who is asking, which is why [a shared cache must not store the authenticated reads](/reverse-proxy#header-and-auth-forwarding). + +The probes (`/livez`, `/readyz`, `/healthz`, and the deprecated `/health` and `/ready`), `/version`, the Prometheus metrics path, and `/v1/ops/*` are tenant-exempt: they ignore the header entirely. + +`X-Tenant-ID` is in the CORS `Access-Control-Allow-Headers` list, so a browser client can send it cross-origin. The SDK sends it through [`options.headers`](/sdk#custom-headers). + +### Upgrading behind a proxy that already sends `X-Tenant-ID` + +`X-Tenant-ID` is a generic name, and some gateways and service meshes stamp one on every request. WaveHouse used to ignore it; now any value other than `0` names an unknown tenant, so **every `/v1` route answers `404 unknown tenant: `** — the SDK's `/v1/health` reachability ping included, while the bare probes stay green. Strip the inbound header at the edge ([header forwarding](/reverse-proxy#header-and-auth-forwarding)) unless you are using it deliberately. + ## ClickHouse Schema WaveHouse uses a **Bring Your Own Schema** model. You create your tables in ClickHouse with whatever columns and engines you need. WaveHouse discovers the schemas automatically via `system.columns` and validates ingest data against them — see [Schema Validation](/api#post-v1ingesttabletable--ingest-data) for the rules a record must satisfy. diff --git a/docs/src/content/docs/reverse-proxy.mdx b/docs/src/content/docs/reverse-proxy.mdx index 187d29ce..9b07239e 100644 --- a/docs/src/content/docs/reverse-proxy.mdx +++ b/docs/src/content/docs/reverse-proxy.mdx @@ -194,7 +194,7 @@ These limit the *whole* request regardless of traffic, so no keepalive extends t - **`Authorization`** — forward verbatim. WaveHouse validates a `Bearer` JWT (resolving the role from it) or an `Operator ` [operator credential](/access-control#operator-key). Most proxies forward `Authorization` unchanged, so it's the transport to prefer for the operator key — the exception to check for is an auth-terminating layer that consumes or rewrites the header (e.g. a gateway doing its own auth). - **`X-Operator-Key`** (optional) — only relevant if you present the [operator key](/access-control#operator-key) via this alias header instead of `Authorization: Operator `. Custom request headers are forwarded by default, but confirm your proxy doesn't strip it — and note nginx silently drops header names containing **underscores** (this one uses hyphens, so it's fine as named). The `Authorization` form needs none of this. -- **`X-Tenant-ID`** (optional) — selects the [tenant](/api#tenant-selection); absent means tenant `0`. It is resolved before authentication and never derived from the token, so if the proxy owns tenant selection it must **replace or strip** a client-supplied value, not add to it: two `X-Tenant-ID` lines are refused with `400`, the same append-vs-replace hazard as `Content-Type` below. The name is hyphenated, so nginx's underscore rule does not apply. +- **`X-Tenant-ID`** — **strip it at the edge unless you are deliberately using it.** It selects the [tenant](/deployment#multi-tenant-deployments) (absent means tenant `0`) and any other value is an unknown tenant, so a gateway or mesh that stamps its own `X-Tenant-ID` turns every `/v1` route into a `404` — the SDK's `/v1/health` reachability ping included. It is resolved before authentication and never derived from the token, so a proxy that owns tenant selection must **replace** a client-supplied value, not add to it: two `X-Tenant-ID` lines are refused with `400`. That is a stricter version of the append-vs-replace hazard of `Content-Type` below — here even two agreeing lines are refused. The name is hyphenated, so nginx's underscore rule does not apply. - **`X-Forwarded-For` / `X-Forwarded-Proto` / `Host`** — set these for your own logs and any upstream that reads them. WaveHouse does not currently derive a client IP from `X-Forwarded-For` (see the caution below); forwarding it is good hygiene and is what the trusted-proxy client-IP work ([#333](https://github.com/Wave-RF/WaveHouse/issues/333)) will consume. :::caution[Don't expose `:8080` directly] @@ -205,6 +205,8 @@ WaveHouse does **not** derive a client IP from forwarded headers — it does no - **CORS** — WaveHouse applies its own CORS from the settings directory's `cors.allowed_origins`. Let one layer own CORS: either pass it through the proxy untouched (recommended), or strip it from WaveHouse and do it at the proxy — not both, or browsers see duplicate `Access-Control-Allow-Origin` headers and reject the response. +- **Response caching** — don't let a shared cache (a CDN or a caching proxy) store responses from the authenticated routes. What a read returns depends on the caller's role and claims, and a pipe read — a `GET`, so cacheable by default — sends no `Cache-Control` and no `Vary: Authorization` to say so. A cache that follows RFC 9111 already refuses to store the response to a request carrying `Authorization`; a credential sent as `?token=` or `X-Operator-Key` is not covered by that rule, so exclude those routes explicitly. The `Vary: X-Tenant-ID` on a [tenant-resolved](/deployment#multi-tenant-deployments) response keys it on the tenant only and says nothing about the caller. + ## Fencing the admin surface Every admin-gated endpoint — raw SQL, pipe inspection, settings reload, schema discovery, DLQ stats — lives under the single `/v1/ops/` prefix, so one proxy rule covers the entire management surface: an nginx `location /v1/ops/ { deny all; }` on the public vhost (serving it only on an internal listener), or an IP allowlist on that prefix. If you deny the prefix outright, keep an internal listener that still serves it — `POST /v1/ops/settings/reload` with the [operator key](/access-control#operator-key) is the one HTTP path that re-adopts a fixed `policies.json` without a restart, and a blanket fence with no internal route removes that break-glass too. Under a [path prefix](#path-prefixes), fence `/v1/ops/` instead. This is belt-and-braces, not a requirement — the server's `RequireAdmin` gate remains the authorization boundary and denies non-admin callers on its own — but keeping the management surface off the public vhost removes it from unauthenticated reach entirely. diff --git a/docs/src/content/docs/sdk/index.mdx b/docs/src/content/docs/sdk/index.mdx index 4a234ff2..e779235c 100644 --- a/docs/src/content/docs/sdk/index.mdx +++ b/docs/src/content/docs/sdk/index.mdx @@ -377,7 +377,7 @@ Your headers are applied *underneath* the SDK's own, and a collision means yours Nothing is ever comma-joined: on a collision the SDK's value stands alone, and two of your own entries differing only in case collapse to the last one — a header joined rather than replaced is how you end up sending `Content-Type: application/json, image/png`. -From a **browser**, a cross-origin custom header must also survive CORS preflight, and WaveHouse allow-lists a fixed set (`Accept`, `Authorization`, `Content-Type`, `Last-Event-ID`, `X-Request-ID`, `X-Tenant-ID`) with no config knob. So any other custom header works server-side, or from a browser when the proxy in front owns CORS — which is the same proxy the header is usually for. [`X-Tenant-ID`](/api#tenant-selection) is on the list, so a browser can select a tenant with `headers: { "X-Tenant-ID": "…" }` directly. +From a **browser**, a cross-origin custom header must also survive CORS preflight, and WaveHouse allow-lists a fixed set (`Accept`, `Authorization`, `Content-Type`, `Last-Event-ID`, `X-Request-ID`, `X-Tenant-ID`) with no config knob. So any other custom header works server-side, or from a browser when the proxy in front owns CORS — which is the same proxy the header is usually for. [`X-Tenant-ID`](/deployment#multi-tenant-deployments) is on the list, so a browser can select a tenant with `headers: { "X-Tenant-ID": "…" }` directly. Headers are static. For a credential that rotates per request, wrap the transport with [`options.fetch`](#supplying-your-own-fetch); a callback form is tracked in [#459](https://github.com/Wave-RF/WaveHouse/issues/459). diff --git a/docs/src/content/docs/sdk/reference.md b/docs/src/content/docs/sdk/reference.md index 24bc2447..32a94614 100644 --- a/docs/src/content/docs/sdk/reference.md +++ b/docs/src/content/docs/sdk/reference.md @@ -56,7 +56,7 @@ On REST, `ABORTED` is the one error raised *by* a backoff rather than by an atte On a stream, a retryable failure is re-dialed on a jittered exponential backoff (capped at 30s, and reset only once a connection has held for a few seconds — so a server that accepts and instantly closes still backs off), with the `status` callback moving `reconnecting` → `live`. -Rejected requests surface the real status and message rather than an opaque connection failure — in a browser going cross-origin, though, only when the rejection passes CORS and the gateway answered whatever preflight the request triggers — `Authorization`, configured `headers`, or `Last-Event-ID` once the stream resumes; a rejected preflight or a response without `Access-Control-Allow-Origin` reaches you as a retryable network error instead — indistinguishable from a drop, and retried. Any `4xx` ends the stream, since repeating the request won't usually talk whatever rejected it round — the exception being a `429` or `408` from a fronting rate limiter, which is transient even though the stream still ends, so catch it and open a new one after a delay ([#469](https://github.com/Wave-RF/WaveHouse/issues/469)). Note that **WaveHouse never rejects a stream for authentication**: `/v1/stream` is ungated, so an expired or missing token resolves to `default_role` and you get a `200` with a filtered view, not a `401`. The 4xx it raises itself are `400` for a missing or empty `table` and, when you send `X-Tenant-ID`, the `400`/`404` of [tenant resolution](/api#tenant-selection); any other `404` or `405` means the request never reached that route, most often a `baseURL` path prefix your proxy didn't strip. Any other 4xx comes from something in front — an auth gateway, a proxy. That silent-downgrade behavior is exactly why `auth` is re-read on every connection attempt, and [#239](https://github.com/Wave-RF/WaveHouse/issues/239) tracks enforcing expiry server-side. `SSE_CONNECT_ERROR` and `SSE_NO_STREAM_BODY` are configuration faults, so fix the cause and start a new stream. +Rejected requests surface the real status and message rather than an opaque connection failure — in a browser going cross-origin, though, only when the rejection passes CORS and the gateway answered whatever preflight the request triggers — `Authorization`, configured `headers`, or `Last-Event-ID` once the stream resumes; a rejected preflight or a response without `Access-Control-Allow-Origin` reaches you as a retryable network error instead — indistinguishable from a drop, and retried. Any `4xx` ends the stream, since repeating the request won't usually talk whatever rejected it round — the exception being a `429` or `408` from a fronting rate limiter, which is transient even though the stream still ends, so catch it and open a new one after a delay ([#469](https://github.com/Wave-RF/WaveHouse/issues/469)). Note that **WaveHouse never rejects a stream for authentication**: `/v1/stream` is ungated, so an expired or missing token resolves to `default_role` and you get a `200` with a filtered view, not a `401`. The 4xx it raises itself are `400` for a missing or empty `table` and, when you send `X-Tenant-ID`, the `400`/`404` of [tenant resolution](/deployment#multi-tenant-deployments); any other `404` or `405` means the request never reached that route, most often a `baseURL` path prefix your proxy didn't strip. Any other 4xx comes from something in front — an auth gateway, a proxy. That silent-downgrade behavior is exactly why `auth` is re-read on every connection attempt, and [#239](https://github.com/Wave-RF/WaveHouse/issues/239) tracks enforcing expiry server-side. `SSE_CONNECT_ERROR` and `SSE_NO_STREAM_BODY` are configuration faults, so fix the cause and start a new stream. `SSE_PARSE_ERROR` is the one code that isn't a connection outcome: it's reported and *skipped*, and the connection keeps reading — one bad frame shouldn't cost you the stream. For an ordinary bad frame its `retryable: true` is therefore vestigial — nothing is re-dialed. Two exceptions, one to each half of that reported-and-skipped rule. A frame the SDK can't turn into a row is skipped but never *reported* — `console.warn` and dropped, with no `error` callback: `data` that isn't valid JSON, a row arriving before any `event: schema` frame, a `row` that is valid JSON but not an array, or a row whose length disagrees with the announced column list. Every one of those is **bounded** — three per cause per connection, then one "further occurrences suppressed" line, with the malformed-schema frame below counted as its own cause. So against a server that never announces a schema you get a handful of lines rather than one per row, and a quiet console is **not** evidence the stream is healthy. A malformed schema frame is the one to watch, because it discards the list rather than keeping a stale one — so every row after it is dropped until the next announcement or a reconnect. The parser's 16 MiB buffer cap is reported but not *skipped*: an overflow terminates the parser, so the transport stops reading and reconnects rather than feeding it again. diff --git a/docs/src/content/docs/sdk/streaming.md b/docs/src/content/docs/sdk/streaming.md index 89e64eb8..0c391ccc 100644 --- a/docs/src/content/docs/sdk/streaming.md +++ b/docs/src/content/docs/sdk/streaming.md @@ -127,7 +127,7 @@ The SSE reader and writer changed in the same release. A **new SDK against an ol `@wavehouse/sdk` publishes to npm independently of the server, so pinning the SDK in a frontend while the backend upgrades on its own schedule (or the reverse) is the normal deployment shape. ::: -A `4xx` is terminal and surfaces through `error` with the real status code rather than an opaque connection failure — in a browser going cross-origin, only when the rejection passes CORS and the gateway answered whatever preflight the request triggers (`Authorization`, configured `headers`, or `Last-Event-ID` once the stream resumes); otherwise it arrives as a retryable network error instead — indistinguishable from a drop, and retried. It won't be an *authentication* rejection from WaveHouse, which leaves `/v1/stream` ungated and answers an expired token with a filtered view rather than a `401`; the 4xx it raises itself are `400` for a missing or empty table name and, when you send `X-Tenant-ID`, the `400`/`404` of [tenant resolution](/api#tenant-selection) — any other `404` or `405` means the request never reached the route, usually a `baseURL` path prefix the proxy didn't strip. Anything else means something in front of it (an auth gateway, a proxy) turned the request away — and note the exception to "retrying wouldn't help": a `429` or `408` from a rate limiter *is* transient, but the stream still ends, so catch it and open a new one after a delay ([#469](https://github.com/Wave-RF/WaveHouse/issues/469)). See [Error Handling](/sdk/reference#error-handling) for every code a stream can report and which ones re-dial. +A `4xx` is terminal and surfaces through `error` with the real status code rather than an opaque connection failure — in a browser going cross-origin, only when the rejection passes CORS and the gateway answered whatever preflight the request triggers (`Authorization`, configured `headers`, or `Last-Event-ID` once the stream resumes); otherwise it arrives as a retryable network error instead — indistinguishable from a drop, and retried. It won't be an *authentication* rejection from WaveHouse, which leaves `/v1/stream` ungated and answers an expired token with a filtered view rather than a `401`; the 4xx it raises itself are `400` for a missing or empty table name and, when you send `X-Tenant-ID`, the `400`/`404` of [tenant resolution](/deployment#multi-tenant-deployments) — any other `404` or `405` means the request never reached the route, usually a `baseURL` path prefix the proxy didn't strip. Anything else means something in front of it (an auth gateway, a proxy) turned the request away — and note the exception to "retrying wouldn't help": a `429` or `408` from a rate limiter *is* transient, but the stream still ends, so catch it and open a new one after a delay ([#469](https://github.com/Wave-RF/WaveHouse/issues/469)). See [Error Handling](/sdk/reference#error-handling) for every code a stream can report and which ones re-dial. Streams go through `options.fetch`, `options.headers`, and `options.fetchOptions` like every other request — which is what lets a stream reach a header-gated origin. A custom `fetch` is asked more of on this path; see [Supplying your own fetch](/sdk#supplying-your-own-fetch). diff --git a/internal/api/pipes.go b/internal/api/pipes.go index 1117e722..e1800391 100644 --- a/internal/api/pipes.go +++ b/internal/api/pipes.go @@ -24,12 +24,12 @@ type PipesHandler struct { // Source yields a tenant's pipes (the store itself in production). Source func(*settings.Store) pipes.Source PolicySource PolicySource // resolves empty role to default_role; may be nil - // Tenants resolves the ?tenant= of the admin reads (List, Get): the ops - // tree is tenant-exempt, so they name their tenant rather than carry one. - Tenants *settings.Registry - CHConn driver.Conn - Cache cache.Cache - sf singleflight.Group + // OpsStore is the store the admin reads (List, Get) serve: /v1/ops is + // tenant-exempt, so they carry no request tenant and read the default one. + OpsStore *settings.Store + CHConn driver.Conn + Cache cache.Cache + sf singleflight.Group // queryTimeout bounds each pipe execution, read per request // (chconn.Manager.QueryTimeout in production) so a settings reload // applies without a restart. @@ -47,28 +47,20 @@ func NewPipesHandler(source func(*settings.Store) pipes.Source, policySource Pol return &PipesHandler{Source: source, PolicySource: policySource, CHConn: conn, Cache: c, queryTimeout: queryTimeout} } -// List returns all named queries of the ?tenant= (admin endpoint). -func (h *PipesHandler) List(w http.ResponseWriter, r *http.Request) { - store, ok := opsStore(w, r, h.Tenants) - if !ok { - return - } +// List returns all named queries (admin endpoint). +func (h *PipesHandler) List(w http.ResponseWriter, _ *http.Request) { w.Header().Set("Content-Type", "application/json") - q := h.Source(store).Pipes() + q := h.Source(h.OpsStore).Pipes() if q == nil { q = []*pipes.NamedQuery{} } _ = json.NewEncoder(w).Encode(q) } -// Get returns a specific named query of the ?tenant= (admin endpoint). +// Get returns a specific named query (admin endpoint). func (h *PipesHandler) Get(w http.ResponseWriter, r *http.Request) { - store, ok := opsStore(w, r, h.Tenants) - if !ok { - return - } name := chi.URLParam(r, "name") - q := h.Source(store).Pipe(name) + q := h.Source(h.OpsStore).Pipe(name) if q == nil { writeJSONError(w, http.StatusNotFound, "pipe not found") return diff --git a/internal/api/pipes_test.go b/internal/api/pipes_test.go index a19a8c19..59dc892b 100644 --- a/internal/api/pipes_test.go +++ b/internal/api/pipes_test.go @@ -48,7 +48,7 @@ func TestPipesHandler_List(t *testing.T) { &pipes.NamedQuery{Name: "recent", SQL: "SELECT * FROM clicks ORDER BY ts DESC LIMIT 10"}, ) h := NewPipesHandler(store, nil, nil, nil, noTimeout) - h.Tenants = testTenants() + h.OpsStore = testStore w := httptest.NewRecorder() r := httptest.NewRequestWithContext(context.Background(), http.MethodGet, "/v1/ops/pipes", nil) @@ -66,7 +66,7 @@ func TestPipesHandler_Get_Found(t *testing.T) { &pipes.NamedQuery{Name: "top_pages", SQL: "SELECT page FROM clicks"}, ) h := NewPipesHandler(store, nil, nil, nil, noTimeout) - h.Tenants = testTenants() + h.OpsStore = testStore w := httptest.NewRecorder() r := pipesRequest(t, http.MethodGet, "/v1/ops/pipes/top_pages", "top_pages", nil) @@ -82,7 +82,7 @@ func TestPipesHandler_Get_NotFound(t *testing.T) { t.Parallel() store := staticPipes() h := NewPipesHandler(store, nil, nil, nil, noTimeout) - h.Tenants = testTenants() + h.OpsStore = testStore w := httptest.NewRecorder() r := pipesRequest(t, http.MethodGet, "/v1/ops/pipes/nope", "nope", nil) @@ -97,7 +97,7 @@ func TestPipesHandler_List_Empty(t *testing.T) { t.Parallel() store := staticPipes() h := NewPipesHandler(store, nil, nil, nil, noTimeout) - h.Tenants = testTenants() + h.OpsStore = testStore w := httptest.NewRecorder() r := pipesRequest(t, http.MethodGet, "/v1/ops/pipes", "", nil) diff --git a/internal/api/router_test.go b/internal/api/router_test.go index 398ddb53..f1955755 100644 --- a/internal/api/router_test.go +++ b/internal/api/router_test.go @@ -337,7 +337,7 @@ func TestNewRouter_RoutesRegistered(t *testing.T) { Version: NewVersionHandler("test", "test", "test"), Schema: NewSchemaHandler(reg), DLQ: NewDLQHandler(emb), - Pipes: &PipesHandler{Source: staticPipes(), PolicySource: staticPolicy(&policy.Policy{}), Tenants: testTenants()}, + Pipes: &PipesHandler{Source: staticPipes(), PolicySource: staticPolicy(&policy.Policy{}), OpsStore: testStore}, AuthMW: func(next http.Handler) http.Handler { return next }, PolicySource: policy.Static(&policy.Policy{}), } diff --git a/internal/api/tenant.go b/internal/api/tenant.go index 82678e0e..6900b44a 100644 --- a/internal/api/tenant.go +++ b/internal/api/tenant.go @@ -4,7 +4,6 @@ import ( "context" "log/slog" "net/http" - "net/url" "github.com/Wave-RF/WaveHouse/internal/policy" "github.com/Wave-RF/WaveHouse/internal/settings" @@ -47,67 +46,38 @@ func requestStore(w http.ResponseWriter, r *http.Request) (*settings.Store, bool return store, ok } -// resolveTenant turns the tenant values a request carries — one header line -// or one query value, none meaning tenant.Default — into that tenant's store. -// A malformed id is a 400 and a well-formed id the registry does not hold is -// a 404. A repeated value is refused rather than picked from, so a value a -// proxy sets can never be shadowed by one the client sent. where names the -// source in the error body. -func resolveTenant(w http.ResponseWriter, tenants *settings.Registry, where string, values []string) (*settings.Store, bool) { - id := tenant.Default - if len(values) > 1 { - writeJSONError(w, http.StatusBadRequest, "invalid "+where+": sent more than once") - return nil, false - } - if len(values) == 1 && values[0] != "" { - parsed, err := tenant.Parse(values[0]) - if err != nil { - writeJSONError(w, http.StatusBadRequest, "invalid "+where+": "+err.Error()) - return nil, false - } - id = parsed - } - store, ok := tenants.For(id) - if !ok { - writeJSONError(w, http.StatusNotFound, "unknown tenant: "+id.String()) - return nil, false - } - return store, true -} - -// TenantMW resolves the request's tenant from the tenant.Header before -// authentication runs and stores it in the request context. Every answer, -// the 400 and 404 included, carries Vary: X-Tenant-ID so a shared cache -// cannot replay one tenant's response to another — added, not set, so the -// CORS Vary: Origin survives. +// TenantMW resolves the request's tenant before authentication runs and +// stores it in the request context: the tenant.Header value, tenant.Default +// when absent. A malformed id is a 400 and a well-formed id the registry does +// not hold is a 404. A repeated header is refused rather than picked from, so +// a value a proxy sets can never be shadowed by one the client sent. Every +// answer, the 400 and 404 included, carries Vary: X-Tenant-ID so a shared +// cache cannot replay one tenant's response to another — added, not set, so +// the CORS Vary: Origin survives. func TenantMW(tenants *settings.Registry) func(http.Handler) http.Handler { return func(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Header().Add("Vary", tenant.Header) - store, ok := resolveTenant(w, tenants, tenant.Header, r.Header.Values(tenant.Header)) + id := tenant.Default + values := r.Header.Values(tenant.Header) + if len(values) > 1 { + writeJSONError(w, http.StatusBadRequest, "invalid "+tenant.Header+": sent more than once") + return + } + if len(values) == 1 && values[0] != "" { + parsed, err := tenant.Parse(values[0]) + if err != nil { + writeJSONError(w, http.StatusBadRequest, "invalid "+tenant.Header+": "+err.Error()) + return + } + id = parsed + } + store, ok := tenants.For(id) if !ok { + writeJSONError(w, http.StatusNotFound, "unknown tenant: "+id.String()) return } next.ServeHTTP(w, r.WithContext(WithStore(r.Context(), store))) }) } } - -// opsTenantParam is the query parameter an ops route takes its tenant from. -// The ops tree is tenant-exempt — it runs no TenantMW and ignores the header -// — so an admin names the tenant explicitly, and none means tenant.Default. -const opsTenantParam = "tenant" - -// opsStore resolves the tenant an ops route addresses from its -// opsTenantParam, with the same answers as TenantMW. The query string is -// parsed strictly: url.Values silently drops a malformed pair, which would -// turn "?tenant=acme;x=1" into the default tenant rather than a 400. -func opsStore(w http.ResponseWriter, r *http.Request, tenants *settings.Registry) (*settings.Store, bool) { - where := "?" + opsTenantParam - values, err := url.ParseQuery(r.URL.RawQuery) - if err != nil { - writeJSONError(w, http.StatusBadRequest, "invalid "+where+": malformed query string") - return nil, false - } - return resolveTenant(w, tenants, where, values[opsTenantParam]) -} diff --git a/internal/api/tenant_test.go b/internal/api/tenant_test.go index 95097ec0..4099ba1f 100644 --- a/internal/api/tenant_test.go +++ b/internal/api/tenant_test.go @@ -9,7 +9,6 @@ import ( "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" - "github.com/Wave-RF/WaveHouse/internal/pipes" "github.com/Wave-RF/WaveHouse/internal/policy" "github.com/Wave-RF/WaveHouse/internal/settings" "github.com/Wave-RF/WaveHouse/internal/stream" @@ -164,48 +163,3 @@ func TestNewRouter_TenantExemptRoutes(t *testing.T) { }) } } - -// The ops tree ignores the tenant header, so the admin pipe reads name their -// tenant with ?tenant= instead — with TenantMW's answers. -func TestPipesHandler_AdminReads_TenantParam(t *testing.T) { - t.Parallel() - tests := []struct { - name string - query string - wantStatus int - wantBody string - }{ - {name: "absent resolves to the default tenant", wantStatus: http.StatusOK}, - {name: "empty resolves to the default tenant", query: "?tenant=", wantStatus: http.StatusOK}, - {name: "explicit default tenant", query: "?tenant=0", wantStatus: http.StatusOK}, - {name: "unknown tenant", query: "?tenant=acme", wantStatus: http.StatusNotFound, wantBody: "unknown tenant: acme"}, - {name: "malformed tenant", query: "?tenant=a.b", wantStatus: http.StatusBadRequest, wantBody: "invalid ?tenant"}, - {name: "repeated parameter", query: "?tenant=0&tenant=0", wantStatus: http.StatusBadRequest, wantBody: "sent more than once"}, - // url.Values would drop the malformed pair and default the tenant. - {name: "malformed query string", query: "?tenant=acme;x=1", wantStatus: http.StatusBadRequest, wantBody: "malformed query string"}, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - t.Parallel() - h := NewPipesHandler(staticPipes(&pipes.NamedQuery{Name: "top_pages", SQL: "SELECT 1"}), nil, nil, nil, noTimeout) - h.Tenants = testTenants() - - w := httptest.NewRecorder() - h.List(w, httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/v1/ops/pipes"+tt.query, nil)) - require.Equal(t, tt.wantStatus, w.Code, "List body: %s", w.Body.String()) - if tt.wantBody != "" { - testutil.AssertJSONErrorResponse(t, w) - } - - w = httptest.NewRecorder() - h.Get(w, pipesRequest(t, http.MethodGet, "/v1/ops/pipes/top_pages"+tt.query, "top_pages", nil)) - require.Equal(t, tt.wantStatus, w.Code, "Get body: %s", w.Body.String()) - if tt.wantBody != "" { - testutil.AssertJSONErrorResponse(t, w) - assert.Contains(t, w.Body.String(), tt.wantBody) - } else { - assert.Contains(t, w.Body.String(), `"top_pages"`) - } - }) - } -} diff --git a/internal/app/app_test.go b/internal/app/app_test.go index dec5bbc5..593188c4 100644 --- a/internal/app/app_test.go +++ b/internal/app/app_test.go @@ -174,6 +174,18 @@ func TestNew_TenantHeaderResolvesAgainstTheRegistry(t *testing.T) { } } +// A tenant the registry cannot resolve must not read as "DLQ off": off is what +// lets the ingest worker ack and drop a message it cannot read, so the miss +// parks instead. The other async getters degrade to their zero value. +func TestAsyncGetters_RegistryMiss(t *testing.T) { + t.Parallel() + tenants := settings.NewRegistry(&settings.Store{}) + unknown := tenant.ID("acme") + + assert.True(t, dlqFor(tenants)(unknown, "events"), "an unknown tenant's failed rows park on the DLQ") + assert.Zero(t, perTenant(tenants, (*settings.Store).GapWindow)(unknown)) +} + func TestNew_DedupeFollowsSettings(t *testing.T) { tests := []struct { name string diff --git a/internal/app/wire.go b/internal/app/wire.go index 98cff2ba..0b8ee208 100644 --- a/internal/app/wire.go +++ b/internal/app/wire.go @@ -84,6 +84,21 @@ func perTenant[T any](tenants *settings.Registry, get func(*settings.Store) T) f } } +// dlqFor adapts the registry to the ingest worker's per-table DLQ switch. A +// miss reads as DLQ on, not as the zero value perTenant would give: off lets +// the worker drop a message it cannot read, and not knowing the tenant is no +// reason to destroy its row. Parked, it survives until the tenant resolves. +func dlqFor(tenants *settings.Registry) func(tenant.ID, string) bool { + return func(id tenant.ID, table string) bool { + store, ok := tenants.For(id) + if !ok { + slog.Error("no settings store for tenant; parking its failed rows on the DLQ", "tenant", id, "table", table) + return true + } + return store.DLQFor(table) + } +} + // wireObservability initializes the OTel pipeline whenever either OTLP push // or Prometheus exposition is wanted — Prometheus-only operation // (Alloy/scrape, no collector) is a first-class mode, and the OTel SDK @@ -355,15 +370,7 @@ func (a *App) wireStreaming() { // drain within the shutdown timeout. func (a *App) wireIngestWorker() { a.add(component{name: "ingest worker", run: func(ctx context.Context) error { - dlqEnabled := func(id tenant.ID, table string) bool { - store, ok := a.tenants.For(id) - if !ok { - slog.Error("no settings store for tenant; treating its DLQ as off", "tenant", id, "table", table) - return false - } - return store.DLQFor(table) - } - stop, failed, err := ingest.StartIngestWorker(ctx, a.mq, a.cache, a.ch.Target, tenant.Default, dlqEnabled) + stop, failed, err := ingest.StartIngestWorker(ctx, a.mq, a.cache, a.ch.Target, tenant.Default, dlqFor(a.tenants)) if err != nil { return err } @@ -499,7 +506,7 @@ func (a *App) wireHTTP(authMW func(http.Handler) http.Handler) { streamHandler.Closing = closing pipesHandler := api.NewPipesHandler(func(s *settings.Store) pipes.Source { return s }, (*settings.Store).Policy, a.ch, a.cache, a.ch.QueryTimeout) - pipesHandler.Tenants = a.tenants + pipesHandler.OpsStore = a.store deps := api.Dependencies{ Ingest: ingestHandler, diff --git a/internal/discovery/discovery.go b/internal/discovery/discovery.go index 02ede3d8..867f38b1 100644 --- a/internal/discovery/discovery.go +++ b/internal/discovery/discovery.go @@ -426,13 +426,26 @@ func (sr *SchemaRegistry) RetryRefresh(ctx context.Context, initialBackoff, maxB } } +// unresolvedRefreshInterval paces StartAutoRefresh while its tenant's interval +// cannot be read. +const unresolvedRefreshInterval = time.Minute + // StartAutoRefresh runs a background goroutine that refreshes schemas // at the configured interval. Blocks until ctx is cancelled. The interval is // re-read after every tick, so a changed setting applies from the next cycle // — an in-flight wait finishes at the old cadence rather than resetting, // which keeps a reload from ever deferring an imminent refresh. +// +// A validated setting is at least a second, so a non-positive interval is a +// tenant the settings registry could not resolve, read as the zero value. +// NewTicker and Reset panic on one, so the loop keeps the cadence it has — +// unresolvedRefreshInterval when it has none yet — and picks the setting up +// on the first tick that resolves. func (sr *SchemaRegistry) StartAutoRefresh(ctx context.Context) { interval := sr.refreshInterval(sr.tenant) + if interval <= 0 { + interval = unresolvedRefreshInterval + } ticker := time.NewTicker(interval) defer ticker.Stop() for { @@ -443,7 +456,7 @@ func (sr *SchemaRegistry) StartAutoRefresh(ctx context.Context) { if err := sr.Refresh(ctx); err != nil { slog.ErrorContext(ctx, "schema auto-refresh failed", "error", err) } - if next := sr.refreshInterval(sr.tenant); next != interval { + if next := sr.refreshInterval(sr.tenant); next > 0 && next != interval { interval = next ticker.Reset(interval) } diff --git a/internal/discovery/discovery_test.go b/internal/discovery/discovery_test.go index 26cb363d..a009b953 100644 --- a/internal/discovery/discovery_test.go +++ b/internal/discovery/discovery_test.go @@ -594,6 +594,52 @@ func TestStartAutoRefresh_ExitsOnContextCancel(t *testing.T) { } } +// TestStartAutoRefresh_UnresolvedIntervalDoesNotPanic pins the zero interval a +// settings-registry miss reads as: time.NewTicker and Ticker.Reset panic on a +// non-positive duration, which would take the process down. The loop keeps +// its cadence instead — at boot and when the interval stops resolving mid-run. +func TestStartAutoRefresh_UnresolvedIntervalDoesNotPanic(t *testing.T) { + t.Parallel() + tests := []struct { + name string + intervals []time.Duration // successive reads; the last repeats + }{ + {name: "unresolved at boot", intervals: []time.Duration{0}}, + {name: "stops resolving after a tick", intervals: []time.Duration{5 * time.Millisecond, 0}}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + conn := &fakeConn{errsThenSuccess: []error{errors.New("transient")}} + var reads atomic.Int32 + interval := func(tenant.ID) time.Duration { + i := int(reads.Add(1)) - 1 + return tt.intervals[min(i, len(tt.intervals)-1)] + } + sr := NewSchemaRegistry(conn, func() string { return "test" }, tenant.Default, interval) + + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan struct{}) + go func() { + defer close(done) + assert.NotPanics(t, func() { sr.StartAutoRefresh(ctx) }) + }() + + if len(tt.intervals) > 1 { + // Two ticks: the zero read after the first must not stop the second. + assert.Eventually(t, func() bool { return conn.calls.Load() >= 2 }, + 2*time.Second, 5*time.Millisecond, "the loop stopped ticking after an unresolved read") + } + cancel() + select { + case <-done: + case <-time.After(2 * time.Second): + t.Fatal("StartAutoRefresh did not return after ctx cancel") + } + }) + } +} + // TestStartAutoRefresh_LogsAndContinuesOnError covers the error branch in // StartAutoRefresh's ticker loop: a failed Refresh logs an ERROR line and // the loop keeps going. Operators rely on this so transient ClickHouse From dc0b3a3374676c2f1641fe97f038dd9fb2f15d7b Mon Sep 17 00:00:00 2001 From: taitelee Date: Mon, 21 Sep 2026 10:31:16 -0400 Subject: [PATCH 11/11] docs(tenant): scope the stray-header 404 to routes outside /v1/ops --- docs/src/content/docs/api.md | 2 +- docs/src/content/docs/deployment.md | 2 +- docs/src/content/docs/reverse-proxy.mdx | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index 27ed88ae..e153827b 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -150,7 +150,7 @@ Status code: `503 Service Unavailable` ### `GET /v1/health` — Liveness ping (public, content-free) -Returns **`200 OK` with an empty body** once the gateway is past boot, or **`503 Service Unavailable`** (also empty) while boot-time schema discovery is still failing. Like every other `/v1` route it [resolves a tenant](/deployment#multi-tenant-deployments) first, so a bad `X-Tenant-ID` answers `400`/`404` before the probe runs. No authentication required and no response body — the caller only branches on the status code, so there's nothing to JSON-encode or cache per request. +Returns **`200 OK` with an empty body** once the gateway is past boot, or **`503 Service Unavailable`** (also empty) while boot-time schema discovery is still failing. Like every `/v1` route outside `/v1/ops/*` it [resolves a tenant](/deployment#multi-tenant-deployments) first, so a bad `X-Tenant-ID` answers `400`/`404` before the probe runs. No authentication required and no response body — the caller only branches on the status code, so there's nothing to JSON-encode or cache per request. This is what the SDK's `wh.sys.health()` calls, and the endpoint to use when choosing among multiple servers in a distributed setup. It mirrors `/livez` under the hood but is intentionally a `/v1` API route rather than a Kubernetes probe path: an operator may filter the bare probe paths (`/livez`, `/readyz`, `/healthz`) out at the reverse proxy since they're internal probes, so the SDK relies on `/v1/health`, which is documented public API surface meant to stay reachable. It does **not** ping ClickHouse — readiness-based load balancing is the proxy/LB's job (via `/readyz`), not the client's. diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index 2ac19c79..358e4ac5 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -357,7 +357,7 @@ The probes (`/livez`, `/readyz`, `/healthz`, and the deprecated `/health` and `/ ### Upgrading behind a proxy that already sends `X-Tenant-ID` -`X-Tenant-ID` is a generic name, and some gateways and service meshes stamp one on every request. WaveHouse used to ignore it; now any value other than `0` names an unknown tenant, so **every `/v1` route answers `404 unknown tenant: `** — the SDK's `/v1/health` reachability ping included, while the bare probes stay green. Strip the inbound header at the edge ([header forwarding](/reverse-proxy#header-and-auth-forwarding)) unless you are using it deliberately. +`X-Tenant-ID` is a generic name, and some gateways and service meshes stamp one on every request. WaveHouse used to ignore it; now any value other than `0` names an unknown tenant, so **every `/v1` route outside `/v1/ops/*` answers `404 unknown tenant: `** — the SDK's `/v1/health` reachability ping included, while the bare probes and the admin surface stay green. Strip the inbound header at the edge ([header forwarding](/reverse-proxy#header-and-auth-forwarding)) unless you are using it deliberately. ## ClickHouse Schema diff --git a/docs/src/content/docs/reverse-proxy.mdx b/docs/src/content/docs/reverse-proxy.mdx index 9b07239e..5f425e3b 100644 --- a/docs/src/content/docs/reverse-proxy.mdx +++ b/docs/src/content/docs/reverse-proxy.mdx @@ -194,7 +194,7 @@ These limit the *whole* request regardless of traffic, so no keepalive extends t - **`Authorization`** — forward verbatim. WaveHouse validates a `Bearer` JWT (resolving the role from it) or an `Operator ` [operator credential](/access-control#operator-key). Most proxies forward `Authorization` unchanged, so it's the transport to prefer for the operator key — the exception to check for is an auth-terminating layer that consumes or rewrites the header (e.g. a gateway doing its own auth). - **`X-Operator-Key`** (optional) — only relevant if you present the [operator key](/access-control#operator-key) via this alias header instead of `Authorization: Operator `. Custom request headers are forwarded by default, but confirm your proxy doesn't strip it — and note nginx silently drops header names containing **underscores** (this one uses hyphens, so it's fine as named). The `Authorization` form needs none of this. -- **`X-Tenant-ID`** — **strip it at the edge unless you are deliberately using it.** It selects the [tenant](/deployment#multi-tenant-deployments) (absent means tenant `0`) and any other value is an unknown tenant, so a gateway or mesh that stamps its own `X-Tenant-ID` turns every `/v1` route into a `404` — the SDK's `/v1/health` reachability ping included. It is resolved before authentication and never derived from the token, so a proxy that owns tenant selection must **replace** a client-supplied value, not add to it: two `X-Tenant-ID` lines are refused with `400`. That is a stricter version of the append-vs-replace hazard of `Content-Type` below — here even two agreeing lines are refused. The name is hyphenated, so nginx's underscore rule does not apply. +- **`X-Tenant-ID`** — **strip it at the edge unless you are deliberately using it.** It selects the [tenant](/deployment#multi-tenant-deployments) (absent means tenant `0`) and any other value is an unknown tenant, so a gateway or mesh that stamps its own `X-Tenant-ID` turns every `/v1` route outside `/v1/ops/*` into a `404` — the SDK's `/v1/health` reachability ping included. It is resolved before authentication and never derived from the token, so a proxy that owns tenant selection must **replace** a client-supplied value, not add to it: two `X-Tenant-ID` lines are refused with `400`. That is a stricter version of the append-vs-replace hazard of `Content-Type` below — here even two agreeing lines are refused. The name is hyphenated, so nginx's underscore rule does not apply. - **`X-Forwarded-For` / `X-Forwarded-Proto` / `Host`** — set these for your own logs and any upstream that reads them. WaveHouse does not currently derive a client IP from `X-Forwarded-For` (see the caution below); forwarding it is good hygiene and is what the trusted-proxy client-IP work ([#333](https://github.com/Wave-RF/WaveHouse/issues/333)) will consume. :::caution[Don't expose `:8080` directly]