feat(mq): a Broker over an external NATS cluster

.github/labeler.yml

@@ -34,6 +34,11 @@
     - any-glob-to-any-file:
       - "internal/cache/**"
 
+"area/coord":
+  - changed-files:
+    - any-glob-to-any-file:
+      - "internal/coord/**"
+
 "area/dedupe":
   - changed-files:
     - any-glob-to-any-file:

.github/workflows/README.md

@@ -44,7 +44,7 @@ Break one of these knowingly or not at all.
 
 2. **A dedicated `coverage` job applies the consolidated gate, polling — not `needs`-ing — the suites.** Each suite (`unit`, `integration`, `e2e`) runs with `COV_DEFER=1` and uploads a `coverage-<suite>` fragment; the `coverage` job runs `make cov` (merge + every threshold gate) over all three — exactly like local `make ci`'s final step. Keeping it a separate job (not folded into e2e's tail) decouples the gate result from the e2e suite's pass/fail. Crucially it is `needs: changes` **only, not the suites**: a `needs` edge is a *scheduling* barrier — GitHub won't pick up a runner, check out, restore caches, or `pnpm install` until the needed jobs finish — so needing the suites would serialize this job's ~50s of setup onto the critical path after the last suite, for nothing (the setup doesn't depend on their results). Instead it starts at run creation, runs its setup in parallel with the suites, and blocks only at the merge by polling for the three fragments with [`scripts/ci/wait-artifact.sh`](../../scripts/ci/wait-artifact.sh) (fails fast if a producer concluded without producing). Tail on the critical path: ~10s, not ~50s. **The aggregator and `docs-deploy` must keep `coverage` *and* every suite in their `needs`** — the suites directly (a suite failure must red the gate even though `coverage` no longer needs them), and `coverage` (else a coverage-gate failure wouldn't block merge or a prod deploy).
 
-3. **e2e builds its own inputs and mirrors local `make test-e2e`.** It compiles the SDK dist + cover binary itself (`make -j test-e2e`, warm per-suffix cache) rather than waiting on a builder job, and runs the suite exactly as a developer does — one orchestrator, one ClickHouse testcontainer, sequential files. The ClickHouse image pulls in the background while caches restore (also in the integration job).
+3. **e2e builds its own inputs and mirrors local `make test-e2e`.** It compiles the SDK dist + cover binary itself (`make -j test-e2e`, warm per-suffix cache) rather than waiting on a builder job, and runs the suite exactly as a developer does — one orchestrator, one ClickHouse and one Redis testcontainer, sequential files. The ClickHouse image pulls in the background while caches restore (also in the integration job).
 
 4. **One change classifier, split into a pure core + a CI wrapper.** The pure allowlist — file list on stdin ⇒ `code`/`docs` — lives in [`scripts/classify-paths.sh`](../../scripts/classify-paths.sh), dependency-free and unit-tested by [`scripts/classify-paths.test.sh`](../../scripts/classify-paths.test.sh) (`make test-classify-paths`, a `verify` leaf) so the allowlists can't silently regress. The `changes` job runs the thin wrapper [`scripts/ci/classify-changes.sh`](../../scripts/ci/classify-changes.sh), which adds the CI-only policy (API file-list fetch + fail-closed: pushes, dispatches, API hiccups ⇒ `code=true`) on top. Keeping the core pure means the local git hooks can share it (`git diff --name-only | scripts/classify-paths.sh`). The `code`/`docs` outputs gate the suites and docs jobs — gate on these, never on workflow-level `paths:` filters, which would orphan the required check (invariant 1).
 

.github/workflows/ci.yml

@@ -280,8 +280,8 @@ jobs:
         with:
           go-cache-suffix: "-e2e-cov"
       # `-j` builds the prereqs (build-ts ∥ build-cover) concurrently,
-      # then runs the orchestrator: ClickHouse testcontainer + the cover
-      # binary + the SDK vitest suite.
+      # then runs the orchestrator: ClickHouse and Redis testcontainers +
+      # the cover binary + the SDK vitest suite.
       - name: Build SDK dist + cover binary, run E2E suite
         run: make -j "$(nproc)" test-e2e COV_DEFER=1
       - name: Upload coverage fragment

.testcoverage.yml

@@ -48,6 +48,15 @@ exclude:
     # HTTP assertions). It is imported only from *_test.go files, never from
     # production code, so there's nothing meaningful to cover.
     - ^internal/testutil/
+    # The coord conformance suite: test helpers every Coordinator's tests
+    # run, imported only from *_test.go like testutil.
+    - ^internal/coord/coordtest/
+    # internal/mq/mqtest/ is the Broker conformance suite: test code that
+    # lives outside *_test.go only so each backend's tests can import it.
+    - ^internal/mq/mqtest/
+    # internal/mq/natstest/ stands up NATS as an operator deploys it, for
+    # tests outside internal/mq; test code, like mqtest.
+    - ^internal/mq/natstest/
     - ^tests/
     # scripts/ holds Go helpers (cov, orchestrator) that drive the build but
     # aren't part of the shipped binary; they show up in `-coverpkg=./...`
@@ -73,11 +82,52 @@ exclude:
       - ^internal/settings/
       - ^cmd/wavehouse/validate\.go$
       - ^cmd/wavehouse/bootstrap\.go$
+      # The DynamoDB dedupe backend: the e2e binary runs Pebble dedupe, so
+      # this file measured 0% there and pulled e2e to 58.6%. The unit
+      # (fake API) and integration (dynamodb-local) suites cover it, and the
+      # merged total still counts it.
+      - ^internal/dedupe/dynamodb\.go$
+      # wireDynamoDedupe and its retry component (internal/app/wire_dynamodb.go):
+      # same reason as dynamodb.go above — the e2e binary never selects
+      # dedupe.backend: dynamodb, so this file measured 0% there and pulled
+      # e2e to 59.7%. The unit and integration suites cover it, and the
+      # merged total still counts it.
+      - ^internal/app/wire_dynamodb\.go$
+      # The in-process cache backend: the e2e stack runs cache.backend=redis
+      # (#613), so the binary carries LocalCache and its version index but e2e
+      # never reaches them. The unit suite and the integration suite's main
+      # app (cache.backend=local) cover them; the merged total still counts them.
+      - ^internal/cache/(local|version_manager)\.go$
+      # What e2e's Redis never makes it run: the cache.redis block's rejection
+      # paths (boot adopts a valid fixture, as with internal/settings above)
+      # and the retry of invalidations the server did not take, which needs an
+      # outage. The unit and integration suites cover both.
+      - ^internal/config/cache_redis\.go$
+      - ^internal/cache/pending\.go$
       # The external-NATS topology spec, verifier and manifest generator
       # (and the `mq manifests` CLI) run against an operator's NATS, which
       # the e2e stack (embedded broker) never has: unit territory, covered
-      # there by a fixture server. Excluding them keeps e2e at ~61%.
+      # there by a fixture server. Excluding them keeps e2e at ~61%. The
+      # external broker is the same, covered by the integration suite.
       - ^internal/mq/nats_topology\.go$
       - ^internal/mq/nats_manifests\.go$
       - ^internal/mq/subject_nats\.go$
+      - ^internal/mq/external\.go$
+      - ^internal/mq/lease\.go$
       - ^cmd/wavehouse/mq\.go$
+      # Their wiring, apart from wire.go for this: the NATS queue and lease
+      # wiring and the retention warning under it.
+      - ^internal/app/wire_nats\.go$
+      # The ops-only listener of a process without the api role, moved
+      # out of wire.go the same way: the e2e binary runs every role.
+      - ^internal/app/wire_ops\.go$
+      # The mq.nats block's checks, as cache_redis.go above: boot adopts the
+      # embedded fixture, so e2e never reads it.
+      - ^internal/config/mq_nats\.go$
+    unit:
+      # The external NATS broker's tests start a server per case, which the
+      # unit suite's 15s per package cannot hold: they are integration-tagged
+      # (make test-integration), and the merged total counts them.
+      - ^internal/mq/external\.go$
+      # The NATS KV leases, the same way.
+      - ^internal/mq/lease\.go$