feat(app): choose the DynamoDB dedupe backend at boot

[EricAndrechek]

Part of #613. Part of the remote-dedupe design, stacked on #628 (feat/dedupe-dynamodb). Commits of its own: feat(app): choose the DynamoDB dedupe backend at boot (ced118c), fix(app): retry a failed DynamoDB table check; refuse no region (a8e43bd), docs(dedupe): … (0b60451).

This branch also carries #618's commits (feat/boot-backends), merged in at 83a6949 because this PR extends its dedupe.backend enum and wireDedupe switch. The resolution: wireMQ(ctx) now switches to wireEmbeddedMQ(ctx), because main's ctx-taking wireMQ met #618's switch. In AGENTS.md, CHANGELOG.md and architecture.md, both sides were kept.

What changes

dedupe.backend: dynamodb selects #628's dedupe.Dynamo: one table that every tenant and every process shares. An id ingested through one pod is then a duplicate through every other pod.

key	env	default	notes
dedupe.backend	WH_DEDUPE_BACKEND	pebble	now also dynamodb
dedupe.lease	WH_DEDUPE_LEASE	30s	→ IngestHandler.DedupeLease. It is also the in-flight 503's Retry-After. Refused when lease + ceil(lease) + 1s > 2m while mq.backend=embedded (so at most 59s) — that mirrors the embedded duplicate window from #629 as a local constant. An explicit 0 is refused rather than read as the default.
dedupe.reserve_concurrency	WH_DEDUPE_RESERVE_CONCURRENCY	64	→ DynamoConfig.ReserveConcurrency; Pebble ignores it. Also sizes the HTTP idle-connection pool, never below the SDK default of 10. An explicit 0 is refused. Has no effect yet, since ingest still sends one id per call — #629's windowed ingest changes that.
dedupe.dynamodb.table	WH_DEDUPE_DYNAMODB_TABLE	—	required when selected
dedupe.dynamodb.region	…_REGION	empty = SDK chain
dedupe.dynamodb.endpoint	…_ENDPOINT	empty	dynamodb-local
dedupe.dynamodb.timeout	…_TIMEOUT	250ms	explicit 0 refused; the table check runs under 10× this (2.5s by default)
dedupe.dynamodb.max_attempts	…_MAX_ATTEMPTS	3	explicit 0 refused
dedupe.dynamodb.retry_mode	…_RETRY_MODE	standard	or adaptive; an empty value is refused
dedupe.dynamodb.create_table	…_CREATE_TABLE	false	refused at config load unless endpoint is set

• Config (internal/config/backends.go): DedupeDynamoDB is added to dedupeBackends. The DedupeDynamoDBConfig sub-block is validated only when the backend is selected. The DedupeDynamoDB name is used for both the enum constant and the struct. Negative durations and counts are refused, and the zero-value refusals above match DynamoConfig. NeedsDataDir already returned false for dedupe once the backend is not pebble. There are no credential keys: the SDK default chain supplies credentials.

• Wiring (internal/app/wire.go, wireDynamoDedupe): NewDynamo is built from the block. Boot checks the table (Dynamo.Check, preceded by CreateTable when create_table is on) whether or not any tenant has dedupe on. Boot is the validator, and a DynamoDB check is one cheap call. A failed check follows the registry's rule for the directory shape, the same rule as a Pebble instance that cannot open:

  • a flat directory refuses boot (dedupe open: …);
  • a nested directory boots, and every tenant with dedupe on answers ErrUnavailable (fails closed). A reload applies stores against the last table-check result — it makes no DynamoDB call itself — and wakes the background retry below.

  A nested directory has no settings watcher, so a check that fails at boot is also retried in the background: the component "dedupe table check" starts at 1s and backs off to 30s, then reconciles the stores once the table passes. A mutex keeps its reconcile and the reload hook's from running at once. Without the retry, a throttle or a credential delay during a rolling deploy would leave dedupe'd ingest failing closed until someone sent a reload. A settings reload also no longer waits on any in-flight DynamoDB call: it waits only for a tenant whose store it actually closes (dedupe switched off, or the tenant removed or rejected) — measured at 0.02 s versus 2.5 s against a hung endpoint before this fix.

• No region refuses boot in both shapes (NewDynamo: awsCfg.Region == "", a small change to feat(dedupe): add a DynamoDB backend, tested but not yet wired #628's dynamodb.go). That config error is certain, so it is caught at boot rather than surfacing as a check failure later.

  The gate is a new dedupe.Factory.Gated(ready) in stores.go: a store opens only once ready returns nil. Dynamo.Tenant stays the Factory, as feat(dedupe): add a DynamoDB backend, tested but not yet wired #628 intended. Without the gate, a tenant's open would succeed for free against a missing table. Stores.Close is the component's release. There are no Pebble gauges (dedupeStats nil).

• Per-tenant dedupe.enabled / id_field / require_id stay in each tenant's config.json. Nothing moved.

Tests

• internal/config/backends_test.go: env (every WH_DEDUPE_* variable), YAML (durations parse), defaults, unknown sub-block keys (dedupe.dynamodb.access_key_id, dedupe.redis) refused, unbound-env knows every new variable, and a validation table. The table covers create_table refused without an endpoint, a missing table, retry mode, negatives, the lease at 2m (ok before this round, refused after) and the current cap boundary, and the block ignored under pebble.

• internal/app/dedupe_dynamodb_test.go (a fake DynamoDB JSON endpoint; AWS env pinned):

  • boot checks the table and never creates it without create_table;
  • a claim and its commit reach the table;
  • create_table creates the table and enables TTL;
  • a missing table refuses flat boot whether dedupe is on or off;
  • nested fails closed (ErrUnavailable; a dedupe-off tenant still gets ErrDisabled) and a reload opens the store once the table exists;
  • after the table appears, the background retry opens the store with no reload (TestRun_DynamoDBDedupeRetriesTheTableCheck);
  • an unresolved region refuses boot, flat and nested;
  • a settings reload makes no DynamoDB call and applies against the last check result (TestReload_DynamoDBDedupeMakesNoTableCall), wakes the background retry (TestRun_DynamoDBDedupeReloadWakesTheRetry), and does not wait on an in-flight commit (TestReload_DynamoDBDedupeDoesNotWaitOnInFlightCommit).

  Mutation checks: with the gate removed, the nested case fails; with the retry loop disabled, the retry test fails.

• internal/dedupe/stores_test.go: Gated fails closed, then opens.

• Integration tests/integration/dedupe_dynamodb_app_test.go: two app.New instances, each with its own data_dir, embedded queue and ingest worker, and dedupe.backend: dynamodb over one dynamodb-local table (both created it with create_table, and the second found it already there). An id accepted by pod A is {"duplicate":true} on pod B, and the reverse. A claim held by a third client makes both pods answer 503 with Retry-After: 7, which shows dedupe.lease reached ingest. After release, the id goes through once. ClickHouse ends with exactly one row per id, each from the pod that accepted it.

• The integration suite's 9 DynamoDB tests (Conformance, Check, Expiry, ThirtyTwoClientsOneID, Throttled, Unreachable, CreateTableNeedsEndpoint, ConfigErrorsAreNotUnavailable, TwoInstancesShareSeenIDs) pass against amazon/dynamodb-local:3.3.1.

• internal/app/wire_dynamodb.go is excluded from the e2e coverage gate only (the e2e binary runs Pebble), matching feat(dedupe): add a DynamoDB backend, tested but not yet wired #628's exclusion for dynamodb.go; unit and integration still cover it.

• make ci is green: unit 93.4%, integration 51.9%, e2e coverage 60.3–60.4% against a 60% floor, Go total 94.7%.

  • A couple of runs hit unrelated unit-test flakes in packages this PR does not change: a TempDir cleanup race in internal/ingest's TestStartIngestWorker_EndToEnd, and the pre-existing internal/mq flake in TestEmbeddedNATS_PacesTheRetriesOfAQueueThatCannotOpen (measured failing 7/15 in isolation on the unmodified feat(dedupe): add a DynamoDB backend, tested but not yet wired #628 base, 8/15 on this branch — it comes from obstruct() making directories under the JetStream streams directory while the server is also writing there). Both cleared on re-run.

Deliberately left to later PRs

• fix(ingest): windowed reserve/publish/commit; 503 when dedupe down #629: windowed ingest, 503 on ErrUnavailable, Nats-Msg-Id and mq.EmbeddedDuplicateWindow. This PR's lease cap is a local constant (config.embeddedDuplicateWindow) until fix(ingest): windowed reserve/publish/commit; 503 when dedupe down #629 lands, at which point it should be replaced by mq.EmbeddedDuplicateWindow directly.
• For an external NATS stream, the operator owns the duplicate window; a corresponding boot check for that broker is separate work. This PR applies the cap only while mq.backend=embedded.
• feat(dedupe): retention per tenant and table, and an expiry sweep #633: retention (dedupe.retention, per tenant). Ingest still commits with retention 0.
• Provisioning a production table is out of scope, along with the real-table benchmark. No AWS resource was created; everything ran against dynamodb-local or a fake endpoint.
• dedupe.Factory.Gated works for any remote backend that checks a shared resource once, and the background table-check retry is a plain run component that a future backend (a Redis-backed dedupe store, say) could reuse.

🤖 Generated with Claude Code

https://claude.ai/code/session_01G6Cz4H5k1spJAPk5CZeGds

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: fe49fe32-e9e3-4663-969f-e57e15cc7ead

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}