fix(dedupe)!: reserve/commit ids, windowed ingest, retention, dynamodb - #625

Merged

EricAndrechek merged 153 commits into

mainfrom

feat/dedupe-reserve

Sep 26, 2026

Commits on Sep 25, 2026

fix(mq): a failed resize restores the ingest stream's own cap; review fixes
taitelee
committed
db2d20a
fix(app): a rejected tenant keeps its replay history; review fixes
taitelee
committed
07c6a91
fix(mq): share the hub bridge's fetch-ahead across tenants; review fixes
taitelee
committed
04aad9f
fix(mq): publish only into a queue the broker recorded open; review fixes
taitelee
committed
57870c4
docs(mq): a consumer that cannot join a queue opened at runtime; review fixes
taitelee
committed
7cdb794
fix(mq): pace publish-side retries of a queue that cannot open; review fixes
taitelee
committed
e199a03
test(mq): one tenant's failed purge stops no other; review fixes
taitelee
committed
55137d8
docs(mq): size a dead-letter stream the shrink guard kept; review fixes
taitelee
committed
060ca9e
feat(config): choose each layer's implementation at boot

EricAndrechek
and
claude
committed
3022b92
docs(config): say coord.backend is reserved; sync the boot-config lists

EricAndrechek
and
claude
committed
fde17ba
docs(config): no backend has a sub-block yet; index backends.go in AGENTS.md

EricAndrechek
and
claude
committed
f129d57
fix(dedupe)!: reserve, commit or release ids keyed by tenant and table

EricAndrechek
and
claude
committed
58d9e77
fix(dedupe): state what Managed guarantees a backend; default a zero lease

EricAndrechek
and
claude
committed
0ec037e
fix(mq): pace a park's reopen, and warn only on a missing consumer; review fixes
taitelee
committed
9344f1b
fix(dedupe): release only claimed claims on a wrong-count answer

EricAndrechek
and
claude
committed
f484511
Merge remote-tracking branch 'origin/mq-tenant-streams' into feat/dedupe-reserve
EricAndrechek
committed
20c28d1
docs(dedupe): link the old-key sweep to #220 rather than promise it

EricAndrechek
and
claude
committed
386ce74
feat(dedupe): DynamoDB backend, conformance-tested on dynamodb-local

EricAndrechek
and
claude
committed
108499f
fix(ingest): windowed reserve/publish/commit; 503 when dedupe is unavailable

EricAndrechek
and
claude
committed
37ef757
fix(ingest): qualify the duplicate-window claims; steadier tests

EricAndrechek
and
claude
committed
71ab824
fix(dedupe): attempt every Commit and Release chunk; keep the breaker honest

EricAndrechek
and
claude
committed
6f7944a
test(dedupe): make the Dynamo failure tests fail without their fix

EricAndrechek
and
claude
committed
ff047d2
docs(ingest): Release gives back definite failures only; tighten wording

EricAndrechek
and
claude
committed
9824663
docs(ingest): say the Pebble benchmark stubs the queue; finish the rename

EricAndrechek
and
claude
committed
99a5eee
Merge origin/feat/boot-backends into feat/dedupe-dynamodb-wiring

EricAndrechek
and
claude
committed
83a6949
feat(app): choose the DynamoDB dedupe backend at boot

EricAndrechek
and
claude
committed
ced118c
feat(dedupe): retention per tenant and table, and an expiry sweep

EricAndrechek
and
claude
committed
b83e670
fix(app): retry a failed DynamoDB table check; refuse no region

EricAndrechek
and
claude
committed
a8e43bd
test(dedupe): pin that a commit mid-sweep-chunk survives; docs wording

EricAndrechek
and
claude
committed
6c74b65
docs(dedupe): name every region source; untangle the dynamodb check clause

EricAndrechek
and
claude
committed
0b60451
Merge remote-tracking branch 'origin/main' into feat/dedupe-reserve
EricAndrechek
committed
18e0b6c
docs(durability): describe the benchmark machine neutrally

EricAndrechek
and
claude
committed
fc4085b
docs(deployment): use neutral tags in the DynamoDB Terraform example

EricAndrechek
and
claude
committed
02d7056
Merge branch 'feat/dedupe-dynamodb' into feat/dedupe-dynamodb-wiring

EricAndrechek
and
claude
committed
841a2f6
Merge branch 'feat/dedupe-windowed-ingest' into feat/dedupe-retention

EricAndrechek
and
claude
committed
9a7f849
fix(dedupe): length-prefix the key's fields; refuse no table name

EricAndrechek
and
claude
committed
d3032cc
feat(settings): a missing dedupe.retention keeps ids forever

EricAndrechek
and
claude
committed
7b9595e
docs(dedupe): Reserve refuses no key; scope the table-name CHANGELOG line

EricAndrechek
and
claude
committed
12741f8
Merge branch 'feat/dedupe-reserve' into feat/dedupe-windowed-ingest
EricAndrechek
committed
a9ed5b8
Merge branch 'feat/dedupe-windowed-ingest' into feat/dedupe-retention
EricAndrechek
committed
dae1b46
Merge branch 'feat/dedupe-reserve' into feat/dedupe-dynamodb
EricAndrechek
committed
e2d97de
Merge branch 'feat/dedupe-dynamodb' into feat/dedupe-dynamodb-wiring
EricAndrechek
committed
c11133d
docs(settings): name dedupe.retention as the one compiled default

EricAndrechek
and
claude
committed
76b6a9e
docs(settings): the last every-key-required comment; rewrap Store's

EricAndrechek
and
claude
committed
bea9e70
refactor(keyenc): one escaping for subject and cache key tokens

EricAndrechek
and
claude
committed
ce14795
Merge branch 'refactor/keyenc' into feat/dedupe-reserve
EricAndrechek
committed
3a02719
refactor(dedupe): key ids as readable escaped text

EricAndrechek
and
claude
committed
2f11973
Merge branch 'feat/dedupe-reserve' into feat/dedupe-windowed-ingest
EricAndrechek
committed
5b723f0
Merge branch 'feat/dedupe-reserve' into feat/dedupe-dynamodb
EricAndrechek
committed
fed9e04
docs(keyenc): name only the keys built from it today

EricAndrechek
and
claude
committed
51db11c
docs(architecture): keyenc is not yet every composite key's escaping

EricAndrechek
and
claude
committed
27f7608
feat(dedupe): key the DynamoDB table by a String pk

EricAndrechek
and
claude
committed
cfdc06c
Merge branch 'feat/dedupe-windowed-ingest' into feat/dedupe-retention
EricAndrechek
committed
935638e
Merge branch 'feat/dedupe-dynamodb' into feat/dedupe-dynamodb-wiring
EricAndrechek
committed
48a6a77
Merge branch 'refactor/keyenc' into feat/dedupe-reserve
EricAndrechek
committed
7b757ad
docs(keyenc): name the dedupe keys among its users

EricAndrechek
and
claude
committed
78e24aa
test(dedupe): pair / with its escape in the keyspace case; review fixes

EricAndrechek
and
claude
committed
35989e4
Merge branch 'feat/dedupe-reserve' into feat/dedupe-dynamodb
EricAndrechek
committed
9f42d99
Merge branch 'feat/dedupe-reserve' into feat/dedupe-windowed-ingest
EricAndrechek
committed
fc0d509
test(dedupe): keep the table/id boundary pair across '/'

EricAndrechek
and
claude
committed
b299d93
Merge branch 'feat/dedupe-reserve' into feat/dedupe-windowed-ingest
EricAndrechek
committed
ca48065
Merge branch 'feat/dedupe-reserve' into feat/dedupe-dynamodb
EricAndrechek
committed
c4ab354
Merge branch 'feat/dedupe-windowed-ingest' into feat/dedupe-retention
EricAndrechek
committed
c5e4f3b
Merge branch 'feat/dedupe-dynamodb' into feat/dedupe-dynamodb-wiring
EricAndrechek
committed
1526c2b
test(dedupe): a legacy value under a current key reads as absent

EricAndrechek
and
claude
committed
5d77211
fix(dedupe): let a sent DynamoDB put answer before its release

EricAndrechek
and
claude
committed
7082907
docs(changelog): the boot-backends entry predates dedupe.dynamodb

EricAndrechek
and
claude
committed
484c72e
test(dedupe): a caller-cancelled put does not reset the breaker

EricAndrechek
and
claude
committed
4c2c2a4
Merge branch 'feat/dedupe-dynamodb' into feat/dedupe-dynamodb-wiring
EricAndrechek
committed
59b0f0c
Merge remote-tracking branch 'origin/main' into refactor/keyenc
EricAndrechek
committed
673f461
refactor(keyenc): keep '-', join keys, count dead letters per table

EricAndrechek
and
claude
committed
2c31eff
Merge branch 'refactor/keyenc' into feat/dedupe-reserve
EricAndrechek
committed
52a8145
docs(keyenc): pin the %2D compatibility to builds since #612

EricAndrechek
and
claude
committed
999e92c
refactor(dedupe): build keys with keyenc.AppendJoin; keep '-'

EricAndrechek
and
claude
committed
28f3886
Merge branch 'refactor/keyenc' into feat/dedupe-reserve
EricAndrechek
committed
deccbf9
test(mq): cover a byte left unescaped in the lenient-token case

EricAndrechek
and
claude
committed
79ab364
Merge branch 'refactor/keyenc' into feat/dedupe-reserve
EricAndrechek
committed
8794bdf

Commits on Sep 26, 2026

docs(dedupe): '-' is kept in the hashing bound and the keyenc listings

EricAndrechek
and
claude
committed
7335888
test(dedupe): pin a pre-#222 collision and the hashed marker's escaping

EricAndrechek
and
claude
committed
bfebcee
test(dedupe): pin the length half of committedLive; document the lease

EricAndrechek
and
claude
committed
f507a2e
test(dedupe): a failed release keeps the publish's error; drop "valid" keys

EricAndrechek
and
claude
committed
155138d
test(dedupe): cover Status.String

EricAndrechek
and
claude
committed
44e0a69
Merge remote-tracking branch 'origin/main' into feat/dedupe-reserve

EricAndrechek
and
claude
committed
a103ab7
Merge remote-tracking branch 'origin/main' into feat/dedupe-reserve

EricAndrechek
and
claude
committed
6ea5dbe
fix(dedupe): a retried DynamoDB put keeps its own claim

EricAndrechek
and
claude
committed
3248e7e
fix(dedupe): jittered DynamoDB retries that fit the call timeout

EricAndrechek
and
claude
committed
8895863
Merge remote-tracking branch 'origin/main' into feat/dedupe-dynamodb-wiring

EricAndrechek
and
claude
committed
e61307b
fix(dedupe): size the DynamoDB client's idle pool to its fan-out

EricAndrechek
and
claude
committed
4b377cb
test(dedupe): race Reserve against a Commit in progress

EricAndrechek
and
claude
committed
da1079e
fix(dedupe): sweep holds the commit lock only to re-check and delete

EricAndrechek
and
claude
committed
e52b336
docs(settings): name dedupe.retention in the last no-defaults claims

EricAndrechek
and
claude
committed
ffa4dd6
fix(dedupe): retry a throttled DynamoDB commit like unprocessed items

EricAndrechek
and
claude
committed
eb5dea9
fix(config): dedupe defaults in defaults(); refuse an explicit zero

EricAndrechek
and
claude
committed
f40c76a
docs(dedupe): scope the DynamoDB Reserve undo to what it guarantees

EricAndrechek
and
claude
committed
c56103e
fix(config): cap dedupe.lease at 59.5s over the embedded queue

EricAndrechek
and
claude
committed
8b616b4
Merge branch 'feat/dedupe-reserve' into feat/dedupe-dynamodb

EricAndrechek
and
claude
committed
4ac7ec8
docs(dedupe): the DynamoDB key example keeps '-' now

EricAndrechek
and
claude
committed
96b9968
fix(app): the dedupe reload hook makes no DynamoDB call

EricAndrechek
and
claude
committed
a694418
fix(app): say what a failed dedupe table check leads to

EricAndrechek
and
claude
committed
9b8ea3c
Merge branch 'feat/dedupe-reserve' into feat/dedupe-windowed-ingest

EricAndrechek
and
claude
committed
29ddf36
Merge branch 'feat/dedupe-dynamodb' into feat/dedupe-dynamodb-wiring

EricAndrechek
and
claude
committed
56727fb
fix(dedupe): never size the DynamoDB idle pool below the SDK default

EricAndrechek
and
claude
committed
0425a55
docs(dedupe): list the short-circuit counter with the DynamoDB metrics

EricAndrechek
and
claude
committed
5c64ace
test(mq): state the lease/duplicate-window invariant as 2*lease+1s

EricAndrechek
and
claude
committed
860d929
test(mq): pin the takeStock boot fix for a stale duplicate window

EricAndrechek
and
claude
committed
6552478
test(mq): add an idempotency-key case to the Broker conformance suite

EricAndrechek
and
claude
committed
67edd57
fix(app,api): correct a stale comment; don't ERROR-log a client-gone Reserve

EricAndrechek
and
claude
committed
a6b2108
test(dedupe): bound the commit-race case under low GOMAXPROCS

EricAndrechek
and
claude
committed
04da801
docs(api): the uncertain-publish caveat covers every publish failure

EricAndrechek
and
claude
committed
f0f39eb
test(api): cover ErrUnavailable in FailedPublishReleasesTheID

EricAndrechek
and
claude
committed
2c27538
docs(dedupe): reword Reserve's on-error guarantee for network backends

EricAndrechek
and
claude
committed
cf24115
perf(dedupe): skip Managed.Apply's write lock when unchanged

EricAndrechek
and
claude
committed
719a3f8
docs(development): note dedupe's Reserve/Commit/Release in the tree

EricAndrechek
and
claude
committed
aea0600
fix(test): remove the broker store after late consumer writes land

EricAndrechek
and
claude
committed
f177f4d
Merge branch 'feat/dedupe-reserve' into feat/dedupe-dynamodb

EricAndrechek
and
claude
committed
1e7e12f
Merge branch 'feat/dedupe-dynamodb' into feat/dedupe-dynamodb-wiring

EricAndrechek
and
claude
committed
97197f2
Merge branch 'feat/dedupe-reserve' into feat/dedupe-windowed-ingest

EricAndrechek
and
claude
committed
fe44591
docs(agents): name Reserve/Commit/Release in the dedupe tree line

EricAndrechek
and
claude
committed
5c45de2
Merge branch 'feat/dedupe-windowed-ingest' into feat/dedupe-retention

EricAndrechek
and
claude
committed
af6205d
fix(config): cap the embedded-mq dedupe lease by lease+ceil(lease)+1s

EricAndrechek
and
claude
committed
3513823
docs(dedupe): sweep the lease cap, the table check's deadline, and what a reload waits on

EricAndrechek
and
claude
committed
2c42f0a
test(app): a reload must not wait behind a tenant's in-flight DynamoDB call

EricAndrechek
and
claude
committed
d0a61e4
test(cmd): keep the boot test's broker store in storedir

EricAndrechek
and
claude
committed
3235598
test(dedupe): replace the sweep-lock timing assertion with structural ones

EricAndrechek
and
claude
committed
5e1ff2d
test(app): drop the concurrent-Reserve half of the reload/commit test

EricAndrechek
and
claude
committed
b16c193
docs(dedupe): the idle-pool floor, and what a reload's wait covers

EricAndrechek
and
claude
committed
674746a
Merge branch 'fix/ingest-teardown-race' into feat/dedupe-reserve

EricAndrechek
and
claude
committed
b97e4d7
Merge branch 'feat/dedupe-reserve' into feat/dedupe-dynamodb

EricAndrechek
and
claude
committed
167496e
test(api): pin the ERROR/DEBUG split on a Reserve failure by exact msg

EricAndrechek
and
claude
committed
51716a2
Merge branch 'feat/dedupe-dynamodb' into feat/dedupe-dynamodb-wiring

EricAndrechek
and
claude
committed
4424338
Merge branch 'feat/dedupe-reserve' into feat/dedupe-windowed-ingest

EricAndrechek
and
claude
committed
d4d50a3
test(dedupe): pin sweepCandidates' read as unlocked from inside its loop

EricAndrechek
and
claude
committed
c4147ec
test: keep this branch's two broker stores in storedir

EricAndrechek
and
claude
committed
2138068
Merge branch 'feat/dedupe-windowed-ingest' into feat/dedupe-retention

EricAndrechek
and
claude
committed
b691d26
test(dedupe): assert the sweep's read hook ran

EricAndrechek
and
claude
committed
a76bbb9
fix(app): move DynamoDB dedupe wiring out of wire.go for the e2e gate

EricAndrechek
and
claude
committed
7f38b9b
fix(dedupe): a caller's cancel no longer strands a DynamoDB claim

EricAndrechek
and
claude
committed
dc9a331
fix(dedupe): name ErrUnavailable "dedupe store unavailable"

EricAndrechek
and
claude
committed
07e4f77
feat(app): refuse boot only for a misconfigured DynamoDB table in use

EricAndrechek
and
claude
committed
138b166
test(dedupe): order the cancel test's puts after the cancel, not a sleep

EricAndrechek
and
claude
committed
23ac57a
Merge branch 'feat/dedupe-dynamodb' into feat/dedupe-dynamodb-wiring

EricAndrechek
and
claude
committed
ade6f7b
fix(dedupe): classify create_table errors; scope the boot rule in docs

EricAndrechek
and
claude
committed
8ac2fca
docs(deployment): a mismatched key schema follows the misconfiguration rule

EricAndrechek
and
claude
committed
6096ddd
Merge branch 'feat/dedupe-windowed-ingest' into feat/dedupe-reserve

EricAndrechek
and
claude
committed
5a8ef7e
Merge branch 'feat/dedupe-dynamodb' into feat/dedupe-reserve

EricAndrechek
and
claude
committed
d5d35f2
fix(dedupe): reconcile the windowed ingest with the configurable lease

EricAndrechek
and
claude
committed
bb35726
docs(dedupe): retention reaches DynamoDB through TTL, not the sweep

EricAndrechek
and
claude
committed
c361832
Merge origin/main (#614 shared cache) into feat/dedupe-reserve

EricAndrechek
and
claude
committed
34c12f0
docs: name the shared dedupe backend where the merge left it out

EricAndrechek
and
claude
committed
6490203