fix(archiver): authenticate every Inbox message recovery anchor on L1 - #172
Draft
spalladino wants to merge 2 commits into
Draft
spalladino wants to merge 2 commits into
spalladino wants to merge 2 commits into
Conversation
A stored message row keeps the L1 height it was first observed at, and recovery accepted any candidate recorded at or below the persisted finality marker as an anchor with no event lookup. That recorded height is never refreshed while the log agrees with the Inbox at the captured head, so a message re-mined to a higher block keeps its old, lower height — which can sit below the marker while the message itself is above it, unfinalized and replaceable. When such a message is later replaced, recovery keeps a prefix L1 no longer has: it rolls back from N to N, refetches, fails to chain the replacement onto the retained prefix, and restarts on the same false anchor. The node then makes no further progress on messages until it is restarted. Every anchor is now a message a bounded event lookup positively found on L1 at the same index and rolling hash. The per-pass lookup budget, the lookup window bounded above by the captured head, the deployment-block fallback and the single-transaction rollback are unchanged, and no new cache or persisted state is introduced. The finality marker is still written and advanced monotonically on authenticated syncs: what is removed is reliance on stale placement, not finality itself. The cost is that a recovery which used to stop early keeps walking backwards, and may prune proposed blocks whose messages sit below the marker; those messages return through ordinary forward ingestion.
…ad backwards addBlock triggers a sync it does not await, so a fixture that adds local blocks and then moves the L1 head backwards leaves a pass in flight that captured the pre-reorg head. Dropping the finalized-height shortcut made that pass do event lookups and a rollback instead of returning at once, so it can now commit after the pass for the new, lower head and leave the old height as the synced one. Two fixtures hit this. The deployment-refill one failed about once in twelve runs before this change and about three in ten after it; the shorter-chain truncation one only became unstable here, at two in twelve. Draining the pending sync at both sites fixes them, fifteen runs clean.
spalladino
added this pull request to stack #188
September 12, 2026 04:52
spalladino
removed this pull request from stack #188
September 15, 2026 22:42
spalladino
added this pull request to stack #217
September 15, 2026 22:43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Recovery no longer trusts a message's recorded L1 height merely because it is below a finality marker. Every anchor
must match an L1 event at the same index and rolling hash.
Reconstructs old PR #25438 — production commit
882af4bc3d09a9ce226bc187d9d669b91b87b635plus the test-draincorrection
32ad4977c3402acf80fe3649688adfff54e49eed.What this fixes
A stored Inbox message row keeps the L1 height it was first observed at. That height is never refreshed while the
local log still agrees with the Inbox at the captured head, because an agreeing log is never refetched. So a message
re-mined to a higher L1 block keeps its old, lower height, and that stale height can sit below the persisted
finality marker while the message itself sits above it — unfinalized and replaceable.
Recovery used to accept any candidate recorded at or below the marker as an anchor with no event lookup. When such a
message is later replaced, recovery keeps a prefix L1 no longer has: it rolls back from N to N, refetches, fails to
chain the replacement onto the retained prefix, and restarts on the same false anchor. The node then makes no further
progress on messages until it is restarted.
Every anchor is now a message a bounded event lookup positively found on L1 at the same index and rolling hash. A
recorded height is a search hint and never evidence in itself, at or below the marker as much as above it.
What is unchanged
The per-pass lookup budget, the lookup window bounded above by the captured head, the deployment-block fallback and
the single-transaction rollback. No new cache and no new persisted state. The finality marker is still written and
advanced monotonically on authenticated syncs; what is removed is reliance on stale placement, not finality.
Accepted cost
A recovery that used to stop early now keeps walking backwards, and may prune proposed blocks whose messages sit
below the marker. Those messages return through ordinary forward ingestion. This is a liveness cost, not a safety
one, and it is not optimized away here.
The test-drain decision
32ad4977c3adds an inline drain to the deployment-refill fixture; the latercc6d63ac94deletes it again, on thegrounds that the shared local-block helper drains the block-triggered sync itself. That helper-side drain is
cd5d1f41d33b9a5c7370c2c896f45a891199c9ab, which belongs to rung FI-S05, two rungs above this one.I read the destination helpers before deciding.
addLocalBlockistarget.addBlock(block, fake.getInboxPrefixRefFor(block))and
addLocalBlocksConsumingends with a plainforloop overaddLocalBlock— neither drains. So32ad4977c3'sinline drain is ported and
cc6d63ac94is not applied. When FI-S05 landscd5d1f41d3, it should move the draininto
addLocalBlocksConsumingand delete both inline drains, which is whatcc6d63ac94does upstream.Two call sites are drained rather than one.
cd5d1f41d3's own message says "Two fixtures hit this, the second failingabout twice in twelve runs", and that is exactly what measuring showed here (numbers below). The second fixture
(
truncates to a genuinely shorter chain once the syncpoint block is positively replaced) was introduced by rungFI-S01 and only became unstable once this rung dropped the shortcut, so draining it belongs here.
Stack position
This is rung 9 of 25 in the Fast Inbox node stack, stacked on
spl/fi-s10-unused-recovery-state.Tests
Red/green. The new fixture
escapes a recovery whose stored messages sit below the finality marker but were re-mined above itfails against the unchanged synchronizer: the store keeps the stale pair instead of the replacement, and atLOG_LEVEL=infothe log shows the livelock verbatim — three identical cycles ofLocal L1 to L2 messages disagree with the Inbox at L1 block 140; recoveringfollowed immediately byAnchoring L1 to L2 message recovery at finalized L1 block 100, rolling back from N to N and never progressing. Withthe fix it passes.
Run from
yarn-project:yarn build— green.yarn format— clean.yarn lint— clean.JEST_MAX_WORKERS=1 yarn workspace @aztec-labs/archiver test— 23 suites, 679 tests, exit 0 (base: 23 / 678;+1 is the new fixture).
JEST_MAX_WORKERS=1 yarn workspace @aztec-labs/archiver test src/archiver-sync.test.ts, repeated:treats a lookup range entirely above the replacement head as a miss and refills from the deploymentflake (advertised synced L1 block), and no other fixture failed;truncates to a genuinely shorter chain once the syncpoint block is positively replacedfailed 2/12 (110n vs105n) — the same hazard, newly exposed by dropping the shortcut;
E2E suites were not run and are left to CI.
Existing fixtures adjusted
short-circuits rollback at the finalized L1 block→looks up a message recorded at the finalized L1 block before anchoring on it; it now removes every checkpoint-3 message so the search walks back past the finalized height, andasserts exactly one lookup at or below block 100 instead of none.
falls back to per-message log queries when finalized block is undefined→finds the common point with per-message log queries when no block is finalized yet(comment only; behavior unchanged).anchors on the finality marker of the last agreed sync, not on a fresher finalized height→advances the persisted finality marker only once the log agrees with L1 again(comment only; behavior unchanged).Deliberately left out
882af4bc3d's hunk ondiscards a block consuming unchanged messages that were re-mined beyond the lookup window.That fixture does not exist in this repo yet: it is introduced by
4ef9a73073, which belongs to rung FI-S05. FI-S05must add it in its post-N7 form, without the
fake.setFinalizedL1BlockNumber(95n)line that882af4bc3ddeletes.cc6d63ac94— see the decision above.[h-49, h+50]search window. That is the next rung; the lookup window is untouched here.DutyBudget, nounverifiable/sentinel-9 status, noR − Pgossip cutoff, no generic Rollup artifact library map, no generic fee-modifier size fix. The diff was grepped for
all of these and is clean.
spartan/or docs content. Developer and operator documentation for thisbehavior stays in
aztec-packages.