Skip to content

fix(archiver): authenticate every Inbox message recovery anchor on L1 - #172

Draft
spalladino wants to merge 2 commits into
spl/fi-s10-unused-recovery-statefrom
spl/fi-n7-authenticate-anchors
Draft

spalladino wants to merge 2 commits into
spl/fi-s10-unused-recovery-statefrom
spl/fi-n7-authenticate-anchors

Conversation

@spalladino

@spalladino spalladino commented Sep 12, 2026

Copy link
Copy Markdown
Collaborator

Recovery no longer trusts a message's recorded L1 height merely because it is below a finality marker. Every anchor
must match an L1 event at the same index and rolling hash.

Reconstructs old PR #25438 — production commit 882af4bc3d09a9ce226bc187d9d669b91b87b635 plus the test-drain
correction 32ad4977c3402acf80fe3649688adfff54e49eed.

What this fixes

A stored Inbox message row keeps the L1 height it was first observed at. That height is never refreshed while the
local log still agrees with the Inbox at the captured head, because an agreeing log is never refetched. So a message
re-mined to a higher L1 block keeps its old, lower height, and that stale height can sit below the persisted
finality marker while the message itself sits above it — unfinalized and replaceable.

Recovery used to accept any candidate recorded at or below the marker as an anchor with no event lookup. When such a
message is later replaced, recovery keeps a prefix L1 no longer has: it rolls back from N to N, refetches, fails to
chain the replacement onto the retained prefix, and restarts on the same false anchor. The node then makes no further
progress on messages until it is restarted.

Every anchor is now a message a bounded event lookup positively found on L1 at the same index and rolling hash. A
recorded height is a search hint and never evidence in itself, at or below the marker as much as above it.

What is unchanged

The per-pass lookup budget, the lookup window bounded above by the captured head, the deployment-block fallback and
the single-transaction rollback. No new cache and no new persisted state. The finality marker is still written and
advanced monotonically on authenticated syncs; what is removed is reliance on stale placement, not finality.

Accepted cost

A recovery that used to stop early now keeps walking backwards, and may prune proposed blocks whose messages sit
below the marker. Those messages return through ordinary forward ingestion. This is a liveness cost, not a safety
one, and it is not optimized away here.

The test-drain decision

32ad4977c3 adds an inline drain to the deployment-refill fixture; the later cc6d63ac94 deletes it again, on the
grounds that the shared local-block helper drains the block-triggered sync itself. That helper-side drain is
cd5d1f41d33b9a5c7370c2c896f45a891199c9ab, which belongs to rung FI-S05, two rungs above this one.

I read the destination helpers before deciding. addLocalBlock is target.addBlock(block, fake.getInboxPrefixRefFor(block))
and addLocalBlocksConsuming ends with a plain for loop over addLocalBlockneither drains. So 32ad4977c3's
inline drain is ported and cc6d63ac94 is not applied. When FI-S05 lands cd5d1f41d3, it should move the drain
into addLocalBlocksConsuming and delete both inline drains, which is what cc6d63ac94 does upstream.

Two call sites are drained rather than one. cd5d1f41d3's own message says "Two fixtures hit this, the second failing
about twice in twelve runs", and that is exactly what measuring showed here (numbers below). The second fixture
(truncates to a genuinely shorter chain once the syncpoint block is positively replaced) was introduced by rung
FI-S01 and only became unstable once this rung dropped the shortcut, so draining it belongs here.

Stack position

This is rung 9 of 25 in the Fast Inbox node stack, stacked on spl/fi-s10-unused-recovery-state.

Tests

Red/green. The new fixture escapes a recovery whose stored messages sit below the finality marker but were re-mined above it fails against the unchanged synchronizer: the store keeps the stale pair instead of the replacement, and at
LOG_LEVEL=info the log shows the livelock verbatim — three identical cycles of
Local L1 to L2 messages disagree with the Inbox at L1 block 140; recovering followed immediately by
Anchoring L1 to L2 message recovery at finalized L1 block 100, rolling back from N to N and never progressing. With
the fix it passes.

Run from yarn-project:

  • yarn build — green.
  • yarn format — clean. yarn lint — clean.
  • JEST_MAX_WORKERS=1 yarn workspace @aztec-labs/archiver test — 23 suites, 679 tests, exit 0 (base: 23 / 678;
    +1 is the new fixture).
  • JEST_MAX_WORKERS=1 yarn workspace @aztec-labs/archiver test src/archiver-sync.test.ts, repeated:
    • base, 12 runs: 11 pass, 1 failure — the documented treats a lookup range entirely above the replacement head as a miss and refills from the deployment flake (advertised synced L1 block), and no other fixture failed;
    • this branch with only the deployment-refill drain, 12 runs: that documented flake gone (0/12), but
      truncates to a genuinely shorter chain once the syncpoint block is positively replaced failed 2/12 (110n vs
      105n) — the same hazard, newly exposed by dropping the shortcut;
    • this branch with both drains, 15 runs: 15/15 pass.

E2E suites were not run and are left to CI.

Existing fixtures adjusted

  • short-circuits rollback at the finalized L1 blocklooks up a message recorded at the finalized L1 block before anchoring on it; it now removes every checkpoint-3 message so the search walks back past the finalized height, and
    asserts exactly one lookup at or below block 100 instead of none.
  • falls back to per-message log queries when finalized block is undefinedfinds the common point with per-message log queries when no block is finalized yet (comment only; behavior unchanged).
  • anchors on the finality marker of the last agreed sync, not on a fresher finalized heightadvances the persisted finality marker only once the log agrees with L1 again (comment only; behavior unchanged).

Deliberately left out

  • 882af4bc3d's hunk on discards a block consuming unchanged messages that were re-mined beyond the lookup window.
    That fixture does not exist in this repo yet: it is introduced by 4ef9a73073, which belongs to rung FI-S05. FI-S05
    must add it in its post-N7 form, without the fake.setFinalizedL1BlockNumber(95n) line that 882af4bc3d deletes.
  • cc6d63ac94 — see the decision above.
  • FI-S12's inclusive [h-49, h+50] search window. That is the next rung; the lookup window is untouched here.
  • Nothing from the seven deferred work families; no DutyBudget, no unverifiable/sentinel-9 status, no R − P
    gossip cutoff, no generic Rollup artifact library map, no generic fee-modifier size fix. The diff was grepped for
    all of these and is clean.
  • No L1/Solidity, Noir, barretenberg, spartan/ or docs content. Developer and operator documentation for this
    behavior stays in aztec-packages.

A stored message row keeps the L1 height it was first observed at, and recovery accepted any candidate recorded
at or below the persisted finality marker as an anchor with no event lookup. That recorded height is never
refreshed while the log agrees with the Inbox at the captured head, so a message re-mined to a higher block
keeps its old, lower height — which can sit below the marker while the message itself is above it, unfinalized
and replaceable.

When such a message is later replaced, recovery keeps a prefix L1 no longer has: it rolls back from N to N,
refetches, fails to chain the replacement onto the retained prefix, and restarts on the same false anchor. The
node then makes no further progress on messages until it is restarted.

Every anchor is now a message a bounded event lookup positively found on L1 at the same index and rolling hash.
The per-pass lookup budget, the lookup window bounded above by the captured head, the deployment-block fallback
and the single-transaction rollback are unchanged, and no new cache or persisted state is introduced. The
finality marker is still written and advanced monotonically on authenticated syncs: what is removed is reliance
on stale placement, not finality itself.

The cost is that a recovery which used to stop early keeps walking backwards, and may prune proposed blocks
whose messages sit below the marker; those messages return through ordinary forward ingestion.
…ad backwards

addBlock triggers a sync it does not await, so a fixture that adds local blocks and then moves the L1 head
backwards leaves a pass in flight that captured the pre-reorg head. Dropping the finalized-height shortcut made
that pass do event lookups and a rollback instead of returning at once, so it can now commit after the pass for
the new, lower head and leave the old height as the synced one.

Two fixtures hit this. The deployment-refill one failed about once in twelve runs before this change and about
three in ten after it; the shorter-chain truncation one only became unstable here, at two in twelve. Draining
the pending sync at both sites fixes them, fifteen runs clean.
@spalladino
spalladino added this pull request to stack #188 September 12, 2026 04:52
@spalladino spalladino changed the title spl/fi n7 authenticate anchors fix(archiver): authenticate every Inbox message recovery anchor on L1 Sep 12, 2026
@spalladino spalladino added the ci-draft Run CI on this draft PR label Sep 12, 2026
@spalladino
spalladino removed this pull request from stack #188 September 15, 2026 22:42
@spalladino
spalladino added this pull request to stack #217 September 15, 2026 22:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-draft Run CI on this draft PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant