Skip to content

A frame nothing witnesses (plan), and a test that only passed here - #176

Merged
torstei merged 2 commits into
mainfrom
plan/frame-witness
Sep 7, 2026
Merged

A frame nothing witnesses (plan), and a test that only passed here#176
torstei merged 2 commits into
mainfrom
plan/frame-witness

Conversation

@torstei

@torstei torstei commented Sep 7, 2026

Copy link
Copy Markdown
Owner

Two things: the design note for the crash-recovery defect the release-day fuzz
found, and a fix for a test of mine that was measuring the host rather than the
code.

The plan — docs/plans/frame-witness.md

A chunk reaches disk as two writes: the compressed frame into the .trunk,
then its 56-byte record into the .rings. Data first, always. A crash between
them leaves a frame no record describes.

The note records the append ordering, where the write window actually lives,
and the four cells that matter:

killed flush crashed stage
wal store recovered, window and all orphan, no witness
non-wal store orphan, window unrecoverable orphan, no witness

Three are broken and on disk they are indistinguishable — which is the
actual defect. Not "the index disagrees with the trunk", but that the stage path
and the non-wal flush path write a frame with no durable witness of what it is,
where the wal path writes one first (.sap.seal, holding exactly that frame's
entries with their wf/wl) and recovers correctly.

Adopting is right for a killed flush and wrong for a crashed stage, which
abort_stage would have discarded — and stage() records its baseline in
memory only, so recovery cannot tell which happened.

Why it is not cosmetic. Left alone, the next append writes at comp_size;
if that frame is shorter than the orphan, the orphan's tail survives spliced
after it and zstd -dc fails on the whole trunk — revoking a documented
recovery path (collapse: stock zstd -dc still recovers the whole survivor)
while timberfs itself reads the store correctly and reports nothing.

The proposal: an on-disk stage marker as the one required change, keep adopting
for wal stores, discard-and-announce for non-wal (a chunk needs a write window,
its only copy died with the process, and a fabricated one poisons the index that
chunk selection, age retention and cursors all run on — while the loss itself is
inside the flush_age contract sap.rs already states). Plus one deterministic
test per cell, keeping the fuzz that found it.

Also written down: why the non-wal path stays (the wal writes every entry twice
— 1.60 GiB of writes becomes 12.8 GiB on a store compressing 7.0×, measured —
and for the file-tailing intakes the source file is still the durable copy), and
that for those intakes the wal is a visibility switch rather than a
durability one, since query --follow tails the sap.

The test fix

a_name_that_resolves_nowhere_says_where_it_looked, from #174, derived the
reading directories — so it exercised the empty-list branch only on a host
without the timberfs package. It passed in CI and locally until 0.32.0 was
installed on the machine it was written on, at which point:

no extractor "nope" — neither a path that exists nor a document in
/usr/lib/timberfs/tally.extractors.d, /tmp/timberfs-tally-…/tally.extractors.d,
which hold timberfs-apache-combined, …, x (the document "t")

The message construction is now no_such_extractor(name, dirs, etc) and the
test passes the directories, so it states its own world. Same flaw I fixed in
the timbergraph twin in #173 (test_a_name_that_resolves_nowhere_says_where_it_looked
replaces the directory list rather than skipping) — the Rust one was left
behind.

No behaviour change: the messages are byte-identical, only their construction
moved.

The plan: a chunk is the frame then its record, data first, so a crash
between them leaves a `.trunk` frame no `.rings` record describes. Which
of the four crash paths can leave one, why a wal store already recovers
its own from the `.sap.seal`, and the on-disk stage marker that makes the
other three decidable — a crashed `stage()` and a killed non-wal flush
look identical on disk and want opposite treatment.

Reproduced against 0.32.0, and the frame is not merely invisible: once a
shorter frame is written over part of it, `zstd -dc` fails on the whole
trunk, which revokes a documented recovery path while timberfs itself
reads the store correctly.

The test fix is the same lesson twice in one evening. #174's
`a_name_that_resolves_nowhere_says_where_it_looked` DERIVED the reading
directories, so it tested a different thing depending on whether the
timberfs package was installed — and it passed everywhere until 0.32.0
landed on the machine it was written on. The message builder now takes
the directories, so the test states its own world. The timbergraph twin
already did this; the Rust one was left behind.
Where the wal writes the bytes again, a witness records where they are —
`{inode, source_offset, buffer_start}`, some two dozen bytes per batch.
`file-intake` is the intake that can, because it holds a path.

It corrects the plan: the write window of a killed non-wal flush is NOT
unrecoverable there. `follow.rs` extracts each line's stamp from the line
deterministically, so re-reading the recorded span and re-extracting
yields the identical window, and such a frame can be adopted rather than
discarded.

The same witness serves the unflushed edge from the source file, under
the reader's OWN permissions — no helper, no setuid, so it grants
nothing `cat` would not — falling back to store-only on a rotated source,
a re-syncing intake, or a reader who cannot open it. Which mode answered
has to be visible: two operators on one incident, one root and one not,
otherwise see different edges and read the difference as data loss.

Also recorded: why it is not urgent. The wal's cost and its benefit move
together — a busy log's edge is already near-current without one and a
quiet log's wal is free — so the only case that needs this is a
high-volume log wanting a sub-second edge. And lowering FLUSH_AGE is no
substitute at the quiet end: one chunk per line is 56 bytes of index
plus a 58-byte frame for a 48-byte line.
@torstei
torstei merged commit 7fab8d4 into main Sep 7, 2026
7 checks passed
@torstei
torstei deleted the plan/frame-witness branch September 7, 2026 06:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant