A frame nothing witnesses (plan), and a test that only passed here - #176
Merged
Conversation
The plan: a chunk is the frame then its record, data first, so a crash between them leaves a `.trunk` frame no `.rings` record describes. Which of the four crash paths can leave one, why a wal store already recovers its own from the `.sap.seal`, and the on-disk stage marker that makes the other three decidable — a crashed `stage()` and a killed non-wal flush look identical on disk and want opposite treatment. Reproduced against 0.32.0, and the frame is not merely invisible: once a shorter frame is written over part of it, `zstd -dc` fails on the whole trunk, which revokes a documented recovery path while timberfs itself reads the store correctly. The test fix is the same lesson twice in one evening. #174's `a_name_that_resolves_nowhere_says_where_it_looked` DERIVED the reading directories, so it tested a different thing depending on whether the timberfs package was installed — and it passed everywhere until 0.32.0 landed on the machine it was written on. The message builder now takes the directories, so the test states its own world. The timbergraph twin already did this; the Rust one was left behind.
Where the wal writes the bytes again, a witness records where they are —
`{inode, source_offset, buffer_start}`, some two dozen bytes per batch.
`file-intake` is the intake that can, because it holds a path.
It corrects the plan: the write window of a killed non-wal flush is NOT
unrecoverable there. `follow.rs` extracts each line's stamp from the line
deterministically, so re-reading the recorded span and re-extracting
yields the identical window, and such a frame can be adopted rather than
discarded.
The same witness serves the unflushed edge from the source file, under
the reader's OWN permissions — no helper, no setuid, so it grants
nothing `cat` would not — falling back to store-only on a rotated source,
a re-syncing intake, or a reader who cannot open it. Which mode answered
has to be visible: two operators on one incident, one root and one not,
otherwise see different edges and read the difference as data loss.
Also recorded: why it is not urgent. The wal's cost and its benefit move
together — a busy log's edge is already near-current without one and a
quiet log's wal is free — so the only case that needs this is a
high-volume log wanting a sub-second edge. And lowering FLUSH_AGE is no
substitute at the quiet end: one chunk per line is 56 bytes of index
plus a 58-byte frame for a 48-byte line.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two things: the design note for the crash-recovery defect the release-day fuzz
found, and a fix for a test of mine that was measuring the host rather than the
code.
The plan —
docs/plans/frame-witness.mdA chunk reaches disk as two writes: the compressed frame into the
.trunk,then its 56-byte record into the
.rings. Data first, always. A crash betweenthem leaves a frame no record describes.
The note records the append ordering, where the write window actually lives,
and the four cells that matter:
Three are broken and on disk they are indistinguishable — which is the
actual defect. Not "the index disagrees with the trunk", but that the stage path
and the non-wal flush path write a frame with no durable witness of what it is,
where the wal path writes one first (
.sap.seal, holding exactly that frame'sentries with their
wf/wl) and recovers correctly.Adopting is right for a killed flush and wrong for a crashed stage, which
abort_stagewould have discarded — andstage()records its baseline inmemory only, so recovery cannot tell which happened.
Why it is not cosmetic. Left alone, the next append writes at
comp_size;if that frame is shorter than the orphan, the orphan's tail survives spliced
after it and
zstd -dcfails on the whole trunk — revoking a documentedrecovery path (
collapse: stock zstd -dc still recovers the whole survivor)while timberfs itself reads the store correctly and reports nothing.
The proposal: an on-disk stage marker as the one required change, keep adopting
for wal stores, discard-and-announce for non-wal (a chunk needs a write window,
its only copy died with the process, and a fabricated one poisons the index that
chunk selection, age retention and cursors all run on — while the loss itself is
inside the
flush_agecontractsap.rsalready states). Plus one deterministictest per cell, keeping the fuzz that found it.
Also written down: why the non-wal path stays (the wal writes every entry twice
— 1.60 GiB of writes becomes 12.8 GiB on a store compressing 7.0×, measured —
and for the file-tailing intakes the source file is still the durable copy), and
that for those intakes the wal is a visibility switch rather than a
durability one, since
query --followtails the sap.The test fix
a_name_that_resolves_nowhere_says_where_it_looked, from #174, derived thereading directories — so it exercised the empty-list branch only on a host
without the timberfs package. It passed in CI and locally until
0.32.0wasinstalled on the machine it was written on, at which point:
The message construction is now
no_such_extractor(name, dirs, etc)and thetest passes the directories, so it states its own world. Same flaw I fixed in
the timbergraph twin in #173 (
test_a_name_that_resolves_nowhere_says_where_it_lookedreplaces the directory list rather than skipping) — the Rust one was left
behind.
No behaviour change: the messages are byte-identical, only their construction
moved.