An extractor that resolves nowhere says where it looked - #174
Merged
Conversation
Found by typing the shipped example's own command line, which was also wrong. A reading directory is listed only if it EXISTS, so on a host where none does the list was empty and the failure read "nor a document in" with nothing after it — no directory named, no hint where the file should go. It now names all three, and says which one the timberfs package owns. The list of what a populated directory holds was the DOCUMENT names, where this lookup takes the FILE stem. Where they differ, printing the document name printed a word that does not resolve. It prints the stem, and says what the document inside is called. Written down in the man page and in tools/README.md too: a provisioning's APPLY names the document, `--extractor` names the file, and keeping them the same is what everything shipped does. The example said `tally --try NAME`; `--try` is a flag and the name goes to `--extractor`. `--etc` claimed `requires = "provision"` and never enforced it — `--try` and `--check` resolve names against it too, which is how a site extractor is tried before anything is provisioned.
torstei
added a commit
that referenced
this pull request
Sep 7, 2026
) * A frame nothing witnesses (plan), and a test that only passed here The plan: a chunk is the frame then its record, data first, so a crash between them leaves a `.trunk` frame no `.rings` record describes. Which of the four crash paths can leave one, why a wal store already recovers its own from the `.sap.seal`, and the on-disk stage marker that makes the other three decidable — a crashed `stage()` and a killed non-wal flush look identical on disk and want opposite treatment. Reproduced against 0.32.0, and the frame is not merely invisible: once a shorter frame is written over part of it, `zstd -dc` fails on the whole trunk, which revokes a documented recovery path while timberfs itself reads the store correctly. The test fix is the same lesson twice in one evening. #174's `a_name_that_resolves_nowhere_says_where_it_looked` DERIVED the reading directories, so it tested a different thing depending on whether the timberfs package was installed — and it passed everywhere until 0.32.0 landed on the machine it was written on. The message builder now takes the directories, so the test states its own world. The timbergraph twin already did this; the Rust one was left behind. * A source witness: one sidecar, recovery and a live edge Where the wal writes the bytes again, a witness records where they are — `{inode, source_offset, buffer_start}`, some two dozen bytes per batch. `file-intake` is the intake that can, because it holds a path. It corrects the plan: the write window of a killed non-wal flush is NOT unrecoverable there. `follow.rs` extracts each line's stamp from the line deterministically, so re-reading the recorded span and re-extracting yields the identical window, and such a frame can be adopted rather than discarded. The same witness serves the unflushed edge from the source file, under the reader's OWN permissions — no helper, no setuid, so it grants nothing `cat` would not — falling back to store-only on a rotated source, a re-syncing intake, or a reader who cannot open it. Which mode answered has to be visible: two operators on one incident, one root and one not, otherwise see different edges and read the difference as data loss. Also recorded: why it is not urgent. The wal's cost and its benefit move together — a busy log's edge is already near-current without one and a quiet log's wal is free — so the only case that needs this is a high-volume log wanting a sub-second edge. And lowering FLUSH_AGE is no substitute at the quiet end: one chunk per line is 56 bytes of index plus a 58-byte frame for a 48-byte line.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by typing the command line the provisioning example shipped in #172 — the
example was wrong, and the error it produced was worse.
The error named nowhere. A reading directory is listed only if it exists,
so on a host where none does the list is empty:
That sentence ends there. No directory named, so nothing tells the reader where
the file was supposed to be. Now:
The "which hold …" list named the wrong thing. It listed the document
names, where this lookup joins
<name>.json— the file stem. The two candiffer on a site's own file, and there the message printed a word that does not
resolve. It prints the stem now, plus the document's name where they differ.
That distinction was nowhere in the docs, so it is in
timberfs(1)andtools/README.mdnow: a provisioning'sAPPLYnames the document,--extractornames the file, and keeping them the same is what everythingshipped does — already enforced for the shipped set by
everything_shipped_carries_the_reserved_prefix.Two smaller ones
packaging/timberfs-tally.conf.examplesaidtimberfs tally --try NAME.--tryis a flag; the name goes to--extractor.--etccarriedrequires = "provision"and never enforced it (verified:--try --extractor … --etc DIRruns).--tryand--checkresolve namesagainst it too, which is exactly how a site extractor is tried before
anything is provisioned — so the attribute was a lie either way. Removed, no
behaviour change.
A test covers both messages.