Skip to content

feat(task): read and check a state file - #208

Draft
greenhat616 wants to merge 1 commit into
feat/hardening-42-resume-routingfrom
feat/hardening-43-resume
Draft

greenhat616 wants to merge 1 commit into
feat/hardening-42-resume-routingfrom
feat/hardening-43-resume

Conversation

@greenhat616

Copy link
Copy Markdown
Owner

C20a (T21), the first of three commits of C20 (resume) and the sixth commit of PR-9d in the download strategy hardening stack. It is stacked on #206 (C33c), which is its base branch. The design notes are in §11.5.5, §11.5.10, §14.30 and §14.30.1.

The implementer proposed, and the lead agreed, to split C20 into three commits:

  • C20a (this PR): read and check a state file. Crate-internal.
  • C20b: the engine resumes from a checked state file. Resume is still internal.
  • C20c: resuming becomes public, with docs and end-to-end tests.

What

Only tests call the new code, so nothing changes behaviour.

Report types (task/resume.rs)

ResumeReport, DroppedRecord, DropReason and RejectReason, as in §11.5.10. They are crate-internal and #[non_exhaustive]. RejectReason::Oversized is new (the implementer's choice): no existing reason fits a file longer than its header allows.

Reader (state_file/read.rs)

read_state reads through the locked handle with positional reads and opens no other handle.

  1. A file shorter than the shortest header (33 bytes) is BadHeader, and nothing is read.
  2. It reads at most the longest header (5153 bytes) and parses it. The header is BadHeader for:
    • a bad magic, version or header CRC;
    • flags ≠ 0;
    • another block size;
    • an unknown validator kind;
    • a validator over 1024 bytes or a source over 4096 bytes;
    • text that is not UTF-8;
    • a cut header;
    • a total of 0 (the implementer's decision).
  3. A file longer than header + (2 × blocks + 1) records is Oversized, and no record is read.
    • A legitimate writer leaves at most one record per block plus a torn tail; the 2× is margin. The derivation is the implementer's, in §14.30.1.
    • There is no absolute cap, because the writer has none: a 3 TiB download can leave about 75 MB of records.
  4. The records are streamed in chunks of 2730 records (65,520 bytes) through one buffer.
    • A torn tail: reported at its start.
    • A bad record CRC: BadRecord with no range.
    • A block past the total, or an index overflow: OutOfBounds.
    • A bad kind, reserved bytes or length: BadRecord with the block's range. Mapping the length to BadRecord is the implementer's decision.
    • Repeats and conflicts (§11.5.5): a repeat of a kept record is dropped silently. On a differing valid record, the kept one, this one and every later record of the block are Conflict.

Checks (resume_check.rs)

  • compare: UnknownSize, ContentCoded, RangeUnsupported, TotalMismatch, SourceMismatch, NoValidator, ValidatorMismatch, in this order.
    • The order (the "not resumable at all" reasons first) is the implementer's decision.
    • C20b compares against the adapter's own retrieve_meta whenever a state file exists, even when the caller passed .meta(...). This is the implementer's reading of §14.27, agreed by the lead.
  • verify: checks cancellation before every block; a block past the partial file's end is BeyondFileLength, without a read; a bad CRC is CrcMismatch. A read error fails the check instead of dropping the block (the implementer's decision). Otherwise a transient or permission error would throw away everything already downloaded.
  • report: merges adjacent kept blocks, totals the kept bytes and sorts the dropped records by position.

StateFile::rewrite

Cut to the header, sync, write the records in batches of at most 2730, then sync. end then points past them. C32's append is unchanged.

Allocation

Everything on the resume path that grows with a state file's record or block count allocates fallibly (try_reserve), and the sorts are unstable and allocation-free. The header buffer, its strings and the chunk buffer have fixed bounds. Engine structures outside C20 are out of scope (a lead decision).

Cached metadata (§14.30)

The adapters cache their probe. So on the same adapter instance, the comparison is against the cached probe. If-Range catches a later change only when the resume still issues a range request. If every block is kept, no request is made. Checking against the current resource therefore needs a fresh adapter. C20c's README will say so.

Not breaking

Every new item is crate-internal. C20c will recheck this against the final public API.

Tests

  • Header: round trips, the exact limits, every bad header, every prefix and every single-bit flip.
  • Records: torn tails of 1–23 bytes, every bad record, out of bounds (including an overflow, and the last block of a u64::MAX total), repeats, six conflict orders, and many records of one block keeping file order.
  • Size:
    • The bound, the bound plus one byte, and a length of u64::MAX.
    • A 3 TiB writer output (3,145,727 records, generated on demand, never written to disk) is accepted, with no read after the header wider than one chunk.
    • A full-length header in 4 KiB reads with Interrupted on every other call.
    • A file shorter than the longest header.
    • Records across a chunk edge, and a torn tail after a full chunk.
  • The locked handle: a read through a real DestinationLock handle.
  • compare, verify and report:
    • every reason;
    • a date in another format;
    • cancellation;
    • a read error;
    • appending to an existing dropped list.
  • rewrite: the step order, batches, an empty rewrite, each step failing, and a failed batch.

Mutations: 51 of 51 caught, among them:

  • the bound or the streaming broken;
  • a conflict misjudged;
  • the sort key without the position;
  • verify's in-place compaction broken;
  • rewrite writing only one batch.

Review

Reviewed with /ccg:review, using the codex backend only (a single-model review).

  1. Round 1 (88). Major: the reader read to EOF before checking the header. It now bounds the file by its header before reading. Minor: the If-Range claim for a fully kept resume was qualified. Suggestion: added the locked-handle test.
  2. Round 2 (89). Major: an absolute 64 MiB cap rejected legitimate files. It was removed, and the records are now streamed. Suggestion: the fragmented-header tests were added.
  3. Round 3 (90). Major: a stable sort allocated without being able to fail. Unstable sorts replaced it. Suggestion: the chunk-edge tests were added.
  4. Round 4 (90). Major: verify, report and rewrite allocated without being able to fail. Now they can fail, and rewrite is batched. The scope rule above is a lead decision. Minor: the wording was corrected. Suggestion: scan is now test-only.
  5. Round 5: PASS (97/100). Its two test suggestions are applied: a record torn inside a rewrite batch, and verify with an existing dropped list.

Checks

All on +nightly-2026-11-06, with --no-fail-fast. Clippy allows unnecessary_lazy_evaluations until #183 merges.

  • cargo fmt --all -- --check.
  • 4 clippy configs with -D warnings.
  • cargo test --workspace --all-features --locked: 28 suites, 934 passed, 1 ignored (the benchmark).
  • cargo test --workspace --locked: 28 suites, 863 passed, 1 ignored.
  • RUSTDOCFLAGS=-Dwarnings cargo doc --no-deps --all-features -p bolt-load: the same 18 errors as its base.

@greenhat616
greenhat616 added this pull request to stack #212 November 11, 2026 15:23
@greenhat616
greenhat616 removed this pull request from stack #212 November 11, 2026 15:25
@greenhat616
greenhat616 added this pull request to stack #190 November 11, 2026 15:25
@greenhat616
greenhat616 force-pushed the feat/hardening-43-resume branch from a9942bb to 1bbc74c Compare November 13, 2026 07:44
@greenhat616
greenhat616 marked this pull request as draft November 13, 2026 08:55
Add the crate-internal pieces that resuming will use, without wiring
them into a task yet, so nothing changes behaviour.

The reader takes the state file through the locked handle. It refuses
a file shorter than any header, reads and checks the header, and
rejects a file longer than its header allows: one record per block
twice over, plus one. Only then does it stream the records in bounded
chunks. A torn tail, a bad record, a record out of bounds and
conflicting records of one block are dropped with a reason, following
the spec's rules.

The checks compare the header with the adapter's metadata, verify each
kept block's CRC against the partial file, and build the report of
what was kept and what was dropped. A block that cannot be read fails
the check rather than being dropped.

StateFile can rewrite itself with the cleaned records: cut to the
header, sync, write the records in bounded batches, sync.

Everything on this path that grows with the number of records
allocates fallibly, so a large state file returns an error instead of
aborting.
@greenhat616
greenhat616 force-pushed the feat/hardening-43-resume branch from 1bbc74c to 461be9b Compare November 13, 2026 22:26
@greenhat616
greenhat616 removed this pull request from stack #190 November 13, 2026 22:28
@greenhat616
greenhat616 added this pull request to stack #215 November 13, 2026 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant