Repository navigation
feat(task): add the state file format and its writer - #204
Draft
greenhat616 wants to merge 1 commit into
Draft
greenhat616 wants to merge 1 commit into
greenhat616 wants to merge 1 commit into
Conversation
This was referenced Nov 10, 2026
greenhat616
force-pushed
the
feat/hardening-40-state-file
branch
from
November 13, 2026 07:44
a5f5c86 to
4180d6f
Compare
greenhat616
marked this pull request as draft
November 13, 2026 08:55
Add the binary state file of the resume design: a header (magic, version, block size, total, the If-Range validator, the source, and a header CRC32C) and fixed 24-byte records of complete blocks, each with its own CRC32C. StateFile writes it only through the destination's locked handle. It writes the header, appends records at the end it keeps, rolls back to that end, and invalidates, and every step is synced before it counts. It is the real receiver for the checkpoints: a failed append rolls back, and a failed rollback stops them. Checkpoints now keep at least 0.5 s between the end of one and the start of the next; the teardown checkpoint does not wait. Measured with the real state file, that cuts the cost of an unthrottled local download from +58% to +16% (teardown alone is +13%), while slower sources barely change. The 8 MiB / 2 s defaults stay. Nothing in production writes a state file yet: resumable downloads switch it on in the next commit, and reading it back comes with resume.
greenhat616
force-pushed
the
feat/hardening-40-state-file
branch
from
November 13, 2026 22:26
4180d6f to
29412b9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
C33a (T21), the third commit of PR-9d (resume) in the download strategy hardening stack. It is stacked on #203, its base branch. C33 was too large for one reviewable commit, so it is split in two. This half adds the state file format and its writer, with no behaviour change. C33b turns it on for resumable downloads. The design notes are in §11.5.5 and §14.29.
What
The format (
task/instance/state_file.rs, §11.5.5)All fields are little-endian.
magic(BOLTLOAD),version1,flags0;block_size(u32) andtotal(u64);source: a u16 length (0xFFFF= none), then the bytes, at most 4096;header_crc: the CRC32C of every header byte before it.index(u64),len(u32),data_crc(u32),kind1 plus 3 zero bytes, thenrecord_crc, the CRC32C of the 20 bytes before it.The writer
StateFileworks through an injectableStateIo. In production that is the destination's locked handle (C31), whichDestinationLock::handle()now exposes inside the crate.write_headersync_allappendend,sync_all, then moveendroll_backset_len(end), thensync_allinvalidateset_len(0), thensync_allStateFileimplements C32'sCheckpointReceiver, so it is the real checkpoint writer. A failed append rolls back, and a failed rollback stops the checkpoints. The file stays valid up toend.handle()andrelease()say that every state-file operation must end before the lock is released. On Windows,seek_writemoves the cursor shared by all clones of the handle.sync_parent_dirsyncs the parent directory on Unix. Windows has no directory sync, which is a known limitation.Checkpoint schedule
C32 left an open item: check its default schedule before checkpoints are switched on. Re-measured with the real
StateFileas the receiver (release build, mmap writer, 5 runs per group, medians, one Windows machine, medium confidence):Not breaking
No public item changes. The
TaskOptionschange is documentation only. Production behaviour is unchanged, because only tests switch checkpoints on.Tests
endis unchanged, and the next append goes to the same offset.end, and the torn tail is dropped by the decoder.InUse, and release removes the file.sync_parent_dir.StateFileas the receiver. It leaves a valid header plus exactly one record per block, with CRCs that match the temporary file and nothing trailing.endmoved before the sync.Review
Reviewed with
/ccg:review, using the codex backend only (a single-model review).Round 1: PASS (94/100). Its two Minors and its suggestion are applied:
handle()andrelease()state the lifetime rule.C33b adds a gated real-handle test for that rule.
Checks
All on
+nightly-2026-11-06, with--no-fail-fast. Clippy allowsunnecessary_lazy_evaluationsuntil #183 merges.cargo fmt --all -- --check.-D warnings.cargo test --workspace --all-features --locked: 28 suites, 867 passed, 1 ignored (the benchmark).cargo test --workspace --locked: 28 suites, 796 passed, 1 ignored.RUSTDOCFLAGS=-Dwarnings cargo doc --no-deps --all-features -p bolt-load: the same 18 errors as its base. It was run before the round-1 fixes, which changed only docs and a test.