Skip to content

feat(task)!: checksum and sync written blocks at checkpoints - #203

Draft
greenhat616 wants to merge 1 commit into
feat/hardening-38-meta-validatorfrom
feat/hardening-39-checkpoints
Draft

greenhat616 wants to merge 1 commit into
feat/hardening-38-meta-validatorfrom
feat/hardening-39-checkpoints

Conversation

@greenhat616

Copy link
Copy Markdown
Owner

C32 (T21), the second commit of PR-9d (resume) in the download strategy hardening stack. It is stacked on #202, which is its base branch. The design notes are in §11.5.5 (only the checkpoint order and rollback; the state file itself is C33), §11.5.6 (block CRCs), §11.5.13 item 2.2 and §14.28.

Why

A resumable download needs to know which bytes of the partial file are really on disk, and in what state. §11.5.6 does this per 1 MiB block, with a CRC32C that is recorded only after the data is synced. C32 builds that mechanism inside the concurrent engine. C33 connects it to the state file, and C20 reads it back.

What

  • Dependency. crc32c 0.6.8, which provides crc32c_append and crc32c_combine. It has no runtime dependencies. 0.6.9 was published on 2026-11-02, too recently to take now; it can be bumped later. Maintenance facts come from crates.io only.
  • Writer sync. FileRangeWriter::sync is new.
    • Its default implementation returns Unsupported, so a writer that cannot sync never lets unsynced data be recorded.
    • Pool, mmap and compio sync inside their workers, through Command::Sync. Those workers hold the attempt's drain, so an unwind waits for a running sync.
    • The new FileWriterError::Sync maps to IoOp::Flush. A sync failure only causes a warn!.
  • CRCs per write. WriteCompletion.crcs holds the CRC32C of each successful write, split at block boundaries. It is computed off the loop thread, and only when checkpoints are on. Failed and partial writes carry nothing.
  • Per-block segments (BlockCrcs). Segments are kept sorted and never adjacent, and merge with both neighbours through crc32c_combine.
    • Caps: 64 segments per block and 16384 in total. Past a cap, the block is read back from the partial file after the sync.
    • A block counts as complete according to C18's written ranges, not the received ones.
  • Checkpoints (Checkpoints, owned by the download loop). They run one at a time, every 8 MiB or every 2 s, plus once at teardown.
    • Order: take the newly complete blocks, sync the writer, read back what needs it, then hand the blocks to the crate-internal CheckpointReceiver (record, roll_back).
    • Success makes the blocks durable.
    • On failure, roll_back runs and the blocks are retried at the next checkpoint. If the rollback fails as well, checkpointing stops for the task, and the engine stops checksumming and keeping segments. Failures only cause a warn! (Q5).
    • The work runs on a blocking thread and holds the attempt's drain (C31's Drained) on every path: return, panic, or being dropped before it runs, read or unread.
  • Teardown. Teardown waits for a running checkpoint on every outcome. It then records the remaining complete blocks on success, failure and cancellation, but not on ResourceChanged, because that data belongs to the old version.
  • Options. TaskOptions::checkpoint_bytes and checkpoint_interval. They have no effect until C33.
  • No receiver, no cost. Without a receiver, nothing is checksummed, scheduled or synced. C33 provides the receiver (ConcurrentTaskInner::checkpoint_to, crate-internal).
  • Singleton mode is out of scope: §11.5.8 sends On through the concurrent engine in C33.

Breaking change

The public file_writer module, which is public for the benchmarks, changes:

  • WriteCompletion gains the field crcs;
  • FileWriterError, which is not #[non_exhaustive], gains the variant Sync;
  • FileRangeWriter gains sync, with a default implementation.

Code that builds a WriteCompletion or matches FileWriterError exhaustively stops compiling. The benchmarks in this repository are unaffected.

Cost

The benchmark is an ignored release test, checkpoint_bench.rs::checkpoint_overhead. It uses the mmap writer and runs 5 times per group. It was run on one Windows machine, so confidence is medium.

Source Default schedule vs off Sync median / max
local, unthrottled, 256 MiB +54% 5.4 / 11.3 ms
100 MiB/s, 256 MiB −0.4% (noise) 6.1 / 13.9 ms
20 MiB/s, 64 MiB −0.1% (noise) 3.6 / 12.6 ms

The CRC itself runs at about 17 GiB/s. The cost comes from syncing: the mmap worker cannot write while it syncs. §14.28 records an open item for C33: before checkpoints are enabled, check the default schedule against these numbers. The options are a minimum spacing between checkpoints, a larger checkpoint_bytes, an mmap flush of only the dirty ranges, and a measurement on slower disks.

Tests

  • CRCs and segments.
    • Frame splitting.
    • Segment merging, including a property test over any split points and any order.
    • Both caps, and the read-back fallback.
  • The checkpoint state machine.
    • Scheduling by bytes and by time, and serialization.
    • Sync, read-back and record failures, with a retry after each.
    • A failed rollback, after which nothing is kept.
    • The teardown checkpoint.
    • Every backend's sync.
  • The download loop.
    • After hedges, clipped frames, restarts and failed writes, every complete block is recorded once, with the file's CRC, and after a sync.
    • Teardown waits for a running checkpoint (record() held by a gate) on ResourceChanged, Cancelled, a failure and Ok.
    • No CRC after checkpoints stop.
    • No CRC and no sync without a receiver.
  • Drain ordering.
    • The receiver and the reader drop before the drain: on a panic, on a future dropped before it is polled, and on an unread completed result.
    • A pool sync held by a gate keeps the drain open. This test waits 0.5 s to check that the drain stays pending, because idle workers exit asynchronously. It can miss a regression, but it never rejects correct code.
  • A real download of 5 MiB + 123 bytes, whose recorded CRCs match the temporary file.
  • Mutations. 20 were written, and all 19 that change behaviour are caught. The one left uncaught removes only the stop flag, which is redundant once writes are ignored after a stop; §14.28 documents it.

Review

Reviewed with /ccg:review, using the codex backend only (a single-model review).

  1. Round 1 (84). Major: on ResourceChanged, teardown dropped a running checkpoint while its blocking work could still run. Fixed: teardown always settles a running checkpoint, and the work holds the drain. Also fixed:
    • a Minor, state growth after a failed rollback;
    • a suggestion to cover a failed read-back;
    • a suggestion to add throttled benchmarks.
  2. Round 2 (90). Major: the closure's captures had no defined drop order on a panic or when dropped before running. Fixed with Drained<Work>.
  3. Round 3 (91). Major: pool sync ran outside the drain. Fixed by routing it through the queue. The unread-result case also got its own test.
  4. Round 4: PASS (97/100). Its Minor, an unread-result test that could fail under valid scheduling, is fixed with a gate. The test then passed 20 times out of 20.

Checks

All on +nightly-2026-11-06, with --no-fail-fast. Clippy allows unnecessary_lazy_evaluations until #183 merges.

  • cargo fmt --all -- --check.
  • 4 clippy configs with -D warnings.
  • cargo test --workspace --all-features --locked: 28 suites, 853 passed, 1 ignored (the benchmark).
  • cargo test --workspace --locked: 28 suites, 782 passed, 1 ignored.
  • RUSTDOCFLAGS=-Dwarnings cargo doc --no-deps --all-features -p bolt-load: the same 18 errors as its base. It was last run in round 3; round 4 changed only a test and design.md.

@greenhat616
greenhat616 added this pull request to stack #212 November 11, 2026 15:23
@greenhat616
greenhat616 removed this pull request from stack #212 November 11, 2026 15:25
@greenhat616
greenhat616 added this pull request to stack #190 November 11, 2026 15:25
@greenhat616
greenhat616 force-pushed the feat/hardening-39-checkpoints branch from cbdec7a to bf77106 Compare November 13, 2026 07:44
@greenhat616
greenhat616 marked this pull request as draft November 13, 2026 08:55
The concurrent engine can now record, at checkpoints, which 1 MiB
blocks of the partial file are complete, with the CRC32C of each, after
the data is synced. This is the mechanism the resume state file (C33)
builds on; nothing turns it on yet.

Each successful write carries the CRC32C of its bytes, split at block
boundaries, computed off the loop thread. Per block, the segments are
merged with crc32c_combine as writes arrive. A block with too many
segments (64, or 16384 in total) is read back from the file instead.

Checkpoints run one at a time, every 8 MiB or 2 s and once at teardown:
take the newly complete blocks, sync the writer, read back what needs
it, and hand the blocks to a crate-internal receiver. On a failure the
receiver rolls back and the blocks are retried at the next checkpoint;
if the rollback fails too, checkpointing stops for the task. Failures
only warn. Teardown waits for a running checkpoint on every outcome,
and its blocking work holds the attempt's drain, so an unwind waits too.

Every writer can sync its data: pool, mmap and compio through their
drained workers. Without a receiver nothing is checksummed, scheduled
or synced. TaskOptions gains checkpoint_bytes and checkpoint_interval.

BREAKING CHANGE: in the public file_writer module, WriteCompletion
gains the field `crcs`, FileWriterError gains the variant `Sync`, and
FileRangeWriter gains `sync`, with a default implementation that
returns an Unsupported error. Code that builds a WriteCompletion or
matches FileWriterError exhaustively must be updated.
@greenhat616
greenhat616 force-pushed the feat/hardening-39-checkpoints branch from bf77106 to e6fb3d2 Compare November 13, 2026 22:25
@greenhat616
greenhat616 removed this pull request from stack #190 November 13, 2026 22:28
@greenhat616
greenhat616 added this pull request to stack #215 November 13, 2026 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant