Skip to content

feat(task): keep a state file for resumable downloads - #205

Draft
greenhat616 wants to merge 1 commit into
feat/hardening-40-state-filefrom
feat/hardening-41-resumable-state
Draft

greenhat616 wants to merge 1 commit into
feat/hardening-40-state-filefrom
feat/hardening-41-resumable-state

Conversation

@greenhat616

Copy link
Copy Markdown
Owner

C33b (T21), the fourth commit of PR-9d (resume) in the download strategy hardening stack. It is stacked on #204 (C33a, the state file format and writer), its base branch.

The implementer proposed, and the lead agreed, to move two parts to a small C33c:

  • routing On with a preferred single connection through the concurrent engine (§11.5.8);
  • discard_partial.

The design notes are in §11.5.3–§11.5.9 and §14.29.2.

What

Resume::{Off, On} stays crate-internal, and tests switch it on. It becomes public in C20, together with reading the state file back. The same holds for kept_for_resume: a lead decision keeps both internal until then.

When a task keeps a state file (§11.5.3)

A task keeps a state file when all of these hold:

  • On;
  • the concurrent mode;
  • a meta.validator (C34);
  • a header that encodes, with a validator of at most 1024 bytes.

Any other On task behaves exactly like Off, apart from the lock. A test compares On and Off field by field for:

  • a preferred single connection;
  • an unknown total;
  • no validator.

Each case runs as a success and as a failure.

On locks with Unsupported::Fail (Q3). Off keeps Proceed.

New start (§11.5.5)

The steps run in Initializing, before info is published:

  1. C31's invalidation of any old state.
  2. Phase B1.
  3. The partial file: truncate, set_len(total), sync_all.
  4. The header, through the locked handle, then sync_all.
  5. Phase C, as soon as the header is synced.
  6. A sync of the parent directory (Unix).
  7. The StateFile becomes C32's checkpoint receiver.
  • Cancellation is checked between completed steps and never raced against a step (C31's rule). In B1 it discards; in C it keeps both files.
  • A failure follows the same rule: B1 discards, C keeps both, with kept_for_resume.

End of task, by phase (§11.5.9)

Outcome Result
Success rename (→ P) → directory sync → empty the state, sync → remove it, directory sync → unlock. Each step after the rename only warns; the result stays Finished.
Failure, cancel, failed rename Both files kept; the state is valid up to the last checkpoint; kept_for_resume = the partial file.
ResourceChanged Discard: invalidate (set_len(0) + sync) → remove the partial file → directory sync → remove the state file → directory sync.
Fallback to one connection Invalidate first, then remove the partial file, then fall back.
Panic (TASK-8) A: nothing. B1: discard. C: keep both. D: remove the temporary file. P: finish the completion steps.

Invalidation failures

An invalidation is two steps, a truncate and a sync, and each can fail on its own. The result is Done, Intact or Unsynced, and a separate keep flag decides which files stay.

  • The truncate fails. The header is intact. A fallback does not happen; the task fails with Io { Lock } and keeps both files with kept_for_resume. A discard keeps the partial file.
  • The truncate succeeds and the sync fails. The state is now empty, and it can come back after a power failure, so this is no usable pair. Both files are kept, the partial file is reported in leftover_temp_file, and kept_for_resume is None.
  • A panic between the two steps. A cut flag is recorded before the sync, so panic cleanup also reports leftover_temp_file, never kept_for_resume.
  • Not falling back when invalidation fails is the implementer's decision. Otherwise the single connection would truncate the partial file under a valid state.

Known limitation

Windows has no directory sync. It relies on NTFS metadata journaling (§11.5.5).

Not breaking

Off takes none of the new steps. It keeps the same lock mode, commit and release, has no checkpoint receiver, and changes no error. A test checks that Off concurrent, Off singleton, On without a validator and On with a single connection take zero file steps and leave only the destination. Every earlier test passes unmodified.

Tests

The tests run real tasks on real files with fault hooks (StepLog), which log every §11.5.5 step and can fail it, gate it or panic at it.

  • The completion order.
  • Failure and cancellation: a valid header plus exactly the checkpointed records, each record's CRC matching its block.
  • ResourceChanged.
  • Each new-start step failing, in B1 and in C.
  • A gated cancellation before and after the header is synced.
  • A failed rename.
  • Each step after the rename failing, checked against §11.5.5's next-On table.
  • The fallback with and without an invalidation failure.
  • Truncate and sync failures of an invalidation, on fallback and on ResourceChanged.
  • A panic at each phase, a panic after the cut, and B1's retried invalidation after a panic.
  • No lock support.
  • A held state write on the real locked handle keeps the lock: a second task gets InUse.
  • On that cannot resume equals Off.

Mutations: 19 of 19 caught. They include:

  • the header written before the partial file is synced;
  • the old state not invalidated first;
  • the partial file deleted when invalidation failed;
  • the start ignoring cancellation;
  • C entered late;
  • an unsynced invalidation reported as resumable;
  • the cut not recorded before a panic.

Review

Reviewed with /ccg:review, using the codex backend only (a single-model review).

  1. Round 1 (82). Three Majors:

    • cancellation was ignored during the start;
    • C was entered after the directory sync rather than after the header sync;
    • a failed fallback invalidation could report an empty state as resumable.

    All three are fixed, together with the suggestion to test panics through real tasks.

  2. Round 2 (90). One Major: a panic between the truncate and the sync of an invalidation. Fixed with the cut flag. The B1 tests asked for in its suggestion are added.

  3. Round 3: PASS (98/100). Its suggestion, a test of B1's retried invalidation after a panic, is applied.

Checks

All on +nightly-2026-11-06, with --no-fail-fast. Clippy allows unnecessary_lazy_evaluations until #183 merges.

  • cargo fmt --all -- --check.
  • 4 clippy configs with -D warnings.
  • cargo test --workspace --all-features --locked: 28 suites, 886 passed, 1 ignored (the benchmark).
  • cargo test --workspace --locked: 28 suites, 815 passed, 1 ignored.
  • The resume and ownership tests: 5 runs in a row, all passed.
  • RUSTDOCFLAGS=-Dwarnings cargo doc --no-deps --all-features -p bolt-load: the same 18 errors as its base. It was last run in round 2; round 3 changed only a test and design.md.

A task with resume on, which stays crate-internal until resume reads
the file back, now keeps the state file of C33a next to its partial
file whenever it can resume: concurrent mode, a usable If-Range
validator, and a header that encodes. It needs the destination lock,
and fails where the file system cannot lock files.

A new start invalidates any old state, creates, sizes and syncs the
partial file, writes and syncs the header, then syncs the directory.
Once the header is synced the pair is resumable. Cancellation is
checked between these steps, never raced against one.

How the task ends depends on its phase:
- Success renames the partial file, then empties, removes and syncs
  the state file. A failure after the rename only warns, and the task
  stays Finished.
- A failure, a cancellation, or a failed rename keeps both files and
  reports the partial file as kept for resume.
- A changed resource discards both files. The state is invalidated
  first; if that fails, the partial file is kept as a leftover.
- A fallback to one connection invalidates the state first, and does
  not fall back if that fails.
- A panic cleans up by phase. A state already cut, but not yet synced,
  is never reported as resumable.

Tasks with resume off, and tasks that cannot resume, take none of these
steps and end exactly as before.
@greenhat616
greenhat616 force-pushed the feat/hardening-41-resumable-state branch from aee9caa to 52ee08a Compare November 13, 2026 22:25
@greenhat616
greenhat616 removed this pull request from stack #190 November 13, 2026 22:28
@greenhat616
greenhat616 added this pull request to stack #215 November 13, 2026 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant