Skip to content

docs(task): design the state file and resume - #200

Draft
greenhat616 wants to merge 1 commit into
test/hardening-35-scanner-tmpfrom
docs/hardening-36-resume-design
Draft

greenhat616 wants to merge 1 commit into
test/hardening-35-scanner-tmpfrom
docs/hardening-36-resume-design

Conversation

@greenhat616

Copy link
Copy Markdown
Owner

The design for T21, resume, the last part of PR-9 in the download strategy hardening stack. This PR changes docs only: no code. It is stacked on #199, which is its base branch. The new section is design §11.5. task-api-spec.md and tasks.md change to match.

Decisions

Maintainer, 2026-11-09 (§3)

  • D15, the API: TaskBuilder::resume(Resume) with Resume { Off (default), On }. The state file is <dest>.boltload, and all its I/O happens in Initializing.
  • D16, layering: the task does the mechanism, and the policy belongs to the caller, as with the restart in Restart a download whose resource changed, above the task engine #195.
    • On resume, the task either resumes or fails with ResumeRejected { reason } without changing either file.
    • Starting over and cleanup belong to a client or job layer.
    • On with no state file, or an empty one, is a fresh start.
  • D17, locking: an exclusive advisory lock on the state file for the whole task, taken by Off tasks too. This fixes 同一目标路径的多个任务共用 .downloading 临时文件 #152. It uses fs4.
  • D18, the format: binary instead of JSON.
    • The header holds a magic, the version, the block size, the total, the selected validator, an optional source and a header CRC.
    • Records are a fixed 24 bytes, each with its own CRC. A torn tail is trimmed to a record boundary.
  • D19, integrity: each 1 MiB block of the partial file has a CRC32C, like a drive or ECC memory. On resume, every kept block is read again and checked. A block that does not match is downloaded again, and the rest is kept.

Threat model

Resume only has to detect when the state file and the partial file disagree, and accidental corruption. Tamper resistance, signing and MACs are explicit non-goals. A future client may sign the state, and the task then reads it unsigned.

Lead (§11.5.12)

  • TaskInfo::resumable tells the caller whether an On task is actually checkpointing.
  • The wait for workers before the lock is released has no bound. Its cost is documented.
  • Off on a filesystem without locks warns and goes on unlocked; On fails.
  • There is no resume feature.
  • A failed checkpoint warns and rolls back. It does not fail the download.

What §11.5 specifies

  • Crash-safe orderings for every path: a fresh start, a checkpoint (D = written \ durable taken before the data sync), the resume writeback, completion (the new phase P), the fallback and discarding. On Unix the parent directory is synced, and the state for a crash between each pair of steps is stated.
  • Retention by phase (A, B1, B2, C, D, P). kept_for_resume is set only for a usable pair. Panics follow the same table (TASK-8).
  • The CRC bookkeeping. CRCs come from the writes as they complete, adjacent segments are merged, and the segment count is capped. Past the cap, a block is read back after the data sync. Only complete blocks are checkpointed.
  • The checks on resume, and the report:
    • ResumeReport and DroppedRecord { position, reason, range };
    • a conflicting block index is rejected for the whole scan;
    • format limits, with checked arithmetic.
  • Single connection: bytes from full_stream are never checkpointed. A one-connection On task runs on the concurrent engine with conditional ranges.
  • Tests:
    • G9: cancel at a random byte and resume, over 20 seeds and 3 clients;
    • corrupted states, CRC mismatches and torn tails;
    • a crash, run as a re-spawned process;
    • the lock across processes;
    • injected failures at each step;
    • a panic in each phase.

Split (§11.5.13 and the tasks.md table)

  1. C31 fix(task)!: lock a destination for the whole task. It fixes 同一目标路径的多个任务共用 .downloading 临时文件 #152.
  2. C34 feat(core): carry the If-Range validator in adapter meta.
  3. C32 feat(task): checksum and sync written blocks at checkpoints, including a bench of the sync and CRC costs.
  4. C33 feat(task): write a state file while downloading. It stays internal and only writes.
  5. C20 feat(task): resume from a checked state file, which adds the public API.

Review

The design was reviewed before any code with /ccg:review, using the codex backend only (a single-model review).

  1. Round 1 (52). It found nine Majors, among them:

    • deleting the live lock file;
    • the reset order;
    • dropped ranges becoming trusted again;
    • torn tails;
    • raw Last-Modified;
    • singleton checkpoints;
    • retention;
    • lock ownership across a panic;
    • same-size modification of the partial.

    The maintainer then revised D16 and D18 and added D19.

  2. Round 2 (80). It found two Majors in completion: the state was not invalidated before it was deleted, and there was no phase after the rename. Both are fixed with phase P.

  3. Round 3: PASS (90/100). Its three Minors and two suggestions are applied: the completion outcomes split per step and for power loss, the writeback phase, TASK-8 following the phase table, the §4.3 initialization order, and the crash-loss wording.

@greenhat616
greenhat616 added this pull request to stack #212 November 11, 2026 15:23
@greenhat616
greenhat616 removed this pull request from stack #212 November 11, 2026 15:25
@greenhat616
greenhat616 added this pull request to stack #190 November 11, 2026 15:25
@greenhat616
greenhat616 force-pushed the docs/hardening-36-resume-design branch from 4c72bc0 to ba270dd Compare November 13, 2026 07:44
@greenhat616
greenhat616 marked this pull request as draft November 13, 2026 08:55
Design §11.5 for T21. The maintainer decided D15 to D19 on 2026-11-09.

- The task does the mechanism and the caller the policy. Resume::On
  resumes or fails with ResumeRejected, and starting over belongs above
  the task.
- An advisory lock on <dest>.boltload covers the whole task, for Off
  too (#152).
- The state file is binary, with a header CRC and fixed-size records
  that each carry a CRC.
- Each 1 MiB block of the partial file has a CRC32C that is checked on
  resume, and a block that fails is downloaded again.

The design also covers:
- crash-safe orderings for every path;
- a retention table by phase;
- the lead's answers to the remaining questions;
- the changes to task-api-spec (STATUS-6, STATUS-12, STATUS-13,
  TASK-8, the initialization sequence);
- the split into C31, C34, C32, C33 and C20.

Tamper resistance is a non-goal.
@greenhat616
greenhat616 force-pushed the docs/hardening-36-resume-design branch from ba270dd to 81a5fa9 Compare November 13, 2026 22:26
@greenhat616
greenhat616 removed this pull request from stack #190 November 13, 2026 22:28
@greenhat616
greenhat616 added this pull request to stack #215 November 13, 2026 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant