Skip to content

fix(task)!: lock a destination for the whole task - #201

Draft
greenhat616 wants to merge 1 commit into
docs/hardening-36-resume-designfrom
fix/hardening-37-destination-lock
Draft

greenhat616 wants to merge 1 commit into
docs/hardening-36-resume-designfrom
fix/hardening-37-destination-lock

Conversation

@greenhat616

Copy link
Copy Markdown
Owner

C31 (T21), the first implementation commit of the resume work in the download strategy hardening stack. It is stacked on #200 (the resume design), which is its base branch. The design notes are in §11.5.4, §11.5.8, §11.5.9, §11.5.13 and §14.26, with task-api-spec TASK-8 and §4.3.

Fixes #152.

Why

Two tasks on the same destination shared one .downloading temporary file (#152). Resume also needs one owner per destination: the state file and the partial file must not change under a task. C31 adds that lock first, for every task, before any resume code exists. Every task is Resume::Off until C20.

What

The lock (task/instance/destination_lock.rs)

  • fs4 1.1.0 (default-features = false, features = ["sync"]). FileExt::try_lock is called explicitly: std's try_lock needs Rust 1.89, and the MSRV is 1.88. same-file and libc (unix) become direct dependencies; both were already in the lock file.
  • Opening. <dest>.boltload is opened with create_new, else the existing file, and created is recorded. It retries on NotFound, and on Windows on a sharing violation or a delete-pending file.
    • Windows reports a delete-pending file as ERROR_ACCESS_DENIED. That is told from a real denial by whether the path's metadata can still be read.
    • That guess can be wrong, so if the retries run out on such a denial, the error is Io { Lock }, not InUse.
  • Locking.
    • Contention is DownloadError::InUse { path: dest }.
    • A file system that cannot lock: a warning, and the task goes on unlocked.
    • Any other error: Io { op: Lock }.
    • A failed acquisition never removes the lock file: without the lock, it may be another task's by then.
  • File identity. After locking, same_file::Handle checks that the path still names the locked file. On a mismatch it unlocks and retries, at most 3 times, then reports InUse. An error in the check itself is Io { Lock }, not contention.
  • A stale lock file with content is emptied (set_len(0) and sync_all) through the locked handle, before the temporary file is touched.
  • Release removes the path, then unlocks. It runs after the last operation on the temporary file.

Ownership outside the unwind (task/instance/ownership.rs)

drive::spawn keeps an Ownership outside catch_unwind. It holds the lock, the temporary path, the phase (Unowned, Transfer, Published: the design's A, D, P), the completed progress once published, and a drain WaitGroup.

  • Locking runs through blocking::unblock and is not raced against cancellation. A cancellation seen after the lock discards the owned temporary file, then releases.
  • Normal ends (success, failure, cancellation, fallback): release runs after teardown and before the terminal status. It sets the phase to Unowned and drops the temporary path under the same mutex, then releases the lock. So a later panic cleanup cannot remove a successor's file.
  • Panics. report_panic cancels, waits for the drain without a bound (design Q2), then cleans up by phase:
    • Unowned: nothing;
    • Transfer: the temporary file is removed, or reported in leftover_temp_file;
    • Published: Finished with the stored completed progress, otherwise Failed(Panicked).
  • Writers.
    • Every Pool, Mmap and Compio worker holds a drain clone until it has closed its file.
    • The singleton's async_fs::File handed writes to a background worker nobody could wait for. It is replaced by DrainedFile: each operation is its own blocking task holding the file and a drain clone, and writes no longer buffer.
  • Drop order. Drained<T> { value, _drain } puts a file and its drain clone in one struct, so the file closes before the drain is told on every path: a return, a worker panic, or a queued task dropped without running. Struct fields drop in declaration order; separate closure captures do not have a specified order.
  • StatusPublisher no longer tracks the temporary file; Ownership does.

Breaking change

  • The lock file. Every task creates <dest>.boltload while it runs and removes it when it ends.
  • A destination in use. A second task on it fails with the new DownloadError::InUse.
  • Errors.
    • Failing to create or lock the lock file is DownloadError::Io with the new IoOp::Lock.
    • An unwritable directory used to report Io { Create } on the temporary file; it now reports Io { Lock } on the lock file.
    • Both enums were already #[non_exhaustive].
  • No lock support. On a file system that cannot lock files, the task warns, goes on unlocked, and leaves an empty lock file behind.

The README, task.md, task-api-spec and §14.26 document this.

Tests

  • Unit tests.
    • destination_lock.rs: contention; a stale file; a crash leftover; a file replaced once and every time; unsupported locking; a failed lock that must not remove the holder's file (barrier-ordered); identity errors; the Windows access-denied classification; running out on a denial.
    • ownership.rs: the lock is held until the workers exit; cleanup by phase; cleanup after release touches nothing.
    • drained_file.rs: a dropped in-flight write holds the drain; Drained drop order on a panic and on a closure dropped unrun.
    • file_writer.rs: every writer kind holds the drain until finalize.
    • drive.rs: panics before and after the rename.
  • task/tests/destination_lock.rs. The first task is held behind an explicit gate, with a one-hour stream idle timeout, so nothing depends on timing.
  • tests/destination_lock.rs, across processes. The test binary re-spawns itself as a child that holds the destination until the parent closes its stdin.
    • The parent's task is InUse, bounded at 60 s.
    • After the child is killed, a new task takes over the destination.
  • Mutations: 16 of 16 caught.
    • No lock taken.
    • No file-ID recheck.
    • Unlock before the drain.
    • No stale-state invalidation.
    • The lock file unlinked too early.
    • A failed acquisition removing its file.
    • release keeping the temporary file owned.
    • The singleton write dropping the drain early.
    • Cancellation keeping the temporary file.
    • A published panic reporting the last sample.
    • An identity error taken as another file.
    • Drained's fields swapped.
    • Running out on a denial reported as InUse.
    • The Pool, Mmap and Compio workers each dropping the drain.

Not covered: a real Windows ACL denial and a real delete-pending file. The tests check the classification with synthesized error codes only.

Review

Reviewed with /ccg:review, using the codex backend only (a single-model review).

  1. Round 1 (65).
    • Three Majors:
      • a failed acquisition could unlink another task's lock;
      • released ownership could later delete a successor's temporary file;
      • the singleton's background writes were outside the drain.
    • Minors:
      • cancellation after locking skipped the discard;
      • Windows delete-pending was misclassified;
      • identity-check errors became InUse;
      • a published panic reported the last sample.
    • A suggestion to replace timing with explicit gates.
    • All were fixed.
  2. Round 2 (82).
    • One Major: drop order on a worker panic or an unrun task relied on unspecified capture order. Fixed with Drained<T>.
    • Minors:
      • a persistent access denial could still become InUse;
      • gated holders kept the 30 s idle timeout;
      • the drain test relied on a sleep.
    • A suggestion to document that panic cleanup and release must not run concurrently.
    • All were fixed.
  3. Round 3: PASS (96/100).
    • Its suggestion to poll the drain directly in the concurrent writer test is applied, extended to every writer kind.
    • Its suggestion of real Windows ACL and delete-pending tests is not done; it is listed above as not covered.

Checks

All on +nightly-2026-11-06, with --no-fail-fast. Clippy allows unnecessary_lazy_evaluations until #183 merges.

  • cargo fmt --all -- --check.
  • 4 clippy configs with -D warnings.
  • cargo test --workspace --all-features --locked: 28 suites, 812 passed.
  • cargo test --workspace --locked: 28 suites, 744 passed.
  • RUSTDOCFLAGS=-Dwarnings cargo doc --no-deps --all-features -p bolt-load: the same 18 errors as its base.

@greenhat616
greenhat616 added this pull request to stack #212 November 11, 2026 15:23
@greenhat616
greenhat616 removed this pull request from stack #212 November 11, 2026 15:25
@greenhat616
greenhat616 added this pull request to stack #190 November 11, 2026 15:25
@greenhat616
greenhat616 force-pushed the fix/hardening-37-destination-lock branch from 29a7c03 to c115b91 Compare November 13, 2026 07:44
@greenhat616
greenhat616 marked this pull request as draft November 13, 2026 08:55
Every task now takes an exclusive advisory lock (fs4) on
<dest>.boltload before it touches the temporary file, and holds it
until teardown is done. A second task on the same destination, in this
process or another, fails with DownloadError::InUse instead of sharing
the .downloading file (#152). The lock is checked against the path's
file ID after locking, and a lock file left by an earlier task is
emptied through the locked handle before the download starts.

The lock lives outside the attempt's unwind boundary, with the temporary
file and the attempt's phase. After a panic, the task cancels its
workers, waits until every worker that holds the temporary file has
closed it, cleans up as the phase requires, then releases the lock. The
singleton file is now written on blocking threads that the panic
handler can wait for, and every writer worker drops its file before its
drain clone on every path.

BREAKING CHANGE: every task creates <dest>.boltload while it runs and
removes it when it ends. A second task on a destination in use fails
with the new DownloadError::InUse, and a failure to create or lock the
lock file is DownloadError::Io with the new IoOp::Lock; an unwritable
directory used to report Io with IoOp::Create on the temporary file. On
a file system that cannot lock files, the task warns, goes on unlocked,
and leaves the empty lock file behind.

Closes #152
@greenhat616
greenhat616 force-pushed the fix/hardening-37-destination-lock branch from c115b91 to 4517da6 Compare November 13, 2026 22:26
@greenhat616
greenhat616 removed this pull request from stack #190 November 13, 2026 22:28
@greenhat616
greenhat616 added this pull request to stack #215 November 13, 2026 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant