Skip to content

feat(task): route resumable single-connection tasks and discard partials - #206

Draft
greenhat616 wants to merge 1 commit into
feat/hardening-41-resumable-statefrom
feat/hardening-42-resume-routing
Draft

greenhat616 wants to merge 1 commit into
feat/hardening-41-resumable-statefrom
feat/hardening-42-resume-routing

Conversation

@greenhat616

@greenhat616 greenhat616 commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

C33c (T21), the last part of C33 and the fifth commit of PR-9d (resume) in the download strategy hardening stack. It is stacked on #205 (C33b), which is its base branch. Together with #204 and #205 it completes C33. The design notes are in §11.5.8, §11.5.5 (discard order) and §14.29.3.

What

Routing (§11.5.8)

A task with Resume::On and prefer_mode(Singleton) is routed to the concurrent engine with max_connections = 1 when all of these hold:

  • the header encodes, which needs a usable validator of at most 1024 bytes;
  • the size is known and non-zero;
  • there is no content coding;
  • the adapter supports ranges.

A routed task then follows the full resumable path from C33b, and TaskInfo::mode reports Concurrent. Otherwise the task stays single-connection and is not resumable. The other cases in which the engine picks one connection already fail §11.5.3, so they never route.

discard_partial (crate-internal until C20, a lead decision)

The order:

  1. Take the lock with Unsupported::Fail. Failure gives InUse or Io { Lock }.
  2. Invalidate the state: truncate it, then sync.
  3. Remove the partial file.
  4. Sync the directory.
  5. Remove the state file, sync the directory, then unlock.

The failures:

  • Invalidation fails (the truncate or the sync): the partial file is kept, and the call returns Io { Lock } on the state file.
  • Removing the partial file fails: Io { Remove } on the partial file. IoOp::Remove is new and additive; the enum is #[non_exhaustive].
  • A missing partial file, or both files missing: Ok.
  • A state file that cannot be removed, or a failed directory sync: only a warning (Q6). This is the implementer's policy, the same as at completion; §11.5.5 shows that every resulting combination is crash-safe.

Runner change: release the input before the stop

begin_stop in runner/run_loop.rs now always drops the runner's input before Draining and before Stopped is published. Before, it did so only while connecting. The manager frees the slot on Stopped, so the previous stream owned by the engine is gone before a retry can take the slot.

  • Nothing reads the stream after begin_stop, and the slot accounting (C28, C29, T8) is unchanged.
  • Effect on ordinary concurrent tasks: a stopped runner (hedge loser, restart, split, cancel) closes its stream at the stop, rather than when its worker exits. Every existing test passes unmodified.
  • Review confirmed that frames, hedge arbitration, split clipping, retry offsets, written ranges and CRC segments are unaffected.

Known limitation: ureq2

The ureq2 adapter runs unblock(request) and the body reads (CrossRuntimeStream / blocking::Unblock) on the blocking pool. A call that is already running keeps its response and socket until it returns, even after the stream is dropped. So a retry after an idle timeout or a cancel can briefly hold a second connection. ureq sets no timeouts by default.

  • Who is affected: every concurrent task, not only routed ones. The behaviour already exists at 8d38f5e, and this PR does not extend it.
  • The fix: it needs an adapter teardown-completion signal, so it is left to a follow-up, tracked in ureq2: a dropped stream can keep its connection open, so a retry briefly overlaps it #207. This is a lead decision; review agreed to defer it.
  • Workaround: the adapter README now documents AgentBuilder::timeout_read, which bounds a stalled socket read, and AgentBuilder::timeout, a deadline for the whole request, together with ureq's DNS caveat.

Not breaking

  • Resume and discard_partial stay crate-internal.
  • An Off task chooses its mode exactly as before.
  • The only public change is the additive IoOp::Remove.

Tests

  • Routing.
    • On + Singleton routed: concurrent mode, at most one range stream, new start → checkpoints → completion, and on failure a valid state.
    • Off + Singleton unchanged: no ranges, no file steps.
    • "On that cannot resume equals Off": 6 cases plus content coding and a 1025-byte validator. Each runs as a success and as a failure, and the mode is compared as well.
    • A 1024-byte validator resumes.
    • A unit test of transfer_options checks one connection.
  • Retry overlap. test_a_retry_waits_for_the_old_stream:
    • The old stream's drop is held at a gate, on a 2-worker runtime.
    • No new range is requested for 3 s, past the 500 ms backoff.
    • Peak live streams stays at 1.
    • After the release, the task finishes correctly.
  • discard_partial.
    • Success, in order.
    • A missing partial file, and both files missing.
    • InUse, which touches nothing.
    • A failed truncate, and a failed sync.
    • A failed removal.
    • Later failures, which are only logged.
    • No lock support.
    • In every case, the lock can be taken again.
  • Mutations. C33c: 14 of 14 caught. C33b's 19 still caught.

Review

Reviewed with /ccg:review, using the codex backend only (a single-model review).

  1. Round 1 (88). One Major: a retry could overlap the previous live stream. It was fixed by the runner change above.
    • The Minor (a contradictory step in task-api-spec §4.3) is fixed.
    • Both suggestions, more non-resumable cases and the discard failures, are applied.
  2. Round 2 (91). The runner change was confirmed sound. One Major remained: ureq2's blocking reads outlive a dropped stream. The lead scoped it out as a pre-existing adapter limitation and documented it. The test runtime now has 2 workers.
  3. Round 3: PASS (96/100). It agreed to defer the ureq2 overlap. Its Minor (what timeout_read really bounds) and its suggestion (a precise wording of the guarantee) are applied.

Checks

All on +nightly-2026-11-06, with --no-fail-fast. Clippy allows unnecessary_lazy_evaluations until #183 merges.

  • cargo fmt --all -- --check.
  • 4 clippy configs with -D warnings.
  • cargo test --workspace --all-features --locked: 28 suites, 899 passed, 1 ignored (the benchmark).
  • cargo test --workspace --locked: 28 suites, 828 passed, 1 ignored.
  • RUSTDOCFLAGS=-Dwarnings cargo doc --no-deps --all-features -p bolt-load: the same 18 errors as its base. It was last run in round 1; later rounds changed only tests, design.md and the adapter README.

A task with resume on that prefers one connection now runs on the
concurrent engine with a single connection, so it can keep a state
file. This happens only when it could resume: a usable validator, a
known non-zero size, no content coding, and range support. Otherwise it
stays a single-connection download, exactly as with resume off.

discard_partial, crate-internal until resume is public, takes the
destination lock and runs the discard order: invalidate the state, and
keep the partial file if that fails; remove the partial file, then the
state file. A failed removal of the partial file is reported with the
new IoOp::Remove. Later failures, such as a state file that cannot be
removed or a failed directory sync, are only logged.

A runner now drops its connection or stream when it begins to stop,
before the stop is published, not only while still connecting. So by
the time its slot is free, the stream the engine owned is gone, and a
retry never overlaps it. Nothing reads the stream after the stop, and
the slot accounting is unchanged.

The ureq2 adapter's blocking reads can still outlive a dropped stream
and hold a second connection for a while. That predates this change
and needs an adapter teardown signal; it is documented as a known
limitation.
@greenhat616
greenhat616 force-pushed the feat/hardening-42-resume-routing branch from 8c30a0c to ff090e6 Compare November 13, 2026 22:25
@greenhat616
greenhat616 removed this pull request from stack #190 November 13, 2026 22:28
@greenhat616
greenhat616 added this pull request to stack #215 November 13, 2026 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant