Skip to content

feat(task): restart silent runners under the doubling strategy - #191

Draft
greenhat616 wants to merge 1 commit into
feat/hardening-28-doubling-capfrom
feat/hardening-29-silent-restart
Draft

greenhat616 wants to merge 1 commit into
feat/hardening-28-doubling-capfrom
feat/hardening-29-silent-restart

Conversation

@greenhat616

Copy link
Copy Markdown
Owner

C29 (T29, PR-7d) of the download strategy hardening stack. It is stacked on #189, the doubling cap, and that branch is its base. The design notes are in §14.19.

Why

T14 (#188, §14.17) found that Doubling waited out the 30 s idle timeout when a runner stalled with plenty of work left (s4b, s4c, about 38 s), and that a stalled runner froze growth through check 5. T13's slow path needs three full windows and three runners, so it never fires in these shapes.

The maintainer decided on 2026-11-08 to add a restart path for silent runners, and later that a slow or silent restart releases a doubling held at the cap (C28).

What

Read activity (runner/mod.rs, run_loop/stream.rs)

  • ReadActivity is shared by the runner and the loop: the last read Instant and an "emitting" flag, behind one mutex.
  • The runner stamps it on each non-empty item. It sets emitting while a push waits on the data channel. When it resumes reading the same item, resume_reading(at) stamps the read and clears emitting under one lock. Terminal pushes only clear emitting.
  • RunnerManager::silent_since is the runner's last read, or its spawn, or now while it is emitting.
  • Accepted frames and writer recovery no longer count. The hedge's last_delivery (§9.6) is unchanged.

Silent path (download_loop/restart.rs), Doubling only

A runner is stopped as Slow when all of these hold:

  • It is Running, not stopping, and not in an undecided pair (may_restart, unchanged from C12).
  • It has at least 2 × min_chunk_size left, so it never competes with the endgame hedge.
  • No frames are queued in its data channel.
  • It has been silent for 4.5 s (SILENT_RESTART_AFTER), and it was already running at a sample 4.5 s ago.
  • The writer is not full and the loop is not congested.

The longest-silent runner goes first, the lower ID on a tie. At most one runner is stopped per tick, and the slow path goes first. The stop is charged like a slow stop: a restart if it delivered data, a charged attempt if not.

Growth (throughput.rs, doubling_strategy.rs)

  • Check 5 lets off a runner that has been silent for hedge_idle (2 s) by the same signal, unless the writer is full or it has queued frames.
  • A silent tick judges no doubling. A doubling held at the limit stays held. One below the limit is dropped and growth goes on.
  • StrategyControl::on_restarted is called by the slow and the silent path after a stop. Doubling drops the pending doubling unjudged, at the limit too. This narrows C28's third known limitation.

Results

10 runs each, alternating. Median / p90 in seconds, Doubling. Full table in §14.19.

Scenario Before (§14.17, §14.18) After Restarts / silent stops
s4a 18.32 12.36 / 12.37 0 / 0 (hedge)
s4b 38.49 13.84 / 13.99 1 / 1
s4c 38.72 13.23 / 13.29 1 / 1
s4 — 11.80 / 12.55 0 / 0
  • No false restarts. s1, s2, s3 and the three no-stall controls have 0 restarts and 0 silent stop requests in every run of both strategies. unmatched is 0 everywhere.
  • G7 (§14.18): s1_max32 −54.8%, peak 32 ≤ 32; s1_max8 −37.9%, 8 ≤ 32; s2 +0.6%, 2 ≤ 2; s3 −3.1%, 4 ≤ 8. All met.
  • s4c duplicates, 49 / 58 KiB. About 32 KiB is the stalled runner's unpushed buffer, dropped on the stop and requested again. The rest is split over-read.
  • Sanity rerun after the last review fixes: s4b 13.93 / 14.00 s, s4c 13.12 / 13.24 s.

Known limitations (§14.19)

  • The read signal is "the stream yielded a non-empty item". It cannot see bytes the adapter or TCP holds back. A stream that only yields empty items does not count as reading.
  • A tick sees only stamps already published. A read that lands right at the 4.5 s boundary can overlap a stop decision. The silence before it was real.
  • A stalled runner with less than 2 × min_chunk_size left is still left to the hedge.
  • The thresholds are not calibrated on real networks. HTTP is not measured.

Dependencies

PR-7b, which switches the default to Doubling, depends on PR-7, on #189 (C28) and on this PR.

Tests

  • Runner:
    • test_read_activity_follows_the_body;
    • test_a_resumed_runner_is_never_seen_silent, which checks every published state across the emit-to-read transition.
  • Download loop (download_loop/tests/silent.rs, explicit time). Covers:
    • the 4.5 s boundary, budget charging with and without data;
    • slow readers, pushing runners, late runners, the 2 × min_chunk_size boundary;
    • hedged runners, a full writer, congestion;
    • one stop per tick, Legacy;
    • a stall measured from the last read across a writer recovery;
    • queued frames;
    • check 5 boundaries;
    • the cap release on both paths.
  • Existing tests: growth.rs covers a small stalled chunk that no longer freezes growth. The T13 tests in restart.rs are adapted to the read signal.
  • Unit tests: in doubling_strategy.rs, throughput.rs and runner_manager.rs, including ID reuse.
  • Integration: tests/silent_restart.rs, a smoke test on a real download. The stalled range is requested again well before the 30 s idle timeout, and the file is correct.
  • Mutations: 28 groups over the whole lib (all features), each failing at least one test. They include:
    • the 4.5 s boundary;
    • the hedge exclusion;
    • writer-full and queued-frame guards;
    • one stop per tick;
    • congestion;
    • Legacy;
    • charging;
    • the spawn stamp;
    • read activity ignored;
    • the emit-to-read lock split back into two;
    • on_restarted removed on each path separately;
    • the C12 may_restart conditions.

Review

Reviewed with /ccg:review, using the codex backend only (a single-model review).

  1. Round 1 (71). A reference rate killed healthy slow runners, the evidence expired, and growth froze below the cap. Replaced by a read-activity signal; a silent tick drops a doubling below the cap.

  2. Round 2 (85). Backpressure from the data channel was seen as silence. Fixed with the queued-frames guard.

  3. Round 3 (89). The guard in may_restart broke C12's median. It moved to the silent path and check 5.

    Between rounds 3 and 4, the loop's side-effect reconstruction of liveness was replaced by the runner-owned ReadActivity. A writer-recovery reset was found to delay stops in s4b and s4c, and accepted frames and writer recovery no longer count.

  4. Round 4 (90). One Major: clearing emitting and stamping the read ran under separate locks, so a tick could see a healthy runner as silent. Fixed with the atomic resume_reading and a deterministic regression.

  5. Round 5: PASS (98/100).

    • Its Minor, a stale line in tasks.md, is fixed.
    • Its suggestion is applied: the boundary-read limitation is documented.

Checks

All on +nightly-2026-11-06. Clippy allows unnecessary_lazy_evaluations until #183 merges.

  • cargo fmt --all -- --check.
  • The 4 clippy configs with -D warnings.
  • cargo test --workspace --all-features --locked: lib 573, integration tests pass.
  • cargo test --workspace --locked: lib 542, integration tests pass.
  • RUSTDOCFLAGS=-Dwarnings cargo doc --no-deps --all-features -p bolt-load: the same 19 errors as its base.
  • The full bench, 10 runs each, plus the s4b and s4c sanity rerun.

Under Doubling, the download loop now stops a running runner that has
read no body for 4.5 s, has at least twice the minimum chunk size left
and has no frames queued in its data channel. It does not wait for full
slow-restart windows or three runners. The runner is stopped as Slow,
with the same budget rules as a slow stop: a restart if it delivered
data, a charged attempt if not. The slow and the silent path share a
tick, and at most one runner is stopped per tick. Legacy is unchanged.

Silence is measured from the runner's own reads. The runner and the
loop share a ReadActivity. The runner stamps it on every non-empty item
and when it resumes reading after a push, and marks itself as emitting
while it waits on the data channel; the stamp and the flag change under
one lock. Accepted frames and writer recovery no longer count as
activity, so a full writer cannot push a stall back. The hedge's own
silence clock is unchanged.

Check 5 of the growth gate no longer waits on a runner that has been
silent for hedge_idle. A tick with such a runner judges no doubling: a
doubling that reached the limit stays held, one below the limit is
dropped and growth goes on. A slow or silent restart drops the pending
doubling, at the limit too, by the maintainer's decision, so the
restarted range can be split again.
@greenhat616
greenhat616 force-pushed the feat/hardening-29-silent-restart branch from 3f2a39b to ed815b5 Compare November 13, 2026 22:25
@greenhat616
greenhat616 removed this pull request from stack #190 November 13, 2026 22:28
@greenhat616
greenhat616 added this pull request to stack #215 November 13, 2026 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant