Skip to content

test(task): fail runners in the split property test as the loop does - #194

Draft
greenhat616 wants to merge 1 commit into
feat/hardening-31-written-offsetfrom
test/hardening-32-proptest-side-fail
Draft

greenhat616 wants to merge 1 commit into
feat/hardening-31-written-offsetfrom
test/hardening-32-proptest-side-fail

Conversation

@greenhat616

Copy link
Copy Markdown
Owner

C30 (PR-9a) of the download strategy hardening stack. It is a test-only change, stacked on #193 (C18), and that branch is its base. The design note is §14.22.

Why

While checking #193, the property test splits_stay_within_the_limit_and_cover_the_file tripped the debug_assert in ChunkPlanner::mark_failed ("the last side of a pair fails, not the first"). The same seed also fails on #192, so #193 did not cause it.

Replaying the shrunk sequence as a fixed test on each commit:

Commits Result
C27 (#187), C13 (#188) Fails earlier, on the "a slot leaked" oracle false positive that C28 fixed
C28 (#189), C29 (#191), C14 (#192) Fails on this assert

C14's Grow op opens many connecting runners in one step, which made the state likely enough for 256 cases to find it.

Cause: a test driver bug, not a product bug

The endgame hedge can pair a connecting runner (§9.6). The test's Op::Fail then called mark_failed on that first side directly.

In the real loop, every failure of a pair side goes through has_other_side and then handle_side_failed, and the partner keeps the chunk. That holds for:

  • Stopped;
  • a rejected spawn;
  • a lost lifecycle.

The other production callers of mark_failed either run only after that check or never target a pair side:

  • the slow and silent restarts exclude undecided pairs;
  • the hedge never picks a stopping runner;
  • release_runner_full exists only in test builds.

Codex verified this independently: no release-build path fails a side of an undecided pair while both sides are live.

What (test code only)

  • Lifecycle helpers. #[cfg(test)] report_started, report_stopped and fail_runner run real lifecycle events through handle_lifecycle.
  • The property test's ops.
    • Fail goes through fail_runner and on_runner.
    • Start and Progress can pick hedges, and Progress can also pick beaten losers.
    • Frames go through on_data_frame, with on_progress and note_delivery; nothing is written.
    • Finish delivers the rest, then reports Finished.
  • The oracle.
    • Plain failures and side failures have separate expectations.
    • The last survivor of a pair is charged once for both sides: !first.restart && !restart && (first.charged || charged). That expectation comes from a failure-class table derived from §8.2, not from the product's helpers.
    • A hedge taking over its chunk keeps the chunk's counters and is not checked as a new connection.
    • A new hedge must not connect while the task is congested.
    • A terminal step is still checked, and only a completed download or a plain congestion may end a sequence.
  • Budgets. The failure budgets are raised so that sequences keep going. The old driver bypassed the failure policy, so no coverage is lost.

Coverage gap

Random sequences rarely reach an undecided pair. In 4096 cases, the Fail ops hit a hedge side once, a pair owner once and a last survivor once, and a takeover happened once.

Pair behaviour is therefore pinned by deterministic tests rather than by biasing the generator. That was the lead's decision; biasing is left for later.

  • the_first_side_of_a_pair_fails_as_a_side, the shrunk sequence, continued through the second failure.
  • a_pair_is_charged_once_for_both_sides: each side failing first, with every pair of Retryable, Timeout and Congestion (18 cases).
  • a_hedge_that_wins_keeps_the_counters_of_its_chunk: a retried range, its hedge wins, and the loser's late frame is dropped.
  • each_pair_keeps_its_own_first_failure: two pairs fail with different classes.
  • the_oracle_rejects_a_hedge_while_congested.

Mutations, all caught

  • Product mutations:
    • the lifecycle without the has_other_side check;
    • the pair charge ignoring the first side;
    • a side failure that records no classification;
    • a beaten side's late frame accepted;
    • a hedge spawned while congested, caught by an existing test.
  • Oracle mutations:
    • the old direct mark_failed;
    • a takeover checked as a new connection;
    • the old survivor lookup;
    • the last side charged alone;
    • the congestion check disabled.

Review

Reviewed with /ccg:review, using the codex backend only (a single-model review).

  1. Round 1 (80). One Major: the last survivor of a pair carries the pair's failure classification, so the plain-branch charge expectation was wrong. Fixed with the class table and the 18-case test.

    • Minor: a terminal step skipped the invariants. Fixed.
    • Suggestions, applied: the progress bookkeeping, and hedge selectors so that hedges start and deliver.
  2. Round 2 (82). Two oracle Majors, both exposed once hedges could deliver:

    • a hedge takeover was checked as a new connection;
    • a failed hedge's survivor was looked up in another pair.

    Both are fixed with deterministic tests. A sweep for owner-keyed state also added the congestion check for new hedges.

  3. Round 3: PASS (96/100). Its Minor (comment wording) and its suggestion (a deterministic test for the congestion check) are applied.

Checks

All on +nightly-2026-11-06. Clippy allows unnecessary_lazy_evaluations until #183 merges.

  • cargo fmt --all -- --check.
  • The 4 clippy configs with -D warnings.
  • cargo test --workspace --all-features --locked and cargo test --workspace --locked pass, run after round 2. After the round-3 items, which only change lib tests: bolt-load lib 576 with all features, 545 with defaults.
  • The property test, run once with 4096 cases: pass.
  • RUSTDOCFLAGS=-Dwarnings cargo doc --no-deps --all-features -p bolt-load: the same 18 errors as its base.

The split property test failed a runner by calling the planner's
mark_failed directly. Since the endgame hedge, that runner can be the
first side of an undecided pair, which the loop never fails that way:
every lifecycle path hands a side's failure to handle_side_failed, so
the partner keeps the chunk. The test tripped the planner's debug
assertion on a state the product handles correctly.

Failures, starts, frames and completions in the property test now go
through the loop's lifecycle and data paths, and the oracle tells a
hedge taking over its chunk from a new connection, charges a pair once
for both sides, and resolves each pair's own survivor. Deterministic
tests cover the shrunk sequence, a pair's charge for every pair of
failure classes, a hedge winning a retried range, two pairs failing
with different classes, and the oracle's congestion check.
@greenhat616
greenhat616 force-pushed the test/hardening-32-proptest-side-fail branch from cc1b41b to 0a16582 Compare November 13, 2026 22:26
@greenhat616
greenhat616 removed this pull request from stack #190 November 13, 2026 22:28
@greenhat616
greenhat616 added this pull request to stack #215 November 13, 2026 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant