Problem
The red-first discipline assumes the executor writes exactly one failing test per cycle. In practice, agent executors strongly prefer to batch-write a whole test file (and sometimes the production module) up front. Observed across several real agent-driven runs:
- On multi-cycle plans, a majority of cycles fired
red_first_violation because their declared test already existed and passed on first run — in the worst runs, more than half the cycles (e.g. 18/32, 9/11).
- Those cycles' only commits are
[refactor]-labelled, yet they introduce new tests and new production behaviour — one observed "refactor" commit added a ~150-line test file plus a new ~55-line production class. Git history semantics become fiction for a third of delivered cycles.
- Several cycles end with
Commits: none, and at least one green commit landed with no corresponding red commit anywhere in the run.
- The sensitivity-mutation fallback carried the entire correctness burden every time (and did catch real assertions), but per-cycle red→green provenance is gone, and the plan's authored
commit_red/commit_green messages are never used.
Small 2-cycle plans run textbook-clean; the collapse correlates with plan size. Enforcement fights the executor's natural (and arguably reasonable) behaviour, and loses.
Proposal: an xfail-batch operating mode
Instead of fighting batch authoring, legalise it and turn the lost signal into a hard failure:
- Authoring cycle (cycle 0): the executor writes all the plan's tests up front, each marked expected-to-fail, plus whatever production stubs are needed for compiled targets to build. The tool verifies every declared target runs and fails-as-expected (each xfail records a real failure) — a stronger, cheaper red proof than today's per-cycle first run.
- Per cycle: remove the expected-failure marker from exactly one test and implement.
advance verifies (a) the target now passes un-marked, (b) every remaining declared target still fails, (c) any unexpected pass elsewhere raises a typed blocker (e.g. over_implementation).
- Over-implementation becomes detectable instead of invisible: strict xfail semantics mean an early-satisfied test fails the suite at exactly the moment the over-implementation happens, instead of surfacing later as a
red_first_violation shrug.
- Honest escape path: some pre-passing is inherent (several cycles that are facets of one branch — a minimal implementation of the first facet unavoidably satisfies the rest). The
over_implementation blocker needs a fold resolution that marks the cycle delivered-by-cycle-N with a recorded justification. Without it, executors will route around the check by weakening tests or adding artificial gating to keep later xfails red.
Runner support
- pytest:
@pytest.mark.xfail(strict=True) — native; XPASS(strict) fails the suite.
- vitest:
test.fails — fails when the test passes.
- XCTest:
XCTExpectFailure — strict by default; lives in the test body, so the un-mark edit is a body edit.
- JUnit4/5: no native strict xfail; needs a small harness-side rule/extension that inverts the result. Weakest link; the mode's adapter contract should let each adapter declare how "expected fail" is spelled and detected.
Design notes
- Compiled targets fail compilation, not assertion, when tests reference not-yet-existing types — hence stubs are a legitimate part of cycle 0. The all-tests-still-fail check is itself the stub/implementation boundary: if a "stub" makes a test pass, it wasn't a stub.
- Commit labelling: cycle 0 could be
[author]; per-cycle commits are green+refactor with no per-cycle red — the red evidence lives in cycle 0's run record.
- Mode selection belongs in the plan contract (per plan or per project). Classic mode stays the default and remains right for small plans; xfail-batch targets the larger plans where collapse actually happens.
- Side benefit: every declared target exists and is collectible from cycle 1, so the
not_found / multiple_new_tests / stub-directive adoption round-trips on new targets disappear entirely (a real cost on slow suites — see the companion issue on target adoption).
Interim mitigations for classic mode (independent of the new mode)
- Detect ahead-implementation and relabel: a commit in a refactor phase that adds new files (especially new test files) should not be labelled
[refactor] — flag it, or label it [adopt].
- Warn at red-commit time when the commit adds tests matching later declared targets.
- When a majority of a run's cycles degrade to sensitivity-only verification, say so loudly in the run summary instead of one event line per cycle.
Problem
The red-first discipline assumes the executor writes exactly one failing test per cycle. In practice, agent executors strongly prefer to batch-write a whole test file (and sometimes the production module) up front. Observed across several real agent-driven runs:
red_first_violationbecause their declared test already existed and passed on first run — in the worst runs, more than half the cycles (e.g. 18/32, 9/11).[refactor]-labelled, yet they introduce new tests and new production behaviour — one observed "refactor" commit added a ~150-line test file plus a new ~55-line production class. Git history semantics become fiction for a third of delivered cycles.Commits: none, and at least one green commit landed with no corresponding red commit anywhere in the run.commit_red/commit_greenmessages are never used.Small 2-cycle plans run textbook-clean; the collapse correlates with plan size. Enforcement fights the executor's natural (and arguably reasonable) behaviour, and loses.
Proposal: an
xfail-batchoperating modeInstead of fighting batch authoring, legalise it and turn the lost signal into a hard failure:
advanceverifies (a) the target now passes un-marked, (b) every remaining declared target still fails, (c) any unexpected pass elsewhere raises a typed blocker (e.g.over_implementation).red_first_violationshrug.over_implementationblocker needs afoldresolution that marks the cycle delivered-by-cycle-N with a recorded justification. Without it, executors will route around the check by weakening tests or adding artificial gating to keep later xfails red.Runner support
@pytest.mark.xfail(strict=True)— native; XPASS(strict) fails the suite.test.fails— fails when the test passes.XCTExpectFailure— strict by default; lives in the test body, so the un-mark edit is a body edit.Design notes
[author]; per-cycle commits are green+refactor with no per-cycle red — the red evidence lives in cycle 0's run record.not_found/multiple_new_tests/ stub-directive adoption round-trips on new targets disappear entirely (a real cost on slow suites — see the companion issue on target adoption).Interim mitigations for classic mode (independent of the new mode)
[refactor]— flag it, or label it[adopt].