Skip to content

advance: red-first collapses when the executor batch-writes tests — add an xfail-batch operating mode #66

Description

@geuben

Problem

The red-first discipline assumes the executor writes exactly one failing test per cycle. In practice, agent executors strongly prefer to batch-write a whole test file (and sometimes the production module) up front. Observed across several real agent-driven runs:

  • On multi-cycle plans, a majority of cycles fired red_first_violation because their declared test already existed and passed on first run — in the worst runs, more than half the cycles (e.g. 18/32, 9/11).
  • Those cycles' only commits are [refactor]-labelled, yet they introduce new tests and new production behaviour — one observed "refactor" commit added a ~150-line test file plus a new ~55-line production class. Git history semantics become fiction for a third of delivered cycles.
  • Several cycles end with Commits: none, and at least one green commit landed with no corresponding red commit anywhere in the run.
  • The sensitivity-mutation fallback carried the entire correctness burden every time (and did catch real assertions), but per-cycle red→green provenance is gone, and the plan's authored commit_red/commit_green messages are never used.

Small 2-cycle plans run textbook-clean; the collapse correlates with plan size. Enforcement fights the executor's natural (and arguably reasonable) behaviour, and loses.

Proposal: an xfail-batch operating mode

Instead of fighting batch authoring, legalise it and turn the lost signal into a hard failure:

  1. Authoring cycle (cycle 0): the executor writes all the plan's tests up front, each marked expected-to-fail, plus whatever production stubs are needed for compiled targets to build. The tool verifies every declared target runs and fails-as-expected (each xfail records a real failure) — a stronger, cheaper red proof than today's per-cycle first run.
  2. Per cycle: remove the expected-failure marker from exactly one test and implement. advance verifies (a) the target now passes un-marked, (b) every remaining declared target still fails, (c) any unexpected pass elsewhere raises a typed blocker (e.g. over_implementation).
  3. Over-implementation becomes detectable instead of invisible: strict xfail semantics mean an early-satisfied test fails the suite at exactly the moment the over-implementation happens, instead of surfacing later as a red_first_violation shrug.
  4. Honest escape path: some pre-passing is inherent (several cycles that are facets of one branch — a minimal implementation of the first facet unavoidably satisfies the rest). The over_implementation blocker needs a fold resolution that marks the cycle delivered-by-cycle-N with a recorded justification. Without it, executors will route around the check by weakening tests or adding artificial gating to keep later xfails red.

Runner support

  • pytest: @pytest.mark.xfail(strict=True) — native; XPASS(strict) fails the suite.
  • vitest: test.fails — fails when the test passes.
  • XCTest: XCTExpectFailure — strict by default; lives in the test body, so the un-mark edit is a body edit.
  • JUnit4/5: no native strict xfail; needs a small harness-side rule/extension that inverts the result. Weakest link; the mode's adapter contract should let each adapter declare how "expected fail" is spelled and detected.

Design notes

  • Compiled targets fail compilation, not assertion, when tests reference not-yet-existing types — hence stubs are a legitimate part of cycle 0. The all-tests-still-fail check is itself the stub/implementation boundary: if a "stub" makes a test pass, it wasn't a stub.
  • Commit labelling: cycle 0 could be [author]; per-cycle commits are green+refactor with no per-cycle red — the red evidence lives in cycle 0's run record.
  • Mode selection belongs in the plan contract (per plan or per project). Classic mode stays the default and remains right for small plans; xfail-batch targets the larger plans where collapse actually happens.
  • Side benefit: every declared target exists and is collectible from cycle 1, so the not_found / multiple_new_tests / stub-directive adoption round-trips on new targets disappear entirely (a real cost on slow suites — see the companion issue on target adoption).

Interim mitigations for classic mode (independent of the new mode)

  • Detect ahead-implementation and relabel: a commit in a refactor phase that adds new files (especially new test files) should not be labelled [refactor] — flag it, or label it [adopt].
  • Warn at red-commit time when the commit adds tests matching later declared targets.
  • When a majority of a run's cycles degrade to sensitivity-only verification, say so loudly in the run summary instead of one event line per cycle.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions