You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The sensitivity check's observed: evidence line — the proof that the mutation actually flipped the target test — captures the wrong output line on some runners, making the verdict unauditable from the log:
pytest + xdist: the captured line is the worker header ([gw3] darwin -- Python 3.x.x ...), not the assertion failure.
XCTest: the captured line is unrelated console noise from the host app (e.g. Socket SO_ERROR [61: Connection refused] from a network poller), not the failing assertion.
JVM/JUnit: works correctly — captured lines show the genuine mutation failure (expected:<...> but was:<...MUTATION...>), which proves the right behaviour is achievable.
Additionally, captured lines are truncated mid-identifier at a fixed width (e.g. -[SomeTests.Rec), so even a correctly-selected line can lose the test identity.
This matters more than it looks: whenever a cycle degrades to sensitivity-only verification (test pre-passing, red_first_violation), the sensitivity evidence is the only proof the test bites. Wrong-line capture means that safety net is unverifiable exactly where it is load-bearing.
Proposal
Make evidence extraction per-adapter: each adapter knows what its failure output looks like (pytest longrepr tail, XCTest XCTAssert... failure lines, JUnit ComparisonFailure). Prefer the assertion/failure line for the target test, filtered to that test's output, falling back to the last non-noise stderr line.
Task file:
tasks/issue-68-sensitivity-evidence.mdProblem
The sensitivity check's
observed:evidence line — the proof that the mutation actually flipped the target test — captures the wrong output line on some runners, making the verdict unauditable from the log:[gw3] darwin -- Python 3.x.x ...), not the assertion failure.Socket SO_ERROR [61: Connection refused]from a network poller), not the failing assertion.expected:<...> but was:<...MUTATION...>), which proves the right behaviour is achievable.Additionally, captured lines are truncated mid-identifier at a fixed width (e.g.
-[SomeTests.Rec), so even a correctly-selected line can lose the test identity.This matters more than it looks: whenever a cycle degrades to sensitivity-only verification (test pre-passing,
red_first_violation), the sensitivity evidence is the only proof the test bites. Wrong-line capture means that safety net is unverifiable exactly where it is load-bearing.Proposal
longreprtail, XCTestXCTAssert...failure lines, JUnitComparisonFailure). Prefer the assertion/failure line for the target test, filtered to that test's output, falling back to the last non-noise stderr line.target_failure).observed: <no assertion line captured>) rather than printing whatever was on the wire.