Skip to content

sensitivity check: observed evidence captures the wrong output line on some runners #68

Description

@geuben

Task file: tasks/issue-68-sensitivity-evidence.md

Problem

The sensitivity check's observed: evidence line — the proof that the mutation actually flipped the target test — captures the wrong output line on some runners, making the verdict unauditable from the log:

  • pytest + xdist: the captured line is the worker header ([gw3] darwin -- Python 3.x.x ...), not the assertion failure.
  • XCTest: the captured line is unrelated console noise from the host app (e.g. Socket SO_ERROR [61: Connection refused] from a network poller), not the failing assertion.
  • JVM/JUnit: works correctly — captured lines show the genuine mutation failure (expected:<...> but was:<...MUTATION...>), which proves the right behaviour is achievable.

Additionally, captured lines are truncated mid-identifier at a fixed width (e.g. -[SomeTests.Rec), so even a correctly-selected line can lose the test identity.

This matters more than it looks: whenever a cycle degrades to sensitivity-only verification (test pre-passing, red_first_violation), the sensitivity evidence is the only proof the test bites. Wrong-line capture means that safety net is unverifiable exactly where it is load-bearing.

Proposal

  • Make evidence extraction per-adapter: each adapter knows what its failure output looks like (pytest longrepr tail, XCTest XCTAssert... failure lines, JUnit ComparisonFailure). Prefer the assertion/failure line for the target test, filtered to that test's output, falling back to the last non-noise stderr line.
  • Don't truncate mid-identifier; if a cap is needed, keep the tail (same reasoning as target_failure truncation keeps the head of the traceback and loses the error #17 — closed — which fixed the identical head-vs-tail problem for target_failure).
  • If no plausible assertion line can be found, say so explicitly (observed: <no assertion line captured>) rather than printing whatever was on the wire.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    hardenedPlan audited by agent-handoff-plan; ready for autonomous execution

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions