Judge the verdict from the app's perspective

[DavertMik]

Pilot's verdict prompt defined "fail" purely by goal-shape — "goal not achieved and no further step toward it is available on the current page" — while "skipped" was scoped to "scenario is irrelevant to the app, OR systematic infrastructure failures". A scenario whose premise the app contradicts while behaving correctly had no bucket, so it landed in fail, and the session analyst's FAILED branch (rule 5, "app contradicted expected outcome → Defect") turned it into a reported product defect.

Planner invents scenarios from the UI, so wrong-premise scenarios are a structural class, not an edge case.

The session that surfaced it

Langfuse trace 2c854bc971fc8c12834f35bc5d360d5f, scenario "Attempt to create a manual plan without a title and verify that submission is rejected."

The form Pilot ended up on labels the field "Title (optional)", and the state it saw reported disabled buttons: No matched tests, Launch, Save — Save is disabled because no tests are selected, not because Title is empty. The app has no missing-title validation at all. Pilot's own reasoning one turn before the verdict wanted to "finish as a mismatch", then emitted fail because that was the nearest available label.

Pilot could already set every status — pass / fail / continue / skipped are all in the schema and each wires to TestResult.* and task.finish(). The gap was the prompt's vocabulary, not the plumbing.

Change

src/ai/pilot.ts — buildVerdictSystemPrompt, plus the decision enum's own description so the two agree:

• "fail" — the app misbehaved: the scenario's action ran against the right target and the app produced a wrong, broken, or missing outcome. Not reaching the goal is not by itself a fail.
• "skipped" — the scenario cannot be judged against this app: the page shows its premise does not hold, the target entity or feature is not the one here, the scenario is irrelevant, or infrastructure failures prevented testing. Still not for "test failed to interact".

Trade-off

An app that claims a constraint (required marker, documented validation) but silently accepts invalid input can now read as skipped instead of fail. The wording mitigates it — fail still applies when the app asserts a constraint and violates it — but it does not eliminate it.

Not in this PR

• Experience recipe A.2 (FLOW: create a new manual run, url /runs/new) was offered while the tester was on /plans, and Pilot's first PROGRESS asserted "the manual-run form is already open" on that /plans page. That entity drift is a second, independent cause.
• session-analyst.ts's "Walk every test" has branches for PASSED and FAILED only — SKIPPED tests fall out of the report rather than being surfaced as untestable premises.

Verification

• bun test tests/integration/ — 148 pass, 1 skip, 0 fail
• bun test tests/unit/ — 1436 pass, 0 fail
• bun run format, bun run lint — clean

Behaviour change is prompt-only, so it needs a regression run against a real app to confirm the verdict flips to skipped on this scenario. Apply the regression label when you want that.

🤖 Generated with Claude Code