Conversation
Pilot's verdict prompt defined "fail" without a prerequisite clause and fenced "skipped" off with NOT for "test failed to interact". A control that is present but has nothing to act on reads as a failed interaction, so it matched the earlier, unqualified "fail" bullet — a pagination test started on page 1 was reported as a failure rather than skipped. Both bullets now partition on one axis, whether the app held the state the scenario presumes, and a line ahead of them makes Pilot settle that question before it chooses. Aligns the verdict with the shared evidence rules, which already route prerequisites unmet to "skipped" but scope it to a missing control. Replayed against the original verdict call: 0/13 skipped before, 16/16 after. Two genuine failures from the same run (a create that was not verified, a copy with no confirmation) stay "fail" with no leakage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
Session
UnknownHeavyBlue357(tracec1406df82226d0698f4198d88c4c1642) ran "Navigate to the previous plans page and verify that a different page of plan rows is displayed" while the Plans list was on page 1. Tester clicked«, goturlChanged:falsewith an empty diff, and stopped. Reported as a failure.The app had done nothing wrong. The Tester's own UI map listed both controls:
There was a next page. Click
», then«, and the scenario passes. It just never got planned that way.Root cause
Pilot's plan was: scroll to the pager → click previous → verify. Plus "Watch for a disabled previous control or unchanged rows; if either occurs, stop and report that the preceding page is unavailable." It planned the click from page 1 and wrote its own surrender clause.
planTestmakes that near-inevitable. Its precondition block is exclusively about creating data viaprecondition()/askApi(), and explicitly says to skip it when "the scenario tests navigation, UI behavior, or viewing". The paragraph immediately below then says "Prefer interacting with the current page over navigating away."So a navigation scenario skips setup by rule, and is then pushed toward the current page. Nothing anywhere asks where the action under test actually starts from.
The change
One paragraph in
planTest, placed right after the "prefer the current page" line so it qualifies it:General by construction: it covers going back a page, the back button, undo, collapsing an expanded panel, clearing a filter, closing, logging out. Pagination is never mentioned.
Measured, not assumed
Replayed the original planning call from the trace — same model, same messages, one paragraph added. Graded on whether the plan reaches a later page before clicking previous:
The single baseline hit is real — one run did reason its way to "If Previous is disabled because this is page 1, click Next once, then click Previous". The correct plan was already within reach; the prompt just never asked for it.
Control — a create scenario ("Create a manual plan with the Run Automated as Manual switch enabled") replayed the same way is unchanged in character: same mix of "already satisfied → verify and finish" and "click New → fill the form". No setup step injected where none is needed.
A second candidate wording scored 6/8 on the same replay; this one won.
Trade-off
Pilot now has to judge, from the page summary and state, whether the scenario's action has somewhere to go — and it can get that wrong in the expensive direction, planning a setup step for an action that was already positioned correctly. The control shows it not firing on a plain create, but it is judgment, not a guarantee.
Verification
bun test tests/unit/— 1436 pass, 0 failbun test tests/integration/— 148 pass, 0 failbun run format,bun run lint:fix— cleanPrompt change, so a regression run is the real confirmation — flagging rather than labelling.
🤖 Generated with Claude Code