Conversation
A scenario that verifies going back was planned from wherever the page happened to be. On the Plans list that meant clicking the previous-page arrow while on page 1: the arrow is inert there, the run reported a failure, and the app had done nothing wrong. The list had a next-page control in the tester's own UI map, so the scenario was executable — it just needed page 2 first. planTest's precondition block only covers creating DATA, and explicitly skips navigation scenarios; the paragraph below it then prefers the current page over navigating away. Nothing asked where the action under test starts from, so Pilot planned the click and a surrender clause. One paragraph in planTest states the missing principle: a reverse or returning action can only be measured from the state it leads back from, so reach that state first. Replayed the original planning call: 1/8 plans set up the starting state before, 7/8 after. A create scenario replayed as control is unchanged — no setup step injected where none is needed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…isites # Conflicts: # CHANGELOG.md
DenysKuchma
approved these changes
Sep 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #228. That PR changed Pilot's verdict to report the run as
skipped— wrong, because the list had more than one page, so the scenario was executable and skipping it would have hidden a real execution gap behind a clean result. The problem was never the verdict.The bug
Session
UnknownHeavyBlue357(tracec1406df82226d0698f4198d88c4c1642) ran "Navigate to the previous plans page and verify that a different page of plan rows is displayed" while the Plans list was on page 1. Tester clicked«, goturlChanged:falsewith an empty diff, and stopped. Reported as a failure.The app had done nothing wrong. The Tester's own UI map listed both controls:
There was a next page. Click
», then«, and the scenario passes. It just never got planned that way.Root cause
Pilot's plan was: scroll to the pager → click previous → verify. Plus "Watch for a disabled previous control or unchanged rows; if either occurs, stop and report that the preceding page is unavailable." It planned the click from page 1 and wrote its own surrender clause.
planTestmakes that near-inevitable. Its precondition block is exclusively about creating data viaprecondition()/askApi(), and explicitly says to skip it when "the scenario tests navigation, UI behavior, or viewing". The paragraph immediately below then says "Prefer interacting with the current page over navigating away."So a navigation scenario skips setup by rule, and is then pushed toward the current page. Nothing anywhere asks where the action under test actually starts from.
The change
One paragraph in
planTest, placed right after the "prefer the current page" line so it qualifies it:General by construction: it covers going back a page, the back button, undo, collapsing an expanded panel, clearing a filter, closing, logging out. Pagination is never mentioned.
Measured, not assumed
Replayed the original planning call from the trace — same model, same messages, one paragraph added. Graded on whether the plan reaches a later page before clicking previous:
The single baseline hit is real — one run did reason its way to "If Previous is disabled because this is page 1, click Next once, then click Previous". The correct plan was already within reach; the prompt just never asked for it.
Control — a create scenario ("Create a manual plan with the Run Automated as Manual switch enabled") replayed the same way is unchanged in character: same mix of "already satisfied → verify and finish" and "click New → fill the form". No setup step injected where none is needed.
A second candidate wording scored 6/8 on the same replay; this one won.
Trade-off
Pilot now has to judge, from the page summary and state, whether the scenario's action has somewhere to go — and it can get that wrong in the expensive direction, planning a setup step for an action that was already positioned correctly. The control shows it not firing on a plain create, but it is judgment, not a guarantee.
Verification
bun test tests/unit/— 1436 pass, 0 failbun test tests/integration/— 148 pass, 0 failbun run format,bun run lint:fix— cleanPrompt change, so a regression run is the real confirmation — flagging rather than labelling.
🤖 Generated with Claude Code