Skip to content

Plan a reverse action from a state it can move from - #230

Open
DavertMik wants to merge 3 commits into
mainfrom
pilot-skipped-prerequisites
Open

DavertMik wants to merge 3 commits into
mainfrom
pilot-skipped-prerequisites

Conversation

@DavertMik

Copy link
Copy Markdown
Contributor

Supersedes #228. That PR changed Pilot's verdict to report the run as skipped — wrong, because the list had more than one page, so the scenario was executable and skipping it would have hidden a real execution gap behind a clean result. The problem was never the verdict.

The bug

Session UnknownHeavyBlue357 (trace c1406df82226d0698f4198d88c4c1642) ran "Navigate to the previous plans page and verify that a different page of plan rows is displayed" while the Plans list was on page 1. Tester clicked «, got urlChanged:false with an empty diff, and stopped. Reported as a failure.

The app had done nothing wrong. The Tester's own UI map listed both controls:

| 'Pagination previous' | link | { role: 'link', text: '«' } | '.pagination .arrow.prev > a' |
| 'Pagination next'     | link | { role: 'link', text: '»' } | '.pagination .arrow.next > a' |

There was a next page. Click », then «, and the scenario passes. It just never got planned that way.

Root cause

Pilot's plan was: scroll to the pager → click previous → verify. Plus "Watch for a disabled previous control or unchanged rows; if either occurs, stop and report that the preceding page is unavailable." It planned the click from page 1 and wrote its own surrender clause.

planTest makes that near-inevitable. Its precondition block is exclusively about creating data via precondition()/askApi(), and explicitly says to skip it when "the scenario tests navigation, UI behavior, or viewing". The paragraph immediately below then says "Prefer interacting with the current page over navigating away."

So a navigation scenario skips setup by rule, and is then pushed toward the current page. Nothing anywhere asks where the action under test actually starts from.

The change

One paragraph in planTest, placed right after the "prefer the current page" line so it qualifies it:

A scenario that verifies a reverse or returning action can only be measured from the state that action leads back from. When the page is not in that state, plan the steps that reach it first — exercising the control from the wrong end proves nothing and is not a defect.

General by construction: it covers going back a page, the back button, undo, collapsing an expanded panel, clearing a filter, closing, logging out. Pagination is never mentioned.

Measured, not assumed

Replayed the original planning call from the trace — same model, same messages, one paragraph added. Graded on whether the plan reaches a later page before clicking previous:

Plans that set up the starting state
Before 1 / 8
After 7 / 8

The single baseline hit is real — one run did reason its way to "If Previous is disabled because this is page 1, click Next once, then click Previous". The correct plan was already within reach; the prompt just never asked for it.

Control — a create scenario ("Create a manual plan with the Run Automated as Manual switch enabled") replayed the same way is unchanged in character: same mix of "already satisfied → verify and finish" and "click New → fill the form". No setup step injected where none is needed.

A second candidate wording scored 6/8 on the same replay; this one won.

Trade-off

Pilot now has to judge, from the page summary and state, whether the scenario's action has somewhere to go — and it can get that wrong in the expensive direction, planning a setup step for an action that was already positioned correctly. The control shows it not firing on a plain create, but it is judgment, not a guarantee.

Verification

  • bun test tests/unit/ — 1436 pass, 0 fail
  • bun test tests/integration/ — 148 pass, 0 fail
  • bun run format, bun run lint:fix — clean

Prompt change, so a regression run is the real confirmation — flagging rather than labelling.

🤖 Generated with Claude Code

DavertMik and others added 3 commits September 18, 2026 01:32
A scenario that verifies going back was planned from wherever the page
happened to be. On the Plans list that meant clicking the previous-page
arrow while on page 1: the arrow is inert there, the run reported a
failure, and the app had done nothing wrong. The list had a next-page
control in the tester's own UI map, so the scenario was executable — it
just needed page 2 first.

planTest's precondition block only covers creating DATA, and explicitly
skips navigation scenarios; the paragraph below it then prefers the
current page over navigating away. Nothing asked where the action under
test starts from, so Pilot planned the click and a surrender clause.

One paragraph in planTest states the missing principle: a reverse or
returning action can only be measured from the state it leads back from,
so reach that state first.

Replayed the original planning call: 1/8 plans set up the starting state
before, 7/8 after. A create scenario replayed as control is unchanged —
no setup step injected where none is needed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants