Skip to content

Plan a reverse action from a state it can move from - #228

Closed
DavertMik wants to merge 1 commit into
mainfrom
pilot-skipped-prerequisites
Closed

DavertMik wants to merge 1 commit into
mainfrom
pilot-skipped-prerequisites

Conversation

@DavertMik

@DavertMik DavertMik commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Replaces an earlier attempt on this branch that changed Pilot's verdict to report the run as skipped. That was wrong — the list had more than one page, so the scenario was executable and skipping it would have hidden a real execution gap behind a clean result. The problem was never the verdict.

The bug

Session UnknownHeavyBlue357 (trace c1406df82226d0698f4198d88c4c1642) ran "Navigate to the previous plans page and verify that a different page of plan rows is displayed" while the Plans list was on page 1. Tester clicked «, got urlChanged:false with an empty diff, and stopped. Reported as a failure.

The app had done nothing wrong. The Tester's own UI map listed both controls:

| 'Pagination previous' | link | { role: 'link', text: '«' } | '.pagination .arrow.prev > a' |
| 'Pagination next'     | link | { role: 'link', text: '»' } | '.pagination .arrow.next > a' |

There was a next page. Click », then «, and the scenario passes. It just never got planned that way.

Root cause

Pilot's plan was: scroll to the pager → click previous → verify. Plus "Watch for a disabled previous control or unchanged rows; if either occurs, stop and report that the preceding page is unavailable." It planned the click from page 1 and wrote its own surrender clause.

planTest makes that near-inevitable. Its precondition block is exclusively about creating data via precondition()/askApi(), and explicitly says to skip it when "the scenario tests navigation, UI behavior, or viewing". The paragraph immediately below then says "Prefer interacting with the current page over navigating away."

So a navigation scenario skips setup by rule, and is then pushed toward the current page. Nothing anywhere asks where the action under test actually starts from.

The change

One paragraph in planTest, placed right after the "prefer the current page" line so it qualifies it:

A scenario that verifies a reverse or returning action can only be measured from the state that action leads back from. When the page is not in that state, plan the steps that reach it first — exercising the control from the wrong end proves nothing and is not a defect.

General by construction: it covers going back a page, the back button, undo, collapsing an expanded panel, clearing a filter, closing, logging out. Pagination is never mentioned.

Measured, not assumed

Replayed the original planning call from the trace — same model, same messages, one paragraph added. Graded on whether the plan reaches a later page before clicking previous:

Plans that set up the starting state
Before 1 / 8
After 7 / 8

The single baseline hit is real — one run did reason its way to "If Previous is disabled because this is page 1, click Next once, then click Previous". The correct plan was already within reach; the prompt just never asked for it.

Control — a create scenario ("Create a manual plan with the Run Automated as Manual switch enabled") replayed the same way is unchanged in character: same mix of "already satisfied → verify and finish" and "click New → fill the form". No setup step injected where none is needed.

A second candidate wording scored 6/8 on the same replay; this one won.

Trade-off

Pilot now has to judge, from the page summary and state, whether the scenario's action has somewhere to go — and it can get that wrong in the expensive direction, planning a setup step for an action that was already positioned correctly. The control shows it not firing on a plain create, but it is judgment, not a guarantee.

Verification

  • bun test tests/unit/ — 1436 pass, 0 fail
  • bun test tests/integration/ — 148 pass, 0 fail
  • bun run format, bun run lint:fix — clean

Prompt change, so a regression run is the real confirmation — flagging rather than labelling.

🤖 Generated with Claude Code

Pilot's verdict prompt defined "fail" without a prerequisite clause and
fenced "skipped" off with NOT for "test failed to interact". A control
that is present but has nothing to act on reads as a failed interaction,
so it matched the earlier, unqualified "fail" bullet — a pagination test
started on page 1 was reported as a failure rather than skipped.

Both bullets now partition on one axis, whether the app held the state
the scenario presumes, and a line ahead of them makes Pilot settle that
question before it chooses. Aligns the verdict with the shared evidence
rules, which already route prerequisites unmet to "skipped" but scope it
to a missing control.

Replayed against the original verdict call: 0/13 skipped before, 16/16
after. Two genuine failures from the same run (a create that was not
verified, a copy with no confirmation) stay "fail" with no leakage.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@DavertMik DavertMik closed this Sep 17, 2026
@DavertMik DavertMik changed the title Skip a scenario the app never had the data for Plan a reverse action from a state it can move from Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant