From ecf1c123808a9d780a41047a756350267fc49a1d Mon Sep 17 00:00:00 2001 From: DavertMik Date: Fri, 18 Sep 2026 01:10:33 +0300 Subject: [PATCH] fix(pilot): skip a scenario the app never had the data for MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pilot's verdict prompt defined "fail" without a prerequisite clause and fenced "skipped" off with NOT for "test failed to interact". A control that is present but has nothing to act on reads as a failed interaction, so it matched the earlier, unqualified "fail" bullet — a pagination test started on page 1 was reported as a failure rather than skipped. Both bullets now partition on one axis, whether the app held the state the scenario presumes, and a line ahead of them makes Pilot settle that question before it chooses. Aligns the verdict with the shared evidence rules, which already route prerequisites unmet to "skipped" but scope it to a missing control. Replayed against the original verdict call: 0/13 skipped before, 16/16 after. Two genuine failures from the same run (a create that was not verified, a copy with no confirmation) stay "fail" with no leakage. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 10 ++++++++++ src/ai/pilot.ts | 13 +++++++++---- 2 files changed, 19 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 706c239..66b2483 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,15 @@ # Changelog +## 2026-09-18 + +### Changes + +- [Pilot] A scenario the app never had the data for is now reported as skipped instead of failed. Asking for + the page before the first one, the page after the last, or a record that was never there used to count as a + failure, because the control was clicked and nothing happened. Pilot now settles whether the app ever held + what the scenario assumed: nothing to act on is skipped, while a control that responded and still produced + the wrong result stays a failure. + ## 2026-09-17 ### Changes diff --git a/src/ai/pilot.ts b/src/ai/pilot.ts index b719905..7e4ecd7 100644 --- a/src/ai/pilot.ts +++ b/src/ai/pilot.ts @@ -118,7 +118,7 @@ export class Pilot implements Agent { } const schema = z.object({ - decision: z.enum(['pass', 'fail', 'continue', 'skipped']).describe('pass = test succeeded, fail = test failed, continue = tester should keep going, skipped = scenario is irrelevant OR systematic execution failures prevented testing'), + decision: z.enum(['pass', 'fail', 'continue', 'skipped']).describe('pass = test succeeded, fail = test failed, continue = tester should keep going, skipped = prerequisites unmet, scenario irrelevant, OR systematic execution failures prevented testing'), reason: z.string().describe('Concise user-facing reason, maximum 1 short sentence and 120 characters. Do NOT repeat the decision status; explain only the evidence. For continue: explain why rejected and suggest alternatives.'), guidance: z.string().nullable().describe('Required for "continue": specific actionable instruction for the tester — what exactly to verify, retry differently, or complete next. Be concrete.'), requestVerification: z @@ -407,9 +407,14 @@ export class Pilot implements Agent { DOM assertion can't be made. Do not pass when Tester achieved only a related navigation/filter/tab/status outcome instead of the requested action, workflow, or entity detail goal. - - "fail": goal not achieved and no further step toward it is available on the current page. - - "skipped": scenario is irrelevant to the app, OR systematic infrastructure failures (LLM errors, - crashes) prevented testing. NOT for "test failed to interact" — that's "fail" or "continue". + Before choosing between "fail" and "skipped", settle whether the app held the state the scenario + presumes: an inert control means a defect only if there was something for it to act on. + - "fail": the app held the data/state the scenario presumes, yet the goal was not achieved and no + further step toward it is available on the current page. + - "skipped": prerequisites unmet — scenario is irrelevant to the app, or the app never held the data + or prior state it presumes, so its control is absent or present with nothing to act on. Also + systematic infrastructure failures (LLM errors, crashes). NOT when the presumed state was present + and the app still failed — that's "fail" or "continue". - "continue": goal incomplete but the control for the NEXT step is present on the current page, or a concrete missing check would change your verdict. Guidance must name that step. If a verify() asserted a state that was ALREADY TRUE before the test, it proves nothing — reject.