Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,18 @@
# Changelog

## 2026-09-15

### Fixes

- [Pilot] A scenario whose goal is that an invalid action must not succeed is no longer failed because the
app blocks it differently than the plan predicted. The plan names one guard — a validation message, say —
but the app may block by disabling the control or refusing the submit, and any of those met the goal.
Pilot told Tester to record such a run as failed, so tests that found the app working correctly were
reported as failures.
- [Navigator] `verify()` can now assert which page is open. There was no URL check among the assertions
Navigator is allowed to write, so every claim about the current URL failed even when the page was on
exactly that address, and Tester was left without evidence that a navigation had worked.

## 2026-09-14

### Changes
Expand Down
22 changes: 22 additions & 0 deletions rules/navigator/verification-actions.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,18 @@ Checks that page title contains expected text.
I.seeInTitle('Dashboard');
</example>

### I.seeInCurrentUrl

I.seeInCurrentUrl(<path>)

Checks that the current URL contains the expected path. Substring match — pass the path, not the full URL.
This is the ONLY way to assert the URL — page text and source are not evidence of it.

<example>
I.seeInCurrentUrl('/dashboard');
I.seeInCurrentUrl('/users/42/edit');
</example>

### I.seeInSource

I.seeInSource(<text>)
Expand Down Expand Up @@ -95,6 +107,16 @@ Checks that an input field does NOT contain the specified value.
I.dontSeeInField('Email', '');
</example>

### I.dontSeeInCurrentUrl

I.dontSeeInCurrentUrl(<path>)

Checks that the current URL does NOT contain the given path.

<example>
I.dontSeeInCurrentUrl('/login');
</example>

### I.dontSeeInSource

I.dontSeeInSource(<text>)
Expand Down
1 change: 1 addition & 0 deletions src/ai/pilot.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1116,6 +1116,7 @@ export class Pilot implements Agent {
Already-achieved detection: if the scenario goal is met in the current state (page_summary, ariaDiff,
state), instruct Tester to verify() and finish(). If goal was already true at the start, propose
different input data so the test is meaningful. If Tester repeats the same successful action, STOP.
A goal of "action must not succeed" is met by any guard that blocks it — the predicted guard is a route, not a requirement.

If needed you should pick the exact item the scenario should act on (from the page, or precondition() one) and pass it to tester

Expand Down
11 changes: 3 additions & 8 deletions src/ai/tester.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1126,14 +1126,9 @@ export class Tester extends TaskAgent implements Agent {
}
}

if (input.status !== null && task.isComplete()) {
if (this.pilot) {
const currentState = this.getCurrentState();
await this.pilot.reviewCompletion(task, currentState, conversation, this.navigator);
} else {
const hasPassed = task.hasAchievedAny();
task.finish(hasPassed ? TestResult.PASSED : TestResult.FAILED);
}
if (input.status !== null && task.isComplete() && !this.pilot) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With Pilot enabled, recording the last expected outcome makes task.hasFinished true via isComplete(), but leaves task.result null. The loop then exits before finalReview() because of the if (task.hasFinished) break guard. Please keep reviewCompletion() here, or separate “all expectations settled” from the terminal state, otherwise these tests finish without a Pilot verdict

const hasPassed = task.hasAchievedAny();
task.finish(hasPassed ? TestResult.PASSED : TestResult.FAILED);
}

const remainingExpectations = task.getRemainingExpectations();
Expand Down
Loading