From 55ab79771e5227ae55270949670bf113188ffe64 Mon Sep 17 00:00:00 2001
From: DavertMik <davert@testomat.io>
Date: Thu, 17 Sep 2026 01:12:55 +0300
Subject: [PATCH] fix(pilot): ask the API whether an action persisted before
 failing on a missing record
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Both descriptions of askApi framed the tool purely as pre-action setup —
"what data already exists", "before precondition()". Nothing told the
model the same tool settles whether its own run's action persisted, so
when a page reported success without rendering the result, Pilot had no
route to the data layer and decided from the screenshot alone.

In trace df450361562bbd7ef4dc36fba307c203 a suite copy submitted fine and
the app showed a "copied to root" notification, but the copy was not
visible in the tree. Pilot read that correctly, then returned text-only
guidance naming a command the Tester has no tool for, and the run ended
as a failure. askApi was in that call's tool list and went unused.

askApi already writes its answer to task notes, which reviewVerdict reads,
so a confirmed write reaches the verdict without new plumbing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 CHANGELOG.md              |  8 ++++++++
 src/ai/fisherman/tools.ts | 14 ++++++++++----
 src/ai/pilot.ts           | 11 ++++++++---
 3 files changed, 26 insertions(+), 7 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 90b359d7..706c2399 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,13 @@
 # Changelog
 
+## 2026-09-17
+
+### Changes
+
+- [Pilot] When the app reports an action succeeded but the record is not visible on the page, Pilot now asks
+  the API whether it was stored instead of failing on what the screenshot shows. A record the API cannot find
+  is still a failure. Needs API access configured; without it, Pilot judges from the page as before.
+
 ## 2026-09-16
 
 ### Configuration
diff --git a/src/ai/fisherman/tools.ts b/src/ai/fisherman/tools.ts
index ac435cbc..3a43fe46 100644
--- a/src/ai/fisherman/tools.ts
+++ b/src/ai/fisherman/tools.ts
@@ -221,13 +221,19 @@ export function createAskApiTool(fisherman: Fisherman | null, task: Test) {
   return {
     askApi: tool({
       description: dedent`
-        Ask what data already exists, changing nothing.
-        Ask a question about existing records: which ones are there, what they are called, whether a particular one exists.
-        Use it before precondition() to see whether suitable data is already available, and whenever a step needs the exact name or id of a record that is already there.
+        Read the app's data over the API, changing nothing.
+        Answers questions about records: which ones are there, what they are called, whether a particular one exists.
+
+        Use it to:
+        - check whether suitable data already exists, before precondition() creates any
+        - get the exact name or id of a record a step must act on
+        - find out whether an action was stored, when the app reported success but the page does not show the result
+        - find out whether data exists at all, when a list or dropdown is empty
+
         It never creates, edits or deletes anything — precondition() does that.
       `,
       inputSchema: z.object({
-        question: z.string().describe('What to find out about data that already exists'),
+        question: z.string().describe('What to find out about the data'),
       }),
       execute: async ({ question }) => {
         tag('info').log(`Ask API: ${question}`);
diff --git a/src/ai/pilot.ts b/src/ai/pilot.ts
index f241fded..b719905a 100644
--- a/src/ai/pilot.ts
+++ b/src/ai/pilot.ts
@@ -1157,9 +1157,14 @@ export class Pilot implements Agent {
 
       YOUR Pilot-only tools, both over the API:
 
-      askApi(question) — ask what data already exists. It changes nothing. Use it to check whether
-      suitable data is already there before creating any, and to get the exact name or id of an existing
-      record a step must act on.
+      askApi(question) — read the app's data over the API. It changes nothing. Use when:
+
+      - Before precondition() — check whether suitable data already exists.
+      - A step needs the exact name or id of an existing record.
+      - The app reported success but the page does not show the result — ask whether it was stored.
+      - A list or dropdown is empty — ask whether the data exists at all.
+
+      The page is not the only witness. A record missing from the page may still exist.
 
       precondition(description) — create FRESH disposable test data. Never request users. Use when: