diff --git a/CHANGELOG.md b/CHANGELOG.md index 90b359d7..706c2399 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,13 @@ # Changelog +## 2026-09-17 + +### Changes + +- [Pilot] When the app reports an action succeeded but the record is not visible on the page, Pilot now asks + the API whether it was stored instead of failing on what the screenshot shows. A record the API cannot find + is still a failure. Needs API access configured; without it, Pilot judges from the page as before. + ## 2026-09-16 ### Configuration diff --git a/src/ai/fisherman/tools.ts b/src/ai/fisherman/tools.ts index ac435cbc..3a43fe46 100644 --- a/src/ai/fisherman/tools.ts +++ b/src/ai/fisherman/tools.ts @@ -221,13 +221,19 @@ export function createAskApiTool(fisherman: Fisherman | null, task: Test) { return { askApi: tool({ description: dedent` - Ask what data already exists, changing nothing. - Ask a question about existing records: which ones are there, what they are called, whether a particular one exists. - Use it before precondition() to see whether suitable data is already available, and whenever a step needs the exact name or id of a record that is already there. + Read the app's data over the API, changing nothing. + Answers questions about records: which ones are there, what they are called, whether a particular one exists. + + Use it to: + - check whether suitable data already exists, before precondition() creates any + - get the exact name or id of a record a step must act on + - find out whether an action was stored, when the app reported success but the page does not show the result + - find out whether data exists at all, when a list or dropdown is empty + It never creates, edits or deletes anything — precondition() does that. `, inputSchema: z.object({ - question: z.string().describe('What to find out about data that already exists'), + question: z.string().describe('What to find out about the data'), }), execute: async ({ question }) => { tag('info').log(`Ask API: ${question}`); diff --git a/src/ai/pilot.ts b/src/ai/pilot.ts index f241fded..b719905a 100644 --- a/src/ai/pilot.ts +++ b/src/ai/pilot.ts @@ -1157,9 +1157,14 @@ export class Pilot implements Agent { YOUR Pilot-only tools, both over the API: - askApi(question) — ask what data already exists. It changes nothing. Use it to check whether - suitable data is already there before creating any, and to get the exact name or id of an existing - record a step must act on. + askApi(question) — read the app's data over the API. It changes nothing. Use when: + + - Before precondition() — check whether suitable data already exists. + - A step needs the exact name or id of an existing record. + - The app reported success but the page does not show the result — ask whether it was stored. + - A list or dropdown is empty — ask whether the data exists at all. + + The page is not the only witness. A record missing from the page may still exist. precondition(description) — create FRESH disposable test data. Never request users. Use when: