Ask the API whether an action persisted before failing on a missing record

[DavertMik]

Problem

Trace df450361562bbd7ef4dc36fba307c203 (session RollingRemarkableCyan648), scenario "Copy an existing suite and verify the copied suite is added to the suites tree."

The copy submitted successfully. see() then reported:

"'Suite Updated by Pilot (Copy)' is not visible in the suites tree. The tree shows 'Suite Updated by Pilot' and a notification stating it was copied to root."

The app itself confirmed the write. Pilot read this correctly in its progress analysis — "Copy submission succeeded and the app reported it copied to root, but the copied suite is not yet visible" — then returned text-only guidance ("reload with I.reloadPage()") and pre-authorised the fail. The Tester searched its tools for reloadPage, found none, and called stop(). Final verdict:

{"decision":"fail","reason":"Copy notification appeared, but no distinct copied suite is visible in the suites tree."}

askApi was in that call's tool list and went unused.

Root cause

Both places the model reads about askApi framed it as pre-action setup only:

• src/ai/pilot.ts:1160 — "what data already exists", "before creating any"
• src/ai/fisherman/tools.ts:223 (the tool's own description) — "Ask what data already exists", "Use it before precondition()", "a record that is already there"

So with the page not rendering the result, Pilot had no route to the data layer and decided from the screenshot alone.

Fix

The tool's stated purpose, in both places it is stated. Pilot's paragraph, 3 lines → 4:

-      askApi(question) — ask what data already exists. It changes nothing. Use it to check whether
-      suitable data is already there before creating any, and to get the exact name or id of an existing
-      record a step must act on.
+      askApi(question) — ask what data exists. It changes nothing. Use it to check whether
+      suitable data is already there before creating any, to get the exact name or id of an existing
+      record a step must act on, and to settle whether this run's action persisted when the app
+      reported success but the page does not show the result — not seeing a record is not proof it is absent.

and the matching clause in the tool description (plus its input-schema describe, which also said "data that already exists").

Visibility stays a concern worth recording, but it stops being the thing that decides the verdict.

No plumbing needed: askApi already does task.addNote(...) on a successful lookup (fisherman/tools.ts:244), and reviewVerdict builds its <notes> block from task.notesToString() (pilot.ts:105). A confirmed write therefore reaches the verdict context on its own.

Trade-off

Pilot spends a Fisherman round-trip at a moment that is common in async UIs. Where API access is not configured, behaviour is unchanged — askApi returns "No API access is configured… Judge from the page instead" and absence still decides the verdict.

Testing

• bun test tests/unit/ — 1436 pass, 0 fail
• bun test tests/integration/ — 148 pass, 1 skip, 0 fail (includes pilot-ask-api.test.ts)
• bun run format, bun run check:fix clean

No test added: this is a prompt clarification with no behavioural branch to pin, and persisted already appears in the shared evidence rules, so an assertion here would only restate the string it asserts.

Worth a replay of the same scenario before merge — expected: one askApi call in Pilot's progress analysis after the copy, guidance naming a check the Tester has a tool for, and a verdict that is not fail. Regression was not run; apply the label if you want it.

Noted while tracing, not fixed here

1. src/ai/pilot.ts:128 — requestVerification's description contradicts itself: "Also use when evidence is insufficient before deciding pass/fail" then "Leave null for 'continue', 'fail', or 'skipped'". pilot.ts:183 gates execution on decision === 'pass', so the first clause is dead text.
2. src/ai/tools.ts:629 — see() always returns "Visual confirmation is valid evidence for test results. Use record() to note the visual findings." True for a sighting, not for an absence, and it routes a negative straight to record(). verify()'s inexpressible branch already carries the right phrasing.
3. Pilot's guidance said I.reloadPage() without the mechanism its own prompt specifies at pilot.ts:1155 ("through form").

🤖 Generated with Claude Code