Exempt running tools from the AI idle timeout

[DavertMik]

Problem

Session UsualNegativeGray57 (trace 4892e780…) failed the scenario "Create a mixed manual and automated run and verify it appears in the runs list." — but the run was created: POST /api/…/runs 200, navigation to /runs/55c2d528. Every form() call still reported success: false, and Pilot voted fail on "Creation timed out".

I.fillField('Title', …) on the mixed-run form took 22.5s (step meta 1789530314391 → 1789530336908). That pushed the roundtrip past the 30s deadline armed in raceWithIdleTimeout, abortAfterIdle fired mid-tool, and withRetry (maxAttempts: 3) replayed the turn. Three invoke_agent spans overlap to the millisecond:

attempt	start	end	level
1	03:44:48.802	03:46:03.796	DEFAULT
2	03:45:18.819 (+30.017s)	03:46:02.515	ERROR This operation was aborted
3	03:45:48.850 (+30.031s)	03:46:25.128	ERROR This operation was aborted

All three issued a creation form() into the one global CodeceptJS recorder. Attempt 1's commands ran (fillField OK → click All tests OK → click Save OK → run created); attempt 2's fillField then ran against the already-navigated detail page and threw. Because the recorder and attachStepLogger's event dispatcher are process-global, attempt 1's await recorder.promise() rejected on attempt 2's error — so action.lastError was set for a tool whose own commands had all succeeded.

The result was one payload containing both outcomes:

attempts: [OK fillField, OK click "All tests", OK click "Save", FAILED fillField]
pageDiff: { urlChanged: true, currentUrl: "/runs/55c2d528",
            requests: [POST /api/…/runs 200, …] }

Every I.step span in that window is duplicated in pairs — two step loggers attached at once. Pilot's notes carry Execution error: AI request timeout verbatim.

Fix

abortAfterIdle measured wall-clock elapsed over the entire roundtrip, tool execution included, and so read a slow tool as an idle model. Its one existing exemption — executionController.isAwaitingInput() — is this same bug already patched narrowly for a single blocking tool. This generalizes it to any tool:

• withIdleExemption() wraps each tool's execute to increment/decrement a toolsRunning counter
• abortAfterIdle re-arms while toolsRunning > 0, so the timer only fires when the model itself is what we are waiting on
• applied once at the generateWithTools parameter, so both the withHarmonyChannelFallback path and the required-tool-choice relax path inherit it

27 lines, one file.

Considered and rejected

• Dropping 'AI request timeout' from retryCondition. It is dead — the generic error.message.includes('timeout') above it already matches — and removing the generic match too would still leave the abort firing mid-fillField, with the zombie form colliding with whatever the next tester iteration sends into the same recorder. The retry amplifies; it is not the cause.
• Re-arming on onStepEnd. Steps bracket model+tools together, so the timer would still have fired at 03:45:48 inside a tool that ended at 03:46:03.
• Reading setActivity state. Activity is also set for the model call itself, so it cannot tell model-idle from tool-busy.
• Touching Pilot or Tester. Both behaved correctly on the inputs they were given. askApi would indeed have proven the run existed, but it is registered on Pilot's supervision conversation while the verdict call has no tools — and the tester's belief was already false two minutes upstream.

Trade-off

A genuinely hung browser tool is no longer bounded by the AI idle timeout, only by Playwright's own setDefaultTimeout(config.action?.timeout ?? DEFAULT_ACTION_TIMEOUT) in Action.executeOnce. That is the right owner for a browser deadline, but a tool that hangs below Playwright's timeout now stalls the turn instead of being killed at 30s.

Verification

• bun test tests/unit/ — 1423 pass, 0 fail
• bun test tests/integration/ — 148 pass, 1 fail (Prima attaches to a live playwright-cli session, pre-existing and environmental; fails identically on main)
• bun run check:fix — clean
• tsc -p tsconfig.json — same 6 pre-existing provider.ts errors as main, none new

Replaying the scenario should show one invoke_agent span per tester iteration with no overlap, zero AI request timeout notes, the first form() returning success: true with POST /runs 200 + urlChanged, and no I.step spans emitted in duplicate pairs. This one needs a regression run to confirm end to end — flagging rather than labelling.

🤖 Generated with Claude Code

[DenysKuchma]

This counter is process-global, so a tool running in one request disables the idle timeout for other concurrent requests too. It should be scoped to the individual generateWithTools call.