Skip to content

fix: recovered Scient Agent turns end completed; Continue, crash and analytics fixes - #965

Merged
yaacovcorcos merged 6 commits into
mainfrom
claude/agent-failure-fixes-20261116
Nov 16, 2026
Merged

yaacovcorcos merged 6 commits into
mainfrom
claude/agent-failure-fixes-20261116

Conversation

@yaacovcorcos

@yaacovcorcos yaacovcorcos commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

What Changed

Fixes from an audit of real Scient Agent failures (Nov 10–16) and what analytics recorded about them.

  • A turn that recovered no longer ends failed. A prompt's own run used up its DNS retry budget, then a background result woke the agent. The later run recovered and finished with stop, but Desktop still marked the turn failed with the old DNS error. The cause: prompt_result reports only the prompt's first run, and Desktop treated it as final. Now, once a later run of the same turn starts, that run's ending decides. One exception: when the later run also fails after a failed prompt, the turn keeps the prompt's structured cause (a usage limit and its reset time). Changed in OmpSessionRuntime.ts.
  • "Continue" works for Scient Agent. Its RPC has no way to continue without a message, so "Continue" (and "Continue with ") after a limit or Stop failed before starting with "cannot resume without a new message". NativeSessionAdapterV2 now offers resumeTurn when a native session declares a continuationPrompt; Scient Agent sends "Continue where you left off."
  • Session-open failures show their cause. Users saw only "Failed to open scient provider session …". The adapter's redacted reason now appears in the message (provider-core failure.ts, in its SCIENT-FORK seam).
  • One retry for the model list. A model-catalog request that timed out during a turn is now asked once more, instead of failing the turn.
  • Renderer crashes. Both main-window crashes on Nov 16 at 09:18 local, and one in a dev app, were the same Chromium failure. Chromium's soft-navigation detection hit a hard check (CHECK_GT) when an input event that changed the in-app route had a timestamp earlier than the page's load start. Soft-navigation detection is now disabled on every platform. The flag is added after Linux's own --disable-features list, because Chromium keeps only the last value.
  • network failure kind. Scient Agent's new network kind ranks above transient, so DNS and refused connections are told apart from provider outages.

Analytics:

  • retryAttempts counts retries on every node of a run. The screenshot run had 17 retries and reported 0, because native retry rows are not on the root node.
  • Runs that fail before their session starts are now reported as provider.turn.failed with started: false. Seven local failures had no event at all.
  • runRef, a one-way hash of the local run id, appears on turn outcomes and on provider.request.completed, so a run's outcome joins its request totals.
  • providerVersion (the agent build) appears on turn outcomes.
  • app.diagnostics adds oldestEventAgeBucket, so an outage shows as delayed delivery rather than missing events.
  • app.health renderer terminations add exitReason and exitCode.
  • None of these fields needs a gateway deploy (revision 10 accepts new fields). The gateway's copy of the conformance corpus is refreshed in fix(events): log why uploads are rejected or fail, without client-chosen names ScientFactory-website#76. Agent side: fix: report usage limits, network and access-policy failures; string-only CCA enums scient-agent#65.

Why

The owner hit many Scient Agent failures. Several were Scient's own defects rather than provider outages, and analytics misreported them: 0 retries, missing startup failures, and no way to link a run's outcome to its request totals.

Documentation Impact

Documentation impact: Updated — docs/internals/product-analytics.md

Checks

  • Targeted server suites (provider/omp, orchestration-v2/Adapters, telemetry, resourceTelemetry, ProviderTurnStartService, NativeQueueHoldPolicy): 2,588 passed. One failure, OmpEventBudget.native.test.ts "sheds only the largest native backlog…", fails the same way on clean origin/main on this Mac.
  • Also passing: provider-core, scient-analytics, contracts, desktop app/window tests, root pnpm run typecheck, the seam check, and the commit hooks (fmt, lint, knip).
  • Each regression test fails without its fix.
  • Review: Codex found two problems over three passes, both fixed; final pass "Fixes verified".
  • Not run: the whole web unit suite (no web code changed), desktop smoke, and live reproduction of the renderer crash.

Platforms

Platform Expected behavior Checks run Unproven
macOS All of the above; soft-navigation detection off Tests and typecheck on macOS arm64 Live crash repro (zero-timestamp click)
Windows Same; flag added on its own CI A Windows run
Linux Flag added after the Wayland feature list, keeping it Pre-ready test with the Linux path A Wayland session

Checklist

  • I explained what changed and why
  • No UI changes

🤖 Generated with Claude Code

yaacovcorcos and others added 6 commits November 16, 2026 18:39
…at failed

A turn whose prompt's own run failed (a DNS outage that spent the retry
budget) could be woken by a background result, recover and finish
normally, and still be marked failed: prompt_result reports only the
prompt's first run, and Desktop treated it as authoritative. A later run
of the same turn now decides the outcome from its own ending.

Analytics:
- retryAttempts counts retry rows on every node of the run; native retry
  rows sit on their own nodes, so the screenshot run's 17 retries became 0.
- A run that fails before its session starts is reported
  (provider.turn.failed started=false) instead of dropped.
- Turn outcomes and provider.request.completed carry runRef, a one-way
  hash of the local run id, so a run's outcome joins its request totals.
- Turn outcomes carry providerVersion (the agent build).
- app.diagnostics adds oldestEventAgeBucket, so an outage shows as late
  delivery rather than missing events.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rt how they end

Both main-window renderer crashes on 2026-11-16 (and one in a dev app)
were Chromium's CHECK_GT in SoftNavigationHeuristics: an input event that
starts an in-app route change carried a timestamp earlier than the page's
load start, which macOS passes through unchecked for posted events.
Scient reads no soft-navigation timing, so --disable-features now adds
SoftNavigationDetection on every platform, merged after the Linux list
(Chromium keeps only the last value).

app.health for a renderer termination now carries exitReason and
exitCode, so a CHECK (SIGTRAP, exit 5) is told apart from memory
pressure or a kill.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Continue after a usage limit or Stop failed before starting on Scient
Agent with "This provider cannot resume without a new message": the OMP
RPC has no message-less continue, so the native-session adapter offered
no resumeTurn. A native session can now declare a continuation prompt,
and Scient Agent's adapter sends one, so the offered Continue works.

Session-open failures showed only "Failed to open <driver> provider
session <id>". The adapter's redacted reason beneath that wrapper now
becomes the category reason, e.g. "The provider session could not be
opened: RPC command get_available_models timed out after 30000 ms. ..."

A turn that needs Scient Agent's model catalog asks again once after a
command timeout instead of failing the turn.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rrors

Scient Agent now reports a `network` kind when no response reached the
provider (DNS, refused, offline). The turn outcome ranks it before
`transient` and analytics keeps it as a failure code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review (Codex): superseding the prompt's evidence dropped its structured
cause, so a turn whose prompt run and follow-up run both hit the same 429
became a generic failure without a reset time. The prompt's evidence now
only gives way when the follow-up run ended normally.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-check (Codex): keeping prompt evidence let a successful prompt_result
mark a turn completed although its later run failed. After a later run,
only that run's ending decides, except that a failed later run after a
failed prompt keeps the prompt's structured cause.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.5 KiB 5.5 KiB 0 B (0.0%) 6.8 KiB ✅
Codex Thread snapshot wire 4.2 KiB 4.2 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.3 KiB 1.3 KiB 0 B (0.0%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 22.3 KiB 22.3 KiB 0 B (0.0%) 24.4 KiB ✅
Codex Live turn messages 1 1 0 (0.0%) 8 ✅
Claude Total thread wire 5.5 KiB 5.5 KiB 0 B (0.0%) 6.8 KiB ✅
Claude Thread snapshot wire 4.2 KiB 4.2 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.3 KiB 1.3 KiB 0 B (0.0%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 22.7 KiB 22.7 KiB 0 B (0.0%) 24.4 KiB ✅
Claude Live turn messages 1 1 0 (0.0%) 8 ✅

Other scenarios

All 64 scenario measurements match main.

All 64 scenario measurements
Provider Scenario Metric Main baseline This PR Impact PR ceiling
Codex short turn resume WebSocket decoded 12.0 KiB 12.0 KiB 0 B (0.0%) 12.7 KiB ✅
Codex short turn resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex long turn running, cold open HTTP decoded 281.7 KiB 281.7 KiB 0 B (0.0%) 302.7 KiB ✅
Codex long turn running, cold open rows 75 75 0 (0.0%) 75 ✅
Codex long turn running, history page HTTP decoded 92.4 KiB 92.4 KiB 0 B (0.0%) 107.4 KiB ✅
Codex long turn running, history page rows 75 75 0 (0.0%) 75 ✅
Codex long turn finished, cold open HTTP decoded 381.5 KiB 381.5 KiB 0 B (0.0%) 410.2 KiB ✅
Codex long turn finished, cold open rows 151 151 0 (0.0%) 151 ✅
Codex long turn finished, socket cold open WebSocket decoded 381.6 KiB 381.6 KiB 0 B (0.0%) 410.2 KiB ✅
Codex long turn finished, socket cold open WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex long turn finished, socket cold open rows 151 151 0 (0.0%) 151 ✅
Codex long turn finished, work page HTTP decoded 186.2 KiB 186.2 KiB 0 B (0.0%) 200.2 KiB ✅
Codex long turn finished, work page rows 150 150 0 (0.0%) 150 ✅
Codex idle thread resume WebSocket decoded 65 B 65 B 0 B (0.0%) 200 B ✅
Codex idle thread resume WebSocket messages 1 1 0 (0.0%) 1 ✅
Codex fleet shell snapshot HTTP decoded 45.5 KiB 45.5 KiB 0 B (0.0%) 53.7 KiB ✅
Codex fleet shell snapshot rows 27 27 0 (0.0%) 27 ✅
Codex fleet shell subscribe WebSocket decoded 65 B 65 B 0 B (0.0%) 200 B ✅
Codex fleet shell subscribe WebSocket messages 1 1 0 (0.0%) 1 ✅
Codex fleet shell resume check HTTP decoded 18 B 18 B 0 B (0.0%) 100 B ✅
Codex fleet shell resume check rows 0 0 0 0 ✅
Codex stale shell reload HTTP decoded 45.5 KiB 45.5 KiB 0 B (0.0%) 53.7 KiB ✅
Codex stale shell reload rows 27 27 0 (0.0%) 27 ✅
Codex stale shell resume WebSocket decoded 140 B 140 B 0 B (0.0%) 400 B ✅
Codex stale shell resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex stale todo list cold open HTTP decoded 72.4 KiB 72.4 KiB 0 B (0.0%) 83.0 KiB ✅
Codex stale todo list cold open rows 75 75 0 (0.0%) 75 ✅
Codex stale todo list cold open nodes 2 2 0 (0.0%) 4 ✅
Codex stale todo list cold open subagents 2 2 0 (0.0%) 4 ✅
Codex stale todo list cold open plans 1 1 0 (0.0%) 2 ✅
Codex stale todo list cold open providerTurns 1 1 0 (0.0%) 2 ✅
Codex stale todo list cold open checkpoints 1 1 0 (0.0%) 2 ✅
Claude short turn resume WebSocket decoded 12.2 KiB 12.2 KiB 0 B (0.0%) 12.7 KiB ✅
Claude short turn resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude long turn running, cold open HTTP decoded 283.3 KiB 283.3 KiB 0 B (0.0%) 302.7 KiB ✅
Claude long turn running, cold open rows 75 75 0 (0.0%) 75 ✅
Claude long turn running, history page HTTP decoded 93.4 KiB 93.4 KiB 0 B (0.0%) 107.4 KiB ✅
Claude long turn running, history page rows 75 75 0 (0.0%) 75 ✅
Claude long turn finished, cold open HTTP decoded 384.1 KiB 384.1 KiB 0 B (0.0%) 410.2 KiB ✅
Claude long turn finished, cold open rows 151 151 0 (0.0%) 151 ✅
Claude long turn finished, socket cold open WebSocket decoded 384.3 KiB 384.3 KiB 0 B (0.0%) 410.2 KiB ✅
Claude long turn finished, socket cold open WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude long turn finished, socket cold open rows 151 151 0 (0.0%) 151 ✅
Claude long turn finished, work page HTTP decoded 188.1 KiB 188.1 KiB 0 B (0.0%) 200.2 KiB ✅
Claude long turn finished, work page rows 150 150 0 (0.0%) 150 ✅
Claude idle thread resume WebSocket decoded 66 B 66 B 0 B (0.0%) 200 B ✅
Claude idle thread resume WebSocket messages 1 1 0 (0.0%) 1 ✅
Claude fleet shell snapshot HTTP decoded 46.1 KiB 46.1 KiB 0 B (0.0%) 53.7 KiB ✅
Claude fleet shell snapshot rows 27 27 0 (0.0%) 27 ✅
Claude fleet shell subscribe WebSocket decoded 66 B 66 B 0 B (0.0%) 200 B ✅
Claude fleet shell subscribe WebSocket messages 1 1 0 (0.0%) 1 ✅
Claude fleet shell resume check HTTP decoded 18 B 18 B 0 B (0.0%) 100 B ✅
Claude fleet shell resume check rows 0 0 0 0 ✅
Claude stale shell reload HTTP decoded 46.1 KiB 46.1 KiB 0 B (0.0%) 53.7 KiB ✅
Claude stale shell reload rows 27 27 0 (0.0%) 27 ✅
Claude stale shell resume WebSocket decoded 140 B 140 B 0 B (0.0%) 400 B ✅
Claude stale shell resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude stale todo list cold open HTTP decoded 73.4 KiB 73.4 KiB 0 B (0.0%) 83.0 KiB ✅
Claude stale todo list cold open rows 75 75 0 (0.0%) 75 ✅
Claude stale todo list cold open nodes 2 2 0 (0.0%) 4 ✅
Claude stale todo list cold open subagents 2 2 0 (0.0%) 4 ✅
Claude stale todo list cold open plans 1 1 0 (0.0%) 2 ✅
Claude stale todo list cold open providerTurns 1 1 0 (0.0%) 2 ✅
Claude stale todo list cold open checkpoints 1 1 0 (0.0%) 2 ✅

Baseline: f9e3944 · PR result: 0a16d87 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.
Bounded compact HTTP snapshot with live cursor.

  • Codex decoded thread snapshot: 88.2 KiB
  • Claude decoded thread snapshot: 88.4 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

Merged via the queue into main with commit 1889349 Nov 16, 2026
52 checks passed
@yaacovcorcos
yaacovcorcos deleted the claude/agent-failure-fixes-20261116 branch November 16, 2026 17:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant