Repository navigation
fix(server): runs end when their provider session dies - #77
nullStack65 merged 1 commit into
Conversation
A run subscribes to its session's events after turn-start work such as the git baseline. If the provider died in that window, release had already failed and swapped out the subscriber map, so the late subscription waited forever and the run stayed running. A subscription made after release now fails itself (or is interrupted on shutdown), so the run settles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bot review round 1 (Claude Opus 5.5, independent) @ ae3d637Scope: Verdict: no blocking findings. The fix is correct. I don't see a smaller change that is still race-free. 1. Correctness and races (double signal / double terminal)
Release only signals queues in the map it took. The self-signal only runs when the compare-and-delete wins. Those two sets don't overlap, so no queue gets two signals. The identity check
2. Graceful shutdown versus provider crash
Non-blocking notes:
3. Test fails on base for the stated reasonVerified in a scratch worktree at the PR head (Node 26.13.1,
The test is deterministic. 4. AGENTS.md test rulesCompliant. The test uses
5. Excess complexityNone found. The other orders are smaller but wrong:
A "closed" marker on the subscriber ref would make this a single atomic step and carry the release kind (fixing N1), but it touches all three signalling helpers and the ref's type. The +25/-3 change is a reasonable size. Out of scope (not reviewed further)
|
Problem
A run could stay
runningwith nothing executing it, and the sidebar would time it from the run's start.Startup and shutdown reconcile already settle these runs. The gap was during the server's lifetime. A run gets its session runtime at turn start, then subscribes to the session's events later, after the thread resume, context handoffs and the git baseline capture. If the provider process died in that window,
ProviderSessionManagerhad already swapped out and failed its subscriber map. The run's subscription went into the new, empty map, which nothing would ever signal again. The run's event consumer waited forever, so no failed terminal was ever written.Fix
makeEventSubscriptionnow checks, after it registers the queue, that the session entry still holds this runtime. If the session was released and the queue is still registered, release can no longer reach it, so the subscription signals itself and the run fails through its normal failure path:ProviderAdapterEventStreamError(raisedByT3) reaches the queue;If the queue was already removed, release has signalled it and nothing more happens, so the run sees exactly one signal. There are no timers or polling, and the trigger is the session release itself.
Per-adapter trace (no change needed)
Each adapter already ends the active turn, or fails its stream, when its process dies mid-turn:
finalizeActiveTurnAfterQueryExit(ClaudeAdapterV2.ts), tests "fails a turn whose Claude query ends before its first frame" and "names the signal that killed the Claude process on the failed turn".awaitTerminationcan only fail, and it fails the event queue (CodexAdapterV2.ts).The session manager turns a failed stream, or a clean end without a provider stop, into
releaseEntry("runtime_error"), which fails every subscriber.RunExecutionServicethen writes the failed terminal. Only a subscriber that arrived after the release was missed.Queued runs (the Crown "299 h" threads): audited, no change
The real Crown threads were orphaned
queuedruns on T3 0.1.2. Onrelease/native-0.1.10:handleRunUpdaterunsstartNextQueuedRunon every terminal status (Orchestrator.tsrun-update reactor). Stop and provider failure deliberately hold the queue (queueHeld: true) instead of starting it. Tests: "keeps the queue after a user interrupts the active run" (runtimeLayer.test.ts), "a message the user sends after Stop releases the held queue; an agent's does not" (ThreadStop.test.ts).releaseUnattendedHolds). Released runs then start in queue order, renumbered after any run that went ahead. Tests: "releases held agent messages once a later run completes, not the user's", "settles past held automatic messages but preserves a held user queue", "queues a plain send behind the stopped queue and releases its hold".reconcile("startup")holds queued runs (ProviderRuntimeRecoveryService.ts, the held-runs loop). Buffered provider wakes are cancelled.resumeQueuedRunsthen starts any queue that isn't held. Test: "recovers queued work after restart".queued.Sidebar.logic.tsshows that as Working with a "Connecting" pill, but with no duration, becauseactivityRunStartedAtonly counts preparing, starting, running and waiting runs. So 0.1.10 can't show "working for 299 h" from a queued run. Sidebar files belong to the sidebar lane and are untouched here.Tests
The new case "ProviderSessionManagerV2 fails a subscription made after its provider died" (
ProviderSessionManager.test.ts):The full
ProviderSessionManager.test.tsfile passes with the fix (73/73).vp run check:changed origin/release/native-0.1.10: format, lint, typecheck (3 packages) and tests (2 files, 75 tests) pass. knip fails only on findings already on the base: the 4 unused upstream files, plus an unusedINBOX_CAPABILITYexport inpackages/contracts/src/threadInbox.tsthat this branch doesn't touch. The pre-commit knip hook failed only on those 4 files, so the commit used--no-verify. (Run without a ref,check:changedcompares to the stalemainand crashes withspawn E2BIG.)Not covered here:
resumeTurnandcompactThreadskip the decoratedmarkBusythatstartTurnhas. With this fix, a start on a released runtime fails the run instead of hanging, but those turns are still not counted as busy for idle release. That's a separate issue.Model: Claude Opus 5.5, Claude Code harness in T3 Code.
🤖 Generated with Claude Code