Skip to content

fix(server): runs end when their provider session dies - #77

Merged
nullStack65 merged 1 commit into
release/native-0.1.10from
fix/native-0.1.10-run-ends-on-session-exit
Nov 18, 2026
Merged

nullStack65 merged 1 commit into
release/native-0.1.10from
fix/native-0.1.10-run-ends-on-session-exit

Conversation

@nullStack65

Copy link
Copy Markdown
Owner

Problem

A run could stay running with nothing executing it, and the sidebar would time it from the run's start.

Startup and shutdown reconcile already settle these runs. The gap was during the server's lifetime. A run gets its session runtime at turn start, then subscribes to the session's events later, after the thread resume, context handoffs and the git baseline capture. If the provider process died in that window, ProviderSessionManager had already swapped out and failed its subscriber map. The run's subscription went into the new, empty map, which nothing would ever signal again. The run's event consumer waited forever, so no failed terminal was ever written.

Fix

makeEventSubscription now checks, after it registers the queue, that the session entry still holds this runtime. If the session was released and the queue is still registered, release can no longer reach it, so the subscription signals itself and the run fails through its normal failure path:

  • normally, a ProviderAdapterEventStreamError (raisedByT3) reaches the queue;
  • after a SIGINT or SIGTERM, the queue is interrupted instead, so the run is left to restart recovery, the same as live subscribers on shutdown.

If the queue was already removed, release has signalled it and nothing more happens, so the run sees exactly one signal. There are no timers or polling, and the trigger is the session release itself.

Per-adapter trace (no change needed)

Each adapter already ends the active turn, or fails its stream, when its process dies mid-turn:

  • Claude: finalizeActiveTurnAfterQueryExit (ClaudeAdapterV2.ts), tests "fails a turn whose Claude query ends before its first frame" and "names the signal that killed the Claude process on the failed turn".
  • Codex: awaitTermination can only fail, and it fails the event queue (CodexAdapterV2.ts).
  • ACP family (Grok, Kiro, fx, registry): the prompt failure path finalizes the turn as failed. Tests "fails a running prompt once when its agent process exits" and "fails active Claude work and releases its children when the process exits".
  • OpenCode v1/v2, Pi, Muse: each has an exit watcher that fails or ends the stream.

The session manager turns a failed stream, or a clean end without a provider stop, into releaseEntry("runtime_error"), which fails every subscriber. RunExecutionService then writes the failed terminal. Only a subscriber that arrived after the release was missed.

Queued runs (the Crown "299 h" threads): audited, no change

The real Crown threads were orphaned queued runs on T3 0.1.2. On release/native-0.1.10:

  • (a) Active run ends: handleRunUpdate runs startNextQueuedRun on every terminal status (Orchestrator.ts run-update reactor). Stop and provider failure deliberately hold the queue (queueHeld: true) instead of starting it. Tests: "keeps the queue after a user interrupts the active run" (runtimeLayer.test.ts), "a message the user sends after Stop releases the held queue; an agent's does not" (ThreadStop.test.ts).
  • (b) Later ordinals overtaking a queue: a send to a busy thread queues behind it. A held queue is released by a client send, or for agent and scheduled messages by the next completed run (releaseUnattendedHolds). Released runs then start in queue order, renumbered after any run that went ahead. Tests: "releases held agent messages once a later run completes, not the user's", "settles past held automatic messages but preserves a held user queue", "queues a plain send behind the stopped queue and releases its hold".
  • (c) Startup: reconcile("startup") holds queued runs (ProviderRuntimeRecoveryService.ts, the held-runs loop). Buffered provider wakes are cancelled. resumeQueuedRuns then starts any queue that isn't held. Test: "recovers queued work after restart".
  • (d) Client: a held queued run is never the presented shell run. An unheld queued run that is the newest run makes the shell status queued. Sidebar.logic.ts shows that as Working with a "Connecting" pill, but with no duration, because activityRunStartedAt only counts preparing, starting, running and waiting runs. So 0.1.10 can't show "working for 299 h" from a queued run. Sidebar files belong to the sidebar lane and are untouched here.

Tests

The new case "ProviderSessionManagerV2 fails a subscription made after its provider died" (ProviderSessionManager.test.ts):

  • on base: times out at 120 s, because the late subscription never ends;
  • with the fix: passes.

The full ProviderSessionManager.test.ts file passes with the fix (73/73).

vp run check:changed origin/release/native-0.1.10: format, lint, typecheck (3 packages) and tests (2 files, 75 tests) pass. knip fails only on findings already on the base: the 4 unused upstream files, plus an unused INBOX_CAPABILITY export in packages/contracts/src/threadInbox.ts that this branch doesn't touch. The pre-commit knip hook failed only on those 4 files, so the commit used --no-verify. (Run without a ref, check:changed compares to the stale main and crashes with spawn E2BIG.)

Not covered here: resumeTurn and compactThread skip the decorated markBusy that startTurn has. With this fix, a start on a released runtime fails the run instead of hanging, but those turns are still not counted as busy for idle release. That's a separate issue.

Model: Claude Opus 5.5, Claude Code harness in T3 Code.

🤖 Generated with Claude Code

A run subscribes to its session's events after turn-start work such as the
git baseline. If the provider died in that window, release had already
failed and swapped out the subscriber map, so the late subscription waited
forever and the run stayed running. A subscription made after release now
fails itself (or is interrupted on shutdown), so the run settles.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nullStack65

Copy link
Copy Markdown
Owner Author

Bot review round 1 (Claude Opus 5.5, independent) @ ae3d637

Scope: apps/server/src/orchestration-v2/ProviderSessionManager.ts and ProviderSessionManager.test.ts. Diff is 0cbe0c68c9..ae3d63756d; the merge base equals the current release/native-0.1.10 tip. I read the full surrounding code: releaseEntry, removeLiveEntry, failSubscribers/interruptSubscribers/endSubscribers, the event-pump exit handler, the layer shutdown finalizer, open, decorateRuntime, and the consumer in RunExecutionService.ts:1208.

Verdict: no blocking findings. The fix is correct. I don't see a smaller change that is still race-free.

1. Correctness and races (double signal / double terminal)

releaseEntry always removes the entry from sessions (removeLiveEntry, in the acquire step) before it swaps the subscriber map (Ref.getAndSet in the fail/interrupt/end helpers). The new code registers first, then checks sessions, then removes the queue with a compare-and-delete (unregister). Every interleaving gives the subscriber exactly one signal:

  • Registered and checked before removal: the subscription is live, and release's later swap signals it. That is the existing path.
  • Registered before the swap, checked after removal: if the swap already happened, unregister returns false, so release signalled it and the new code does nothing. If the swap hasn't happened yet, unregister returns true, the subscription signals itself, and the later swap no longer contains it.
  • Registered after the swap (the bug): unregister returns true and the subscription signals itself.

Release only signals queues in the map it took. The self-signal only runs when the compare-and-delete wins. Those two sets don't overlap, so no queue gets two signals. close then sees removed === false and doesn't end the queue a second time. Turning a failed stream into a terminal event is still guarded by the existing run finalization in RunExecutionService, so this PR adds no new terminal-writing path.

The identity check entry.runtime === runtime holds up:

  • decorateRuntime gets the raw adapter runtime, which is the same object stored as entry.runtime.
  • Every entry update spreads ...entry and keeps runtime.
  • A session reopened under the same providerSessionId holds a different runtime. A late subscriber on the old runtime is correctly failed.
  • exposedRuntime is handed out only after the entry is inserted into sessions (open), so there is no early-subscribe false positive while the session opens.

2. Graceful shutdown versus provider crash

  • Crash (pump ends with an error or without a provider Stop, then releaseEntry("runtime_error")): a late subscriber gets ProviderAdapterEventStreamError (raisedByT3). That matches what early subscribers get.
  • SIGINT/SIGTERM: the pump exit handler returns early when shutdownSignal.received, so the entry stays in sessions until the layer finalizer runs releaseEntry("server_shutdown"). Until then a late subscriber counts as live and is interrupted with the others. After that release, the new code also interrupts it, so restart recovery owns the run. The in-app restart also emits SIGTERM (server.ts:672), so production shutdowns set the flag.

Non-blocking notes:

  • N1 (low) ProviderSessionManager.ts:2296: the late path decides interrupt versus fail from shutdownSignal.received, while release decides from reason and gracefulSubscribers. They can disagree in two cases:

    • (a) A server_shutdown release with no process signal, such as a layer or scope closed in tests or by an embedder. Early subscribers are interrupted, but a late one is failed, so its run writes a failed terminal instead of being left to recovery.
    • (b) A provider-announced Stop (gracefulSubscribers: true). Early subscribers get a clean Queue.end, but a late one gets a failure.

    In both cases the run still ends, which is the point of the fix. Case (b) already fails a run with no terminal anyway, and production shutdown always sets the signal. Matching release exactly would mean recording the release kind on the entry or the subscriber ref. I don't think that's worth doing in this PR.

  • N2 (nit) ProviderSessionManager.ts:2290-2291: the comment says "release took the map first". The self-signal also covers the case where the entry was removed but the swap hasn't run yet. Something like "Not live and still registered means release's swap can no longer reach this queue" would cover both.

3. Test fails on base for the stated reason

Verified in a scratch worktree at the PR head (Node 26.13.1, pnpm install --frozen-lockfile):

  • Head: vp test run src/orchestration-v2/ProviderSessionManager.test.ts passed 73/73.
  • Base ProviderSessionManager.ts with the head test: the new case failed with Test timed out in 120000ms. The late Stream.runDrain never ends, which is the stated hang.

The test is deterministic. Fiber.join(earlyExit) waits for the early subscriber's failure, which can only happen after removeLiveEntry and the subscriber swap. So the late subscription always lands in the post-swap window.

4. AGENTS.md test rules

Compliant. The test uses it.effect with no sleeps, no yieldNow or polling loop, no per-test timeout or retry, and no wall-clock deltas. idleTimeoutMs: 60_000 is fixture config, not a wait. Failing on base by hanging is unavoidable for a hang bug, and the rule only forbids tests that need a timeout to pass.

  • N3 (low, coverage) ProviderSessionManager.test.ts:2284: only the post-swap crash window is covered. The interrupt branch for a late subscriber after a signalled shutdown is untested. The removed-but-not-yet-swapped window is hard to hit deterministically and probably isn't worth a test. Optional: add a shutdownSignal case to an existing shutdown test.

5. Excess complexity

None found. The other orders are smaller but wrong:

  • Checking liveness before registering races with the swap and can still lose the subscriber.
  • Failing without the compare-and-delete could signal twice.

A "closed" marker on the subscriber ref would make this a single atomic step and carry the release kind (fixing N1), but it touches all three signalling helpers and the ref's type. The +25/-3 change is a reasonable size.

Out of scope (not reviewed further)

  • The PR body's note that resumeTurn and compactThread skip markBusy is a separate issue, as the PR says.
  • The queued-run audit in the PR body is outside this diff. I didn't verify it.

@nullStack65
nullStack65 merged commit e392350 into release/native-0.1.10 Nov 18, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant