Skip to content

fix(server): a failed child spawn no longer crashes the server - #313

Merged
andrewcai8 merged 3 commits into
mainfrom
fix/server-spawn-error-crash
Nov 22, 2026
Merged

andrewcai8 merged 3 commits into
mainfrom
fix/server-spawn-error-crash

Conversation

@andrewcai8

Copy link
Copy Markdown
Owner

Why

A failed child spawn killed the whole server. On the production home server (runtime-v0.1.141, 2026-11-21 22:16) the machine hit fs.file-max, every spawn failed with ENFILE, and the process died:

[cause]: TypeError: Cannot read properties of undefined (reading '4')
    at spawnClaudeProcessGroup (.../runtimeLayer.mjs)
node:events:505
    throw er; // Unhandled 'error' event
Error: spawn /bin/sh ENFILE

spawnClaudeProcessGroup never attached an 'error' listener. After a failed spawn child.stdio is undefined, so child.stdio[4] threw a TypeError (the turn failed), then Node emitted 'error' on the ChildProcess on the next tick with no listener, which is fatal.

What changed

  • spawnClaudeProcessGroup listens for 'error' right after spawn, and throws a clear error when the spawn has no pid instead of reading child.stdio. The errno (ENFILE, EMFILE, EAGAIN, ...) only arrives with the event, so the listener copies code and cause onto the thrown error.
  • The same fault in three other long-lived paths: the Claude capability probe (spawnClaudeProbeThroughEnv), the Namespace CLI runners (spawnNsc, executeCli), and the service launcher's runtime spawn. Each touched child.stdio, child.stdout or child.stdin before attaching an error listener. With EMFILE/ENFILE those are undefined, not null.

A failed spawn now fails only the turn or operation that asked for it.

Scope

Every raw node:child_process spawn in apps/server/src was checked. Already safe: SpawnHost, spawnHostProcess, SpawnWorker, windowsVerbatimSpawn, speech/native.ts, the systemd daemon-reload spawn, cli/triage.ts. All attach an error listener synchronously after spawn. execFile/execFileSync/spawnSync call sites report errors through their callback or return value. Effect ChildProcessSpawner users are untouched. Script strings written to disk or sent over SSH (AgentDeviceShim, sshDeviceScript, deviceToolMaintenance) run in other processes.

Tradeoffs

The thrown error's message is generic because Node reports the errno one tick after spawn returns, so code is filled in on the error object a tick later. The three extra sites got no new tests: reproducing EMFILE needs file-descriptor exhaustion.

Verification

  • New tests in ClaudeAgentSdkQueryRunner.test.ts fail on the test commit alone (expected [ …(1) ] to deeply equal []) and pass with the fix. One mocks a spawn that fails with no pid; the other uses an already-aborted signal.
  • vp test run on the touched test files, vp run typecheck in apps/server, and vp run check:changed all pass.

🤖 Generated with Claude Code

andrewcai8 and others added 3 commits November 21, 2026 22:59
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
spawnClaudeProcessGroup attached no 'error' listener and read child.stdio[4]
on a child that never started, so ENFILE/EMFILE threw a TypeError and then
Node emitted an unhandled 'error' that killed the process. Listen first and
throw a clear error when the spawn has no pid.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tdio

Same fault as the Claude CLI spawn: with EMFILE/ENFILE a child has no stdio,
so code that reads it before listening throws and then the child's 'error'
goes unhandled. Covers the Claude probe, the Namespace CLI runners and the
service launcher's runtime spawn.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted Author is trusted by repo permissions or the VOUCHED list. size:M labels Nov 22, 2026
@andrewcai8

Copy link
Copy Markdown
Owner Author

Verifier verdict (round 1): PASS+NOTES

Independent check of head a227d2e3be. No blocking findings.

1. Correctness

  • spawnClaudeProcessGroup (ClaudeAdapterV2.ts:434-445): the 'error' listener goes on synchronously after spawn, before anything that can throw. A child with no pid now throws an ordinary Error, and the errno (code, cause) is added on the next tick. A started child takes the same path as before. The new listener is added alongside existing ones, and the caller still listens with child.on("error", releaseAgent) (ClaudeAdapterV2.ts:1672), so it does not swallow anything that used to reach a handler.
  • spawnNsc and executeCli: the same reject listener now attaches before the first stdio access, and pid === undefined returns early. The 'error' event still rejects. spawnNsc still removes the deadline listener. executeCli returns before it registers its abort listener, so nothing leaks.
  • spawnClaudeProbeThroughEnv: the no-op listener is added alongside any others, so the SDK's own listeners still fire.
  • serviceLauncher: the truthiness guard skips appendOutput(undefined). The 'error' listener inside the Promise executor attaches in the same tick, so the launch rejects normally.

2. Tests

  • Head: both new tests pass.
  • On origin/main's apps/server/src with the head's test file, both fail for the stated reason. The test with no pid fails with expected [ …(1) ] to deeply equal [] (an uncaught spawn /nonexistent/t3-shell ENOENT). The already-aborted signal test fails the same way with an uncaught AbortError.
  • The reported diagnostic ClaudeAgentSdkQueryRunner.test.ts:983:35 ... 'signal' is missing does not reproduce at this head: line 983 passes signal: new AbortController().signal. It looks stale, from an earlier revision. vp run typecheck in apps/server exits 0 with no error TS lines. apps/server/tsconfig.json includes src, so test files are typechecked, and CI's vpr typecheck ("Build and Typecheck") is green on this PR.

3. Missed sites

None found. I checked every raw node:child_process spawn in non-test apps/server/src: SpawnHost, spawnHostProcess (including the canStopReading probe), SpawnWorker, windowsVerbatimSpawn, speech/native (both), serviceLauncher's daemon-reload, and cli/triage. Each attaches 'error' synchronously and guards stdio access. cli/uninstall goes through awaitDetachedSpawn. The execFile users (noCopyOnWrite, DocumentText, skillBundleSync, namespaceSdkRunner) are safe because Node's execFile attaches its own error handler and guards child.stdout/child.stderr. spawnSync and execFileSync return their errors or throw them.

4. Comments

Every new comment is one line and states a constraint, not the fix. vp lint on the six files is clean.

Notes (non-blocking, follow-up at most)

  • When spawnClaudeProcessGroup throws, stderrClosed never resolves, so the failure path waits the full CLAUDE_STDERR_DRAIN_TIMEOUT (1s, ClaudeAdapterV2.ts:382) before it fails the turn. main had the same wait after its TypeError. It is bounded and harmless.
  • The already-aborted signal test calls spawnClaudeProcessGroup directly. In production the caller's listener at :1672 already covered a started child, so the real crash this PR fixes is the no-pid path, and the first test covers it.

Commands

git checkout origin/fix/server-spawn-error-crash   # a227d2e3be
vp i
cd apps/server && vp test run src/orchestration-v2/Adapters/ClaudeAgentSdkQueryRunner.test.ts -t "unhandled 'error' event"   # 2 passed
git checkout origin/main -- apps/server/src && git checkout a227d2e3be -- apps/server/src/orchestration-v2/Adapters/ClaudeAgentSdkQueryRunner.test.ts
cd apps/server && vp test run src/orchestration-v2/Adapters/ClaudeAgentSdkQueryRunner.test.ts -t "unhandled 'error' event"   # 2 failed (uncaught ENOENT / AbortError)
git checkout a227d2e3be -- apps/server/src && git reset
cd apps/server && vp run typecheck   # exit 0, no error TS
vp lint <6 changed files>            # exit 0
grep -rnE "\b(spawn|fork)\(" --include='*.ts' apps/server/src   # site sweep

@github-actions

Copy link
Copy Markdown

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ The exact PR base did not have a successful artifact. Baseline uses the latest successful main measurement shown below.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Codex Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 21.5 KiB 21.5 KiB 0 B (0.0%) 24.4 KiB ✅
Codex Live turn messages 1 1 0 (0.0%) 8 ✅
Claude Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Claude Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 21.9 KiB 21.9 KiB 0 B (0.0%) 24.4 KiB ✅
Claude Live turn messages 1 1 0 (0.0%) 8 ✅

Other scenarios

4 of 64 scenario measurements changed or failed.

Provider Scenario Metric Main baseline This PR Impact PR ceiling
Codex fleet shell snapshot HTTP decoded 42.4 KiB 42.4 KiB +20 B (+0.0%) 53.7 KiB ✅
Codex stale shell reload HTTP decoded 42.4 KiB 42.4 KiB +20 B (+0.0%) 53.7 KiB ✅
Claude fleet shell snapshot HTTP decoded 43.0 KiB 43.0 KiB +20 B (+0.0%) 53.7 KiB ✅
Claude stale shell reload HTTP decoded 43.0 KiB 43.0 KiB +20 B (+0.0%) 53.7 KiB ✅
All 64 scenario measurements
Provider Scenario Metric Main baseline This PR Impact PR ceiling
Codex short turn resume WebSocket decoded 10.5 KiB 10.5 KiB 0 B (0.0%) 12.7 KiB ✅
Codex short turn resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex long turn running, cold open HTTP decoded 281.7 KiB 281.7 KiB 0 B (0.0%) 302.7 KiB ✅
Codex long turn running, cold open rows 75 75 0 (0.0%) 75 ✅
Codex long turn running, history page HTTP decoded 92.4 KiB 92.4 KiB 0 B (0.0%) 107.4 KiB ✅
Codex long turn running, history page rows 75 75 0 (0.0%) 75 ✅
Codex long turn finished, cold open HTTP decoded 381.5 KiB 381.5 KiB 0 B (0.0%) 410.2 KiB ✅
Codex long turn finished, cold open rows 151 151 0 (0.0%) 151 ✅
Codex long turn finished, socket cold open WebSocket decoded 381.6 KiB 381.6 KiB 0 B (0.0%) 410.2 KiB ✅
Codex long turn finished, socket cold open WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex long turn finished, socket cold open rows 151 151 0 (0.0%) 151 ✅
Codex long turn finished, work page HTTP decoded 186.2 KiB 186.2 KiB 0 B (0.0%) 200.2 KiB ✅
Codex long turn finished, work page rows 150 150 0 (0.0%) 150 ✅
Codex idle thread resume WebSocket decoded 65 B 65 B 0 B (0.0%) 200 B ✅
Codex idle thread resume WebSocket messages 1 1 0 (0.0%) 1 ✅
Codex fleet shell snapshot HTTP decoded 42.4 KiB 42.4 KiB +20 B (+0.0%) 53.7 KiB ✅
Codex fleet shell snapshot rows 27 27 0 (0.0%) 27 ✅
Codex fleet shell subscribe WebSocket decoded 65 B 65 B 0 B (0.0%) 200 B ✅
Codex fleet shell subscribe WebSocket messages 1 1 0 (0.0%) 1 ✅
Codex fleet shell resume check HTTP decoded 18 B 18 B 0 B (0.0%) 100 B ✅
Codex fleet shell resume check rows 0 0 0 0 ✅
Codex stale shell reload HTTP decoded 42.4 KiB 42.4 KiB +20 B (+0.0%) 53.7 KiB ✅
Codex stale shell reload rows 27 27 0 (0.0%) 27 ✅
Codex stale shell resume WebSocket decoded 140 B 140 B 0 B (0.0%) 400 B ✅
Codex stale shell resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex stale todo list cold open HTTP decoded 72.4 KiB 72.4 KiB 0 B (0.0%) 83.0 KiB ✅
Codex stale todo list cold open rows 75 75 0 (0.0%) 75 ✅
Codex stale todo list cold open nodes 2 2 0 (0.0%) 4 ✅
Codex stale todo list cold open subagents 2 2 0 (0.0%) 4 ✅
Codex stale todo list cold open plans 1 1 0 (0.0%) 2 ✅
Codex stale todo list cold open providerTurns 1 1 0 (0.0%) 2 ✅
Codex stale todo list cold open checkpoints 1 1 0 (0.0%) 2 ✅
Claude short turn resume WebSocket decoded 10.6 KiB 10.6 KiB 0 B (0.0%) 12.7 KiB ✅
Claude short turn resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude long turn running, cold open HTTP decoded 283.3 KiB 283.3 KiB 0 B (0.0%) 302.7 KiB ✅
Claude long turn running, cold open rows 75 75 0 (0.0%) 75 ✅
Claude long turn running, history page HTTP decoded 93.4 KiB 93.4 KiB 0 B (0.0%) 107.4 KiB ✅
Claude long turn running, history page rows 75 75 0 (0.0%) 75 ✅
Claude long turn finished, cold open HTTP decoded 384.2 KiB 384.2 KiB 0 B (0.0%) 410.2 KiB ✅
Claude long turn finished, cold open rows 151 151 0 (0.0%) 151 ✅
Claude long turn finished, socket cold open WebSocket decoded 384.3 KiB 384.3 KiB 0 B (0.0%) 410.2 KiB ✅
Claude long turn finished, socket cold open WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude long turn finished, socket cold open rows 151 151 0 (0.0%) 151 ✅
Claude long turn finished, work page HTTP decoded 188.1 KiB 188.1 KiB 0 B (0.0%) 200.2 KiB ✅
Claude long turn finished, work page rows 150 150 0 (0.0%) 150 ✅
Claude idle thread resume WebSocket decoded 66 B 66 B 0 B (0.0%) 200 B ✅
Claude idle thread resume WebSocket messages 1 1 0 (0.0%) 1 ✅
Claude fleet shell snapshot HTTP decoded 43.0 KiB 43.0 KiB +20 B (+0.0%) 53.7 KiB ✅
Claude fleet shell snapshot rows 27 27 0 (0.0%) 27 ✅
Claude fleet shell subscribe WebSocket decoded 66 B 66 B 0 B (0.0%) 200 B ✅
Claude fleet shell subscribe WebSocket messages 1 1 0 (0.0%) 1 ✅
Claude fleet shell resume check HTTP decoded 18 B 18 B 0 B (0.0%) 100 B ✅
Claude fleet shell resume check rows 0 0 0 0 ✅
Claude stale shell reload HTTP decoded 43.0 KiB 43.0 KiB +20 B (+0.0%) 53.7 KiB ✅
Claude stale shell reload rows 27 27 0 (0.0%) 27 ✅
Claude stale shell resume WebSocket decoded 140 B 140 B 0 B (0.0%) 400 B ✅
Claude stale shell resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude stale todo list cold open HTTP decoded 73.4 KiB 73.4 KiB 0 B (0.0%) 83.0 KiB ✅
Claude stale todo list cold open rows 75 75 0 (0.0%) 75 ✅
Claude stale todo list cold open nodes 2 2 0 (0.0%) 4 ✅
Claude stale todo list cold open subagents 2 2 0 (0.0%) 4 ✅
Claude stale todo list cold open plans 1 1 0 (0.0%) 2 ✅
Claude stale todo list cold open providerTurns 1 1 0 (0.0%) 2 ✅
Claude stale todo list cold open checkpoints 1 1 0 (0.0%) 2 ✅

Baseline: 9e65e61 · PR result: a227d2e · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@andrewcai8
andrewcai8 merged commit fb671ec into main Nov 22, 2026
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M vouch:trusted Author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant