Skip to content

fix(server): starting processes on Linux no longer freezes the server again - #295

Merged
andrewcai8 merged 1 commit into
mainfrom
fix/restore-spawn-host
Nov 16, 2026
Merged

andrewcai8 merged 1 commit into
mainfrom
fix/restore-spawn-host

Conversation

@andrewcai8

@andrewcai8 andrewcai8 commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Since the upstream sync (#294), the mini PC's server stalls for 2–6 s at a time, about a minute of every hour. A 45 s CPU profile of the live server showed 75% of the main thread inside child_process.spawn. The stall log agrees: per 30 s, 10.5 s of system CPU and about 630k minor page faults. Before the sync it was 2–3 s and 125–166k. The server starts about 30 processes a second (git, tar, gzip, claude). Node 24.22 bundles libuv 1.52, which forks each one on the main thread and copies the 7 GB server's page tables every time.

#294 deleted the fork's SpawnHost (#269) because upstream's pingdotgg#30209 looked like it covered this. It doesn't. NonBlockingSpawner only moves the chdir and PATH checks off the main thread. On Linux with GNU env -C, SpawnWorker.layerServer still hands spawning to Node's own spawner on the main thread.

The fix restores SpawnHost, its host process, the __spawn-host entry and the process-metrics grouping, all unchanged from before the sync. SpawnWorker.layerServer now returns SpawnHost.layer on that branch. NonBlockingSpawner still wraps it, so hung-mount protection stays. SpawnHost.layer is Node's spawner everywhere except Linux before libuv 1.53, so macOS doesn't change and Windows keeps its workers.

Verification: typecheck, lint, format and knip pass. The SpawnHost tests only run on Linux, so CI runs them. On a Mac, SpawnWorker.test.ts fails one case on /private/var vs /var, which is unrelated to this change. After it deploys, I'll re-profile the mini PC and compare the stall log.

Follow-ups (both pre-existing in SpawnHost before #294, found by the independent review):

  • Run upstream's NUL-character and shell-with-args checks before handing a command to the host, so those fail as badArgument instead of a defect or a silently different command.
  • Redact Redacted args in the host's spawnargs error cause, as upstream's spawner does.

Model: Claude Opus 5.5 in Claude Code.

🤖 Generated with Claude Code

… again

The upstream sync deleted SpawnHost, assuming upstream's NonBlockingSpawner
replaced it. It does not: on Linux with GNU env -C, upstream still calls
child_process.spawn on the main thread, and Node's libuv 1.52 forks there,
copying the server's page tables. On the mini PC spawning took 75% of the
main thread and the event loop stalled 2-6 s at a time.

SpawnWorker.layerServer now picks SpawnHost on that branch. NonBlockingSpawner
still wraps it, so hung-mount checks stay off the main thread too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted Author is trusted by repo permissions or the VOUCHED list. size:XL labels Nov 16, 2026
@github-actions

Copy link
Copy Markdown

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ The exact PR base did not have a successful artifact. Baseline uses the latest successful main measurement shown below.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Codex Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 21.5 KiB 21.5 KiB 0 B (0.0%) 24.4 KiB ✅
Codex Live turn messages 1 1 0 (0.0%) 8 ✅
Claude Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Claude Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 21.9 KiB 21.9 KiB +1 B (+0.0%) 24.4 KiB ✅
Claude Live turn messages 1 1 0 (0.0%) 8 ✅

Other scenarios

No main baseline exists yet for these 64 scenario measurements.

All 64 scenario measurements
Provider Scenario Metric Main baseline This PR Impact PR ceiling
Codex short turn resume WebSocket decoded — 10.5 KiB — 12.7 KiB ✅
Codex short turn resume WebSocket messages — 2 — 2 ✅
Codex long turn running, cold open HTTP decoded — 281.7 KiB — 302.7 KiB ✅
Codex long turn running, cold open rows — 75 — 75 ✅
Codex long turn running, history page HTTP decoded — 92.4 KiB — 107.4 KiB ✅
Codex long turn running, history page rows — 75 — 75 ✅
Codex long turn finished, cold open HTTP decoded — 381.5 KiB — 410.2 KiB ✅
Codex long turn finished, cold open rows — 151 — 151 ✅
Codex long turn finished, socket cold open WebSocket decoded — 381.6 KiB — 410.2 KiB ✅
Codex long turn finished, socket cold open WebSocket messages — 2 — 2 ✅
Codex long turn finished, socket cold open rows — 151 — 151 ✅
Codex long turn finished, work page HTTP decoded — 186.2 KiB — 200.2 KiB ✅
Codex long turn finished, work page rows — 150 — 150 ✅
Codex idle thread resume WebSocket decoded — 65 B — 200 B ✅
Codex idle thread resume WebSocket messages — 1 — 1 ✅
Codex fleet shell snapshot HTTP decoded — 42.4 KiB — 53.7 KiB ✅
Codex fleet shell snapshot rows — 27 — 27 ✅
Codex fleet shell subscribe WebSocket decoded — 65 B — 200 B ✅
Codex fleet shell subscribe WebSocket messages — 1 — 1 ✅
Codex fleet shell resume check HTTP decoded — 18 B — 100 B ✅
Codex fleet shell resume check rows — 0 — 0 ✅
Codex stale shell reload HTTP decoded — 42.4 KiB — 53.7 KiB ✅
Codex stale shell reload rows — 27 — 27 ✅
Codex stale shell resume WebSocket decoded — 140 B — 400 B ✅
Codex stale shell resume WebSocket messages — 2 — 2 ✅
Codex stale todo list cold open HTTP decoded — 72.4 KiB — 83.0 KiB ✅
Codex stale todo list cold open rows — 75 — 75 ✅
Codex stale todo list cold open nodes — 2 — 4 ✅
Codex stale todo list cold open subagents — 2 — 4 ✅
Codex stale todo list cold open plans — 1 — 2 ✅
Codex stale todo list cold open providerTurns — 1 — 2 ✅
Codex stale todo list cold open checkpoints — 1 — 2 ✅
Claude short turn resume WebSocket decoded — 10.6 KiB — 12.7 KiB ✅
Claude short turn resume WebSocket messages — 2 — 2 ✅
Claude long turn running, cold open HTTP decoded — 283.3 KiB — 302.7 KiB ✅
Claude long turn running, cold open rows — 75 — 75 ✅
Claude long turn running, history page HTTP decoded — 93.4 KiB — 107.4 KiB ✅
Claude long turn running, history page rows — 75 — 75 ✅
Claude long turn finished, cold open HTTP decoded — 384.2 KiB — 410.2 KiB ✅
Claude long turn finished, cold open rows — 151 — 151 ✅
Claude long turn finished, socket cold open WebSocket decoded — 384.3 KiB — 410.2 KiB ✅
Claude long turn finished, socket cold open WebSocket messages — 2 — 2 ✅
Claude long turn finished, socket cold open rows — 151 — 151 ✅
Claude long turn finished, work page HTTP decoded — 188.1 KiB — 200.2 KiB ✅
Claude long turn finished, work page rows — 150 — 150 ✅
Claude idle thread resume WebSocket decoded — 66 B — 200 B ✅
Claude idle thread resume WebSocket messages — 1 — 1 ✅
Claude fleet shell snapshot HTTP decoded — 43.0 KiB — 53.7 KiB ✅
Claude fleet shell snapshot rows — 27 — 27 ✅
Claude fleet shell subscribe WebSocket decoded — 66 B — 200 B ✅
Claude fleet shell subscribe WebSocket messages — 1 — 1 ✅
Claude fleet shell resume check HTTP decoded — 18 B — 100 B ✅
Claude fleet shell resume check rows — 0 — 0 ✅
Claude stale shell reload HTTP decoded — 43.0 KiB — 53.7 KiB ✅
Claude stale shell reload rows — 27 — 27 ✅
Claude stale shell resume WebSocket decoded — 140 B — 400 B ✅
Claude stale shell resume WebSocket messages — 2 — 2 ✅
Claude stale todo list cold open HTTP decoded — 73.4 KiB — 83.0 KiB ✅
Claude stale todo list cold open rows — 75 — 75 ✅
Claude stale todo list cold open nodes — 2 — 4 ✅
Claude stale todo list cold open subagents — 2 — 4 ✅
Claude stale todo list cold open plans — 1 — 2 ✅
Claude stale todo list cold open providerTurns — 1 — 2 ✅
Claude stale todo list cold open checkpoints — 1 — 2 ✅

Baseline: 33cfae0 · PR result: 3088811 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@andrewcai8
andrewcai8 merged commit c8de4ba into main Nov 16, 2026
32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL vouch:trusted Author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant