You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug]: Spawning git forks the whole server process, taking a median 8 to 32 ms from fork to exec under agent load #31422
I searched existing issues and did not find a duplicate.
I included enough detail to reproduce or investigate the problem.
Area
apps/server
Steps to reproduce
Run the server on Linux with many agent threads working in git worktree projects at once (here 7 to 156 runs working).
Record process forks and execs for 15 to 20 seconds with perf record -a -e sched:sched_process_fork -e sched:sched_process_exec, and pair each fork by the server's main thread with its child's exec.
Sample the server's main thread with perf record -g -t <main thread id> for 20 seconds.
Time GET / from loopback in a loop.
Expected behavior
Starting a git child does not hold up the event loop for a noticeable time, and status and pull request lookups do not starve HTTP, WebSocket, and MCP handling.
Actual behavior
The server spawned 13 to 24 processes a second, 95 to 98% of them git.
Measured from fork to the child's exec, a spawn took a median of 8 to 32 ms, and 18 to 26 ms in a 30-minute run after a restart.
libuv forks the whole Node process. _Fork from node::ProcessWrap::Spawn was on the stack in 33% of main-thread CPU samples, with copy_page_range as the leaf. The main thread slept in pipe_read in 77 of 300 state samples, which fits waiting on the exec pipe while the child releases its copy of the address space during exec (unmap_page_range in 58% of the child's samples before exec).
Summed fork-to-exec intervals covered up to 44% of elapsed time in 15 to 20 second windows, and 35 to 41% across the 30-minute run.
The median GET / from loopback rose with the number of runs working: 93 ms with 7, 289 ms with 27, 762 ms with 86, and about 1 s with 124 to 156.
In the server traces, the top call chains for git included ThreadPullRequestServiceV2.synchronize calling GitManager.branchPullRequest, which the server enqueues on checkpoint.captured and on terminal run.updated events, and 37% of git calls were GitVcsDriver.readConfigValue.
On the same machine, fork plus exec of /bin/true from a test process holding 1.6 GB of touched private anonymous memory took a median of 199 ms in 4 KB pages and 4 ms in 2 MB pages (AnonHugePages 1.68 GB). The server held 0.99 to 1.70 GB of resident memory, with 20 to 34 MB of page tables, during these measurements.
A server restart lowered the fork-to-exec median to 8 ms at first, but within 6 minutes the resumed runs raised the spawn rate and GET / was back above 0.6 s.
Impact
With many agents running, the web UI and MCP tool calls stall; McpHttpServer.authenticateRequest spans reached 26 s.
Version or commit
0.1.8-nightly.20261104.4165
Environment
Linux 6.12.113 (Debian 13), Node v26.12.1 inside the t3 binary, 80-thread Xeon without AVX2, 512 GB of memory, a 32 GB state database.
Logs or stack traces
Main-thread CPU samples, 20 s at 499 Hz: 33% under _Fork from node::ProcessWrap::Spawn, with copy_page_range as the leaf. Child CPU samples between fork and exec: 58% unmap_page_range under unmap_vmas. Fork-to-exec intervals from sched_process_fork and sched_process_exec are a separate, elapsed-time measure.
Workaround
No lasting workaround found; a restart helped for minutes. #30838 and #24099 would cut the number of git calls. A proposed mitigation is running git from a small long-lived helper process, so each fork copies little memory.
Claude Opus 5.5 (high effort) wrote this; GPT-6.1-Sol (Codex, high effort) reviewed it.
On today's nightly I profiled the server's main thread while it stalled, and it spent 14.2 s of a 32 s profile inside spawn from node:child_process. That's 0.1.10-nightly.20261116.4761 (6b7027b), headless t3 serve on a 32 vCPU Linux VM, and the web UI often didn't load for me.
The server's stall log shows it isn't a short burst. In each full hour from 14:00 to 21:00 UTC, the log recorded between 42 and 114 stalls. The logged samples averaged a utilization of 0.84 to 1.00 per hour, and the longest delay in each hour was 6 to 8 s.
The server's own trace spans say who started each git process. Of 387 such spans in its trace files, 164 came from pull request sync (ThreadPullRequestServiceV2.synchronize), 124 from checkpoint capture when a run starts or ends, and 99 from repository detection for the git status (detectGitRepositoryForStatus).
Many are repeats of the same read. In 15 s of strace -e execve, the server ran git config --get-all remote.origin.url 5 times, git remote -v 4 times and git config --get-regexp ^remote\..*\.gh-resolved$ 4 times.
#31488 moves these launches into a small helper process on Linux, and #30838 answers the repeated config reads from a snapshot. Both target exactly this work. I'm running #31488 on this server next and will post its before and after numbers there.
Claude Opus 5.5 wrote this from measurements it ran on my setup.
Same problem on the v0.1.10 stable release, on a remote headless t3 serve. The client is the macOS desktop app connected over an SSH tunnel. The desktop app shows "requests are slow" all the time.
Environment:t3 v0.1.10, Ubuntu 26.04.1 LTS, kernel 7.0.0-38, 20 vCPU, 40 GB RAM, no CPU steal. 11 projects, about 25 worktrees, 2–3 agent runs working. After 9 h the server had RSS 1.04–1.1 GB, VmPTE 22–24 MB, VmSize 27 GB, 2836 mappings and AnonHugePages: 0. THP is set to madvise.
What the server's own trace spans show
span (server side)
n
p50
p90
max
NonBlockingSpawner.spawn
10,199
115 ms
449 ms
2.1 s
ws.rpc.vcs.refreshStatus
55
7.5 s
12.1 s
33.6 s
ws.rpc.server.refreshProviders
25
2.5 s
7.3 s
24.9 s
ws.rpc.vcs.listRefs
131
1.3 s
2.5 s
6.4 s
ws.rpc.vcs.refreshBranch
335
1.0 s
1.5 s
2.9 s
There is a spawn floor of about 100 ms even when idle. I bucketed the NonBlockingSpawner.spawn minimum by 10-minute windows across 2 hours. It never fell below 95–101 ms, including quiet windows with fewer than 60 spawns.
One 9.6 s refreshStatus was about 30 sequential git commands. These were git config, rev-parse, symbolic-ref, status --porcelain=2, remote get-url and similar. Each took 220–340 ms, and almost all of it was inside NonBlockingSpawner.spawn. The same commands run by hand in that worktree take 0–2 ms. On top of that, 3.6 s of the refresh had no child span longer than 150 ms, inside VcsStatusBroadcaster.refreshRemoteAndLocalTotals before readRemoteStatus.
What the desktop client sees:EnvironmentRpc.request p50 69 ms (the network RTT is 66 ms), p90 1.5 s, p99 10.6 s, max 43 s. Even server.reportClientActivity sometimes took more than 5 s.
Benchmarks on the same host (system Node v22.22.1)
child_process.spawn('/usr/bin/git', ['--version']), time to the spawn event:
RSS of test process
spawn p50
81 MB
2.3 ms
568 MB
21 ms
1.19 GB
41 ms
I also copied what NonBlockingSpawner does on Linux at about 1 GB RSS: fs.access on the cwd, then on each PATH candidate up to /usr/bin (6 candidates in this server's PATH), then a spawn through /usr/bin/env -C <cwd> -- git ....
Access checks p50 0.7 ms, fork+exec p50 43 ms.
The result was the same while the libuv pool was busy with dns.lookup calls (0.6 ms and 44.6 ms).
So the access checks and trampoline added in #36327 are not the cost by themselves. Fork cost that grows with RSS explains about 40 ms. The real server's floor is about 110 ms, roughly 2.5× a test process with the same RSS, and I couldn't account for the rest. It could be the larger page tables and mappings, or a delay before the main thread handles the spawn result. NonBlockingSpawner.spawn has no child spans, so the trace can't separate prepare, native spawn and delivery. A child span around each would make this much easier to diagnose.
On a server with a busy disk, one fork can take seconds even at ~1 GB RSS, because it waits on another thread's page fault. This is 0.1.11 stable, headless t3 serve, Linux 7.2.15, btrfs on dm-crypt on a DRAM-less QLC NVMe.
During a stall, the main thread spent 13.4 s in uninterruptible sleep inside the fork for a spawn: clone → dup_mmap → __vma_start_write → __vma_start_exclude_readers. At the same moment a SQLite WorkerThread was in a write fault on the mapped statev2.sqlite-shm: btrfs_page_mkwrite → folio_wait_writeback. The fork has to lock each of the parent's mappings, so it waits until that fault finishes, which takes as long as btrfs writeback. Freezes of 2.2 s and 3.6 s in the same hour had the same stacks.
So here a fork costs as much as the slowest page fault another thread has in flight, not only what RSS predicts. That applies to every spawn that still forks the server, including the provider launches mentioned in #31488. This server forks about 60 times a minute: ~58 spawns per finished turn and ~55 per vcs.refreshStatus.
Drafted with Claude Opus 5.5 (Claude Code) on behalf of @lingolooper-simon.
Activity
test(server): reproduce #31422, ProcessRunner forks the server for ev…
nitedani commentedon Nov 16, 2026
On today's nightly I profiled the server's main thread while it stalled, and it spent 14.2 s of a 32 s profile inside
spawnfromnode:child_process. That's0.1.10-nightly.20261116.4761(6b7027b), headlesst3 serveon a 32 vCPU Linux VM, and the web UI often didn't load for me.The server's stall log shows it isn't a short burst. In each full hour from 14:00 to 21:00 UTC, the log recorded between 42 and 114 stalls. The logged samples averaged a utilization of 0.84 to 1.00 per hour, and the longest delay in each hour was 6 to 8 s.
The server's own trace spans say who started each
gitprocess. Of 387 such spans in its trace files, 164 came from pull request sync (ThreadPullRequestServiceV2.synchronize), 124 from checkpoint capture when a run starts or ends, and 99 from repository detection for the git status (detectGitRepositoryForStatus).Many are repeats of the same read. In 15 s of
strace -e execve, the server rangit config --get-all remote.origin.url5 times,git remote -v4 times andgit config --get-regexp ^remote\..*\.gh-resolved$4 times.#31488 moves these launches into a small helper process on Linux, and #30838 answers the repeated config reads from a snapshot. Both target exactly this work. I'm running #31488 on this server next and will post its before and after numbers there.
mahirozdin commentedon Nov 17, 2026
Same problem on the v0.1.10 stable release, on a remote headless
t3 serve. The client is the macOS desktop app connected over an SSH tunnel. The desktop app shows "requests are slow" all the time.Environment:
t3 v0.1.10, Ubuntu 26.04.1 LTS, kernel 7.0.0-38, 20 vCPU, 40 GB RAM, no CPU steal. 11 projects, about 25 worktrees, 2–3 agent runs working. After 9 h the server had RSS 1.04–1.1 GB,VmPTE22–24 MB,VmSize27 GB, 2836 mappings andAnonHugePages: 0. THP is set tomadvise.What the server's own trace spans show
NonBlockingSpawner.spawnws.rpc.vcs.refreshStatusws.rpc.server.refreshProvidersws.rpc.vcs.listRefsws.rpc.vcs.refreshBranchNonBlockingSpawner.spawnminimum by 10-minute windows across 2 hours. It never fell below 95–101 ms, including quiet windows with fewer than 60 spawns.refreshStatuswas about 30 sequential git commands. These weregit config,rev-parse,symbolic-ref,status --porcelain=2,remote get-urland similar. Each took 220–340 ms, and almost all of it was insideNonBlockingSpawner.spawn. The same commands run by hand in that worktree take 0–2 ms. On top of that, 3.6 s of the refresh had no child span longer than 150 ms, insideVcsStatusBroadcaster.refreshRemoteAndLocalTotalsbeforereadRemoteStatus.EnvironmentRpc.requestp50 69 ms (the network RTT is 66 ms), p90 1.5 s, p99 10.6 s, max 43 s. Evenserver.reportClientActivitysometimes took more than 5 s.Benchmarks on the same host (system Node v22.22.1)
child_process.spawn('/usr/bin/git', ['--version']), time to thespawnevent:I also copied what
NonBlockingSpawnerdoes on Linux at about 1 GB RSS:fs.accesson the cwd, then on each PATH candidate up to/usr/bin(6 candidates in this server's PATH), then a spawn through/usr/bin/env -C <cwd> -- git ....dns.lookupcalls (0.6 ms and 44.6 ms).So the access checks and trampoline added in #36327 are not the cost by themselves. Fork cost that grows with RSS explains about 40 ms. The real server's floor is about 110 ms, roughly 2.5× a test process with the same RSS, and I couldn't account for the rest. It could be the larger page tables and mappings, or a delay before the main thread handles the
spawnresult.NonBlockingSpawner.spawnhas no child spans, so the trace can't separate prepare, native spawn and delivery. A child span around each would make this much easier to diagnose.Notes
env -Cthis build spawns from the main thread, so theSpawnWorkerpool from fix(server): isolate Linux process launches from the RPC event loop #30209 doesn't help here.lingolooper-simon commentedon Nov 20, 2026
On a server with a busy disk, one fork can take seconds even at ~1 GB RSS, because it waits on another thread's page fault. This is 0.1.11 stable, headless
t3 serve, Linux 7.2.15, btrfs on dm-crypt on a DRAM-less QLC NVMe.During a stall, the main thread spent 13.4 s in uninterruptible sleep inside the fork for a spawn:
clone→dup_mmap→__vma_start_write→__vma_start_exclude_readers. At the same moment a SQLiteWorkerThreadwas in a write fault on the mappedstatev2.sqlite-shm:btrfs_page_mkwrite→folio_wait_writeback. The fork has to lock each of the parent's mappings, so it waits until that fault finishes, which takes as long as btrfs writeback. Freezes of 2.2 s and 3.6 s in the same hour had the same stacks.So here a fork costs as much as the slowest page fault another thread has in flight, not only what RSS predicts. That applies to every spawn that still forks the server, including the provider launches mentioned in #31488. This server forks about 60 times a minute: ~58 spawns per finished turn and ~55 per
vcs.refreshStatus.Drafted with Claude Opus 5.5 (Claude Code) on behalf of @lingolooper-simon.