Repository navigation
Description
Before submitting
- I searched existing issues and did not find a duplicate.
- I included enough detail to reproduce or investigate the problem.
Area
apps/server
Steps to reproduce
Run many Claude agents (5 to 8) in parallel on one server over several days, each in its own worktree, one long-lived orchestrator thread, keep the worktrees after the work is merged.
Expected behavior
- Settled threads (and the worktrees of finished, merged work) should not be polled by git; polling them on demand, or an opt-in cleanup of worktrees whose branch is merged and whose tree is clean, would keep the process count proportional to the active work.
- The state database should have a retention or compaction policy for old events and projections, and an incremental way to read the free-page count instead of PRAGMA freelist_count on a multi-GB file. A supported way to archive or retire a very long thread without losing the link targets of its child threads and PRs would also help.
- The per-turn process-table read should be throttled or shared (one poll for all turns, or a short TTL, or reading /proc directly) so its cost does not scale with concurrent turns.
- A host with 2 cores and 16 GB running a handful of agents should not be starved for minutes by server-side polling.
Actual behavior
On a small multi-agent host (2 cores / 16 GB, Linux under gVisor, t3 v0.1.11-nightly.20261119.4928, up to 8 live Claude agents, one long-lived orchestrator thread plus many short worker threads) the server load grows with history and with finished work, and the host froze three times in one day.
-
State database: statev2.sqlite is 6.6-6.9 GB.
t3 doctor --dry-runreports 6.86 GiB with only 607 MiB reclaimable, so compaction would not help much; the size is real history (about 6.0 M orchestration_events, about 434 k projection turn items, about 364 k nodes, about 61 k messages, plus an FTS table). About 870 SQL statements and about 500 transactions per minute; projection queries take 40-80 ms, sql.transaction averages 180 ms, PRAGMA freelist_count took up to 2.7 s. One long-lived thread holds about 18 k items and is the largest single load. -
Git polling of finished work: every thread gets a worktree and they stay after the work is merged. We had 656 worktrees (112 GB) and the server started about 1900 git child processes in 25 minutes, across worktrees whose threads were long settled.
-
Host freezes: server event-loop-stalls.ndjson shows event-loop stalls of 195 s, 222 s and 63 s within 55 minutes (with under 2 s of CPU in those windows, so the whole container was starved, not busy), after the load went up. In the two days before that there was one stall over 30 s.
-
pspolling per Claude turn (also reported in On macOS every open Claude session runs ps -A once a second, which floods endpoint security with process launches #37582): about 2ps -A -o pid=,ppid=,pgid=,lstart=spawns per second with only 3 active Claude agents.
Impact
Blocks work completely
Version or commit
t3 v0.1.11-nightly.20261119.4928
Environment
Linux, gVisor sandbox, 2 cores / 16 GB, no swap
Logs or stack traces
- Settled threads (and the worktrees of finished, merged work) should not be polled by git; polling them on demand, or an opt-in cleanup of worktrees whose branch is merged and whose tree is clean, would keep the process count proportional to the active work.
- The state database should have a retention or compaction policy for old events and projections, and an incremental way to read the free-page count instead of PRAGMA freelist_count on a multi-GB file. A supported way to archive or retire a very long thread without losing the link targets of its child threads and PRs would also help.
- The per-turn process-table read should be throttled or shared (one poll for all turns, or a short TTL, or reading /proc directly) so its cost does not scale with concurrent turns.
- A host with 2 cores and 16 GB running a handful of agents should not be starved for minutes by server-side polling.Screenshots, recordings, or supporting files
No response
Workaround
No response
Activity