Skip to content

Long-running multi-agent hosts: state DB growth, git polling of finished worktrees #40677

Open

Description

@RayFungHK

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

Run many Claude agents (5 to 8) in parallel on one server over several days, each in its own worktree, one long-lived orchestrator thread, keep the worktrees after the work is merged.

Expected behavior

  • Settled threads (and the worktrees of finished, merged work) should not be polled by git; polling them on demand, or an opt-in cleanup of worktrees whose branch is merged and whose tree is clean, would keep the process count proportional to the active work.
  • The state database should have a retention or compaction policy for old events and projections, and an incremental way to read the free-page count instead of PRAGMA freelist_count on a multi-GB file. A supported way to archive or retire a very long thread without losing the link targets of its child threads and PRs would also help.
  • The per-turn process-table read should be throttled or shared (one poll for all turns, or a short TTL, or reading /proc directly) so its cost does not scale with concurrent turns.
  • A host with 2 cores and 16 GB running a handful of agents should not be starved for minutes by server-side polling.

Actual behavior

On a small multi-agent host (2 cores / 16 GB, Linux under gVisor, t3 v0.1.11-nightly.20261119.4928, up to 8 live Claude agents, one long-lived orchestrator thread plus many short worker threads) the server load grows with history and with finished work, and the host froze three times in one day.

  1. State database: statev2.sqlite is 6.6-6.9 GB. t3 doctor --dry-run reports 6.86 GiB with only 607 MiB reclaimable, so compaction would not help much; the size is real history (about 6.0 M orchestration_events, about 434 k projection turn items, about 364 k nodes, about 61 k messages, plus an FTS table). About 870 SQL statements and about 500 transactions per minute; projection queries take 40-80 ms, sql.transaction averages 180 ms, PRAGMA freelist_count took up to 2.7 s. One long-lived thread holds about 18 k items and is the largest single load.

  2. Git polling of finished work: every thread gets a worktree and they stay after the work is merged. We had 656 worktrees (112 GB) and the server started about 1900 git child processes in 25 minutes, across worktrees whose threads were long settled.

  3. Host freezes: server event-loop-stalls.ndjson shows event-loop stalls of 195 s, 222 s and 63 s within 55 minutes (with under 2 s of CPU in those windows, so the whole container was starved, not busy), after the load went up. In the two days before that there was one stall over 30 s.

  4. ps polling per Claude turn (also reported in On macOS every open Claude session runs ps -A once a second, which floods endpoint security with process launches #37582): about 2 ps -A -o pid=,ppid=,pgid=,lstart= spawns per second with only 3 active Claude agents.

Impact

Blocks work completely

Version or commit

t3 v0.1.11-nightly.20261119.4928

Environment

Linux, gVisor sandbox, 2 cores / 16 GB, no swap

Logs or stack traces

- Settled threads (and the worktrees of finished, merged work) should not be polled by git; polling them on demand, or an opt-in cleanup of worktrees whose branch is merged and whose tree is clean, would keep the process count proportional to the active work.
- The state database should have a retention or compaction policy for old events and projections, and an incremental way to read the free-page count instead of PRAGMA freelist_count on a multi-GB file. A supported way to archive or retire a very long thread without losing the link targets of its child threads and PRs would also help.
- The per-turn process-table read should be throttled or shared (one poll for all turns, or a short TTL, or reading /proc directly) so its cost does not scale with concurrent turns.
- A host with 2 cores and 16 GB running a handful of agents should not be starved for minutes by server-side polling.

Screenshots, recordings, or supporting files

No response

Workaround

No response

Activity

  1. added
    bugSomething is broken or behaving incorrectly.
    needs-triageIssue needs maintainer review and initial categorization.
    on Nov 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething is broken or behaving incorrectly.needs-triageIssue needs maintainer review and initial categorization.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions