You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug]: Checkpoint capture hits the 30 s git timeout in large workspaces and never converges; every killed attempt leaves a tmp_objdir-bulk-fsync-* directory behind #26227
T3 Code Nightly 0.1.3-nightly.20261025.3669, macOS (Apple Silicon), git 2.53.0
Workspace: a plain folder that is a git repository with an unborn HEAD (no commits), used only as a shared coordination workspace; checkpoints are the only git writes
What happens
The workspace received about 170,000 new files within a few hours (generated snapshot copies, about 2.8 GB).
Since then no checkpoint capture has succeeded. The last checkpoint commit under refs/t3 is from the time right before that; before it the workspace produced 220 to 635 successful checkpoints per day.
The server trace shows VcsProcessTimeoutError: VCS process timed out in GitVcsDriver.checkpoints.captureCheckpoint: git (<workspace>) after 30000ms for 57 of 59 capture attempts within about 26 minutes. In the source, the git add -A call in captureCheckpoint passes no timeoutMs, so VcsProcess uses DEFAULT_TIMEOUT_MS = 30_000 with timeoutBehavior: "error".
Because the capture runs with core.fsyncMethod=batch on macOS, git writes new objects into .git/objects/tmp_objdir-bulk-fsync-* and only migrates them at the end. A killed attempt leaves that directory behind and none of its objects reach the object store, so the next attempt has to write the same objects again and is killed again.
Result after six days: 2,997 leftover tmp_objdir-bulk-fsync-* directories (264 to 826 new per day). A sample of 30 had on average about 7,600 files and 52 MB each (up to about 19,500 files and 135 MB), roughly 23 million files and 150 GB in total by extrapolation. There was no pack file and 7,937 loose refs under refs/t3. The constant capture attempts kept one git add running almost continuously and the leftover files kept fseventsd, Spotlight and Time Machine busy.
Expected
A capture that cannot finish in time should not leave quarantine object directories behind (clean up tmp_objdir-* on timeout or interruption), or should not be hard-killed while it is still making progress.
The capture should be able to converge, for example by keeping objects written so far, scaling the timeout with the workspace size, or reusing the previous checkpoint index instead of hashing the whole tree with a fresh index each time.
Optionally a per-project way to exclude paths from checkpoints or to disable checkpoints for non-code workspaces.
Workaround used locally, and why it is not enough
Moved the leftover directories out of .git/objects (they are never referenced), ran git pack-refs --all, excluded .git from Time Machine, and added .git/info/exclude entries for the bulk snapshot folders. That halved the number of files in the capture (about 58,000 to 31,500), but captures still fail: 49 of 51 attempts in the next 25 minutes timed out, and no new checkpoint ref was written. The remaining files total about 6 GB. Hashing them once with git hash-object --stdin-paths (read-only, no writes) takes 44 s on this machine (Apple Silicon, 10 cores, load about 16). Because each capture starts from a fresh index, every attempt has to read and hash all of it again, although only about 1,600 files (78 MB) changed since the last successful checkpoint. Excluding media files and archives as well (down to 2.7 GB) still needs 21 s for hashing alone, close to the limit. Failed attempts are retried about every 30 s while threads are active, so the leftover directories grow by up to about 120 per hour.
Related: #24083 (checkpoint refs never pruned), #17825 (hung checkpoint capture), #13741 (full checkpoint index rebuild).
Reacted by John Colvin, dl-pt, EvilHumphrey, Deepankar Mahapatro, Joel Tony, Sanzhar Danybayev, sanzhar-foody-ai, LetterXbox and Simon
Another macOS data point with the same failure mode, on the 0.1.2 stable line. This one is a media folder, so each leaked directory is much larger.
Environment
T3 Code 0.1.2 desktop. It's a rebuild that only changes UI strings; the server/VCS code is identical to upstream. macOS 27.0.1, Apple Silicon, git 2.54.0 (Apple Git-157)
Workspace: a non-code project folder of about 60 GiB and 40,000 files, mostly .mp4, .mov and .wav (82 files over 100 MiB), with no .gitignore. It had become a git repository with an unborn HEAD (no commits, no index, no refs) shortly before the problem started. None of the agent session logs contain a git init, so the in-app "Initialize Git" action is the most likely cause, but I can't confirm it because the server logs from that time had already rotated.
What happened
Each turn spawned git -C <workspace> -c core.fsmonitor=false … -c core.fsync=objects,reference -c core.fsyncMethod=fsync -c core.fsyncMethod=batch add -A -- . against a .git/t3-checkpoint-index-* and was killed at 30 s (VcsProcessTimeoutError … GitVcsDriver.checkpoints.captureCheckpoint … after 30000ms). The trace had rotated except for the last few minutes; all 4 captures in that window timed out at 30.0 s. No checkpoint ever succeeded: there were no refs/t3 refs, and the real object store held only 73 loose objects (2.3 MiB) and no packs.
70 tmp_objdir-bulk-fsync-* directories accumulated over about 16 hours, most of them during roughly 6 hours of active use. With several threads active, two sometimes appeared in the same minute. Each was about 1.5 GiB (a sampled one held 585 objects), about 94 GiB in total. Free disk space fell to about 8 GB before I noticed.
Each attempt gets about 1.5 GiB into the roughly 60 GiB tree before it is killed, and the next attempt starts over, so it can never converge.
An ordinary code repository on the same machine also had one 94 MiB tmp_objdir-bulk-fsync-* left from a single interrupted capture. This leak isn't limited to unusual workspaces; it is just much faster in them.
Recovery
I stopped the running capture and deleted the whole .git. With no commits there was nothing to lose, and the workspace files were untouched. That brought free space back to about 98 GiB.
Another macOS data point. The issue is still present on 0.1.4 stable.
Environment
Incident build: 0.1.3-nightly.20261026.3710. I also checked the installed 0.1.4 bundle (/Applications/T3 Code (Beta).app): it has the same captureCheckpoint / stageFiles / cleanupTempIndex code.
macOS, APFS, core.fsyncMethod=batch on the capture command.
Workspace: an ordinary code repo with a normal HEAD, not unborn. Agents had written large untracked build and data output into it.
What happened (2026-10-26)
Free disk fell to about 0.39 GiB. The machine crashed three times in 24 hours. I did not inspect the panic logs, so I can't say that disk pressure caused every crash.
lsof showed one git process writing both <repo>/.git/t3-checkpoint-index-<id>.lock and <repo>/.git/objects/tmp_objdir-bulk-fsync-<x>/….
The repo's .git was 46.1 GB. After I removed the leaked tmp_objdir-bulk-fsync-* dirs it was 1.4 GB, and free space came back to about 43 GiB. Other large directories (Codex worktrees) did not change in size over the same samples, so the checkpoint writer was the active drain.
Code notes (bundled names, 0.1.3 nightly; same code in 0.1.4)
stageFiles runs git add -A -- . against the temporary index. Nothing checks the size of the untracked input first.
cleanupTempIndex removes only the temp index and its .lock. It never removes a tmp_objdir-bulk-fsync-* created by the attempt that timed out, so every killed attempt leaks one.
The darwin writeout-only re-run in VcsProcess.runUnbounded (gated by canRetryFullFsync) is a second full add -A, but only for the two fsync/lock stderr patterns. The isTransientGitExit path in VcsProcess.run retries up to 2 more times. Neither path runs on a timeout, so the leak comes from capture being attempted again turn after turn, not from these retries.
Suggested fix, in addition to the quarantine cleanup in #15400
Before git add, measure the selected untracked input (respecting ignore rules) plus tracked changes. If it is over a budget, skip the capture or mark it incomplete. Don't edit user ignore files, and don't claim recovery coverage for content that was skipped.
After a timeout or kill, once the child process has exited, remove only the tmp_objdir-* that this attempt created.
Don't retry a full stage after a size or timeout failure. Back off instead (this matches the back-off suggested above).
Test idea: use a small synthetic repo with a small byte limit and force repeated stage timeouts. Assert that no new tmp_objdir-* remain, that a concurrent writer's files and the live index survive, and that a normal checkpoint still restores the source.
Another macOS data point, on 0.1.5 stable, in an ordinary code repository (normal HEAD, actively developed), not an unborn or media folder. Here the tracked files alone exceed the timeout.
Environment
T3 Code 0.1.5 server (background service), macOS 27.0.1, Apple M1 Max, git 2.54.0 (Apple Git-157)
Workspace: 32,252 tracked files, 21.3 GB (mostly committed screenshots), plus 1 to 2 GB of untracked output that agents write. Threads run in the project's main checkout.
What happened
In about 18 hours (Oct 29 15:49 to Oct 30 10:00), 44 tmp_objdir-bulk-fsync-* directories built up in .git/objects, about 41 GB, roughly 1 GB per turn.
The trace shows the same failure: VcsProcessTimeoutError: VCS process timed out in GitVcsDriver.checkpoints.captureCheckpoint: git (<workspace>) after 30000ms. While it ran, git add -A -- . (with core.fsyncMethod=batch) showed up as a child of t3 serve.
Hashing the tracked files from an index freshly read from HEAD (git status, git update-index --refresh) takes about 100 s here. If capture starts from a fresh index each time, as the report above reads the source, it can't finish in 30 s in this repo even with no untracked files at all.
Removing the untracked output made each leftover smaller but didn't stop them. Reusing the previous checkpoint index, or cleaning up tmp_objdir-* when the timeout fires, would fix it for us.
Windows data point for the same failure mode, on 0.1.5 stable (desktop). It's the first one here that isn't macOS: core.fsyncMethod=batch is also applied on win32, so NTFS leaks the same tmp_objdir-bulk-fsync-* directories. It adds two details: the trigger was T3's own scratch folder, and the temp index .lock files leak too.
What happened
Threads with no project run in ~/.t3/scratch. On 2026-10-20 something ran git init in ~ (unborn HEAD, no commits, no remote). From then on, git rev-parse --show-toplevel in ~/.t3/scratch resolved to ~, so every capture ran git add -A -- . for the scratch folder against ~/.git.
The scratch folder holds about 200k files and 1.5 GB of agent output: downloads, an 888 MB partial file and a 289 MB executable. There's no .gitignore, because it isn't meant to be a repo. Hashing it doesn't fit in 30 s.
The last checkpoint refs (7 under refs/t3/orchestration-v2/checkpoints/…) are from 2026-10-22 19:06 local. No capture succeeded after that, so revert didn't work for any scratch thread for 8 days.
Diagnosis (source at v0.1.5, 6d450a4284)
Same as the original report:
stageFiles in apps/server/src/vcs/GitVcsDriver.ts (~L1040) runs add -A with durableWrite (core.fsyncMethod=batch on darwin and win32, ~L814) and no timeoutMs. So VcsProcess kills it at DEFAULT_TIMEOUT_MS = 30_000 (VcsProcess.ts:64).
Nothing in apps/server/src cleans up objects/tmp_objdir-* after a timeout or interruption.
Windows-specific:
cleanupTempIndex runs in Effect.ensuring, but its remove(…).pipe(Effect.ignore) left 6 stale 0-byte t3-checkpoint-index-<uuid>.lock files in ~/.git, aged 31–199 h.
My guess, not confirmed: the timed-out git.exe still holds the lock when the cleanup runs, Windows refuses the delete, and the error is swallowed.
Measured durations go up to 45.8 s for a 30 s timeout, so the kill isn't immediate.
main at 2e860972ae (2026-10-30) still has no tmp_objdir cleanup, and the capture still uses the default timeout. #21315 and #21329 reduce git process count and scan cost, but don't change this.
Steps to reproduce (Windows)
git init in your home folder (or have any repo at ~).
Use threads with no project, so the workspace is ~/.t3/scratch, and let agents write a few GB of files there.
Each turn: VcsProcessTimeoutError … captureCheckpoint: git (~\.t3\scratch) after 30000ms, plus one new ~/.git/objects/tmp_objdir-bulk-fsync-* directory, and sometimes a leftover t3-checkpoint-index-*.lock.
Version
T3 Code (Beta) 0.1.5 desktop (server 0.1.5, per the triage context). Triage CLI 0.1.5.
Environment
Windows 11 Pro 26H2 (10.0.26300), NTFS on NVMe, Git for Windows 2.55.0.windows.5, Ryzen 9 7950X3D, 63 GB RAM. Desktop app with a local server; several agent threads running at once.
Evidence
Leaked directories: 948 tmp_objdir-bulk-fsync-* in ~/.git/objects, the first dated 2026-10-22 19:12. Per day: 80 / 68 / 110 / 183 / 161 / 200 / 130. The count rose from 932 to 948 within about 40 minutes today.
Total size:~/.git was 48 GB in about 9.7 million files. A sample of 7 directories held 5.6k–13.6k files and 31–92 MB each. git count-objects saw only 1,096 loose objects; everything else sat in the leaked directories.
Trace: 119 VcsProcessTimeoutError spans for GitVcsDriver.checkpoints.captureCheckpoint in the retained trace files, all between 20:23 and 21:02 UTC on 2026-10-30, about 3 per minute with 6–7 threads active. Durations ran from 30,136 to 45,773 ms.
Windows side effects:
Defender (MsMpEng) averaged about one full core over 2 days, with samples at 171–218%.
The NTFS MFT on C: grew to 13.9 GB.
A git add was running at about 98% of a core most of the time.
Related issues
#26227 (this issue), #3646 and #15400 (tmp_pack litter from failed checkpoints), #1228 (checkpoint timeout when running from the home directory), #24083.
Fix applied or workaround
Moved the accidental ~/.git out of the profile. ~/.t3/scratch is no longer inside a repo, so captures there now fail fast as "not a git repository" instead of timing out. After later turns, T3 had not re-created a .git in ~ or in the scratch folder.
One leaked tmp_objdir stayed locked by another process and needs deleting after a reboot.
Suggestions beyond the original report:
Retry the .lock cleanup after the child has fully exited, at least on Windows.
Consider not checkpointing T3's own scratch folder into an ancestor repo it doesn't own, such as ~. Anyone with a repo at their home folder would otherwise hit this on every scratch thread.
Filed by
Claude Opus 5.5 (claude-opus-5-5) in Claude Code, following the t3 triage 0.1.5 playbook, on the affected machine.
Another macOS case on October 31: my disk fell to about 0.5 GiB free. I found 273 abandoned .git/objects/tmp_objdir-bulk-fsync-* directories holding 82.5 GiB and 161,853 files.
Environment: T3 Code Nightly 0.1.4-nightly.20261027.3770, macOS 26.4, APFS, Apple Silicon arm64, Git 2.55.0. The workspace has an unborn HEAD, contains generated media, and its 494 refs are all T3 checkpoints.
The retained server traces contain six capture timeouts at approximately 30 seconds, reported as VcsProcessTimeoutError in GitVcsDriver.checkpoints.captureCheckpoint. Each interval matches the modification time of one abandoned directory. A Git object hash also confirmed that the same original video had been copied into multiple abandoned directories.
I tested the interruption mechanism separately in disposable repositories on this machine:
git add with core.fsync=objects,reference and core.fsyncMethod=batch left a tmp_objdir-bulk-fsync-* directory after SIGTERM. SIGKILL did the same.
With core.fsyncMethod=fsync, SIGTERM left no bulk-fsync directory.
With batch and no interruption, git add completed without leaving a temporary directory.
After checking that the old directories were not in use, I removed only those 273 directories. All 494 checkpoint refs remained identical, the normal object count stayed at 11,674, and git fsck --connectivity-only --no-dangling passed before and after. Original workspace files were preserved.
Linking #23715 as the relevant proposed fix. I can help validate its batch-fsync timeout and cancellation cleanup on native macOS arm64 with Git 2.55.0, along with successful capture and restore. I have not tested that PR yet. The affected T3 build above is the one investigated; I have not reproduced this incident on a newer release.
Another Windows data point, from a normal checkout with HEAD (not an unborn repo), on desktop 0.1.4-nightly.20261028.3841.
What we saw. The repo has about 300k tracked paths. At the worst point it also had about 9,000 nonignored untracked files (0.9 GB, none over 50 MB). .git/objects had collected 349 tmp_objdir-bulk-fsync-* folders, 39.7 GB in total. On a busy day, about 82 were created in one hour. We assume most came from checkpoint git add -A runs killed at the 30 s limit. We didn't match each folder to a trace.
Why it seems to happen. As far as we can read the installed build, each capture makes a fresh UUID temp index, seeds it from the workspace index, and removes it afterwards. The workspace index doesn't hold the untracked files, so a fresh capture index doesn't keep their cached metadata from the last capture. That fits the cold and warm difference we measured. (#23624 now covers skip-worktree and assume-unchanged entries; ours is the untracked files that aren't in the workspace index.) We replayed the add on a copy of the index, with a temp object dir:
First trial: 15.5 s cold, then 1.1 s reusing the same temp index.
Second trial, about 15 minutes later under heavier load: 29.6 s cold, right at the 30 s limit.
These are git add timings, not whole captures. Refreshing the real index (update-index --refresh) didn't improve this replay. Reusing one temp index is evidence of the direction, not a safe implementation.
What helped. We hid untracked files over 1.5 MB with .git/info/exclude. In T3's own trace (GitVcsDriver.checkpoints.captureCheckpoint), captures went from a 32.5 s median with 23 of 29 failed (19 minutes before) to an 11.0 s median with 0 of 9 failed (5 minutes after). New leftover folders dropped from about 82 an hour to about one. Later we cut the untracked files from about 9,000 to under 200. A sparse worktree replayed in 2.9 to 5.7 s, against 17.6 to 22.1 s on the full checkout.
What would help us. The proposals already in this thread match what we need. Our addition is the measured cold and warm comparison above.
Keep the capture index between captures, or seed it from the previous one, so untracked files already hashed aren't hashed again every turn. This matches the index-reuse idea earlier in this thread.
When a capture can't finish in time, skip it with a visible notice instead of retrying every turn.
Environment: Windows 11 Pro, Git 2.54.0.windows.1, T3 Code desktop 0.1.4-nightly.20261028.3841 (server bundle SHA-256 0a3cfede...7464). We haven't tied the running backend version to logs yet, and we haven't tried 0.1.7. Reading the latest nightly's source, it still uses a fresh capture index and the 30 s staging default, but we haven't reproduced the incident on it. Happy to run a check on a newer nightly.
Additional incident on macOS: about 1.28 TB of abandoned checkpoint objects nearly filled a 2 TB system drive. Adding the measurements here because they match this issue.
Environment
T3 Code desktop: 0.1.7 installed at diagnosis.
macOS 27.0.1 (26A434), Apple Silicon, APFS.
Git 2.55.0.
Investigation date: 4 November 2026. The accumulation began on 30 October; I have not established which T3 version created every older directory.
Approximately 1,279,754,428,416 allocated bytes (1.28 TB) attributable to those directories. This was calculated from the measured object-directory total minus all non-temporary object entries, rather than extrapolated from a sample.
The other Git objects occupied approximately 9.04 GB.
552 of the temporary directories were created on 4 November.
Free space fell to approximately 160 MB, and other local services reported No space left on device.
Three sampled temporary directories contained 5,210 common Git object names. Matching examples had the same sizes but different inodes and a link count of one, consistent with repeated separate copies. One sample alone occupied approximately 1.61 GB.
Correlation with T3 checkpoint failures
The retained T3 server traces contained 82 checkpoint attempts, all failed:
70VcsProcessTimeoutError failures at the 30,000 ms limit.
12VcsProcessExitError failures with exit code 128.
74 of the temporary-directory creation timestamps fell inside these failed GitVcsDriver.checkpoints.captureCheckpoint intervals.
Representative error, with the private path removed:
VcsProcessTimeoutError: VCS process timed out in
GitVcsDriver.checkpoints.captureCheckpoint:
git (<workspace>) after 30000ms
The timestamp comparison supports the same checkpoint leak described above. Retained logs do not cover the full accumulation period, so I am not claiming an individual trace match for all 1,025 directories.
Impact and requested fix
This exhausted disk capacity and disrupted work outside T3. T3 has been stopped. A completed cleanup and a fixed release have not yet been verified.
Please prioritise preventing each failed or interrupted capture from leaving its temporary object storage behind. Cleanup should remove only storage owned by that capture, after its Git child has exited, and preserve concurrent writers. A regression test should force repeated capture timeouts/cancellation and confirm that temporary storage does not accumulate. Backoff or a visible suspension after repeated failures would also limit the continuing I/O load.
No private workspace names, file names, file contents, absolute paths, thread identifiers, or full logs are included in this report.
Another macOS data point. The leak is still present on the 2026-11-03 nightly, and it used about 449 GB in 25 hours.
Environment
T3 Code (Nightly) 0.1.7-nightly.20261103.4119, macOS 26.6.2, git 2.55.0
Workspace: a video production project with a normal HEAD (one commit, 2 tracked files). Agents write renders into it, and several threads run at once.
Files that git add -A must process: 18,008 untracked, non-ignored files, 31.5 GB, of which 25.9 GB is rendered video output.
What happened
.git/objects held 457 tmp_objdir-bulk-fsync-* directories, 448.9 GB in total. The real object store held 632 MiB.
The first directory appeared on Nov 3 at 06:36 and the newest on Nov 4 at 07:55. The directories had a median size of 1.05 GB and a maximum of 1.68 GB. The peak was 52 new directories in one hour.
Free disk space fell to 12 GB on a 1 TB disk. The growth over the last 2 hours measured was 36.7 GB.
The last checkpoint ref is from Nov 2 at 22:17, so no capture has succeeded in this project since then.
Link between timeouts and directories
The retained trace covers 06:59 to 07:56 on Nov 4. It has 53 VcsProcessTimeoutError failures from GitVcsDriver.checkpoints.captureCheckpoint in this workspace. In the same window, 53 directories appeared, and each one appeared within 35 seconds before a failure.
Note on #29259
As I read that PR, it stops the leak but does not change the timeout. This workspace would still have no checkpoints and would still spend 30 seconds on each failed capture. The back-off or size budget suggested above would still be needed for this type of project.
I observe the same issue on macOS Apple Silicon with T3 0.1.7 using Codex. Repeated checkpoint timeouts leave eight tmp_objdir-bulk-fsync-* directories behind, and block each prompt for 30 seconds before it reaches Codex. #28996 is the same issue.
Another Windows data point, with a workaround that has held.
Environment: T3 Code Nightly (currently 0.1.8.0; the build at the time of the incidents wasn't recorded), Windows 11, NTFS, git 2.55.0.windows.5. An ordinary repo with commits, where other agents commit to main.
What happened: about 5.6 GB of untracked, non-ignored files made captureCheckpoint hit the 30-second timeout. Each killed git add left a tmp_objdir-bulk-fsync-* folder of about 1 GB and about 3,400 files. It happened three times: 2026-10-29 (size not recorded), 2026-11-01 (376.7 GB) and 2026-11-04 (352.6 GB in 358 folders, up to about 37 GB an hour).
Workaround: adding the idle bulky folders to .git/info/exclude brought the files to save under 1 GB. After that, 377 checkpoints were recorded with no leftover folders, saves took about 3 seconds, and the trace logs showed no failures. Those logs only reach back about 4 hours here. It only works for folders that don't need saving, and it doesn't fix the timeout.
For #29259: on Windows, each killed attempt here cost about 1 GB, so a fix that removes the batch folder would have prevented all of this.
Linux data point, on 0.1.5 stable. On Linux the leak takes a different form: there are no tmp_objdir-bulk-fsync-* directories, and the killed captures leave .git/objects/pack/tmp_pack_* files instead. Here they filled the disk that also holds the T3 data directory, and T3 Connect stopped answering.
What happened
Diagnosed and written by Claude Opus 5.5, based on a t3 triage session.
The mobile app showed "T3 Code on omarchy sent an unexpected answer". The desktop app had hung for about 11 minutes before that.
Diagnosis
The workspace is an ordinary repo with commits. It holds about 54 GB of untracked, non-ignored training checkpoints. Each optimizer.pt is about 839 MB.
captureCheckpoint runs git add -A with the default DEFAULT_TIMEOUT_MS = 30_000 (apps/server/src/vcs/VcsProcess.ts:64). The trace shows VcsProcessTimeoutError ... captureCheckpoint ... after 30000ms.
GitVcsDriver.ts adds core.fsyncMethod=batch only on darwin and win32 (apps/server/src/vcs/GitVcsDriver.ts:819, v0.1.5). On Linux, Git does not use a tmp_objdir quarantine. Files above core.bigFileThreshold (512 MiB by default) go through bulk-checkin, which streams them into .git/objects/pack/tmp_pack_*. A killed git add leaves that file behind, and the next capture writes it again.
From 2026-11-05 11:47 to 2026-11-06 15:36, 171 tmp_pack_* files built up, 107 GB in total. They arrived in pairs every 15 minutes, about 700 MB each.
The disk reached 0 bytes free. The T3 data directory is on the same disk, so the server could not write. The last capture failed with exit 128 instead of a timeout, and the server trace stopped there.
With the disk full, cloudflared stayed connected, but the origin answered 77 relay requests with HTTP 500. The mobile app reported invalid_response and endpoint_request_failed with the tunnel marked "ok". The desktop hang probably has the same cause. I did not confirm that.
Steps to reproduce
On Linux, open a project whose repo has a few untracked, non-ignored files larger than 512 MiB, enough that git add -A takes longer than 30 seconds.
Run turns in a thread in that project.
Watch .git/objects/pack/. Each timed-out capture leaves a tmp_pack_* file that is never cleaned up.
Version
T3 Code 0.1.5 desktop (AUR t3code-bin 0.1.5-1), host server 0.1.5. Mobile app 2.2.0.
Environment
Arch Linux (Omarchy), kernel 7.2.8, x64, git 2.56.0, ext4.
Evidence
VcsProcessTimeoutError: VCS process timed out in GitVcsDriver.checkpoints.captureCheckpoint: git (<repo>) after 30000ms # 2026-11-06 02:12
VcsProcessExitError: VCS process failed in GitVcsDriver.checkpoints.captureCheckpoint: git (<repo>) exited with 128 # 2026-11-06 15:36, disk full
$ ls <repo>/.git/objects/pack/tmp_pack_* | wc -l
171 # 107 GB, mtimes 11-05 11:47 to 11-06 15:36
$ df -h <data disk>
917G 870G 0 100%
cloudflared_tunnel_response_by_code{status_code="500"} 77
Fix applied
Deleted the tmp_pack_* files while no Git process was running. That freed 107 GB.
Added *.pt, *.safetensors, *.onnx, *.bin, checkpoint-*/ and similar patterns to the repo's .git/info/exclude. git status now finishes in milliseconds and captures stop timing out.
Cleaning up on timeout should cover objects/pack/tmp_pack_* on Linux as well as tmp_objdir-*.
Sent from Claude Opus 5.5 (Claude Code, via t3 triage) for janosch.
Confirmed on macOS/Apple Silicon with 0.1.8-nightly.20261104.4165: 790 leftover tmp_objdir-bulk-fsync-* directories occupied 27.4 GB.
Excluding specific generated QA/source copies through .git/info/exclude restored successful checkpointing. This workaround reduces checkpoint coverage.
Windows checkpoint-coverage trap worth knowing when validating a fix here: a capture can succeed quickly and still omit all non-ignored untracked files.
Why: before staging, captureCheckpoint asks filterIgnoredPaths(input.cwd, ["."]) whether the workspace root itself is ignored. If it is, it stages with git add -u instead of -A (v0.1.7 L1105 and L1177; same on main, L1111/L1183). Git answers "ignored" for . when a root-level ignore source has a blank line ending in CRLF. Git's parser skips an empty LF line, but for an otherwise empty CRLF line it strips the \r and adds an empty pattern, and that pattern matches ..
Repro (Git 2.56.0.windows.4, Git Bash, in a disposable directory):
Ordinary untracked listing is unaffected; only asking git about . itself trips. T3's NUL-delimited form of the probe gives the same answer, and replaying the selected staging command omits new.txt with -u.
Scope, as far as we tested:
.git/info/exclude (the file this thread suggests for the workaround) and a user-wide core.excludesFile trigger it the same way, so if an exclude workaround made your captures fast, git check-ignore -v --no-index . will tell you whether it did so by switching to -u.
A line of only spaces triggers it even with LF endings. A blank line in a subdirectory's .gitignore doesn't.
core.autocrlf=true (Git for Windows' installer default) checks a LF .gitignore out with CRLF endings unless .gitattributes pins it (for example .gitignore text eol=lf). We haven't tested other Git builds or operating systems.
Why it matters for this issue: changing nothing but line endings can switch which staging branch runs, so fast, successful capture spans don't show that untracked capture works. Check that a non-ignored sentinel file actually appears in the checkpoint. In our own repo every checkpoint tree we sampled held only the tracked files. We haven't established what caused the capture timeouts we hit earlier.
A worse consequence on restore:restoreCheckpoint runs git restore --source <checkpoint> --worktree --staged -- . and then git clean -fd -- . (v0.1.7 L1317, L1346). When the capture took the -u branch, the clean can delete non-ignored untracked files, including ones that existed before the turn and that the agent never touched. We reproduced this in a scratch repo with T3's command sequence, not by driving T3. That data-loss path is now tracked separately in #32360.
Sent via Claude Opus 5.5 in Claude Code. Other AI models reviewed it before posting.
Another macOS case, on 0.1.7. It adds a trigger and a second kill path not covered above, and a test of #29259 on 0.1.7.
The project is a plain folder holding 14 repositories and 28 worktrees, plus about 7.8 GB of notes and outputs outside them. Between 2 and 5 November, Initialize Git in the thread details card made it a repository with no commits five times, because the card shows a thread's linked PRs only in git projects (#27626). Each time, every turn left a tmp_objdir-bulk-fsync-* of about 800 MB, up to 177 GB in total.
Checkpoints can't do anything useful in such a folder. git add -A stores each nested repository as a gitlink, so the agent's work inside them never reaches the checkpoint. The capture only hashes everything around them, and here that never fits in 30 s.
There is also a second kill path. When a nested repository has no commits, the first add -A fails, and the retry that excludes it runs under CHECKPOINT_RECOVERY_TIMEOUT (5 s, still on main). On a large tree that retry is killed every time and leaks as well. In a synthetic copy (770 MB in 31,360 files, one nested repository without commits), every turn took this path: the first add failed after about 21 s, the retry was killed at 5 s, and each turn left 147 MB.
I ported #29259 to v0.1.7, reading the environment from process.env because 0.1.7 has no HostProcess.Environment. Its tests pass there, including the ones for a killed add and for large-file packs. In a commit-less repository of 30,000 files, captures killed at 4 s left 118 MB each before the port and nothing after it. As noted above, turns still spend the timeout.
For the folder-of-repositories case, capture could skip a repository that has no commits when one of its untracked folders holds a .git, and record a missing checkpoint as it does for a folder that isn't a repository. The check is one git ls-files --others --exclude-standard --directory, which took 0.04 s on the real folder.
If you want the git UI on such a folder before a fix ships, a repository-local excludes file that ignores everything works: printf '/*\n' > .git/t3-ignore-all && git config core.excludesFile "$PWD/.git/t3-ignore-all". Checkpoints then commit an empty tree in about 50 ms. Don't put /* in .git/info/exclude instead: ripgrep reads that file too, so agents' search tools would see no files at the root. With this setup git check-ignore --no-index . exits 1, so the -u branch described above doesn't apply. To undo it, delete the whole .git. Removing only the setting turns it back into the repository that leaks.
FWIW, I just ran into this on my Windows machine, where one (large) repo with a lot of turns had accumulated 52 GB of these temporary directories over just a week:
Another macOS data point, on 0.1.7 stable (desktop), with two triggers not covered above. In both, the workspace is small enough for the fast capture path, but capture silently falls back to the slow full-rehash path.
Environment: T3 Code 0.1.7, macOS, Apple Silicon, git 2.54.0 (Apple Git-157). An ordinary repo with a normal HEAD and about 200k tracked files, where several agents commit to main in the same checkout.
What we saw: 66 of 73 captures in the retained traces timed out at 30 s, and ProviderTurnStartService.start took 92–124 s. The chat showed "streaming" with no output for that whole time.
Trigger 1: more than 1 MB of skip-worktree entries disables the fast path
captureCheckpoint copies the user's index and collects flagged entries from ls-files -v (skip-worktree S and lowercase assume-unchanged). Once they pass FLAGGED_INDEX_ENTRIES_MAX_BYTES (1 MiB), it sets fallback = true and rebuilds the temp index with a plain read-tree HEAD. That index has no cached stat data, so git add -A re-reads and re-hashes every tracked file, LFS clean filters included.
Our repo had 15,815 skip-worktree entries (1.7 MB). They were archived files that had already been deleted on disk and were hidden from git status with update-index --skip-worktree. Measured on the same repo:
Path
Time
Fallback (read-tree HEAD + add -A)
81.4 s
Fast path (copied index + read-tree --reset HEAD + add -A)
8.8 s
Fast path, tracked files only (add -u)
0.5 s
Fix on our side: we committed the deletions, which left 0 flagged entries. Captures now succeed in 1–10 s.
Trigger 2: a plain git read-tree <commit> in user tooling makes the next capture slow
We had a sync script that moved HEAD and then ran git read-tree <new-commit>. A plain read-tree drops cached stat data from the user's index, and capture copies that index, so the next capture re-hashed everything. With git read-tree --reset <new-commit>, the same step afterwards took 0.8 s instead of 81 s. Tools that sync the checkout in the background can cause this without anyone seeing it.
Suggestions
When the flagged-entry cap is hit, log it at warn level, or keep the copied index's stat data for the unflagged entries instead of a full read-tree HEAD. Today the fallback is silent and costs about 10× the fast path.
Windows data point on 0.1.9-nightly.20261110.4465: a 4 GB workspace filled a 118 GB system drive, and a second, tighter timeout is involved
Adding this because it differs from the reports above in three ways: the workspace is small (4 GB), the system drive is small (118 GB), and the kill appears to come from a 5-second budget rather than the 30 s VcsProcess default.
Environment
T3 Code (Nightly) 0.1.9-nightly.20261110.4465, desktop, Windows 11, Git for Windows 2.53.0, NTFS on NVMe
Workspace: a plain folder (C:\Users\<user>\Downloads) containing a .git with an unborn HEAD — no commits, no remote, no .gitignore
483 files / 3.98 GB, 415 untracked, of which 119 binaries account for 3.77 GB (896 MB zip, 453 MB mp4, 423 MB exe, 362 MB mp4, …)
System drive: 118 GB total; free space fell to 40 MB
What happened
git rev-list --all --count = 0, so headExists is false and every capture takes the full-add path.
33 tmp_objdir-bulk-fsync-* directories, 19.43 GB, created between 2026-11-11 09:46 and 2026-11-12 20:51 (~35 h).
Each held 128–215 files, 93–672 MB; average 0.59 GB. That is ~15% of the workspace per killed attempt, and 4.9× the workspace size leaked in total.
git count-objects -vH reported only 34 objects / 13.65 MiB, so the leak is invisible to normal git tooling — that is why it went unnoticed until the drive was full.
Two stale t3-checkpoint-index-<uuid>.lock files were present, matching the Windows report above.
A second, tighter timeout
The reports above attribute the kill to DEFAULT_TIMEOUT_MS = 30_000 in VcsProcess. The installed 0.1.9-nightly bundle has an additional budget in CheckpointService.ts:
So baseline capture is interrupted after 5 s, not 30 s. A 4 GB tree cannot be hashed and written in 5 s, so this path fails even in a workspace that would comfortably finish inside the 30 s git timeout. If this budget is the timer that fires, raising or scaling the 30 s default would not help this case.
I could not confirm from the outside which of the two timers killed each individual attempt (no retained traces on this install), so I am flagging it rather than asserting it.
Mechanism confirmed locally (matches archuity)
In a disposable repo with 30 × 30 MB files:
git -c core.fsync=objects,reference -c core.fsyncMethod=batch add -A -- . created tmp_objdir-bulk-fsync-4xsnF0 at 0.3 s and removed it on successful completion.
Killing the same process with TerminateProcess right after the directory appeared left tmp_objdir-bulk-fsync-TShaEt behind permanently.
Self-reinforcing loop
Because the leak is what fills the disk, the failure gets worse over time: less free space → slower writes → less progress per 5 s (or 30 s) → killed again → another directory. At the point I found it, the leak was still adding ~0.6 GB per attempt while the drive had 40 MB left. Removing the directories restored 19.5 GB.
Workaround that worked here
Removing .git/objects/tmp_objdir-* was safe: 0 commits, no refs, git count-objects unaffected. Beyond that, the same remedies as above apply — a .gitignore for the 3.77 GB of binaries (so git add only needs ~215 MB) and an initial commit so headExists is true and the incremental path is used.
Suggestion
If the 5 s baseline budget is intentional, it needs the same size-awareness as the 30 s default, or baseline capture should be skippable/deferred above some workspace size. A folder whose .git has an unborn HEAD is arguably the worst case for this bug: there is no incremental path at all, so it can never converge.
Linux data point on 0.1.9-nightly.20261110.4478: about 670 GB of tmp_pack_* / tmp_obj_* overnight, from a few large untracked videos
Environment: Ubuntu 24.04, t3 serve, git 2.43, ext4. Ordinary repo with a normal HEAD, about 7,100 tracked files (0.5 GB), and several agent threads in the same checkout.
What we saw
Untracked, non-ignored content had grown to about 67k files and 13.4 GB, mostly video frames (.jpg) plus a few 0.5–2.4 GB .mp4 renders.
The server trace shows 729 checkpoint capture failed errors with VcsProcessTimeoutError ... captureCheckpoint: git (<repo>) after 30000ms, and 661 checkpoint baseline capture failed; starting provider without a baseline.
On Linux the leftovers differ from the macOS bulk-fsync case. Each killed git add -A left a .git/objects/pack/tmp_pack_* (files over core.bigFileThreshold are streamed straight into a pack) and/or loose .git/objects/??/tmp_obj_*. The same ~564 MB file was re-streamed about every 30 s. In about 22 hours that came to 941 tmp_pack_* (503 GB) plus 883 tmp_obj_* (169 GB), and the disk reached 99%.
Measured cause of the 30 s overrun. Reproducing the capture by hand on the same tree:
GIT_INDEX_FILE=<new> git read-tree HEAD && GIT_INDEX_FILE=<new> git add -A -- . took 1 min 54 s, because a new index has no stat data, so every file is rehashed.
Running the same git add -A again on that index took 0.15 s.
So keeping the checkpoint index between captures (per workspace), instead of starting from an empty one each time, would turn this into a sub-second operation for most turns. After we moved the frames out of the checkout, a cold capture took 7.6 s.
Suggestions, in addition to those above
Keep the checkpoint temp index between captures so git's stat cache works.
On timeout or failure, delete tmp_pack_* / tmp_obj_* created during the attempt, or run git add with -c core.bigFileThreshold large enough to avoid streaming to packs, or skip untracked files over a size limit.
Back off after repeated timeouts in the same workspace instead of retrying every turn.
Workaround we use: a 10-minute timer that adds untracked files over 100 MB to .git/info/exclude and deletes tmp_* under .git/objects older than an hour that no process has open.
I can confirm this on macOS with an existing HEAD. When checkpoint staging reaches the 30-second timeout, Git's tmp_objdir-bulk-fsync-* directories can remain across subsequent attempts. The residual directories contained loose blobs, including objects that never reached the main object database.
Can T3 track the temporary object directory it owns for each checkpoint and clean it up after the child exits, including on timeout, while preserving objects from concurrent Git operations?
Another macOS data point, plus a measurement on why the capture never converges under batch mode.
Environment
T3 Code Nightly 0.1.10-nightly.20261111.4529, macOS, Apple Silicon, APFS, git 2.54.0 (Apple Git-157). No core.fsync or core.fsyncMethod in the repository config.
A repository with about 18,000 untracked media files (1.4 GB). Nine threads ran in it as a shared checkout.
What happened
In about six hours, 74 tmp_objdir-bulk-fsync-* directories built up in .git/objects, 68 MB to 605 MB each, 34 GB together. They appeared one or two per minute while threads were active.
git count-objects -v showed 1,102 loose objects, 83 MB, so none of those objects reached the object store. No git process was running when I looked.
The retained server trace covers 23:02 to 23:29. It has 52 GitVcsDriver.checkpoints.captureCheckpoint spans: 25 failed with VcsProcessTimeoutError at about 30 s, 26 succeeded, 1 was interrupted. The last timeout is at 23:29, the same minute as the newest directory.
After I wrote the untracked files into the object store once with git hash-object -w, no new directory appeared over the next 20 minutes with ten threads running.
Why git leaves the directory tmp-objdir.c registers only an atexit cleanup and no signal handler. Any signal ends git add without removing its batch directory. In disposable repositories with T3's flags (-c core.fsync=objects,reference -c core.fsyncMethod=fsync -c core.fsyncMethod=batch add -A -- .), each run on a fresh repository with 300 MB of untracked files:
Convergence
I replayed T3's capture loop: a fresh temp index each turn, SIGTERM at the timeout, SIGKILL 1 s later. The repository had 3,000 untracked 50 KB random files (152 MB), and I used a 4 s timeout in place of 30 s:
Under batch, a killed attempt throws away everything it wrote, so a workspace whose first capture exceeds the timeout never gets a checkpoint and leaks one directory per turn. With plain fsync, the method Linux already uses here, a killed attempt leaves no directory, and the objects it wrote count for the next attempt. The capture finished on turn 10.
The cost is speed. A full git add -A of those 3,000 files took 10.5 s with batch and 37.2 s with per-object fsync on this machine.
As I read #23715, a killed capture's private store is removed, which stops the leak. Its objects are removed with it, so the next attempt starts from zero. The non-convergence noted above for #29259 would remain. Keeping completed objects from a killed attempt, for example by staging without batch, or by publishing a private store's finished objects even when the add fails, would let these workspaces get a checkpoint after a few turns.
Not checked: Windows, and whether the per-object fsync cost is acceptable for normal turns.
Written by claude-opus-5-5 in Claude Code (T3 Code).
Another macOS data point, from an ordinary mobile app repo. About 280 GB in 6 days, and free space fell to 1.3 GB on a 1 TB drive before I noticed.
Environment
T3 Code (Nightly) 0.1.9-nightly.20261110.4478, macOS 26, Apple Silicon, git 2.50.1 (Apple Git-155)
An Expo / React Native monorepo with a normal HEAD. Its packed history is only 2.4 GB.
What happened
A local iOS build landed in an untracked folder that wasn't ignored (output/manual-iphone-build, 5.5 GB of .app, .dylib and derived data).
From then on every checkpoint capture timed out. The retained traces show VcsProcessTimeoutError: VCS process timed out in GitVcsDriver.checkpoints.captureCheckpoint ... after 30000ms, and each killed attempt left a 600-850 MB tmp_objdir-bulk-fsync-* behind.
562 leftover directories, about 282 GB. 485 of them appeared after the build folder did, up to 212 in one day, including overnight runs at 1-3 AM. The repo's .git grew from 15 GB to 285 GB.
The last successful checkpoint under refs/t3 was from right before the build folder appeared. So for 6 days rewind was silently broken too, and the only symptom I saw was a full disk.
Recovery
I deleted the leftover directories (git fsck came back clean) and added /output/ to .git/info/exclude. Before that there were 25 new directories in 30 minutes; since then there have been none.
Glad #23715 landed, and I'm updating to a v0.1.10 nightly. Two requests beyond the cleanup fix:
Show a visible warning when checkpoint capture keeps timing out, ideally naming the large untracked paths. Right now it fails silently until the disk is full.
Windows data point: 421 GB in .git/objects from an empty repo that never got a checkpoint. On Windows both leak forms showed up: tmp_pack_* files and tmp_objdir-bulk-fsync-* directories.
Environment
Windows 11 build 22631.6199, T3 Code desktop app (Nightly), Claude provider
t3 --version: t3 v0.1.9-nightly.20261110.4478
git --version: git version 2.56.0.windows.4 (Git for Windows, ucrt64)
Workspace
Project root D:\Vaults: a plain folder that was an empty Git repository (unborn HEAD, 0 commits, no remote). Its .git was created on 2026-11-09.
The folder held about 500 GB: Obsidian vaults plus Rust repositories with large target/ folders.
What I measured before cleanup
.git/objects reached 421 GB in 21,784 files:
420 tmp_pack_* files in objects/pack, about 0.7 GB each, about 261 GB in total
248 tmp_objdir-bulk-fsync-* directories, 0.1 to 1.2 GB each, mostly 0.6 to 0.7 GB
git rev-list --all --count = 0, so no checkpoint was ever written.
tmp_pack_* files by day: Nov 11: 91 (60 GB), Nov 12: 154 (98 GB), Nov 13: 158 (93 GB), Nov 14: 15 (9.5 GB). New ones kept appearing on Nov 16 with each new thread.
Caught live on Nov 16: starting a thread spawned git.exe -C D:\vaults -c core.fsmonitor=false -c sparse.expectFilesOutsideOfPatterns=false -c core.splitIndex=false ..., and a new tmp_objdir-bulk-fsync-* directory appeared in the same second.
About #23715
The build above (nightly 20261110.4478) predates #23715, which merged on 2026-11-14. So this data does not test that fix. Its notes say runtime verification was on Linux. It is still open whether its per-capture private object store also removes the Windows tmp_objdir-bulk-fsync-* directories.
Workaround
I deleted the .git folder (it had no commits) and pointed the projects at the real repository subfolders. That freed 421 GB.
Suggestions
On timeout or kill, clean up the tmp_objdir-* and tmp_pack_* that the attempt created.
Skip or back off checkpoints when HEAD is unborn or the workspace is very large.
Another measured macOS data point, on T3 CLI/server v0.1.9, observed 2026-11-16. This matches the temporary-object-directory failure mode described here.
Environment and impact
Apple silicon Mac mini; macOS 27.2.1 (26B131); Git 2.54.0 (Apple Git-157); local APFS; approximately 245 GB internal volume.
The workspace has a normal HEAD and existing commits/checkpoint refs (not an unborn repository).
Found 175.git/objects/tmp_objdir-bulk-fsync-* directories, totaling 166,045,278,208 allocated bytes (166.05 GB / 154.64 GiB). Many individual directories were roughly 0.9–1.15 GB. Their timestamps spanned the same day.
Free space fell to around 1.5 GB during investigation and an audit-output write hit ENOSPC. macOS Storage initially attributed large totals to System Data and later Documents; the filesystem audit located the dominant usage in these Git temporary directories, rather than Time Machine snapshots, swap, or update staging.
Local evidence
In the retained server traces, all 37GitVcsDriver.checkpoints.captureCheckpoint spans failed; 36 explicitly mentioned a timeout. Sample durations were approximately 30,018–30,029 ms, matching the 30,000 ms deadline.
A large generated Kitesurf/Bazel build-cache directory was inside the workspace and was not ignored. Nested source node_modules was already ignored, but that separate cache was not. No cache paths were in the real tracked index.
git fsck --connectivity-only --no-dangling passed, and neither objects/info/alternates nor http-alternates existed. No process had the approved temporary directories open when cleanup was checked.
Recovery and workaround
Added one narrow local .git/info/exclude rule for that generated build-cache directory, retaining the cache and its running service. No AGENTS.md changes were needed.
After exclusion, nonignored untracked-path enumeration took about 0.1 s (3,097 paths). A capture-style staging test using a separate temporary index and object directory, read-tree HEAD, then git -c core.fsmonitor=false -c core.fsync=objects,reference -c core.fsyncMethod=batch add -A -- ., completed successfully in 1.512 s. The real index and refs were not changed by that test.
Revalidated and removed 172 of the approved temporary directories: 5,807,026 files and 163,687,632,896 allocated bytes. Three candidates were retained after a file changed during validation. Repository connectivity checks passed again afterward. Normal objects, commits, refs, and working files were preserved.
This supports two distinct problems: large unignored generated input triggers capture timeouts, while cancellation leaves enough temporary Git objects to fill the disk through repeated attempts. The ignore rule mitigates our local trigger; this is not a claim that the cancellation/cleanup defect is fixed in v0.1.9, or that a subsequent live T3 checkpoint has yet been verified.
Windows data point on 0.1.10 stable (desktop), the first release with untracked-blob reuse (#37482). Reuse cannot help a workspace whose first capture already times out.
Environment
T3 Code desktop 0.1.10 (server 0.1.10, confirmed via /.well-known/t3/environment), Windows 11 Pro (build 26300), NTFS, Git for Windows 2.56.0.windows.2
Two affected projects, both ordinary repos with a normal HEAD:
Threads with no project (scratch dir) were unaffected, which isolated the cause to these workspaces.
What the user saw
Every turn in either project, with Claude, Codex or any other provider:
"Starting Claude" for several seconds before the provider started.
After the reply arrived, the thread stayed on Working for about 35 s more.
A follow-up message sent during that window was queued rather than sent.
Trace evidence (server.trace.ndjson, every turn in a ~30 min window)
ProviderTurnStartService.start ~7.4 s
RunExecutionService.captureBaseline ~6–7 s -> captureCheckpoint FAIL
RunFinalizationService.finalize ~35 s
orchestrationV2.checkpointCapture ~35 s
GitVcsDriver.checkpoints.captureCheckpoint FAIL
GitVcsDriver.checkpoints.reuseUntrackedBlobs 0 ms
GitVcsDriver.gitCommand (git add -A) 31.2 s
VcsProcessTimeoutError: VCS process timed out in
GitVcsDriver.checkpoints.captureCheckpoint: git (<repo>) after 30000ms
One post-turn capture took 68 s because it waited behind another capture in the same workspace.
Why 0.1.10's reuse never kicks in
In apps/server/src/vcs/checkpointUntrackedBlobs.ts (v0.1.10), reuse() deletes the workspace's in-memory snapshot up front ("A capture that fails after this point hashes everything once more, then remembers again"). A snapshot is only recorded after a capture succeeds. Here the very first git add -A over ~1 GB of media never finishes within the 30 s DEFAULT_TIMEOUT_MS. So there is never a snapshot to reuse, reuseUntrackedBlobs runs in 0 ms every time, and each turn rehashes the full gigabyte. The fix only helps workspaces that manage at least one successful capture.
Leak
Repo A had 7 leftover directories in .git/objects (six tmp_objdir-bulk-fsync-*, plus one tmp_objdir-t3-checkpoint-*), each 370–594 MB, 3.4 GB in total, within about 4 hours. Repo B had one (17 MB).
Workaround that worked
Added media patterns to each repo's .git/info/exclude (local only, not committed). For repo A: *.mp4 *.mov *.webm *.wav *.mp3 *.png *.jpg *.jpeg *.webp. For repo B: the same audio/video patterns plus images scoped to its media folders. This took the untracked sets from 968 MB to 14 MB and from 944 MB to 6 MB.
Deleted the leftover tmp_objdir-* directories while no git process was running. git fsck --connectivity-only was clean afterwards.
Result: the first capture after the change took 15.6 s and succeeded; the hashed blobs were remembered from then on. Every capture since has succeeded in 1.2–3.4 s in both repos, with fresh refs/t3 checkpoints, no new tmp_objdir-* directories, and replies that settle immediately.
Suggestions (agreeing with the issue body)
Bootstrap the blob reuse on a timeout: keep the blobs that were already hashed, or let the first capture in a workspace run without the 30 s cap, so the second capture can converge.
Same issue here, on an older build, with a repo that has a normal history (not an unborn HEAD):
T3 Code 0.1.2-nightly.20261021.3473, macOS (Apple Silicon), git 2.50.1 (Apple Git-155)
The repo has regular commits. The slow part is untracked content (node_modules, a temp folder, and several GB of generated data exports), which git add -A in captureCheckpoint hashes on every turn.
The server trace shows VcsProcessTimeoutError … captureCheckpoint … after 30000ms.
Over 6 days this left 437 tmp_objdir-bulk-fsync-* directories, each with about 14,500 files and about 600 MB, so 171.6 GB in total. The actual object store is 3.7 GB. The disk went to 99% full.
Even after cleanup, new directories keep appearing (14 in one hour).
Removing the stale dirs is safe: git fsck --connectivity-only was clean afterwards.
+1 for cleaning up the tmp objdir when the capture is killed, and/or making the timeout configurable or skipping checkpoints for slow workspaces.
Activity
markshan00 commentedon Oct 28, 2026
Another macOS data point with the same failure mode, on the 0.1.2 stable line. This one is a media folder, so each leaked directory is much larger.
Environment
.mp4,.movand.wav(82 files over 100 MiB), with no.gitignore. It had become a git repository with an unborn HEAD (no commits, no index, no refs) shortly before the problem started. None of the agent session logs contain agit init, so the in-app "Initialize Git" action is the most likely cause, but I can't confirm it because the server logs from that time had already rotated.What happened
git -C <workspace> -c core.fsmonitor=false … -c core.fsync=objects,reference -c core.fsyncMethod=fsync -c core.fsyncMethod=batch add -A -- .against a.git/t3-checkpoint-index-*and was killed at 30 s (VcsProcessTimeoutError … GitVcsDriver.checkpoints.captureCheckpoint … after 30000ms). The trace had rotated except for the last few minutes; all 4 captures in that window timed out at 30.0 s. No checkpoint ever succeeded: there were norefs/t3refs, and the real object store held only 73 loose objects (2.3 MiB) and no packs.tmp_objdir-bulk-fsync-*directories accumulated over about 16 hours, most of them during roughly 6 hours of active use. With several threads active, two sometimes appeared in the same minute. Each was about 1.5 GiB (a sampled one held 585 objects), about 94 GiB in total. Free disk space fell to about 8 GB before I noticed.tmp_objdir-bulk-fsync-*left from a single interrupted capture. This leak isn't limited to unusual workspaces; it is just much faster in them.Recovery
I stopped the running capture and deleted the whole
.git. With no commits there was nothing to lose, and the workspace files were untouched. That brought free space back to about 98 GiB.Notes on fixes
tmp_objdirpath. That would stop the leak, but on a workspace like this the capture would still burn 30 s of I/O on every turn.dpreil-ydlp commentedon Oct 29, 2026
Another macOS data point. The issue is still present on 0.1.4 stable.
Environment
/Applications/T3 Code (Beta).app): it has the samecaptureCheckpoint/stageFiles/cleanupTempIndexcode.core.fsyncMethod=batchon the capture command.What happened (2026-10-26)
lsofshowed onegitprocess writing both<repo>/.git/t3-checkpoint-index-<id>.lockand<repo>/.git/objects/tmp_objdir-bulk-fsync-<x>/…..gitwas 46.1 GB. After I removed the leakedtmp_objdir-bulk-fsync-*dirs it was 1.4 GB, and free space came back to about 43 GiB. Other large directories (Codex worktrees) did not change in size over the same samples, so the checkpoint writer was the active drain.Code notes (bundled names, 0.1.3 nightly; same code in 0.1.4)
stageFilesrunsgit add -A -- .against the temporary index. Nothing checks the size of the untracked input first.cleanupTempIndexremoves only the temp index and its.lock. It never removes atmp_objdir-bulk-fsync-*created by the attempt that timed out, so every killed attempt leaks one.writeout-onlyre-run inVcsProcess.runUnbounded(gated bycanRetryFullFsync) is a second fulladd -A, but only for the two fsync/lock stderr patterns. TheisTransientGitExitpath inVcsProcess.runretries up to 2 more times. Neither path runs on a timeout, so the leak comes from capture being attempted again turn after turn, not from these retries.Suggested fix, in addition to the quarantine cleanup in #15400
git add, measure the selected untracked input (respecting ignore rules) plus tracked changes. If it is over a budget, skip the capture or mark it incomplete. Don't edit user ignore files, and don't claim recovery coverage for content that was skipped.tmp_objdir-*that this attempt created.Test idea: use a small synthetic repo with a small byte limit and force repeated stage timeouts. Assert that no new
tmp_objdir-*remain, that a concurrent writer's files and the live index survive, and that a normal checkpoint still restores the source.klinerj commentedon Oct 30, 2026
Another macOS data point, on 0.1.5 stable, in an ordinary code repository (normal HEAD, actively developed), not an unborn or media folder. Here the tracked files alone exceed the timeout.
Environment
What happened
tmp_objdir-bulk-fsync-*directories built up in.git/objects, about 41 GB, roughly 1 GB per turn.VcsProcessTimeoutError: VCS process timed out in GitVcsDriver.checkpoints.captureCheckpoint: git (<workspace>) after 30000ms. While it ran,git add -A -- .(withcore.fsyncMethod=batch) showed up as a child oft3 serve.git status,git update-index --refresh) takes about 100 s here. If capture starts from a fresh index each time, as the report above reads the source, it can't finish in 30 s in this repo even with no untracked files at all.Removing the untracked output made each leftover smaller but didn't stop them. Reusing the previous checkpoint index, or cleaning up
tmp_objdir-*when the timeout fires, would fix it for us.mateodelcon1 commentedon Oct 30, 2026
Windows data point for the same failure mode, on 0.1.5 stable (desktop). It's the first one here that isn't macOS:
core.fsyncMethod=batchis also applied on win32, so NTFS leaks the sametmp_objdir-bulk-fsync-*directories. It adds two details: the trigger was T3's own scratch folder, and the temp index.lockfiles leak too.What happened
~/.t3/scratch. On 2026-10-20 something rangit initin~(unborn HEAD, no commits, no remote). From then on,git rev-parse --show-toplevelin~/.t3/scratchresolved to~, so every capture rangit add -A -- .for the scratch folder against~/.git..gitignore, because it isn't meant to be a repo. Hashing it doesn't fit in 30 s.refs/t3/orchestration-v2/checkpoints/…) are from 2026-10-22 19:06 local. No capture succeeded after that, so revert didn't work for any scratch thread for 8 days.Diagnosis (source at
v0.1.5,6d450a4284)Same as the original report:
stageFilesinapps/server/src/vcs/GitVcsDriver.ts(~L1040) runsadd -AwithdurableWrite(core.fsyncMethod=batchon darwin and win32, ~L814) and notimeoutMs. SoVcsProcesskills it atDEFAULT_TIMEOUT_MS = 30_000(VcsProcess.ts:64).apps/server/srccleans upobjects/tmp_objdir-*after a timeout or interruption.Windows-specific:
cleanupTempIndexruns inEffect.ensuring, but itsremove(…).pipe(Effect.ignore)left 6 stale 0-bytet3-checkpoint-index-<uuid>.lockfiles in~/.git, aged 31–199 h.git.exestill holds the lock when the cleanup runs, Windows refuses the delete, and the error is swallowed.mainat2e860972ae(2026-10-30) still has notmp_objdircleanup, and the capture still uses the default timeout. #21315 and #21329 reduce git process count and scan cost, but don't change this.Steps to reproduce (Windows)
git initin your home folder (or have any repo at~).~/.t3/scratch, and let agents write a few GB of files there.VcsProcessTimeoutError … captureCheckpoint: git (~\.t3\scratch) after 30000ms, plus one new~/.git/objects/tmp_objdir-bulk-fsync-*directory, and sometimes a leftovert3-checkpoint-index-*.lock.Version
T3 Code (Beta) 0.1.5 desktop (server 0.1.5, per the triage context). Triage CLI 0.1.5.
Environment
Windows 11 Pro 26H2 (10.0.26300), NTFS on NVMe, Git for Windows 2.55.0.windows.5, Ryzen 9 7950X3D, 63 GB RAM. Desktop app with a local server; several agent threads running at once.
Evidence
tmp_objdir-bulk-fsync-*in~/.git/objects, the first dated 2026-10-22 19:12. Per day: 80 / 68 / 110 / 183 / 161 / 200 / 130. The count rose from 932 to 948 within about 40 minutes today.~/.gitwas 48 GB in about 9.7 million files. A sample of 7 directories held 5.6k–13.6k files and 31–92 MB each.git count-objectssaw only 1,096 loose objects; everything else sat in the leaked directories.VcsProcessTimeoutErrorspans forGitVcsDriver.checkpoints.captureCheckpointin the retained trace files, all between 20:23 and 21:02 UTC on 2026-10-30, about 3 per minute with 6–7 threads active. Durations ran from 30,136 to 45,773 ms.MsMpEng) averaged about one full core over 2 days, with samples at 171–218%.git addwas running at about 98% of a core most of the time.Related issues
#26227 (this issue), #3646 and #15400 (
tmp_packlitter from failed checkpoints), #1228 (checkpoint timeout when running from the home directory), #24083.Fix applied or workaround
~/.gitout of the profile.~/.t3/scratchis no longer inside a repo, so captures there now fail fast as "not a git repository" instead of timing out. After later turns, T3 had not re-created a.gitin~or in the scratch folder.tmp_objdirstayed locked by another process and needs deleting after a reboot..lockcleanup after the child has fully exited, at least on Windows.~. Anyone with a repo at their home folder would otherwise hit this on every scratch thread.Filed by
Claude Opus 5.5 (
claude-opus-5-5) in Claude Code, following thet3 triage0.1.5 playbook, on the affected machine.Yclpaiva commentedon Oct 31, 2026
Another macOS case on October 31: my disk fell to about 0.5 GiB free. I found 273 abandoned .git/objects/tmp_objdir-bulk-fsync-* directories holding 82.5 GiB and 161,853 files.
Environment: T3 Code Nightly 0.1.4-nightly.20261027.3770, macOS 26.4, APFS, Apple Silicon arm64, Git 2.55.0. The workspace has an unborn HEAD, contains generated media, and its 494 refs are all T3 checkpoints.
The retained server traces contain six capture timeouts at approximately 30 seconds, reported as VcsProcessTimeoutError in GitVcsDriver.checkpoints.captureCheckpoint. Each interval matches the modification time of one abandoned directory. A Git object hash also confirmed that the same original video had been copied into multiple abandoned directories.
I tested the interruption mechanism separately in disposable repositories on this machine:
After checking that the old directories were not in use, I removed only those 273 directories. All 494 checkpoint refs remained identical, the normal object count stayed at 11,674, and git fsck --connectivity-only --no-dangling passed before and after. Original workspace files were preserved.
Linking #23715 as the relevant proposed fix. I can help validate its batch-fsync timeout and cancellation cleanup on native macOS arm64 with Git 2.55.0, along with successful capture and restore. I have not tested that PR yet. The affected T3 build above is the one investigated; I have not reproduced this incident on a newer release.
archuity commentedon Nov 3, 2026
Another Windows data point, from a normal checkout with HEAD (not an unborn repo), on desktop 0.1.4-nightly.20261028.3841.
What we saw. The repo has about 300k tracked paths. At the worst point it also had about 9,000 nonignored untracked files (0.9 GB, none over 50 MB).
.git/objectshad collected 349tmp_objdir-bulk-fsync-*folders, 39.7 GB in total. On a busy day, about 82 were created in one hour. We assume most came from checkpointgit add -Aruns killed at the 30 s limit. We didn't match each folder to a trace.Why it seems to happen. As far as we can read the installed build, each capture makes a fresh UUID temp index, seeds it from the workspace index, and removes it afterwards. The workspace index doesn't hold the untracked files, so a fresh capture index doesn't keep their cached metadata from the last capture. That fits the cold and warm difference we measured. (#23624 now covers skip-worktree and assume-unchanged entries; ours is the untracked files that aren't in the workspace index.) We replayed the add on a copy of the index, with a temp object dir:
These are
git addtimings, not whole captures. Refreshing the real index (update-index --refresh) didn't improve this replay. Reusing one temp index is evidence of the direction, not a safe implementation.What helped. We hid untracked files over 1.5 MB with
.git/info/exclude. In T3's own trace (GitVcsDriver.checkpoints.captureCheckpoint), captures went from a 32.5 s median with 23 of 29 failed (19 minutes before) to an 11.0 s median with 0 of 9 failed (5 minutes after). New leftover folders dropped from about 82 an hour to about one. Later we cut the untracked files from about 9,000 to under 200. A sparse worktree replayed in 2.9 to 5.7 s, against 17.6 to 22.1 s on the full checkout.What would help us. The proposals already in this thread match what we need. Our addition is the measured cold and warm comparison above.
t3-checkpoint-index-*.lockwas also left from 10-28, though the existing finalizer should remove it. We can't say why it stayed.Environment: Windows 11 Pro, Git 2.54.0.windows.1, T3 Code desktop 0.1.4-nightly.20261028.3841 (server bundle SHA-256 0a3cfede...7464). We haven't tied the running backend version to logs yet, and we haven't tried 0.1.7. Reading the latest nightly's source, it still uses a fresh capture index and the 30 s staging default, but we haven't reproduced the incident on it. Happy to run a check on a newer nightly.
romanolivera commentedon Nov 4, 2026
Additional incident on macOS: about 1.28 TB of abandoned checkpoint objects nearly filled a 2 TB system drive. Adding the measurements here because they match this issue.
Environment
Measured disk usage
.git/objects/tmp_objdir-bulk-fsync-*.No space left on device.Three sampled temporary directories contained 5,210 common Git object names. Matching examples had the same sizes but different inodes and a link count of one, consistent with repeated separate copies. One sample alone occupied approximately 1.61 GB.
Correlation with T3 checkpoint failures
The retained T3 server traces contained 82 checkpoint attempts, all failed:
VcsProcessTimeoutErrorfailures at the 30,000 ms limit.VcsProcessExitErrorfailures with exit code 128.GitVcsDriver.checkpoints.captureCheckpointintervals.Representative error, with the private path removed:
The timestamp comparison supports the same checkpoint leak described above. Retained logs do not cover the full accumulation period, so I am not claiming an individual trace match for all 1,025 directories.
Impact and requested fix
This exhausted disk capacity and disrupted work outside T3. T3 has been stopped. A completed cleanup and a fixed release have not yet been verified.
Please prioritise preventing each failed or interrupted capture from leaving its temporary object storage behind. Cleanup should remove only storage owned by that capture, after its Git child has exited, and preserve concurrent writers. A regression test should force repeated capture timeouts/cancellation and confirm that temporary storage does not accumulate. Backoff or a visible suspension after repeated failures would also limit the continuing I/O load.
No private workspace names, file names, file contents, absolute paths, thread identifiers, or full logs are included in this report.
lyndon050516 commentedon Nov 4, 2026
Another macOS data point. The leak is still present on the 2026-11-03 nightly, and it used about 449 GB in 25 hours.
Environment
git add -Amust process: 18,008 untracked, non-ignored files, 31.5 GB, of which 25.9 GB is rendered video output.What happened
.git/objectsheld 457tmp_objdir-bulk-fsync-*directories, 448.9 GB in total. The real object store held 632 MiB.Link between timeouts and directories
The retained trace covers 06:59 to 07:56 on Nov 4. It has 53
VcsProcessTimeoutErrorfailures fromGitVcsDriver.checkpoints.captureCheckpointin this workspace. In the same window, 53 directories appeared, and each one appeared within 35 seconds before a failure.Note on #29259
As I read that PR, it stops the leak but does not change the timeout. This workspace would still have no checkpoints and would still spend 30 seconds on each failed capture. The back-off or size budget suggested above would still be needed for this type of project.
dl-pt commentedon Nov 5, 2026
I observe the same issue on macOS Apple Silicon with T3 0.1.7 using Codex. Repeated checkpoint timeouts leave eight
tmp_objdir-bulk-fsync-*directories behind, and block each prompt for 30 seconds before it reaches Codex. #28996 is the same issue.Chiptonk commentedon Nov 6, 2026
Another Windows data point, with a workaround that has held.
Environment: T3 Code Nightly (currently 0.1.8.0; the build at the time of the incidents wasn't recorded), Windows 11, NTFS, git 2.55.0.windows.5. An ordinary repo with commits, where other agents commit to
main.What happened: about 5.6 GB of untracked, non-ignored files made
captureCheckpointhit the 30-second timeout. Each killedgit addleft atmp_objdir-bulk-fsync-*folder of about 1 GB and about 3,400 files. It happened three times: 2026-10-29 (size not recorded), 2026-11-01 (376.7 GB) and 2026-11-04 (352.6 GB in 358 folders, up to about 37 GB an hour).Workaround: adding the idle bulky folders to
.git/info/excludebrought the files to save under 1 GB. After that, 377 checkpoints were recorded with no leftover folders, saves took about 3 seconds, and the trace logs showed no failures. Those logs only reach back about 4 hours here. It only works for folders that don't need saving, and it doesn't fix the timeout.For #29259: on Windows, each killed attempt here cost about 1 GB, so a fix that removes the batch folder would have prevented all of this.
janoschbl commentedon Nov 6, 2026
Linux data point, on 0.1.5 stable. On Linux the leak takes a different form: there are no
tmp_objdir-bulk-fsync-*directories, and the killed captures leave.git/objects/pack/tmp_pack_*files instead. Here they filled the disk that also holds the T3 data directory, and T3 Connect stopped answering.What happened
Diagnosed and written by Claude Opus 5.5, based on a
t3 triagesession.The mobile app showed "T3 Code on omarchy sent an unexpected answer". The desktop app had hung for about 11 minutes before that.
Diagnosis
optimizer.ptis about 839 MB.captureCheckpointrunsgit add -Awith the defaultDEFAULT_TIMEOUT_MS = 30_000(apps/server/src/vcs/VcsProcess.ts:64). The trace showsVcsProcessTimeoutError ... captureCheckpoint ... after 30000ms.GitVcsDriver.tsaddscore.fsyncMethod=batchonly ondarwinandwin32(apps/server/src/vcs/GitVcsDriver.ts:819, v0.1.5). On Linux, Git does not use atmp_objdirquarantine. Files abovecore.bigFileThreshold(512 MiB by default) go through bulk-checkin, which streams them into.git/objects/pack/tmp_pack_*. A killedgit addleaves that file behind, and the next capture writes it again.tmp_pack_*files built up, 107 GB in total. They arrived in pairs every 15 minutes, about 700 MB each.cloudflaredstayed connected, but the origin answered 77 relay requests with HTTP 500. The mobile app reportedinvalid_responseandendpoint_request_failedwith the tunnel marked "ok". The desktop hang probably has the same cause. I did not confirm that.Steps to reproduce
git add -Atakes longer than 30 seconds..git/objects/pack/. Each timed-out capture leaves atmp_pack_*file that is never cleaned up.Version
T3 Code 0.1.5 desktop (AUR
t3code-bin0.1.5-1), host server 0.1.5. Mobile app 2.2.0.Environment
Arch Linux (Omarchy), kernel 7.2.8, x64, git 2.56.0, ext4.
Evidence
Fix applied
tmp_pack_*files while no Git process was running. That freed 107 GB.*.pt,*.safetensors,*.onnx,*.bin,checkpoint-*/and similar patterns to the repo's.git/info/exclude.git statusnow finishes in milliseconds and captures stop timing out.Cleaning up on timeout should cover
objects/pack/tmp_pack_*on Linux as well astmp_objdir-*.Sent from Claude Opus 5.5 (Claude Code, via
t3 triage) for janosch.davidalexander commentedon Nov 6, 2026
Confirmed on macOS/Apple Silicon with
0.1.8-nightly.20261104.4165: 790 leftovertmp_objdir-bulk-fsync-*directories occupied 27.4 GB.Excluding specific generated QA/source copies through
.git/info/excluderestored successful checkpointing. This workaround reduces checkpoint coverage.EvilHumphrey commentedon Nov 7, 2026
Windows checkpoint-coverage trap worth knowing when validating a fix here: a capture can succeed quickly and still omit all non-ignored untracked files.
Why: before staging,
captureCheckpointasksfilterIgnoredPaths(input.cwd, ["."])whether the workspace root itself is ignored. If it is, it stages withgit add -uinstead of-A(v0.1.7 L1105 and L1177; same onmain, L1111/L1183). Git answers "ignored" for.when a root-level ignore source has a blank line ending in CRLF. Git's parser skips an empty LF line, but for an otherwise empty CRLF line it strips the\rand adds an empty pattern, and that pattern matches..Repro (Git 2.56.0.windows.4, Git Bash, in a disposable directory):
Ordinary untracked listing is unaffected; only asking git about
.itself trips. T3's NUL-delimited form of the probe gives the same answer, and replaying the selected staging command omitsnew.txtwith-u.Scope, as far as we tested:
.git/info/exclude(the file this thread suggests for the workaround) and a user-widecore.excludesFiletrigger it the same way, so if an exclude workaround made your captures fast,git check-ignore -v --no-index .will tell you whether it did so by switching to-u..gitignoredoesn't.core.autocrlf=true(Git for Windows' installer default) checks a LF.gitignoreout with CRLF endings unless.gitattributespins it (for example.gitignore text eol=lf). We haven't tested other Git builds or operating systems.Why it matters for this issue: changing nothing but line endings can switch which staging branch runs, so fast, successful capture spans don't show that untracked capture works. Check that a non-ignored sentinel file actually appears in the checkpoint. In our own repo every checkpoint tree we sampled held only the tracked files. We haven't established what caused the capture timeouts we hit earlier.
A worse consequence on restore:
restoreCheckpointrunsgit restore --source <checkpoint> --worktree --staged -- .and thengit clean -fd -- .(v0.1.7 L1317, L1346). When the capture took the-ubranch, the clean can delete non-ignored untracked files, including ones that existed before the turn and that the agent never touched. We reproduced this in a scratch repo with T3's command sequence, not by driving T3. That data-loss path is now tracked separately in #32360.Sent via Claude Opus 5.5 in Claude Code. Other AI models reviewed it before posting.
ridhitb commentedon Nov 7, 2026
Another macOS case, on 0.1.7. It adds a trigger and a second kill path not covered above, and a test of #29259 on 0.1.7.
The project is a plain folder holding 14 repositories and 28 worktrees, plus about 7.8 GB of notes and outputs outside them. Between 2 and 5 November, Initialize Git in the thread details card made it a repository with no commits five times, because the card shows a thread's linked PRs only in git projects (#27626). Each time, every turn left a
tmp_objdir-bulk-fsync-*of about 800 MB, up to 177 GB in total.Checkpoints can't do anything useful in such a folder.
git add -Astores each nested repository as a gitlink, so the agent's work inside them never reaches the checkpoint. The capture only hashes everything around them, and here that never fits in 30 s.There is also a second kill path. When a nested repository has no commits, the first
add -Afails, and the retry that excludes it runs underCHECKPOINT_RECOVERY_TIMEOUT(5 s, still on main). On a large tree that retry is killed every time and leaks as well. In a synthetic copy (770 MB in 31,360 files, one nested repository without commits), every turn took this path: the first add failed after about 21 s, the retry was killed at 5 s, and each turn left 147 MB.I ported #29259 to v0.1.7, reading the environment from
process.envbecause 0.1.7 has noHostProcess.Environment. Its tests pass there, including the ones for a killed add and for large-file packs. In a commit-less repository of 30,000 files, captures killed at 4 s left 118 MB each before the port and nothing after it. As noted above, turns still spend the timeout.For the folder-of-repositories case, capture could skip a repository that has no commits when one of its untracked folders holds a
.git, and record a missing checkpoint as it does for a folder that isn't a repository. The check is onegit ls-files --others --exclude-standard --directory, which took 0.04 s on the real folder.If you want the git UI on such a folder before a fix ships, a repository-local excludes file that ignores everything works:
printf '/*\n' > .git/t3-ignore-all && git config core.excludesFile "$PWD/.git/t3-ignore-all". Checkpoints then commit an empty tree in about 50 ms. Don't put/*in.git/info/excludeinstead: ripgrep reads that file too, so agents' search tools would see no files at the root. With this setupgit check-ignore --no-index .exits 1, so the-ubranch described above doesn't apply. To undo it, delete the whole.git. Removing only the setting turns it back into the repository that leaks.Written by Claude Opus 5.5 in Claude Code.
edemaine commentedon Nov 9, 2026
FWIW, I just ran into this on my Windows machine, where one (large) repo with a lot of turns had accumulated 52 GB of these temporary directories over just a week:
kaijiabo commentedon Nov 12, 2026
Another macOS data point, on 0.1.7 stable (desktop), with two triggers not covered above. In both, the workspace is small enough for the fast capture path, but capture silently falls back to the slow full-rehash path.
Environment: T3 Code 0.1.7, macOS, Apple Silicon, git 2.54.0 (Apple Git-157). An ordinary repo with a normal HEAD and about 200k tracked files, where several agents commit to
mainin the same checkout.What we saw: 66 of 73 captures in the retained traces timed out at 30 s, and
ProviderTurnStartService.starttook 92–124 s. The chat showed "streaming" with no output for that whole time.Trigger 1: more than 1 MB of skip-worktree entries disables the fast path
captureCheckpointcopies the user's index and collects flagged entries fromls-files -v(skip-worktreeSand lowercase assume-unchanged). Once they passFLAGGED_INDEX_ENTRIES_MAX_BYTES(1 MiB), it setsfallback = trueand rebuilds the temp index with a plainread-tree HEAD. That index has no cached stat data, sogit add -Are-reads and re-hashes every tracked file, LFS clean filters included.Our repo had 15,815 skip-worktree entries (1.7 MB). They were archived files that had already been deleted on disk and were hidden from
git statuswithupdate-index --skip-worktree. Measured on the same repo:read-tree HEAD+add -A)read-tree --reset HEAD+add -A)add -u)Fix on our side: we committed the deletions, which left 0 flagged entries. Captures now succeed in 1–10 s.
Trigger 2: a plain
git read-tree <commit>in user tooling makes the next capture slowWe had a sync script that moved HEAD and then ran
git read-tree <new-commit>. A plainread-treedrops cached stat data from the user's index, and capture copies that index, so the next capture re-hashed everything. Withgit read-tree --reset <new-commit>, the same step afterwards took 0.8 s instead of 81 s. Tools that sync the checkout in the background can cause this without anyone seeing it.Suggestions
read-tree HEAD. Today the fallback is silent and costs about 10× the fast path.WhiteBalance2800K commentedon Nov 13, 2026
Windows data point on 0.1.9-nightly.20261110.4465: a 4 GB workspace filled a 118 GB system drive, and a second, tighter timeout is involved
Adding this because it differs from the reports above in three ways: the workspace is small (4 GB), the system drive is small (118 GB), and the kill appears to come from a 5-second budget rather than the 30 s
VcsProcessdefault.Environment
C:\Users\<user>\Downloads) containing a.gitwith an unborn HEAD — no commits, no remote, no.gitignoreWhat happened
git rev-list --all --count= 0, soheadExistsis false and every capture takes the full-add path.tmp_objdir-bulk-fsync-*directories, 19.43 GB, created between 2026-11-11 09:46 and 2026-11-12 20:51 (~35 h).git count-objects -vHreported only 34 objects / 13.65 MiB, so the leak is invisible to normal git tooling — that is why it went unnoticed until the drive was full.t3-checkpoint-index-<uuid>.lockfiles were present, matching the Windows report above.A second, tighter timeout
The reports above attribute the kill to
DEFAULT_TIMEOUT_MS = 30_000inVcsProcess. The installed 0.1.9-nightly bundle has an additional budget inCheckpointService.ts:So baseline capture is interrupted after 5 s, not 30 s. A 4 GB tree cannot be hashed and written in 5 s, so this path fails even in a workspace that would comfortably finish inside the 30 s git timeout. If this budget is the timer that fires, raising or scaling the 30 s default would not help this case.
I could not confirm from the outside which of the two timers killed each individual attempt (no retained traces on this install), so I am flagging it rather than asserting it.
Mechanism confirmed locally (matches archuity)
In a disposable repo with 30 × 30 MB files:
git -c core.fsync=objects,reference -c core.fsyncMethod=batch add -A -- .createdtmp_objdir-bulk-fsync-4xsnF0at 0.3 s and removed it on successful completion.TerminateProcessright after the directory appeared lefttmp_objdir-bulk-fsync-TShaEtbehind permanently.Self-reinforcing loop
Because the leak is what fills the disk, the failure gets worse over time: less free space → slower writes → less progress per 5 s (or 30 s) → killed again → another directory. At the point I found it, the leak was still adding ~0.6 GB per attempt while the drive had 40 MB left. Removing the directories restored 19.5 GB.
Workaround that worked here
Removing
.git/objects/tmp_objdir-*was safe: 0 commits, no refs,git count-objectsunaffected. Beyond that, the same remedies as above apply — a.gitignorefor the 3.77 GB of binaries (sogit addonly needs ~215 MB) and an initial commit soheadExistsis true and the incremental path is used.Suggestion
If the 5 s baseline budget is intentional, it needs the same size-awareness as the 30 s default, or baseline capture should be skippable/deferred above some workspace size. A folder whose
.githas an unborn HEAD is arguably the worst case for this bug: there is no incremental path at all, so it can never converge.ethernetz commentedon Nov 13, 2026
Linux data point on 0.1.9-nightly.20261110.4478: about 670 GB of
tmp_pack_*/tmp_obj_*overnight, from a few large untracked videosEnvironment: Ubuntu 24.04,
t3 serve, git 2.43, ext4. Ordinary repo with a normal HEAD, about 7,100 tracked files (0.5 GB), and several agent threads in the same checkout.What we saw
.jpg) plus a few 0.5–2.4 GB.mp4renders.checkpoint capture failederrors withVcsProcessTimeoutError ... captureCheckpoint: git (<repo>) after 30000ms, and 661checkpoint baseline capture failed; starting provider without a baseline.git add -Aleft a.git/objects/pack/tmp_pack_*(files overcore.bigFileThresholdare streamed straight into a pack) and/or loose.git/objects/??/tmp_obj_*. The same ~564 MB file was re-streamed about every 30 s. In about 22 hours that came to 941tmp_pack_*(503 GB) plus 883tmp_obj_*(169 GB), and the disk reached 99%.Measured cause of the 30 s overrun. Reproducing the capture by hand on the same tree:
GIT_INDEX_FILE=<new> git read-tree HEAD && GIT_INDEX_FILE=<new> git add -A -- .took 1 min 54 s, because a new index has no stat data, so every file is rehashed.git add -Aagain on that index took 0.15 s.So keeping the checkpoint index between captures (per workspace), instead of starting from an empty one each time, would turn this into a sub-second operation for most turns. After we moved the frames out of the checkout, a cold capture took 7.6 s.
Suggestions, in addition to those above
tmp_pack_*/tmp_obj_*created during the attempt, or rungit addwith-c core.bigFileThresholdlarge enough to avoid streaming to packs, or skip untracked files over a size limit.Workaround we use: a 10-minute timer that adds untracked files over 100 MB to
.git/info/excludeand deletestmp_*under.git/objectsolder than an hour that no process has open.test(server): interrupted checkpoint capture must not leak batch obje…
lhagenmayer commentedon Nov 13, 2026
I can confirm this on macOS with an existing HEAD. When checkpoint staging reaches the 30-second timeout, Git's
tmp_objdir-bulk-fsync-*directories can remain across subsequent attempts. The residual directories contained loose blobs, including objects that never reached the main object database.Can T3 track the temporary object directory it owns for each checkpoint and clean it up after the child exits, including on timeout, while preserving objects from concurrent Git operations?
spiky02plateau commentedon Nov 13, 2026
Another macOS data point, plus a measurement on why the capture never converges under batch mode.
Environment
core.fsyncorcore.fsyncMethodin the repository config.What happened
tmp_objdir-bulk-fsync-*directories built up in.git/objects, 68 MB to 605 MB each, 34 GB together. They appeared one or two per minute while threads were active.git count-objects -vshowed 1,102 loose objects, 83 MB, so none of those objects reached the object store. No git process was running when I looked.GitVcsDriver.checkpoints.captureCheckpointspans: 25 failed withVcsProcessTimeoutErrorat about 30 s, 26 succeeded, 1 was interrupted. The last timeout is at 23:29, the same minute as the newest directory.git hash-object -w, no new directory appeared over the next 20 minutes with ten threads running.Why git leaves the directory
tmp-objdir.cregisters only anatexitcleanup and no signal handler. Any signal endsgit addwithout removing its batch directory. In disposable repositories with T3's flags (-c core.fsync=objects,reference -c core.fsyncMethod=fsync -c core.fsyncMethod=batch add -A -- .), each run on a fresh repository with 300 MB of untracked files:So no grace period lets git clean up.
Convergence
I replayed T3's capture loop: a fresh temp index each turn, SIGTERM at the timeout, SIGKILL 1 s later. The repository had 3,000 untracked 50 KB random files (152 MB), and I used a 4 s timeout in place of 30 s:
Under batch, a killed attempt throws away everything it wrote, so a workspace whose first capture exceeds the timeout never gets a checkpoint and leaks one directory per turn. With plain
fsync, the method Linux already uses here, a killed attempt leaves no directory, and the objects it wrote count for the next attempt. The capture finished on turn 10.The cost is speed. A full
git add -Aof those 3,000 files took 10.5 s with batch and 37.2 s with per-object fsync on this machine.As I read #23715, a killed capture's private store is removed, which stops the leak. Its objects are removed with it, so the next attempt starts from zero. The non-convergence noted above for #29259 would remain. Keeping completed objects from a killed attempt, for example by staging without batch, or by publishing a private store's finished objects even when the add fails, would let these workspaces get a checkpoint after a few turns.
Not checked: Windows, and whether the per-object fsync cost is acceptable for normal turns.
Written by claude-opus-5-5 in Claude Code (T3 Code).
sanzhardanybayev commentedon Nov 15, 2026
That's a real bug and it's very annoying. Are there any plans to address it?
sanzhar-foody-ai commentedon Nov 15, 2026
Another macOS data point, from an ordinary mobile app repo. About 280 GB in 6 days, and free space fell to 1.3 GB on a 1 TB drive before I noticed.
Environment
What happened
output/manual-iphone-build, 5.5 GB of.app,.dyliband derived data).VcsProcessTimeoutError: VCS process timed out in GitVcsDriver.checkpoints.captureCheckpoint ... after 30000ms, and each killed attempt left a 600-850 MBtmp_objdir-bulk-fsync-*behind..gitgrew from 15 GB to 285 GB.refs/t3was from right before the build folder appeared. So for 6 days rewind was silently broken too, and the only symptom I saw was a full disk.Recovery
I deleted the leftover directories (
git fsckcame back clean) and added/output/to.git/info/exclude. Before that there were 25 new directories in 30 minutes; since then there have been none.Glad #23715 landed, and I'm updating to a v0.1.10 nightly. Two requests beyond the cleanup fix:
dl-pt commentedon Nov 15, 2026
Time has been reduced to 5s but that obviously isn't a fix. #34849
dl-pt commentedon Nov 15, 2026
Makes the app completely unusable. It's clearly just vibe-coded slop.
aniketbhoir-git commentedon Nov 15, 2026
Windows data point: 421 GB in
.git/objectsfrom an empty repo that never got a checkpoint. On Windows both leak forms showed up:tmp_pack_*files andtmp_objdir-bulk-fsync-*directories.Environment
t3 --version:t3 v0.1.9-nightly.20261110.4478git --version:git version 2.56.0.windows.4(Git for Windows, ucrt64)Workspace
D:\Vaults: a plain folder that was an empty Git repository (unborn HEAD, 0 commits, no remote). Its.gitwas created on 2026-11-09.target/folders.What I measured before cleanup
.git/objectsreached 421 GB in 21,784 files:tmp_pack_*files inobjects/pack, about 0.7 GB each, about 261 GB in totaltmp_objdir-bulk-fsync-*directories, 0.1 to 1.2 GB each, mostly 0.6 to 0.7 GBgit count-objects -vH: count 245, packs 0, size-pack 0, garbage 418, size-garbage 260.88 GiBgit rev-list --all --count= 0, so no checkpoint was ever written.tmp_pack_*files by day: Nov 11: 91 (60 GB), Nov 12: 154 (98 GB), Nov 13: 158 (93 GB), Nov 14: 15 (9.5 GB). New ones kept appearing on Nov 16 with each new thread.git.exe -C D:\vaults -c core.fsmonitor=false -c sparse.expectFilesOutsideOfPatterns=false -c core.splitIndex=false ..., and a newtmp_objdir-bulk-fsync-*directory appeared in the same second.About #23715
The build above (nightly 20261110.4478) predates #23715, which merged on 2026-11-14. So this data does not test that fix. Its notes say runtime verification was on Linux. It is still open whether its per-capture private object store also removes the Windows
tmp_objdir-bulk-fsync-*directories.Workaround
I deleted the
.gitfolder (it had no commits) and pointed the projects at the real repository subfolders. That freed 421 GB.Suggestions
tmp_objdir-*andtmp_pack_*that the attempt created.Status questions for maintainers
v0.1.10-nightly.20261114.4695) cover the Windows and macOStmp_objdir-bulk-fsync-*case and the Linuxtmp_obj_*files, or onlytmp_pack_*? fix(server): failed checkpoints no longer leak Git's batch object directory #29259, which targeted the batch directory directly, was closed without merging.LetterXbox commentedon Nov 16, 2026
Another measured macOS data point, on T3 CLI/server v0.1.9, observed 2026-11-16. This matches the temporary-object-directory failure mode described here.
Environment and impact
.git/objects/tmp_objdir-bulk-fsync-*directories, totaling 166,045,278,208 allocated bytes (166.05 GB / 154.64 GiB). Many individual directories were roughly 0.9–1.15 GB. Their timestamps spanned the same day.ENOSPC. macOS Storage initially attributed large totals to System Data and later Documents; the filesystem audit located the dominant usage in these Git temporary directories, rather than Time Machine snapshots, swap, or update staging.Local evidence
GitVcsDriver.checkpoints.captureCheckpointspans failed; 36 explicitly mentioned a timeout. Sample durations were approximately 30,018–30,029 ms, matching the 30,000 ms deadline.node_moduleswas already ignored, but that separate cache was not. No cache paths were in the real tracked index.git fsck --connectivity-only --no-danglingpassed, and neitherobjects/info/alternatesnorhttp-alternatesexisted. No process had the approved temporary directories open when cleanup was checked.Recovery and workaround
.git/info/excluderule for that generated build-cache directory, retaining the cache and its running service. No AGENTS.md changes were needed.read-tree HEAD, thengit -c core.fsmonitor=false -c core.fsync=objects,reference -c core.fsyncMethod=batch add -A -- ., completed successfully in 1.512 s. The real index and refs were not changed by that test.This supports two distinct problems: large unignored generated input triggers capture timeouts, while cancellation leaves enough temporary Git objects to fill the disk through repeated attempts. The ignore rule mitigates our local trigger; this is not a claim that the cancellation/cleanup defect is fixed in v0.1.9, or that a subsequent live T3 checkpoint has yet been verified.
simplywebgroup commentedon Nov 17, 2026
Windows data point on 0.1.10 stable (desktop), the first release with untracked-blob reuse (#37482). Reuse cannot help a workspace whose first capture already times out.
Environment
/.well-known/t3/environment), Windows 11 Pro (build 26300), NTFS, Git for Windows 2.56.0.windows.2.mp4= 477 MB, 307.png= 357 MB, plus.jpg/.wav).png= 568 MB, 25.mp4= 233 MB, audio)What the user saw
Every turn in either project, with Claude, Codex or any other provider:
Trace evidence (server.trace.ndjson, every turn in a ~30 min window)
One post-turn capture took 68 s because it waited behind another capture in the same workspace.
Why 0.1.10's reuse never kicks in
In
apps/server/src/vcs/checkpointUntrackedBlobs.ts(v0.1.10),reuse()deletes the workspace's in-memory snapshot up front ("A capture that fails after this point hashes everything once more, then remembers again"). A snapshot is only recorded after a capture succeeds. Here the very firstgit add -Aover ~1 GB of media never finishes within the 30 sDEFAULT_TIMEOUT_MS. So there is never a snapshot to reuse,reuseUntrackedBlobsruns in 0 ms every time, and each turn rehashes the full gigabyte. The fix only helps workspaces that manage at least one successful capture.Leak
Repo A had 7 leftover directories in
.git/objects(sixtmp_objdir-bulk-fsync-*, plus onetmp_objdir-t3-checkpoint-*), each 370–594 MB, 3.4 GB in total, within about 4 hours. Repo B had one (17 MB).Workaround that worked
.git/info/exclude(local only, not committed). For repo A:*.mp4 *.mov *.webm *.wav *.mp3 *.png *.jpg *.jpeg *.webp. For repo B: the same audio/video patterns plus images scoped to its media folders. This took the untracked sets from 968 MB to 14 MB and from 944 MB to 6 MB.tmp_objdir-*directories while no git process was running.git fsck --connectivity-onlywas clean afterwards.Result: the first capture after the change took 15.6 s and succeeded; the hashed blobs were remembered from then on. Every capture since has succeeded in 1.2–3.4 s in both repos, with fresh
refs/t3checkpoints, no newtmp_objdir-*directories, and replies that settle immediately.Suggestions (agreeing with the issue body)
Diagnosed and written by Claude Opus 5.5 (Claude Code) from a
t3 triagesession (triage CLI 0.1.10).louis-sp15 commentedon Nov 19, 2026
Same issue here, on an older build, with a repo that has a normal history (not an unborn HEAD):
node_modules, a temp folder, and several GB of generated data exports), whichgit add -AincaptureCheckpointhashes on every turn.VcsProcessTimeoutError … captureCheckpoint … after 30000ms.tmp_objdir-bulk-fsync-*directories, each with about 14,500 files and about 600 MB, so 171.6 GB in total. The actual object store is 3.7 GB. The disk went to 99% full.git fsck --connectivity-onlywas clean afterwards.+1 for cleaning up the tmp objdir when the capture is killed, and/or making the timeout configurable or skipping checkpoints for slow workspaces.