Skip to content

With a thread open and the app idle, Windows starts ~32 git processes every 30 s; half of them re-read local status that nothing changed #39986

Closed

Description

@juliusmarminge

What is slow

While a thread is open, the server polls its repository every 30 s (DEFAULT_AUTOMATIC_GIT_FETCH_INTERVAL). Each poll runs VcsStatusBroadcaster.refreshRemoteStatus, which reads remote status and then always reads full local status again (refreshLocalStatusCore), even when the poll fetched nothing and no ref moved. On Windows that is about 32 git processes and over a second of git work every 30 s while the user is doing nothing, and that CPU competes with whatever they do next.

Measured

  • Windows Server 2025, 4 logical CPUs, Defender on. Commit f2ec853fbb, server from source, production web in headless Chromium. One thread open (a repo with 20 files, one commit, no remote), no input for 180 s.
  • 194 git processes in 180 s, 7.4 s of summed git process time, 0 git fetch.
  • 6 polls, refreshRemoteStatus 1.14–1.59 s each:
    • readRemoteStatus: 15 git processes per poll, ~470 ms;
    • refreshLocalStatusCore → GitWorkflowService.localStatus: 16.5 git processes per poll, 612–740 ms (99 over 180 s);
    • 2 resolveBaseBranchForNoUpstream calls per poll (13 git processes), counted inside the two lines above.
  • Linux, same setup: the same polls run.

How to measure it again

  1. Build the web app (cd apps/web && vp build). Seed a few projects with devReplay threads: start T3CODE_DEV_SEED=0 node src/bin.ts serve --base-dir <home> --port <p> --host 127.0.0.1 --dev-url http://localhost:5733 in apps/server, wait ~40 s, run node seed.cjs <apps/server dir> <home> <reposDir> 3 3 3 (script below). Stop the server and its dev-replay-agent.ts processes.
  2. Restart without --dev-url, with every child process logged: SPAWNLOG=<abs>/spawn.ndjson T3CODE_DEV_SEED=0 node --require <abs>/spawnlog.cjs src/bin.ts serve --base-dir <home> --port <p> --host 127.0.0.1.
  3. Pair (node src/bin.ts pair --ttl 1h --base-dir <home>, take the http...pair#token=... URL) and run node idle.cjs <node_modules/.pnpm/playwright@*/node_modules/playwright> <pairUrl> 1 180 idle.json.
  4. Count lines in spawn.ndjson with at between idleStart and idleEnd. For the split, wrap idle.json's window as [{"at": idleStart, "idx": 0}] and run tracecount.cjs with window 180000 over <home>/userdata/logs/server.trace.ndjson*.
// spawnlog.cjs: --require hook that logs every child process (JSON lines to $SPAWNLOG) when it exits
const cp = require("child_process"); const fs = require("fs"); const out = process.env.SPAWNLOG;
const log = (o) => { try { fs.appendFileSync(out, JSON.stringify(o) + "\n"); } catch {} };
for (const name of ["spawn", "execFile", "exec", "spawnSync", "execFileSync", "execSync", "fork"]) { const orig = cp[name];
  cp[name] = function (cmd, args) { const at = Date.now(); const r = orig.apply(this, arguments);
    const rec = { fn: name, at, cmd: String(cmd), args: (Array.isArray(args) ? args : []).map(String).slice(0, 12) };
    if (r && typeof r.on === "function") { r.on("exit", (c) => { rec.ms = Date.now() - at; rec.code = c; log(rec); }); r.on("error", (e) => { rec.err = String(e && e.code); log(rec); }); } else { rec.ms = Date.now() - at; log(rec); }
    return r; }; }
require("module").syncBuiltinESMExports();
// idle.cjs <playwrightDir> <pairUrl> <row> <idleSec> <out.json>: open one thread, then stay idle; writes the idle window (epoch ms)
const [pw, pairUrl, row, idleSec, out] = process.argv.slice(2); const { chromium } = require(pw); const fs = require("node:fs");
(async () => { const b = await chromium.launch(); const p = await (await b.newContext({ viewport: { width: 1600, height: 1000 } })).newPage();
  await p.goto(pairUrl); await p.waitForTimeout(15000); await p.locator("[data-thread-item]").nth(+row).click(); await p.waitForTimeout(15000);
  const m = { idleStart: Date.now() }; await p.waitForTimeout(+idleSec * 1000); m.idleEnd = Date.now(); console.log(JSON.stringify(m)); fs.writeFileSync(out, JSON.stringify(m)); await b.close(); })();
// seed.cjs <serverDir> <home> <reposDir> <projects> <threadsPerProject> [concurrency]
const { execFileSync, execFile } = require("node:child_process"); const fs = require("node:fs"); const path = require("node:path");
const [serverDir, home, reposDir, nP, nT, conc = "4"] = process.argv.slice(2);
const run = (a) => new Promise((res, rej) => execFile(process.execPath, ["src/bin.ts", ...a], { cwd: serverDir, env: { ...process.env, T3CODE_DEV_SEED: "0" }, maxBuffer: 1 << 26 }, (e, o, er) => e ? rej(new Error(String(er || e.message))) : res(o)));
const retry = async (a) => { try { return await run(a); } catch { return await run(a); } };
(async () => { const jobs = [];
  for (let i = 0; i < +nP; i++) { const dir = path.join(reposDir, `proj-${i}`); fs.mkdirSync(dir, { recursive: true });
    for (let f = 0; f < 20; f++) fs.writeFileSync(path.join(dir, `file${f}.txt`), `hello ${i} ${f}\n`);
    execFileSync("git", ["init", "-q", "-b", "main"], { cwd: dir }); execFileSync("git", ["add", "."], { cwd: dir });
    execFileSync("git", ["-c", "user.email=a@b.c", "-c", "user.name=seed", "commit", "-qm", "init"], { cwd: dir });
    const pid = /Added project (\S+)/.exec(await retry(["project", "add", dir, "--base-dir", home]))[1];
    for (let t = 0; t < +nT; t++) jobs.push([pid, `task ${i}-${t}`]); }
  let next = 0; await Promise.all(Array.from({ length: +conc }, async () => { while (next < jobs.length) { const [pid, m] = jobs[next++];
    await retry(["thread", "start", "--base-dir", home, "--provider", "devReplay", "--model", "todo-list", "--wait", "--json", pid, m]); } }));
  console.log("DONE", jobs.length); })();
// tracecount.cjs <windows.json> <windowMs> <trace.ndjson>...: per window, git processes and named spans (duration/git processes under it/exit)
const fs = require("node:fs"); const [winFile, W, ...traces] = process.argv.slice(2);
const sp = new Map(); for (const f of traces) for (const l of fs.readFileSync(f, "utf8").split("\n")) { try { const r = JSON.parse(l); if (r.type === "effect-span") sp.set(r.spanId, r); } catch {} }
const kids = new Map(); for (const s of sp.values()) { const k = kids.get(s.parentSpanId) || []; k.push(s); kids.set(s.parentSpanId, k); }
const st = (s) => Number(BigInt(s.startTimeUnixNano) / 1000000n);
const git = (s) => { let n = 0; const q = [s]; while (q.length) { const x = q.pop(); if (x.name === "runGitCommand") n++; q.push(...(kids.get(x.spanId) || [])); } return n; };
const NAMES = ["resolveBaseBranchForNoUpstream", "GitWorkflowService.localStatus", "GitWorkflowService.remoteStatus", "VcsStatusBroadcaster.refreshRemoteStatus"];
for (const w of JSON.parse(fs.readFileSync(winFile, "utf8"))) { const a = w.at, b = a + +W; const ins = [...sp.values()].filter((s) => st(s) >= a && st(s) < b);
  const o = { at: a, gitProcesses: ins.filter((s) => s.name === "runGitCommand").length };
  for (const n of NAMES) o[n] = ins.filter((s) => s.name === n).map((s) => `${Math.round(s.durationMs)}ms/${git(s)}git/${s.exit._tag[0]}`);
  console.log(JSON.stringify(o)); }

Where the time goes

apps/server/src/vcs/VcsStatusBroadcaster.ts: makeRemoteRefreshLoop → refreshRemoteStatusIfEnabled → refreshRemoteStatus, which runs readRemoteStatus, then yield* refreshLocalStatusCore(cwd) with no condition (comment: "Base refs and merge bases can move without changing divergence counts"). The sibling refreshRemoteAndLocalTotals in the same file already re-reads local status only when workflow.readRefSnapshot(cwd) changed.

Why it is avoidable

Local status can only be made stale by the poll if the poll moved a ref. When no fetch ran or refs are unchanged, the re-read returns what the head watcher and the last read already published. Open PR #38949 adds this ref-snapshot guard to the first subscribe and explicitly leaves the 30-s poll as it is.

What a fix has to show

The idle measurement, before and after, 180 s with one thread open:

  • GitWorkflowService.localStatus under refreshRemoteStatus when refs did not move: 6 → 0 per 180 s. Git processes in the window: 194 → about 110 or fewer (the ref snapshot itself costs about 2 git processes per poll).
  • Must stay: the remote read each poll (fetch and divergence); a local re-read in any poll whose fetch moved a ref (an origin/* update must still refresh the Changes totals); local changes still reach the client through the head watcher.
  • Overlaps with the base-branch lookup finding (2 lookups per poll are inside these reads). Count the two separately.

Found by the automated Windows performance sweep on f2ec853.


Found by the automated Windows performance sweep on f2ec853.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions