bun test is non-deterministic on a clean clone: 3 failures then 0, same command

[CarringtonCreative]

bun test — the documented Tier 1 gate — produces different results run to run on an unmodified clone. Same command, same tree, no changes in between: 3 failures, then 0.

Reproduce (pristine v1.61.0.0, no modifications)

git clone --depth 1 https://github.com/garrytan/gstack.git && cd gstack
bun install
bun test browse/test/ test/ make-pdf/test/ \
  --ignore 'test/skill-e2e-*.test.ts' --ignore test/skill-llm-eval.test.ts \
  --ignore test/skill-routing-e2e.test.ts --ignore test/codex-e2e.test.ts \
  --ignore test/gemini-e2e.test.ts

(That's the test script from package.json, minus the trailing slop:diff.)

Run 1 — 3 failures:

(fail) --list > handles missing proposals file                                        [5006.61ms]
(fail) gstack-community-dashboard > connects to Supabase when config exists           [6887.86ms]
(fail) gstack-gbrain-install D5 detect-first > --dry-run falls through to fresh clone
       when no valid clone detected                                                   [7388.56ms]

Run 2, identical command, nothing changed — 0 failures.

A third run of just those three files together produced yet another failing set — only gstack-gbrain-install D5 detect-first > rejects a pre-existing path that lacks a valid gbrain package.json, with the other two green.

Each file is deterministic in isolation

Two runs each, same clone:

test/distill-apply.test.ts          11 pass  0 fail   (x2)
test/telemetry.test.ts              40 pass  0 fail   (x2)
test/gbrain-detect-install.test.ts  16 pass  0 fail   (x2)

So the tests themselves are fine. Something about running the suite together isn't.

What it isn't

Not shared-state pollution — at least not for distill-apply, which is hermetic:

beforeEach(() => {
  stateRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-apply-'));
  ...
});
afterEach(() => { fs.rmSync(stateRoot, { recursive: true, force: true }); });

// and in run():
env.GSTACK_STATE_ROOT = stateRoot;
delete env.GSTACK_HOME;

Fresh temp dir per test, state root redirected into it, GSTACK_HOME stripped from the child env. Another test writing to ~/.gstack can't reach it.

Not network or missing config — handles missing proposals file asserts on a hermetic temp dir and touches nothing external.

Hypothesis (not proven)

All three spawn subprocesses via spawnSync, and every failure carried a multi-second duration — 5.0s, 6.9s, 7.4s — against sub-second in isolation. Under bun's concurrent file execution, dozens of files each spawning children may be saturating the machine enough for these to time out or for a spawn to fail.

I'm flagging that as a guess. What I've established is the non-determinism and that it isn't state or environment; I haven't proven the contention mechanism.

Worth noting one failure was Expected: 0, Received: 3 — a real exit code from the binary, not an obvious timeout artifact. So contention may not explain all of them.

Why it matters

CONTRIBUTING lists bun test as the free gate that "runs on every commit." A gate that fails ~randomly trains people to re-run until green, which is how a real regression gets waved through. It also cost me a while on an unrelated PR (#2531) working out whether I'd broken something.

Possible directions

• Run the subprocess-spawning files serially (bun's --concurrency or splitting them into their own pass)
• Give the spawnSync calls explicit generous timeouts so contention produces a clear timeout rather than a confusing assertion failure
• Capture stderr/status in the failure message on those assertions, so the next person sees why the child failed

Happy to take a crack at any of these if you have a preference.

Environment: macOS (Darwin 25.5.0), bun 1.3.13, pristine clone at 94993f7.

Related: #2532 (a separate, deterministic isolation problem in test/host-config.test.ts).

[CarringtonCreative]

Ran the identical command 5 more times on the same pristine clone. 7 samples total, and the data narrows this a lot — including away from the hypothesis I posted above.

Frequency

Run	Fails	Which
1	3	--list > handles missing proposals file (5.0s) · gstack-community-dashboard > connects to Supabase (6.9s) · D5 detect-first > --dry-run falls through to fresh clone (7.4s)
2	0
3	1	D5 detect-first > --dry-run reuses a pre-existing ~/git/gbrain-shaped clone (0.67s)
4	0
5	0
6	1	D5 detect-first > rejects a pre-existing path that lacks a valid gbrain package.json (0.70s)
7	0

3 of 7 runs failed (43%). gstack-gbrain-install D5 detect-first is in every failing run — a different one of its three tests each time. The file passes 16/16 in isolation, repeatedly.

Correcting my hypothesis above

I guessed resource contention from the multi-second durations. That looks wrong for the dominant case: runs 3 and 6 failed in ~0.7s, which is a logical failure, not a timeout. And run 1 is the least trustworthy sample — I had other work running on the machine concurrently, which likely explains the 5-7s outliers there. distill-apply and telemetry each appeared only in that run and never again in 6 further samples.

Actual cause: PATH leaks into the D5 tests

test/gbrain-detect-install.test.ts defines a deliberate guard:

// Minimal PATH with POSIX tools + homebrew (for jq/git/curl) but no user-bin
// dirs — this keeps `gbrain` out of PATH deterministically across dev machines
const SAFE_PATH = '/usr/bin:/bin:/usr/sbin:/sbin:/opt/homebrew/bin:/usr/local/bin';

But run() doesn't apply it:

const env = {
  ...process.env,          // <-- PATH comes through untouched
  GSTACK_HOME: tmpHome,
  HOME: tmpHomeReal,
  ...(opts.env || {}),
};

GSTACK_HOME and HOME are pinned, so those are safe. PATH is not — and all three D5 tests call run(INSTALL, ['--dry-run']) with no opts.env, so none of them get SAFE_PATH. Meanwhile there are ~20 process.env.PATH = ... assignments across the suite's other test files. If those share a process with this one, gstack-gbrain-install sees a PATH the test never intended, finds (or stops finding) a gbrain binary, and takes a different branch.

That fits every observation: isolation-clean, fails only alongside other files, a different test in the block each time, sub-second logical failures.

I'll flag the one thing I have not proven: that bun shares process.env across concurrently-executing test files. I haven't caught a specific polluting file in the act. The circumstantial case is strong but that link is inferred.

Suggested fix

Default SAFE_PATH in run() so every test gets the guard the file already intends:

const env = {
  ...process.env,
  PATH: SAFE_PATH,
  GSTACK_HOME: tmpHome,
  HOME: tmpHomeReal,
  ...(opts.env || {}),
};

opts.env still wins, so the tests that deliberately prepend a fake-gbrain dir keep working.

Happy to submit that, validated across ~10 full-suite runs to show the 43% goes to 0 — say the word and I'll open it.

[CarringtonCreative]

Found the actual cause, and it is not the PATH leak I proposed above. Retracting that — the suggested SAFE_PATH default would also have broken these tests on any machine where bun lives at ~/.bun/bin (the official installer's location), since SAFE_PATH doesn't include it. Glad I checked before sending it.

The real cause: --dry-run makes a network call

The failing assertion is expect(r.status).toBe(0) receiving 3. In bin/gstack-gbrain-install, fail() exits 3, and one of its callers runs before any dry-run early-exit:

if ! $VALIDATE_ONLY; then
  check_prereq bun ...
  check_prereq git ...
  if ! curl -s --head --max-time 10 https://github.com >/dev/null 2>&1; then
    fail "cannot reach https://github.com. Check your network and try again."
  fi
fi

--dry-run sets DRY_RUN, not VALIDATE_ONLY, so a dry run performs a live request to github.com — then exits without ever cloning.

Direct proof

60 concurrent --dry-run invocations against temp HOME/GSTACK_HOME, outside the test harness entirely:

exit=3 :: gstack-gbrain-install: cannot reach https://github.com. Check your network and try again.
...  (9 of 60)

~15% failure rate, no test runner involved. Under a full bun test the suite spawns plenty of concurrent processes, the curl loses a race for sockets/DNS, and whichever D5 test is executing at that moment goes red. That explains everything: passes in isolation, different test each time, sub-second failures, ~43% per suite run.

Fix

Skip the reachability check under --dry-run — it plans, it never clones, so it doesn't need the network:

if ! $DRY_RUN && ! curl -s --head --max-time 10 https://github.com >/dev/null 2>&1; then

Same 60-concurrent experiment with the guard: 0 of 60. Real installs still get the fail-fast offline check.

Running the full suite 10x to confirm the 43% goes to zero; PR to follow.

One broader note

telemetry > connects to Supabase when config exists — one of the other failures in my first sample — is likely the same shape: a real network call inside the free Tier-1 suite. I haven't investigated it and am not claiming it, but it may be worth a look while this is fresh.

[CarringtonCreative]

Validation done, PR up: #2540.

Documented Tier 1 gate on a clean clone — 10 runs, 0 failures, against the 43% (3 of 7) measured on the same clone before the change. Combined with the 9/60 → 0/60 direct experiment, that's the mechanism, a causal test, and a statistical result all pointing the same way.