Skip to content

fix(test-helpers): kill the whole watch process group so teardown stops leaking orphans - #77

Merged
LanNguyenSi merged 1 commit into
masterfrom
task/c71de504-watch-test-flake
Aug 7, 2026
Merged

fix(test-helpers): kill the whole watch process group so teardown stops leaking orphans#77
LanNguyenSi merged 1 commit into
masterfrom
task/c71de504-watch-test-flake

Conversation

@LanNguyenSi

Copy link
Copy Markdown
Owner

watch-mirror-delete.test.ts flaked under suite load and the runner stalled long after every test had gone green. The cause is not the test content.

spawnWatch's child is the tsx launcher, not the process that runs src/main.ts. tsx spawns a second real node process with stdio: inherit, so that grandchild holds the same stdout/stderr pipe fds. SIGKILL cannot be caught or relayed, so the deadline path killed the launcher and left the grandchild reparented to pid 1, still holding the pipe open. Nothing reading it ever saw the stream close. (SIGINT and SIGTERM happen to be relayed by tsx today, so the graceful path was not the leak. That is not something to rely on, and the comments say so.)

spawnWatch now spawns detached, making the launcher its own process group leader, and stopWatchProcessGroup signals the whole group via process.kill(-pid, sig). That is PID-scoped: it can only ever reach descendants of that one spawnWatch call, never a name pattern. Graceful SIGINT with a 2s grace period, then SIGKILL. runWatchTick's teardown always attempts it rather than trusting the launcher's exit state.

Review found three gaps in the first cut, all fixed here:

  • Detaching costs the free cleanup the old code got from sitting in the runner's own process group, where a Ctrl-C on npm test reached the children too. Measured: post-change the same signal left 2 survivors. A module-level registry plus exit/SIGINT/SIGTERM handlers now SIGKILL any still-tracked group as a last resort.
  • Signalling can throw on a non-ESRCH errno. The throw happened before reject(...) in the deadline timer, so it became an uncaught exception that killed the whole test file instead of failing one test, and in the finally it replaced the real error. The deadline path now rejects before it signals, and EPERM is treated like ESRCH.
  • No regression guard. Reverting either half of the contract is silent: kill(-pid) then throws ESRCH, which the helper swallows, so teardown degrades to a no-op with every existing test still green. The new watch-teardown-guard test pins the group actually dying, using the parent-side pipe close as the observable. Both revert arms were mutation-tested outside the repo and each fails this guard.

Evidence. The reviewer reproduced both arms under one instrument: the pre-change helper hung the runner for the full 10-minute cap, the fixed helper exited in 24.9s with zero survivors. The implementer's own negative control (both files reverted, heavy synthetic load) reproduced the failing test plus two simultaneous ppid-1 orphans. After the fix: 158 tests green across repeated runs, zero orphans in every run, typecheck clean.

TICK_TIMEOUT_MS stays at 20000ms; no clean-condition run came near it. Raising it for CI's weaker runner would need CI-side evidence we do not have.

One acceptance criterion is not met and is worth recording rather than glossing: the spec asked for a wall time around 30s, and runs land at 41-60s. The reviewer traced that to run-schedule-escalation.test.ts (about 29-59s on its own), which already behaves that way on origin/master. Not attributable to this change.

Refs: task c71de504

…ps leaking orphans

spawnWatch's child is the tsx launcher, not the process that runs
src/main.ts: tsx spawns a second real node process with stdio inherit,
so that grandchild holds the same stdout/stderr pipe fds. SIGKILL
cannot be caught or relayed, so the deadline path killed the launcher
and left the grandchild reparented to pid 1, still holding the pipe
open. Nothing reading it saw the stream close, which stalled test-file
teardown long after all tests had gone green. (SIGINT and SIGTERM
happen to be relayed by tsx today, so the graceful path was not the
leak; that is not something to rely on.)

spawnWatch now spawns detached, making the launcher its own process
group leader, and stopWatchProcessGroup signals the whole group via
process.kill(-pid, sig): PID-scoped, so it can only reach descendants
of that one spawnWatch call, never a name pattern. Graceful SIGINT with
a 2s grace period, then SIGKILL. runWatchTick's teardown always
attempts it instead of trusting the launcher's exit state.

Detaching costs the free cleanup the old code got from sitting in the
runner's own process group, where a Ctrl-C reached the children too, so
a module-level registry plus exit/SIGINT/SIGTERM handlers SIGKILL any
still-tracked group as a last resort. Signalling can also throw on a
non-ESRCH errno, so the deadline path now rejects before it signals and
EPERM is treated like ESRCH; otherwise a throw from a timer callback
killed the whole test file instead of failing one test, or replaced the
real error in the finally.

New watch-teardown-guard test pins the group actually dying, using the
parent-side pipe close as the observable. Reverting either half of the
contract is otherwise silent: kill(-pid) then throws ESRCH, which the
helper swallows, so teardown degrades to a no-op with every existing
test still green. Both revert arms were mutation-tested outside the
repo and each fails this guard.

Measured: the pre-change helper hung the runner for the full 10-minute
cap under the same instrument where the fixed helper exits in 24.9s
with zero survivors. Suite 158 tests green across repeated runs, zero
orphans, typecheck clean. TICK_TIMEOUT_MS stays at 20000ms.

Refs: task c71de504

Co-authored-by: Lan Nguyen Si <contact@lan-nguyen-si.de>
@LanNguyenSi LanNguyenSi closed this Aug 6, 2026
@LanNguyenSi LanNguyenSi reopened this Aug 6, 2026
@LanNguyenSi LanNguyenSi closed this Aug 7, 2026
@LanNguyenSi LanNguyenSi reopened this Aug 7, 2026
@LanNguyenSi
LanNguyenSi merged commit 6c916bc into master Aug 7, 2026
3 of 5 checks passed
@LanNguyenSi
LanNguyenSi deleted the task/c71de504-watch-test-flake branch August 7, 2026 05:01
@LanNguyenSi LanNguyenSi added review:tests-pass merge-approval gate prerequisite review:checklist-complete merge-approval gate prerequisite review:comments-resolved merge-approval gate prerequisite review:scope-matches-task merge-approval gate prerequisite review:evidence-logged merge-approval gate prerequisite labels Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review:checklist-complete merge-approval gate prerequisite review:comments-resolved merge-approval gate prerequisite review:evidence-logged merge-approval gate prerequisite review:scope-matches-task merge-approval gate prerequisite review:tests-pass merge-approval gate prerequisite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant