fix(test-helpers): kill the whole watch process group so teardown stops leaking orphans - #77
Merged
Merged
Conversation
…ps leaking orphans spawnWatch's child is the tsx launcher, not the process that runs src/main.ts: tsx spawns a second real node process with stdio inherit, so that grandchild holds the same stdout/stderr pipe fds. SIGKILL cannot be caught or relayed, so the deadline path killed the launcher and left the grandchild reparented to pid 1, still holding the pipe open. Nothing reading it saw the stream close, which stalled test-file teardown long after all tests had gone green. (SIGINT and SIGTERM happen to be relayed by tsx today, so the graceful path was not the leak; that is not something to rely on.) spawnWatch now spawns detached, making the launcher its own process group leader, and stopWatchProcessGroup signals the whole group via process.kill(-pid, sig): PID-scoped, so it can only reach descendants of that one spawnWatch call, never a name pattern. Graceful SIGINT with a 2s grace period, then SIGKILL. runWatchTick's teardown always attempts it instead of trusting the launcher's exit state. Detaching costs the free cleanup the old code got from sitting in the runner's own process group, where a Ctrl-C reached the children too, so a module-level registry plus exit/SIGINT/SIGTERM handlers SIGKILL any still-tracked group as a last resort. Signalling can also throw on a non-ESRCH errno, so the deadline path now rejects before it signals and EPERM is treated like ESRCH; otherwise a throw from a timer callback killed the whole test file instead of failing one test, or replaced the real error in the finally. New watch-teardown-guard test pins the group actually dying, using the parent-side pipe close as the observable. Reverting either half of the contract is otherwise silent: kill(-pid) then throws ESRCH, which the helper swallows, so teardown degrades to a no-op with every existing test still green. Both revert arms were mutation-tested outside the repo and each fails this guard. Measured: the pre-change helper hung the runner for the full 10-minute cap under the same instrument where the fixed helper exits in 24.9s with zero survivors. Suite 158 tests green across repeated runs, zero orphans, typecheck clean. TICK_TIMEOUT_MS stays at 20000ms. Refs: task c71de504 Co-authored-by: Lan Nguyen Si <contact@lan-nguyen-si.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
watch-mirror-delete.test.tsflaked under suite load and the runner stalled long after every test had gone green. The cause is not the test content.spawnWatch's child is the tsx launcher, not the process that runssrc/main.ts. tsx spawns a second real node process withstdio: inherit, so that grandchild holds the same stdout/stderr pipe fds. SIGKILL cannot be caught or relayed, so the deadline path killed the launcher and left the grandchild reparented to pid 1, still holding the pipe open. Nothing reading it ever saw the stream close. (SIGINT and SIGTERM happen to be relayed by tsx today, so the graceful path was not the leak. That is not something to rely on, and the comments say so.)spawnWatchnow spawns detached, making the launcher its own process group leader, andstopWatchProcessGroupsignals the whole group viaprocess.kill(-pid, sig). That is PID-scoped: it can only ever reach descendants of that onespawnWatchcall, never a name pattern. Graceful SIGINT with a 2s grace period, then SIGKILL.runWatchTick's teardown always attempts it rather than trusting the launcher's exit state.Review found three gaps in the first cut, all fixed here:
npm testreached the children too. Measured: post-change the same signal left 2 survivors. A module-level registry plusexit/SIGINT/SIGTERM handlers now SIGKILL any still-tracked group as a last resort.reject(...)in the deadline timer, so it became an uncaught exception that killed the whole test file instead of failing one test, and in thefinallyit replaced the real error. The deadline path now rejects before it signals, and EPERM is treated like ESRCH.kill(-pid)then throws ESRCH, which the helper swallows, so teardown degrades to a no-op with every existing test still green. The newwatch-teardown-guardtest pins the group actually dying, using the parent-side pipe close as the observable. Both revert arms were mutation-tested outside the repo and each fails this guard.Evidence. The reviewer reproduced both arms under one instrument: the pre-change helper hung the runner for the full 10-minute cap, the fixed helper exited in 24.9s with zero survivors. The implementer's own negative control (both files reverted, heavy synthetic load) reproduced the failing test plus two simultaneous ppid-1 orphans. After the fix: 158 tests green across repeated runs, zero orphans in every run, typecheck clean.
TICK_TIMEOUT_MSstays at 20000ms; no clean-condition run came near it. Raising it for CI's weaker runner would need CI-side evidence we do not have.One acceptance criterion is not met and is worth recording rather than glossing: the spec asked for a wall time around 30s, and runs land at 41-60s. The reviewer traced that to
run-schedule-escalation.test.ts(about 29-59s on its own), which already behaves that way onorigin/master. Not attributable to this change.Refs: task c71de504