Skip to content

fix(console,sdk): tear down console/dev children so they cannot spin at 100% CPU - #7297

Merged
eladb merged 2 commits into
mainfrom
fix/6861-console-dev-script-cpu
Sep 7, 2026
Merged

eladb merged 2 commits into
mainfrom
fix/6861-console-dev-script-cpu

Conversation

@eladb

@eladb eladb commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #6861 — Console dev script orphaned node processes at 100% CPU.

Root cause

Detached simulator sandboxes were not reliably cleaned up on stop:

  • Simulator.stop() refused to run while starting; console fires start without awaiting it.
  • Sandbox.cleanup() only SIGTERM without SIGKILL escalation.
  • Console HTTP close never terminated the tRPC WebSocketServer; dev.mjs had no force-exit timeout.

Fix

  • Allow stop-during-start; escalate sandbox cleanup to SIGKILL; close WSS before HTTP server; harden scripts/dev.mjs shutdown.

Test plan

  • SDK sandbox-cleanup + stop-while-starting tests
  • Existing cleanup + service tests
  • WSS close unblocks server.close
  • Manual: console app dev Ctrl+C leaves no 100% CPU orphans

…at 100% CPU

Root cause for #6861: simulator sandboxes are forked with detached:true so
Ctrl+C does not immediately kill them. Several gaps then left those children
orphaned (or the parent hung) while developing the console:

1. Simulator.stop() refused to run while status was "starting", and the console
   fires simulator.start() without awaiting it — so stopping mid-boot skipped
   cleanup of already-spawned detached sandboxes.
2. Sandbox.cleanup() only sent SIGTERM and dropped the handle; busy-loop
   children that ignore SIGTERM survived forever at ~100% CPU.
3. Console HTTP close never terminated the tRPC WebSocketServer, so a
   promisified server.close() could hang; the app/dev script also had no
   force-exit timeout and ignored a second Ctrl+C.

Fixes: allow stop-during-start with abort + shared stop promise; escalate
sandbox cleanup to process-group SIGKILL after a grace period; close WSS
clients before the HTTP server; harden scripts/dev.mjs shutdown (timeout +
second-signal force exit). Regression tests cover sandbox SIGKILL escalation
and stop-while-starting with a long-running Service.

Fixes #6861
@eladb
eladb requested a review from a team as a code owner September 6, 2026 16:39
@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown

Thanks for opening this pull request! 🎉
Please consult the contributing guidelines for details on how to contribute to this project.
If you need any assistance, don't hesitate to ping use over Discord.

Adversarial review of #7297 found that the SDK's new "no handle" branch
in stopResource never actually reaped the detached sandbox child — the
test only checked the SDK's _running state, not whether the child was
gone. This commit closes that gap and tightens a few related corners.

- simulator: in stopResource, when no handle is recorded in state yet,
  recover the resource via HandleManager.tryFindHandleByPath (new) and
  call resource.cleanup() so the sandbox is reaped even if its init()
  never returns. Deallocate the handle and deregister the policy.
- simulator: add tryFindHandleByPath on HandleManager for the inverse
  handle-by-path lookup.
- simulator: in startResource, bail out of writing to state[path].attrs
  if a stop is in progress between the init() and save() awaits, so a
  concurrent stopResource doesn't leave a TypeError in its wake.
- simulator: reset _stopRequested defensively at the top of update(),
  so a leaked flag from a previous aborted start cannot silently skip
  starting resources on hot-reload.
- sandbox: in onChildError, use killProcessTree (group kill) for
  consistency with cleanup(), so grandchildren are reaped too.
- dev.mjs: drop optional chaining on forceTimer.unref() — setTimeout
  always returns a Timeout.
- test/simulator/stop-while-starting: handler now writes its PID to a
  file (path passed via Service.env), and the test asserts
  process.kill(pid, 0) throws after stop. The previous test would
  have passed even if the orphan bug persisted.

Co-Authored-By: Claude <noreply@anthropic.com>
@eladb

eladb commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

Pushed 27df3e338 with the fixes from the adversarial review. The headline issue: the new regression test in this PR only checked that the SDK's _running state machine reported "stopped" — it never verified the detached sandbox child was actually reaped. So the bug the PR was meant to fix (the "no handle" branch in stopResource short-circuiting without calling resource.cleanup()) would have persisted undetected.

What 27df3e338 does:

  • Reap the sandbox in the "no handle" branch (simulator.ts:529-553): the resource is mid-init when HANDLE_ATTRIBUTE hasn't been written to state yet, but its handle IS in _handles.paths. New HandleManager.tryFindHandleByPath(path) does the inverse lookup so stopResource can call resource.cleanup() (the path that reaches Service.stop() → sandbox.cleanup()), then deallocates the handle and deregisters the policy. This is the actual fix for Console: the dev script sometimes doesn't close and consumes 100% CPU #6861.
  • Stop-guards in startResource (simulator.ts:1054-1079): bail out of writing to state[path].attrs if a stop is in progress between the init() and save() awaits, so a concurrent stopResource doesn't leave a TypeError: Cannot set properties of undefined in its wake.
  • Defensive _stopRequested = false in update() (simulator.ts:417-420): a leaked flag from a previous aborted start would otherwise silently skip starting resources on hot-reload.
  • killProcessTree in onChildError (sandbox.ts:332-343): matches the cleanup() path so grandchildren are reaped too, not just the immediate child.
  • Strengthened test (stop-while-starting.test.ts): the handler now writes its own PID to a file (path passed via Service.env), and the test polls until the file exists, calls sim.stop(), and asserts process.kill(pid, 0) throws. This would have caught the orphan bug. forceTimer.unref?.() → forceTimer.unref() in dev.mjs while I was there.

The four low-severity findings (rewording a misleading comment, removing the beforeExit listener — confirmed unnecessary, a handle-map leak after mid-init teardown, an inconsistency between onChildError and cleanup() kill style) are folded into the same commit.

Pre-existing prettier warnings on sandbox.ts line 108 and simulator.ts line 184 are untouched — they're not from this PR. Local tsc/eslint/vitest runs are blocked by the missing .gen/ in fresh clones (see #7266); CI will be the real test.

@eladb
eladb enabled auto-merge (squash) September 7, 2026 07:24
@eladb
eladb merged commit 8407333 into main Sep 7, 2026
7 of 10 checks passed
@eladb
eladb deleted the fix/6861-console-dev-script-cpu branch September 7, 2026 07:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Console: the dev script sometimes doesn't close and consumes 100% CPU

1 participant