fix(HUD-1598): terminate workspace shell process trees

[jdchawla29]

Summary

• launch workspace shell commands in dedicated POSIX process groups
• terminate the whole group with SIGTERM, wait on a bounded deadline, then use SIGKILL if descendants remain
• run the same cleanup when a command finishes normally, times out, is cancelled, or its SSH channel closes
• add deterministic coverage that starts a shell child, closes the SSH-backed session, and verifies the child is reaped

Root cause

Workspace shells execute through AsyncSSH on the workspace server. Teardown stopped or lost only the immediate shell process, so longer-running descendants could be re-parented and continue running. Descendants holding inherited pipes could also prevent the output relay from completing, delaying cleanup indefinitely.

Impact

On supported POSIX execution paths, workspace session teardown now cleans up builds, simulators, servers, and other child processes instead of leaving them behind.

Validation

• uv run pytest -q hud/environment/tests/test_capability_backing.py hud/tests/test_graders.py::TestBashGrader — 16 passed
• uv run ruff format --check hud/environment/workspace.py hud/utils/process.py hud/environment/tests/test_capability_backing.py
• uv run ruff check hud/environment/workspace.py hud/utils/process.py hud/environment/tests/test_capability_backing.py
• uv run pyright hud/environment/workspace.py hud/utils/process.py hud/environment/tests/test_capability_backing.py

---

Note

Medium Risk
Changes subprocess and SSH teardown paths used by all workspace shells and other ProcessGroup callers; behavior is broader but covered by targeted POSIX tests and fixes known orphan-process leaks.

Overview
Workspace SSH command handling now tears down the whole POSIX process group when a command finishes, times out, or the client closes the SSH channel, instead of waiting only on the top-level shell and leaving background children running.

In hud/utils/process.py, group shutdown uses a bounded SIGTERM deadline, polls until the group is gone via killpg(..., 0), then escalates to SIGKILL; ProcessGroup.communicate() always runs terminate in a finally block.

In workspace.py, the relay loop races subprocess exit against process.channel.wait_closed(), runs shared cleanup in finally, and treats asyncssh.Error on stdin like a broken pipe.

A new POSIX regression test asserts that closing an SSH session reaps a long-running shell child.

^{Reviewed by Cursor Bugbot for commit c49c657. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit c49c657. Configure here.}

[cursor]

Channel close skips process exit

Medium Severity

When the SSH channel closes before the workspace subprocess finishes, _handle_process returns early after teardown but never calls process.exit(). Every other completion path in this handler sends an exit status; this new branch leaves the AsyncSSH session handler unfinished.

 

^{Reviewed by Cursor Bugbot for commit c49c657. Configure here.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c49c657c93

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Bound output draining after terminating the process group

When a descendant creates a new session/process group while retaining stdout or stderr (for example, setsid sleep 120 &), sub.terminate() cannot kill it, so these reader tasks never reach EOF and the SSH handler remains stuck here even after its channel closes or its one-hour timeout fires. Cancel or bound the output readers during teardown, or use a teardown boundary that also contains descendants which re-group themselves; otherwise the new cleanup is only a partial workaround for common daemonized children.

AGENTS.md reference: AGENTS.md:L44-L45

Useful? React with 👍 / 👎.