Skip to content

fix(rustyclawd): replace Bash-tool wall-clock cap with idle-liveness reaping (#2607)#2650

Merged
rysweet merged 1 commit into
mainfrom
feat/issue-2607-resolve-simard-issue-2607-there-must-be-no-wall-cl
Jul 6, 2026
Merged

fix(rustyclawd): replace Bash-tool wall-clock cap with idle-liveness reaping (#2607)#2650
rysweet merged 1 commit into
mainfrom
feat/issue-2607-resolve-simard-issue-2607-there-must-be-no-wall-cl

Conversation

@rysweet

@rysweet rysweet commented Jul 6, 2026

Copy link
Copy Markdown
Owner

Summary

Resolves #2607no wall-clock timeout may kill in-flight agent/LLM work.

The #2607 audit swept every tokio::time::timeout / wait_timeout / recv_timeout on a child/agent/LLM path. Exactly one remaining wall-clock kill on local agentic work was found: the RustyClawd adapter's Bash tool. This PR replaces it with idle-liveness, mirroring the canonical meeting agent proxy (src/meeting_backend/agent_proxy.rs).

The Signal parity piece of #2607 (signal_agent_mode() → Meeting idle-liveness proxy) already landed on main in #2624; this PR closes out the remaining audit item.

The bug

The Bash arm wrapped child.wait_with_output() in tokio::time::timeout(120s, …) and spawned with ProcessSpawnConfig::default() (no isolation). A command still streaming output at 120 s was SIGKILLed mid-flight → ClientError::Timeout. Worse, because the child was neither process-group isolated nor signalled on future-drop, it was orphaned — left running detached while the tool call reported failure.

The fix (idle-liveness)

  • Spawn isolated via ProcessSpawnConfig::with_isolation() (setsid): the child leads its own process group (pid == pgid) so the reaper can SIGKILL the whole subtree.
  • Stream stdout/stderr over a channel; every chunk resets last_activity. A command that keeps producing output runs unbounded regardless of total runtime. Only sustained silence for the whole window reaps the child, killing the whole process group (numeric-PID libc::kill(-pgid, SIGKILL), shell-free signal policy) so no orphan survives. The error honestly identifies an IDLE reap, never a productive kill.
  • New env knob SIMARD_RUSTYCLAWD_IDLE_LIVENESS_SECS (seconds) sets the window; 0 = fully unbounded escape hatch. Unset/malformed falls back to the model-supplied per-call timeout (ms, default 120 s), now reinterpreted as an idle window rather than a total budget.

Tests (the three #2607 acceptance scenarios + resolver)

  • bash_producing_output_past_the_window_is_never_killed_2607 — a command emitting output past the window runs to completion, all output captured.
  • bash_idle_child_is_reaped_with_honest_error_and_no_orphan_2607 — a genuinely idle child is reaped after the window with an honest "idle" error, and no PID retains the command's marker (whole process group reaped, no orphan).
  • bash_env_zero_disables_idle_reaping_2607SIMARD_RUSTYCLAWD_IDLE_LIVENESS_SECS=0 disables reaping.
  • Pure resolve_idle_window cases: positive override wins, 0None, unset/malformed → per-call fallback.

Audit — wall-clock caps on agent work (keep vs replace)

Location Semantics Verdict
base_type_rustyclawd/tool_executor.rs — Bash arm (was) wall-clock 120 s on a tool child; killed productive work and orphaned it REPLACE → idle-liveness (this PR)
meeting_backend/agent_proxy.rs — idle window idle-liveness (resets per chunk; 0 = unbounded) KEEP — canonical template
meeting_backend/agent_proxy.rsrecv_timeout(200 ms) poll tick, not a turn cap KEEP
signal_conversation/channel.rssignal_agent_mode() routes Signal turns through the Meeting idle-liveness proxy KEEP (already merged, #2624)
engineer_loop/agent_spawn.rswait_with_output() unbounded natural wait KEEP — already compliant
agent_supervisor heartbeat staleness idle-based (SIGTERM on stale heartbeat), not wall-clock KEEP
terminal_session idle detection idle-based, work-process aware KEEP
meeting_backend/close_guard.rs graceful post-turn meeting-close budget (shutdown, not a turn) KEEP
copilot_task_submit/orchestration.rs bounded polling of a remote GitHub agent job KEEP — remote poll, not local agent I/O
operator_commands_dashboard/* VM/SSH/tmux/websocket infra ops KEEP — non-agentic infra
bridge_subprocess/subprocess.rs, update_check.rs channel-framing / update-check reads KEEP — non-agentic

Keeps are deliberate. The rule targets agent turns and LLM subprocesses; a remote-job poll, a channel-framing read, or a graceful-shutdown budget is legitimately bounded and stays bounded.

Docs

New reference docs/reference/rustyclawd-bash-tool-idle-liveness.md (behavior, configuration, mechanism, full audit, invariants), linked from docs/index.md and docs/reference/base-type-adapters.md. mkdocs build --strict passes.

Verification

  • cargo test --lib base_type_rustyclawd::tool_executor — 24 passed.
  • cargo clippy --release -- -D warnings and full --all-targets --all-features --locked clippy — clean.
  • cargo fmt --all -- --check — clean.
  • mkdocs build --strict — clean.
  • Rebased onto latest origin/main; diff is exactly the 5 intended files.

Closes #2607

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

…reaping (#2607)

The RustyClawd adapter's `Bash` tool wrapped `child.wait_with_output()` in a
`tokio::time::timeout(120s, …)` and spawned without isolation. That wall-clock
cap SIGKILLed a still-producing command at 120s and, because the child was
neither process-group isolated nor signalled on drop, orphaned it — exactly the
class of bug issue #2607 targets.

Replace the cap with idle-liveness, mirroring `meeting_backend::agent_proxy`:

- Spawn via `ProcessSpawnConfig::with_isolation()` (setsid) so the child leads
  its own process group (pid == pgid) and the reaper can kill the whole subtree.
- Stream stdout/stderr over a channel; every chunk resets `last_activity`. A
  command that keeps producing output runs unbounded regardless of total
  runtime. Only sustained silence for the whole window reaps the child, and the
  error honestly identifies an IDLE reap (not a productive kill).
- New `SIMARD_RUSTYCLAWD_IDLE_LIVENESS_SECS` (seconds) sets the window; `0`
  disables reaping entirely (fully unbounded escape hatch). Unset/malformed
  falls back to the model-supplied per-call `timeout` (ms, default 120s), now
  reinterpreted as an idle window rather than a total budget.

Tests (the three #2607 acceptance scenarios) plus pure resolver cases:
- a command emitting output past the window is never killed;
- a genuinely idle child is reaped after the window with an honest error and no
  orphaned process group survives;
- `SIMARD_RUSTYCLAWD_IDLE_LIVENESS_SECS=0` disables reaping.

Docs: new reference `rustyclawd-bash-tool-idle-liveness.md` (behavior, config,
mechanism, and the full audit of every wall-clock cap on agent/LLM work,
classified keep/replace), linked from the index and base-type-adapters.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown

📊 Coverage Summary

Generated by cargo llvm-cov --workspace --summary-only (nightly, excluding test files)

Module Lines Covered Coverage
Total 140496 116585 83.0%

Coverage data from CI run. Test files matching tests?/ are excluded from line counts.

@rysweet rysweet merged commit 3335aac into main Jul 6, 2026
17 checks passed
@rysweet rysweet deleted the feat/issue-2607-resolve-simard-issue-2607-there-must-be-no-wall-cl branch July 6, 2026 06:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

No wall-clock timeouts on agent turns/subprocesses — reap on idle-liveness only (audit + signal parity)

1 participant