perf(sandbox): idle openshell-sandbox wakes ~890 times/s (orphan-reaper thread)

[btrzupek]

Summary

While a sandbox is idle (no client connected, nothing running in it), the in-sandbox openshell-sandbox process stays busy. Its openshell-orphan-reaper thread wakes about 890 times per second and its main thread about 100 times per second. One idle sandbox costs about 2 % of a CPU core, which is roughly 30 times the cost of the gateway alone.

Environment

• OpenShell v0.1.1, installed at user level
• Podman compute driver (rootless Podman 5.4.2)
• Debian 13, Linux kernel 6.18, AMD Ryzen AI (x86_64)
• One sandbox (demo, default workspace, base image nvcr.io/nvidia/base/ubuntu:24.04), created and then left idle

Measurements

I sampled cgroup cpu.stat and memory.current every 30 s for 10 minutes with no client connected:

State	CPU	Memory
Gateway alone, no sandbox	0.07 % of one core (about 2.7 s per hour)	15 MB cgroup, flat
Gateway + one idle sandbox	gateway 0.14 %, supervisor 0.10 %, sandbox 1.94 %, total 2.2 % of one core (about 78 s per hour)	gateway 60 MB, supervisor 44 MB, sandbox 20 MB

I measured wakeups from the per-thread voluntary_ctxt_switches and nonvoluntary_ctxt_switches in /proc/<pid>/task/<tid>/status, over the same window:

• openshell-orphan-reaper thread in openshell-sandbox: about 890 per second
• main thread of openshell-sandbox: about 100 per second
• gateway (all tokio workers together): about 25 per second

Expected

An idle sandbox should cost close to nothing between events. managed_children.rs appears to sleep 50 ms per loop iteration, which would be about 20 wakeups per second, so the observed rate suggests the sleep is not taking effect (or something else wakes the thread at millisecond scale). Waiting on SIGCHLD, a pidfd, or waitpid without polling would avoid the periodic wakeups altogether.

Impact

On a workstation that keeps a sandbox around between uses, this is a steady power and CPU cost. For now I stop the sandbox when it is not in use.

I'm happy to collect more data (for example perf trace, or strace -f -c on the reaper thread) if that would help.

---

Written with Claude Code.

[drew]

📋 triage-agent

Triage Assessment

Classification: validated-bug

Summary

The reported idle CPU cost is credible and identifies a real source of periodic idle work in openshell-sandbox. The exact share of CPU and context switches caused by the orphan reaper has not been independently measured here.

Investigation

In v0.1.1 and the current source, the sandbox starts openshell-orphan-reaper when it is PID 1 (boundary_server.rs). The reaper scans every 50 ms; each scan enumerates /proc/self/task and reads each task's children file (managed_children.rs). This work continues while the sandbox is idle. The 50 ms sleep implies about 20 scans/s, but 890 context switches/s for the thread does not establish 890 scans/s or a broken sleep: one scan can involve multiple blocking operations. A per-thread syscall/profile sample would distinguish scan cadence from work within a scan. The managed-child registry also means a replacement using a general blocking waitpid needs to preserve explicit waiters' ownership of their children.

I did not reproduce the Podman measurement locally. v0.1.1 is the latest stable release, and no later change to this reaper path is present in the current checkout. I found no duplicate; #2895 concerns a separate gvproxy process in the VM driver.

Impact Signals

• Affected users/scope: At least the reported rootless Podman workstation with one idle sandbox; scope across other drivers is unknown.
• Regression: Unknown.
• Workaround: Stop the sandbox between uses, at the cost of losing an always-ready sandbox.
• Evidence quality: Medium. The report includes a 10-minute cgroup sample, per-thread switch counts, and environment details; raw syscall traces and independent reproduction are unavailable.

Human Decision Required

Decide whether OpenShell should address this issue. If yes, apply state:accepted, associate it with a roadmap item, or both. If no, close it as not planned with a rationale. A useful follow-up before choosing an implementation is a per-thread strace/perf sample showing sleep cadence, /proc scan cost, task count, and the main thread's activity. Define a measurable idle CPU target for the fix.

[drew]

🏗️ build-plan

Implementation Plan

Issue type: perf
Complexity: Medium
Confidence: High for eliminating periodic idle reaper scans; CPU savings need measurement

Summary

Wake the sandbox PID 1 orphan reaper on SIGCHLD via an eventfd. Keep the existing managed-child registry and child-by-child scan so explicit waiters retain their exit status. A low-frequency safety sweep will recover from missed notifications or transient scan errors without the present 50 ms idle polling.

Scope

• crates/openshell-sandbox/src/managed_children.rs: install an async-signal-safe notifier, wait for events, preserve registry-guarded reaping, and add focused tests.
• architecture/sandbox.md: document the PID 1 reaping and managed-child ownership invariant.
• architecture/plans/3756-idle-orphan-reaper.md: working plan for this implementation.

Implementation Steps

1. Add a nonblocking, close-on-exec eventfd and SIGCHLD handler before runtime threads start. The handler only writes a counter to the descriptor.
2. Scan once at startup, then on notification. Drain the eventfd before scanning so notifications arriving during a scan cause another pass. Keep a low-frequency recovery timeout and error retry.
3. Preserve per-PID waitpid(WNOHANG) under the registry guard; cover unmanaged reaping, managed waiter ownership, burst exits, and idle waiting in tests.
4. Document the invariant and verify the sandbox path.

Test Plan

• Unit tests: Isolated Linux process tests in managed_children.rs for event wakeup, unmanaged child reaping, managed child ownership, and burst/coalesced exits; retain existing spawn/register race tests.
• Integration tests: Focused sandbox crate tests plus local idle CPU/context-switch comparison if a suitable container is available.
• E2E tests: Run the sandbox Podman lane if available; otherwise use the available Docker sandbox lane and report the limit.
• Gates: mise run pre-commit and mise run test.

Risks & Open Questions

• SIGCHLD is process-wide. The handler must be async-signal-safe and child launch must restore signal disposition/masks.
• Signals can coalesce; a scan must inspect all direct children. The low-frequency sweep bounds risk from an unobserved adopted zombie.
• The issue also reports main-thread wakeups. This plan addresses the reaper thread; total CPU improvement must be measured.
• Existing procfs access stays the same. No new cross-label /proc/<pid>/exe access is introduced for SELinux or AppArmor hosts.

Documentation Impact

Update the existing sandbox architecture overview. No gateway config or published user workflow changes.

---

Revision 1 — initial plan

[drew]

🏗️ build-from-issue-agent

Implementation Complete

PR: #3757

The sandbox PID 1 orphan reaper now wakes on child-exit notifications, with a 30-second recovery sweep. Its managed-child registry and targeted waits remain intact. An isolated process test covers idle scans, managed child waitability, and burst orphan exits; the sandbox architecture overview is updated.

mise run pre-commit and mise run test passed locally. The local Podman E2E run was stopped at the user's request; the PR is labeled test:e2e, and the GitHub Branch E2E workflow has been rerun for remaining verification.

The issue will close when the PR is merged.