Skip to content

perf(sandbox): wake orphan reaper on child exit - #3757

Merged
drew merged 1 commit into
mainfrom
codex/perf-3756-idle-orphan-reaper
Sep 28, 2026
Merged

drew merged 1 commit into
mainfrom
codex/perf-3756-idle-orphan-reaper

Conversation

@drew

@drew drew commented Sep 28, 2026

Copy link
Copy Markdown
Collaborator

🏗️ build-from-issue-agent

Summary

Wake the sandbox PID 1 orphan reaper when children exit, so an idle sandbox no longer scans procfs every 50 ms. Keep a 30-second recovery sweep and preserve managed-child wait ownership.

Related Issue

Closes #3756

Changes

  • Install an async-signal-safe SIGCHLD notifier backed by a nonblocking eventfd before workload threads start.
  • Scan all direct children after notifications, retaining the registry guard and targeted waitpid(WNOHANG) behavior.
  • Add an isolated process test for idle scans, managed child waitability, and burst orphan reaping.
  • Document the PID 1 reaping invariant in the sandbox architecture overview.

Deviations from Plan

None.

Testing

  • mise run pre-commit passes
  • mise run test passes, including the new reaper test
  • Podman E2E: started locally, then stopped at user request to run remaining checks on the PR

The PR checks will provide the remaining verification. I have not measured the total idle CPU reduction in a deployed Podman sandbox; the reporter's main thread wakeups remain outside this change.

Checklist

Closes #3756

Replace the idle 50 ms procfs scan with SIGCHLD notifications and a 30 second recovery sweep. Preserve managed-child wait ownership and cover idle and exit behavior in isolated tests.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@drew drew added the test:e2e Requires end-to-end coverage label Sep 28, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for d448cd2. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@drew

drew commented Sep 28, 2026

Copy link
Copy Markdown
Collaborator Author

🏗️ build-from-issue-agent

Manual Podman smoke test

Tested commit d448cd2d9 on ARM64 Linux with an isolated rootless Podman gateway and locally built sandbox image.

  1. Created a sandbox from nvcr.io/nvidia/base/ubuntu:24.04 using openshell sandbox create --detach.
  2. Sampled the openshell-orphan-reaper thread's host /proc context-switch counters for five idle seconds: 0 switches.
  3. Used openshell sandbox exec to launch a background sleep 1 and captured its sandbox PID (67). Two seconds later, another CLI exec confirmed /proc/67 was absent, so the exited orphan was reaped.
  4. Ran a follow-up CLI exec successfully, then deleted the sandbox. The isolated gateway harness cleaned up its containers.

Result: passed. This is a focused smoke check of idle wakeups, orphan cleanup, and continued sandbox use; it is not a full idle CPU benchmark.

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

The initial code review found no blocking issues. The SIGCHLD eventfd wakeup preserves managed-child wait ownership, retains a recovery sweep, and includes focused regression coverage and architecture documentation.

Blocking findings:

  • No blocking findings remain

Carried findings:

  • None

Non-blocking suggestions:

  • None
Gator metadata
  • Validation: Implements validated issue #3756 as a focused sandbox-runtime performance fix; the maintainer author has repository admin authority.
  • Docs: Direct Fern documentation is not needed because this changes internal PID 1 reaping behavior without changing a CLI, API, configuration field, or user workflow; the sandbox architecture overview is updated.
  • Checks: Current-head Branch Checks, Helm Lint, Trivy Changes, and required status gates are green.
  • E2E: test:e2e is applied, the E2E Label Help rerun instruction was followed, and OpenShell / E2E is green on the current head.
  • Head SHA: d448cd2d9ac9d3f323a7578360a4f9e01e27af48
  • Base SHA: 8934b74a851696c70fd3445afd6ad59530bd8a52
  • Merge base SHA: 8934b74a851696c70fd3445afd6ad59530bd8a52
  • Patch ID: c2e4005e8c55e5eee70d480cd8b739eaa29f9ade
  • Gator payload: 9
  • Review mode: initial
  • Previous reviewed SHA: none
  • Review budget exhausted: no
  • Maintainer decision required: no
  • Next state: gator:watch-pipeline

@johntmyers johntmyers added gator:watch-pipeline Gator is monitoring PR CI/CD status gator:approval-needed Gator completed review; maintainer approval needed and removed gator:watch-pipeline Gator is monitoring PR CI/CD status labels Sep 28, 2026
@drew
drew added this pull request to the merge queue Sep 28, 2026
@johntmyers johntmyers added gator:merge-ready and removed gator:approval-needed Gator completed review; maintainer approval needed labels Sep 28, 2026
Merged via the queue into main with commit 6648bd0 Sep 28, 2026
203 of 205 checks passed
@drew
drew deleted the codex/perf-3756-idle-orphan-reaper branch September 28, 2026 03:19
@johntmyers

Copy link
Copy Markdown
Collaborator

gator-agent

Monitoring Complete

Monitoring is complete because this PR has merged.

Final status: The current head passed Gator review and required E2E checks, received maintainer approval, and merged from gator:merge-ready.

Gator metadata
  • Head SHA: d448cd2d9ac9d3f323a7578360a4f9e01e27af48
  • Gator payload: 9

I removed the active gator:* label because there is nothing left for gator to monitor on this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(sandbox): idle openshell-sandbox wakes ~890 times/s (orphan-reaper thread)

2 participants