Skip to content

fix(sandbox): finish starting container recovery - #7879

Closed
sandl99 wants to merge 7 commits into
mainfrom
fix/double-onboard-start-recovery
Closed

fix(sandbox): finish starting container recovery#7879
sandl99 wants to merge 7 commits into
mainfrom
fix/double-onboard-start-recovery

Conversation

@sandl99

@sandl99 sandl99 commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

Sandbox start could stop after Docker reported a restarted container as running with health still starting, before the existing gateway recovery probe could finish the operation.
This change treats only that transient state as recoverable and continues to the gateway and host-forward checks.

Changes

  • Continue the existing start recovery flow when the restarted container is running and its Docker health check is still starting.
  • Carry the parsed Docker runtime and health state as structured recovery metadata so control flow does not depend on diagnostic wording.
  • Preserve immediate failure for stopped, exited, unhealthy, or otherwise unexpected Docker recovery results.
  • Add behavior coverage for both the transient running-and-starting state and terminal readiness failures.
  • Record the escaped-test gap from E2E run 30510639059: the earlier unit coverage did not exercise the exact transient recovery result that the live double-onboard target returned.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: Existing lifecycle and command pages already state that start repairs the agent runtime and host forwards. They do not classify the transient running-and-starting state as terminal.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: PASS with no findings; the exact transient match retains existing registration, driver, runtime, and gateway-probe boundaries.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: no-docs-needed
  • Evidence: Existing sandbox lifecycle and command documentation already describes the corrected start recovery contract. The exact-head review found no documentation gap; the added live E2E assertion preserves the observed terminal diagnostic as literal test evidence.
  • Agent: Codex Desktop

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit: Not applicable; scripts/prepare-dgx-station-host.sh is unchanged.
  • Station profile/scenario: Not applicable.
  • Result: Not applicable.
  • Supporting evidence: Not applicable.

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: the action and Docker recovery suites passed 39 tests after the runtime change; the final test-only diagnostic decoupling rerun passed all 16 action tests.
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: Not run; the focused action test covers the two changed recovery branches.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: San Dang sdang@nvidia.com

Summary by CodeRabbit

  • Bug Fixes
    • Improved sandbox startup recovery when the container is running but health is still “starting”.
    • Terminal container failures now stop recovery cleanly without unnecessary gateway checks.
    • OpenShell readiness retries now tolerate additional diagnostic output.
    • Recovery diagnostics now include container runtime and health status.
  • Tests
    • Added coverage for sandbox recovery, readiness retries, and successful restart behavior.

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
@sandl99 sandl99 self-assigned this Jul 30, 2026
@coderabbitai

coderabbitai Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

startSandbox now continues gateway recovery when Docker is running but health remains starting. Terminal Docker failures still stop gateway probing. Recovery results expose structured readiness states, and OpenShell retries accept appended diagnostics.

Changes

Sandbox recovery

Layer / File(s) Summary
Return structured Docker readiness
src/lib/onboard/docker-driver-sandbox-recovery.ts, src/lib/onboard/docker-driver-sandbox-recovery.test.ts
Recovery polling tracks runtime and health separately and returns the final readiness state for successful and failed recovery paths.
Handle running-but-starting recovery
src/lib/actions/sandbox/start.ts, src/lib/actions/sandbox/start.test.ts, test/e2e/live/double-onboard.test.ts
Startup continues gateway probing for running containers with health=starting. Terminal readiness failures return exit code 1 without probing. The end-to-end test checks restart output.
Broaden OpenShell retry matching
src/lib/actions/sandbox/process-recovery.ts, src/lib/actions/sandbox/process-recovery.test.ts
Transient OpenShell Error-phase responses remain retryable when appended diagnostics follow the expected error text.

Estimated code review effort: 2 (Simple) | ~10 minutes

Sequence Diagram(s)

sequenceDiagram
  participant startSandbox
  participant DockerRecovery
  participant GatewayProbe
  startSandbox->>DockerRecovery: recover sandbox
  DockerRecovery-->>startSandbox: runtime running, health starting
  startSandbox->>GatewayProbe: probe gateway health
  GatewayProbe-->>startSandbox: gateway readiness result
Loading

Possibly related PRs

Suggested labels: area: sandbox, platform: container, bug-fix, area: e2e

Suggested reviewers: apurvvkumaria, cv

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the sandbox container recovery fix, which is the main change in the pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/double-onboard-start-recovery

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 3cd3e78 in the fix/double-onboard-s... branch remains at 96%, unchanged from commit b675ed3 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 3cd3e78 in the fix/double-onboard-s... branch remains at 81%, unchanged from commit b675ed3 in the main branch.

Show a code coverage summary of the most impacted files.
File main b675ed3 fix/double-onboard-s... 3cd3e78 +/-
src/lib/domain/.../connect-env.ts 97% 89% -8%
src/lib/actions...ess-recovery.ts 79% 79% 0%
src/lib/sandbox...rce-identity.ts 88% 88% 0%
src/lib/inferen...ompatibility.ts 94% 95% +1%
src/lib/actions...andbox/start.ts 89% 90% +1%
src/lib/onboard...box-recovery.ts 89% 90% +1%

Updated July 31, 2026 05:11 UTC

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
src/lib/actions/sandbox/start.ts (1)

36-43: 🎯 Functional Correctness | 🔵 Trivial | 🏗️ Heavy lift

Use a typed recovery state instead of matching diagnostic text.

startSandbox now uses one exact human-readable recovery.detail string as a control-flow discriminator. Any wording change in recoverDockerDriverSandbox would silently turn this transient state back into a terminal failure. Add an explicit readiness/status field and keep detail solely for diagnostics.

As per path instructions, actions should orchestrate while pure decisions belong in domain modules; this avoids a brittle cross-layer string contract.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/actions/sandbox/start.ts` around lines 36 - 43, Replace the exact
recovery.detail comparison in isRunningContainerStillStarting with an explicit
typed readiness/status field returned by recoverDockerDriverSandbox. Update the
recovery result and recoverDockerDriverSandbox to set that field for the
running/starting transient state, and have startSandbox use it for control flow
while retaining detail only for diagnostics.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/lib/actions/sandbox/start.ts`:
- Around line 36-43: Replace the exact recovery.detail comparison in
isRunningContainerStillStarting with an explicit typed readiness/status field
returned by recoverDockerDriverSandbox. Update the recovery result and
recoverDockerDriverSandbox to set that field for the running/starting transient
state, and have startSandbox use it for control flow while retaining detail only
for diagnostics.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a0d2eba0-f7f5-45c9-899c-6adae5798a43

📥 Commits

Reviewing files that changed from the base of the PR and between 4dcb89e and 253afb6.

📒 Files selected for processing (2)
  • src/lib/actions/sandbox/start.test.ts
  • src/lib/actions/sandbox/start.ts

@sandl99

sandl99 commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator Author

Security Review

Verdict: PASS. The change is safe to merge from a security perspective. It carries Docker's parsed runtime and health state as internal structured recovery metadata, and only runtime=running plus health=starting continues into the existing gateway and host-forward recovery probe. All other recovery failures remain fail-closed.

No findings.

Category Verdict Reason
Secrets and credentials PASS The diff adds no credentials, secrets, or credential handling.
Input validation and data sanitization PASS The structured values come from the existing bounded Docker inspect result; the change adds no parser or shell boundary.
Authentication and authorization PASS Existing sandbox registration, driver, and Docker-runtime gates remain unchanged.
Dependencies and third-party libraries PASS The diff adds no dependency or version change.
Error handling and logging PASS Terminal recovery results still exit nonzero. Diagnostic text remains logging-only and contains no new sensitive data.
Cryptography and data protection PASS The diff does not change cryptography, storage, transport, or credential persistence.
Configuration and security headers PASS The diff does not change container configuration, permissions, ports, policies, or HTTP behavior.
Security testing PASS Tests prove typed transient continuation independent of diagnostic wording and terminal fail-closed behavior.
System security PASS The change does not bypass readiness; the transient state proceeds to the existing gateway and host-forward probe, whose failure still propagates.

Files reviewed at head 24ec527d6:

  • src/lib/actions/sandbox/start.ts
  • src/lib/actions/sandbox/start.test.ts
  • src/lib/onboard/docker-driver-sandbox-recovery.ts
  • src/lib/onboard/docker-driver-sandbox-recovery.test.ts

Signed-off-by: San Dang <sdang@nvidia.com>
@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: Review the warnings below.
Findings: 0 blockers · 1 warning · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 1 warning · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Model comparison: normalized findings differ; normalized E2E selections differ; Nemotron reported the same number of blockers, 1 fewer warning, the same number of suggestions.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: double-onboard, onboard-repair, onboard-resume, ubuntu-repo-docker-post-reboot-recovery, cloud-onboard

1 warning · 0 suggestions

Warnings

Warnings do not block.

PRA-1 Warning — Document the contract for bypassing the Docker readiness deadline

  • Location: src/lib/actions/sandbox/start.ts:36
  • Category: docs
  • Problem: The start action continues after Docker readiness times out when the container reports running/starting, but the new exception does not document the source-system limitation or the condition for removing it.
  • Impact: A later change can treat a Docker readiness timeout as generally recoverable and invoke gateway recovery for a container that cannot become usable.
  • Recommendation: Document that Docker health can remain starting after the fixed recovery deadline, that the recover-action gateway probe is the authoritative follow-up check, and the upstream readiness behavior that would permit removal of this exception.
  • Verification: Read startSandbox and the default probe implementation in src/lib/actions/sandbox/connect.ts to confirm that a rejected probe returns a failing command result rather than a successful start.
  • Test coverage: Existing coverage: src/lib/actions/sandbox/start.test.ts tests that startSandbox propagates a rejected gateway probe rather than reporting success. No additional regression test is required for this finding.
  • Evidence: src/lib/actions/sandbox/start.ts:36-41 permits only a failed recovery with readiness runtime=running and health=starting. src/lib/actions/sandbox/start.ts:107-125 continues to probe the gateway after that exception. src/lib/onboard/docker-driver-sandbox-recovery.ts:252-258 returns the running/starting state after the Docker readiness deadline. src/lib/actions/sandbox/start.test.ts:251-256 contains "propagates a probe rejection instead of reporting success ([All Platforms][CLI&UX] nemoclaw sandbox has no stop or pause command — only destroy is available #6026)" and expects startSandbox to reject when probeSandbox rejects. F-001 concerns the same running/starting path whose authoritative readiness boundary is probeSandbox.

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

sandl99 added 2 commits July 30, 2026 12:43
Signed-off-by: San Dang <sdang@nvidia.com>
Keep the exact same-sandbox phase-Error refusal retryable when OpenShell appends stderr diagnostics.

The read-only probe remains bounded by the managed-health guard and existing deadline.

Unrelated sandbox and terminal-phase failures remain definitive.

Refs #7227

Signed-off-by: San Dang <sdang@nvidia.com>
@prekshivyas

Copy link
Copy Markdown
Collaborator

Trusted E2E result for exact PR head 64532878500e33c5eb5b6fe6006bbc35bfcf1dbc:

Job Result Duration
onboard-repair passing 4m33s
onboard-resume passing 6m25s
double-onboard failing — fix still needed, not a retryable flake 9m28s job / 8m32s test
Overall trusted E2E failing 10m27s

Evidence: trusted E2E run 30604344756, double-onboard job, artifact. The dispatch receipt selected double-onboard,onboard-repair,onboard-resume and bound them to the SHA above.

double-onboard reached the intended running but still starting; continuing with gateway recovery path, then nemoclaw e2e-double-b start exited 1 because the recreated sandbox failed the managed health guard, so the primary dashboard/API host forward was not started. Setup, checkout, Docker authentication, artifact upload, and cleanup all passed. The controller correctly skipped its hosted-runner retry.

Conclusion: this is product-path evidence, not runner/cloud flake evidence. Do not merge yet; update the recovery implementation and rerun the trusted E2E on the new exact head.

@sandl99 sandl99 closed this Jul 31, 2026
@prekshivyas

Copy link
Copy Markdown
Collaborator

Trusted E2E follow-up for exact PR head 3cd3e78ab0df69d02cc420b3d42f609c30626bdd against base b675ed361a02d2d602900e1f301ba117af7effaf:

Job Result Duration
onboard-repair passing 6m12s
ubuntu-repo-docker-post-reboot-recovery failing — code-path failure, not retryable flake 4m56s job / ~4m10s test
onboard-resume cancelled after PR closure; no verdict 7m05s
double-onboard cancelled after PR closure; no verdict 7m13s
Overall trusted E2E child cancelled after PR closure 8m08s

Evidence: controller 30605590819, trusted E2E child 30606139059, failed recovery job, artifact. The job checkout and NEMOCLAW_E2E_EXPECTED_SHA both confirm the exact PR head above.

The recovery target onboarded successfully, stopped the labeled sandbox container to simulate the reboot boundary, restarted the OpenShell gateway service, and then failed the required user-visible status proof with Failure layer: sandbox_container_stopped — sandbox container exists but is not running. The artifact confirms docker stop succeeded and the subsequent nemoclaw ... status exited 1. This is product-path evidence; do not retry it as infrastructure noise.

PR #7879 was closed unmerged by its author at 05:19:23 UTC while the run was active, which caused the remaining two jobs and report job to be cancelled. Therefore this run does not establish that double-onboard is fixed on the new head.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants