feat(onboard): bind managed create to runtime providers

[ericksoa]

Summary

Bind managed sandbox creation to the pluggable runtime-provider bundle without activating buildless support in production. The candidate Docker surface owns create lifecycle and routing, while central onboarding accepts the same contract from an MXC-style provider and keeps every current production bundle bootstrap-unsupported.

This slice also makes the existing Docker compatibility path retain rollback authority through readiness, GPU-proof, and applicable local-inference gates. A failed late rollback is propagated and reported instead of being treated as success.

Related Issue

Part of #7744.

Changes

• Add one provider-neutral managed-create contract covering create lifecycle, routing, runtime patches, authority storage, and runtime snapshots.
• Add an inert Docker candidate surface that composes the transactional bootstrap adapter; central orchestration does not select Docker, Podman, or MXC by switch.
• Carry identity-bound held startup through OpenClaw, Hermes, and DCode launch construction without putting encoded profiles in sandbox argv.
• Make GPU creation and host-local inference use the provider transaction, including exact rollback before fallback/failure and commit only after applicable readiness and inference gates.
• Preserve the pre-patch container through late qualification, propagate rolledBack: false, and emit manual-cleanup guidance only when exact rollback does not complete.
• Protect the abstraction with registry, MXC-style provider, all-agent launch, GPU lifecycle, local-inference, rollback-failure, and compatibility-path tests.
• Update inference, commands, troubleshooting, and internal architecture documentation to match the exact rollback behavior.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification:
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Exact-tree local review covers provider neutrality, immutable rollback authority, all-agent held startup, late rollback outcome propagation, GPU/local-inference commit gates, and production dormancy. Public exact-head advisors and protected E2E remain required before merge.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: docs-updated
• Evidence: Clean candidate 7ed953c7d0934b4c8781b125a4ce57757591781d and append-only graft 48526666b6b975d6e6173f7f2bc7a76ab0f46361 share exact tree 048dc1f0dd8e0dbbe8de6511bbac75b18828dc61. Independent exact-diff review covered provider neutrality, the MXC-style path, production dormancy, rollback wording, and the four changed documentation files. Final signed+DCO qualification child 34b43764dd37885b88d1b931861cdf6de92b508b changes only the stale local-NIM docs contract assertion, so every reviewed documentation blob remains unchanged; the exact assertion now passes 23/23.
• Agent: Codex Desktop

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit:
• Station profile/scenario:
• Result:
• Supporting evidence:

Verification

• Exact base/head: c63e7eb5ec07448e26a6cd8617a24601216bec35 / 34b43764dd37885b88d1b931861cdf6de92b508b.
• Review budget: 30 paths, +1,842/-313.
• Stable patch ID: dad9c0a58cb8b696a0a29c787ff116a0438190b0.
• Binary diff SHA-256: 91d6f4f300ace2ce4f7110e397b09f9f153a936bf1cde28140219f857e2c4f2e.
• Clean exact-base reconstruction: signed+DCO commit 7ed953c7d0934b4c8781b125a4ce57757591781d, direct child of merged feat(onboard): add transactional Docker bootstrap adapter #8128 main c63e7eb5ec07448e26a6cd8617a24601216bec35; graft tree equality is exact at 048dc1f0dd8e0dbbe8de6511bbac75b18828dc61. Final test-only qualification child 34b43764dd37885b88d1b931861cdf6de92b508b has tree cc3b68897023975c70225ddde2559a3d44357e5d.
• Append-only ancestry: signed+DCO graft 48526666b6b975d6e6173f7f2bc7a76ab0f46361 has preserved prior live head c14be060337d19318be9324e883cf4ddf5040d72 as first parent and clean reconstruction 7ed953c7d0934b4c8781b125a4ce57757591781d as second parent. Signed+DCO child 34b43764dd37885b88d1b931861cdf6de92b508b is a normal fast-forward. No remote history is rewritten.
• Durable refs: backup/podman-stack/pr8040-live-head-c14be0603, backup/podman-stack/pr8040-clean-7ed953c7d, backup/podman-stack/pr8040-graft-48526666b, backup/podman-stack/pr8040-final-34b43764d, and hold/podman-stack/pr8041-base-34b43764d.
• Exact c63-based focused behavior suite passed 164/164: 141 CLI tests and 23 integration tests. The hardened central-source boundary passed 8/8. Final docs contract test passed 23/23 after updating its stale local-NIM wording assertion.
• CLI/plugin builds, all three typecheck surfaces, Biome, repository architecture, source-shape, test-size, conditional scan, documentation preparation and route checks, diff checks, gitleaks, markdownlint, and the complete commit-time pre-commit suite passed.
• Documentation validation passed across OpenClaw, Hermes, and DCode variants.
• Production provider construction still returns unsupported bootstrap/recovery surfaces; no selectable runtime or buildless activation is added.

Risk Plan

• Primary risk: central orchestration acquiring a runtime-specific branch, or committing compatibility-path changes before late readiness/GPU/inference qualification and losing exact rollback authority.
• Containment: one injected provider bundle owns managed create; MXC-style tests forbid Docker-specific central routing; commit is deferred through applicable gates; unsuccessful exact rollback is propagated and reported.
• Regression risk: the compatibility path changes failure/recovery timing. Focused lifecycle and local-inference tests cover successful commit, pre-commit failure, exact rollback, rollback failure, and manual-cleanup reporting.
• Rollback: the managed-create candidate remains unregistered; the compatibility-path change can be reverted independently if qualification exposes a regression.

Stack

• Base: merged PR3.10B2 feat(onboard): add transactional Docker bootstrap adapter #8128 at main commit c63e7eb5ec07448e26a6cd8617a24601216bec35.
• This slice: PR3.11 feat(onboard): bind managed create to runtime providers #8040, branch feat/managed-bootstrap-provider-create, exact head 34b43764dd37885b88d1b931861cdf6de92b508b.
• Next: PR3.12A feat(onboard): persist managed bootstrap transactions #8041 on protected handoff hold/podman-stack/pr8041-base-34b43764d.
• Buildless and Podman support remain disabled until the complete all-agent, multiarch, GPU/local-inference, recovery, installer, and protected-E2E activation gates in [Epic] Support native Podman with buildless managed onboarding #7744 pass.

Signed-off-by: Aaron Erickson aerickson@nvidia.com

Summary by CodeRabbit

• New Features

  • Added managed startup and provider-driven sandbox creation support.
  • Added Docker GPU onboarding with native and compatibility routing.
  • Added durable sandbox identity resolution for OpenShell environments.

• Bug Fixes

  • GPU verification now completes before changes are finalized.
  • Failed readiness, GPU, or inference checks attempt rollback and report recovery issues.
  • Improved diagnostics and error propagation during sandbox recreation.

• Documentation

  • Expanded GPU rollback, diagnostics, cleanup, and local NVIDIA NIM guidance.

[ericksoa]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

♻️ Duplicate comments (1)

src/lib/onboard/docker-gpu-sandbox-create.ts (1)

376-461: 🗄️ Data Integrity & Integration | 🔴 Critical | ⚡ Quick win

commitAfterReady can resolve successfully after a failed commit or rollback.

Three branches in commitAfterReady report a failure only through onPatchFailureExit and then return (or fall off the end of the async IIFE) without throwing:

• Line 378-395 (needsSupervisorWait branch): calls onPatchFailureExit(...), then return;.
• Line 404-433 (managedBootstrapCutover.commit() catch branch): calls onPatchFailureExit(...), then return;.
• Line 439-449 (backup-removal-failure branch): calls onPatchFailureExit(...) with no following throw or return.

onPatchFailureExit defaults to printDockerGpuPatchFailureAndExit, which calls process.exit(1) and never returns. A managed-bootstrap override of onPatchFailureExit can return normally instead of exiting. When that happens, await patch.commitAfterReady() resolves without error even though the commit or rollback failed, and any caller (including a commit-after-verify wrapper) proceeds as if onboarding succeeded.

Compare this to verifyGpuOrExit (lines 505-512, 531-538), which always throws the original failure directly, independent of what onPatchFailureExit does. commitAfterReady should follow the same pattern on all three branches.

A prior review on an earlier commit of this file flagged exactly this defect for the commit-catch and backup-removal branches, and the thread states it was fixed in a descendant PR (#8077). That fix is not reflected in this file as shown here — the same missing-throw pattern is still present, and the needsSupervisorWait branch has the identical defect but was not covered by that earlier discussion. Confirm whether this PR should carry the fix directly, or whether it intentionally waits on the descendant PR; either way, the pattern remains exploitable in the code under review right now.

🐛 Proposed fix: throw after reporting on all three branches

       if (needsSupervisorWait) {
         const error = new Error(
           "Managed startup cannot commit before the recreated OpenShell supervisor reconnects.",
         );
         const rollbackError = await rollbackAfterFailure();
+        const failure = rollbackError
+          ? new Error(`${error.message} Rollback failed: ${rollbackError.message}`)
+          : error;
         onPatchFailureExit(
           options.sandboxName,
-          rollbackError
-            ? new Error(`${error.message} Rollback failed: ${rollbackError.message}`)
-            : error,
+          failure,
           {
             runCaptureOpenshell: options.deps.runCaptureOpenshell,
             dockerCapture: options.deps.dockerCapture,
             additionalSummaryLines: routeAdapter.additionalSummaryLines,
           },
         );
-        return;
+        throw failure;
       }
       ...
             onPatchFailureExit(options.sandboxName, failure, {
               runCaptureOpenshell: options.deps.runCaptureOpenshell,
               dockerCapture: options.deps.dockerCapture,
               additionalSummaryLines: routeAdapter.additionalSummaryLines,
               context: {
                 ...failureContext(),
                 rolledBack: rollbackError === null,
               },
             });
-            return;
+            throw failure;
           }
         }
         const finalizeOutcome = result
           ? finalizeBackup({ result, supervisorReady: true }, options.deps)
           : null;
         cutoverFinalized = true;
         if (!finalizeOutcome || finalizeOutcome.backupRemoved) return;
+        const backupFailure = new Error(
+          "Managed startup passed Ready, but its rollback backup could not be removed.",
+        );
         onPatchFailureExit(
           options.sandboxName,
-          new Error("Managed startup passed Ready, but its rollback backup could not be removed."),
+          backupFailure,
           {
             runCaptureOpenshell: options.deps.runCaptureOpenshell,
             dockerCapture: options.deps.dockerCapture,
             additionalSummaryLines: routeAdapter.additionalSummaryLines,
             context: failureContext(),
           },
         );
+        throw backupFailure;

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/onboard/docker-gpu-sandbox-create.ts` around lines 376 - 461, Update
commitAfterReady so each failure path throws its original failure after calling
onPatchFailureExit: the needsSupervisorWait branch should throw error, the
managedBootstrapCutover.commit catch should rethrow failure, and the
backup-removal failure branch should throw the constructed Error. Preserve the
existing reporting and rollback behavior while ensuring the method rejects even
when onPatchFailureExit returns normally.

🧹 Nitpick comments (1)

src/lib/onboard/docker-gpu-sandbox-create-lifecycle.test.ts (1)

87-167: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add coverage for commitAfterReady rejecting on a failed commit or rollback.

The shown tests exercise rollbackManagedStartupAfterCreateFailure and verifyGpuOrExit, but no test calls commitAfterReady() while needsSupervisorWait is still true, or while a managedBootstrapCutover.commit() mock rejects, to confirm the returned promise rejects. Add such a test once the fix on src/lib/onboard/docker-gpu-sandbox-create.ts (lines 376-461) lands, so a regression to the silent-return pattern is caught.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/onboard/docker-gpu-sandbox-create-lifecycle.test.ts` around lines 87
- 167, Add test coverage for
createDockerGpuSandboxCreatePatch.commitAfterReady() that keeps
needsSupervisorWait true and verifies the promise rejects when the
managedBootstrapCutover.commit() mock rejects. Assert the rejection error and
preserve the existing successful commit and rollback failure coverage.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Duplicate comments:
In `@src/lib/onboard/docker-gpu-sandbox-create.ts`:
- Around line 376-461: Update commitAfterReady so each failure path throws its
original failure after calling onPatchFailureExit: the needsSupervisorWait
branch should throw error, the managedBootstrapCutover.commit catch should
rethrow failure, and the backup-removal failure branch should throw the
constructed Error. Preserve the existing reporting and rollback behavior while
ensuring the method rejects even when onPatchFailureExit returns normally.

---

Nitpick comments:
In `@src/lib/onboard/docker-gpu-sandbox-create-lifecycle.test.ts`:
- Around line 87-167: Add test coverage for
createDockerGpuSandboxCreatePatch.commitAfterReady() that keeps
needsSupervisorWait true and verifies the promise rejects when the
managedBootstrapCutover.commit() mock rejects. Assert the rejection error and
preserve the existing successful commit and rollback failure coverage.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 9a2057f1-34eb-4dfd-afc1-e18ddf79700a

📥 Commits

Reviewing files that changed from the base of the PR and between 982be3a and c14be06.

📒 Files selected for processing (14)

• ci/source-shape-test-budget.json
• docs/inference/verify-inference-route.mdx
• docs/reference/commands.mdx
• docs/reference/troubleshooting.mdx
• src/lib/onboard.ts
• src/lib/onboard/docker-gpu-local-inference.test.ts
• src/lib/onboard/docker-gpu-local-inference.ts
• src/lib/onboard/docker-gpu-sandbox-create-lifecycle.test.ts
• src/lib/onboard/docker-gpu-sandbox-create.ts
• src/lib/onboard/managed-bootstrap/README.md
• src/lib/onboard/managed-bootstrap/runtime-create.ts
• src/lib/onboard/sandbox-gpu-create-flow.ts
• src/lib/onboard/sandbox-gpu-create-run-attempt.ts
• test/runtime-provider-source-shape.test.ts

💤 Files with no reviewable changes (1)

• src/lib/onboard/managed-bootstrap/runtime-create.ts

🚧 Files skipped from review as they are similar to previous changes (6)

• src/lib/onboard.ts
• src/lib/onboard/docker-gpu-local-inference.test.ts
• test/runtime-provider-source-shape.test.ts
• src/lib/onboard/managed-bootstrap/README.md
• src/lib/onboard/sandbox-gpu-create-flow.ts
• src/lib/onboard/sandbox-gpu-create-run-attempt.ts

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-8040.docs.buildwithfern.com/nemoclaw

[ericksoa]

Maintainer disposition for advisor PRA-1 on exact head 34b4376: this is a false positive; no code change is appropriate. Production Local NVIDIA NIM selection sets state.provider to vllm-local in src/lib/onboard.ts:3121, and that exact state value is passed to verifyGpuSandboxLocalInferenceAndCommitAfterReady at src/lib/onboard.ts:2767-2769. vllm-local is already included in LOCAL_INFERENCE_PROVIDERS, so Local NIM receives the HTTP 2xx post-ready gate and rollback behavior. nvidia-nim is the remote NVIDIA Endpoints identity; adding it to the local-provider set would incorrectly apply host-local probing to a cloud provider. Existing setup and recovery tests explicitly assert the Local NIM vllm-local identity and distinguish the legacy/cloud nvidia-nim alias. The finding is therefore evidence-dispositioned as non-applicable. The separate reviewed-npm-audit failure remains external to this patch.