Skip to content

fix(rebuild): remove obsolete sandbox images - #8039

Merged
senthilr-nv merged 9 commits into
mainfrom
codex/fix-rebuild-image-cleanup
Aug 1, 2026
Merged

fix(rebuild): remove obsolete sandbox images#8039
senthilr-nv merged 9 commits into
mainfrom
codex/fix-rebuild-image-cleanup

Conversation

@senthilr-nv

@senthilr-nv senthilr-nv commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

Rebuild now removes the obsolete owned sandbox image only after the same-name replacement is registered and its exact journaled generation and live identity are proven. Interrupted resume preserves shared-image ownership state, so cleanup retains shared images. This restores the documented cleanup behavior without weakening the recreate journal added by #7788.

Changes

  • Capture the source registry entry before journaled same-name replacement.
  • Persist the minimal secret-free source-workload cleanup receipt so interrupted replacements resume cleanup against the original image.
  • Preserve the source workload's sharing state through checkpoint validation and resume cleanup.
  • Reload the same recreate transaction after replacement creation and bind cleanup to its exact recorded live identity.
  • Record post-create transaction and cleanup failures as repair failures while preserving the original error.
  • Remove only a provider-owned obsolete image after the replacement generation and live identity are proven.
  • Retain shared images, reused images, foreign generations, and workloads whose ownership cannot be proven.
  • Use the active launcher name in cleanup guidance, including nemohermes gc.
  • Add lifecycle-boundary regressions for removal, reuse, sharing, missing identity, foreign generation, and journal ordering.

Product scope: this restores the existing supported and documented rebuild image-cleanup contract; it does not add a new integration, configuration, or product surface.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: Existing docs/reference/commands.mdx already documents automatic rebuild image cleanup and the gc recovery path; this change restores that contract.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Final exact-head nine-category review PASS: fix(rebuild): remove obsolete sandbox images #8039 (comment)
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: no-docs-needed
  • Evidence: Existing docs/manage-sandboxes/recover-rebuild-sandboxes.mdx:172-195 already requires matching live identity and registry generation and fails closed on identity mismatch; docs/reference/commands.mdx:284-287 routes interrupted replacements to that contract; exact-head writer review found no command, configuration, output, or workflow change; focused tests passed 98/98, and git diff --check passed.
  • Agent: Codex Desktop

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit: Not applicable
  • Station profile/scenario: Not applicable
  • Result: Not applicable
  • Supporting evidence: Not applicable

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: Focused tests passed 98/98 at the exact final tree; regressions cover shared-state loss, same-generation replacement identity mismatch, and post-create cleanup failure recording; independent exact-head writer review passed.
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: Not a broad runtime or harness change; npm run validate:pr passed repository checks, CLI type-checking, commitlint, secret scanning, and all applicable hooks.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Senthil Ravichandran senthilr@nvidia.com

Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
@coderabbitai

coderabbitai Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Sandbox recreation now checkpoints the source workload, creates the replacement, and retires the replaced workload. Cleanup validates ownership, generation, identity, provider authority, and retention rules.

Changes

Sandbox recreation cleanup

Layer / File(s) Summary
Source workload checkpoint contract
src/lib/state/onboard-checkpoint-types.ts, src/lib/state/onboard-checkpoint.ts, src/lib/onboard/sandbox-recreate-transaction.ts, related tests
Recreate journals store and validate source workload metadata. Older journals remain compatible.
Provider-aware workload retirement
src/lib/onboard/sandbox-recreate-transaction.ts, src/lib/onboard/runtime-provider/replaced-workload.test.ts
retireReplacedSandboxWorkload removes only an owned source workload with a proven replacement. Tests cover reuse, sharing, retention, authority, and provider errors.
Source entry propagation and retirement
src/lib/onboard/machine/handlers/sandbox.ts
The recreation flow captures and propagates the source registry entry. It retires the source after replacement creation and reports cleanup status.
Dependency wiring and recreate-flow validation
src/lib/onboard.ts, src/lib/onboard/machine/handlers/sandbox-test-fixtures.ts, src/lib/onboard/machine/core-flow-phases.test.ts, src/lib/onboard/machine/handlers/sandbox-recreate-journal.test.ts
Wires cliName and retirement dependencies. Tests verify creation-before-retirement ordering, interrupted retirement retry, and journal cleanup.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant SandboxRecreateFlow
  participant SandboxRegistry
  participant RecreationJournal
  participant RuntimeProvider
  SandboxRecreateFlow->>SandboxRegistry: read source entry
  SandboxRecreateFlow->>RecreationJournal: prepare recreation with source entry
  RecreationJournal-->>SandboxRecreateFlow: preparation result
  SandboxRecreateFlow->>SandboxRegistry: create replacement
  SandboxRecreateFlow->>RuntimeProvider: retire replaced workload
  RuntimeProvider-->>SandboxRecreateFlow: cleanup result
Loading

Possibly related PRs

Suggested labels: platform: container

Suggested reviewers: apurvvkumaria, cv

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 7.69% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: removing obsolete sandbox images during rebuilds.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/fix-rebuild-image-cleanup

Comment @coderabbitai help to get the list of available commands.

@senthilr-nv

Copy link
Copy Markdown
Collaborator Author

Sensitive-path security review — PASS at 8c6c17ac

  1. Authentication and authorization — PASS. No authentication surface changes; destructive cleanup requires the existing runtime-provider ownership authority.
  2. Secrets and credentials — PASS. No credential values, environment secrets, tokens, or new persistence paths are introduced.
  3. Input validation and injection — PASS. Cleanup accepts registry-owned structured entries; it adds no shell construction, command interpolation, or user-controlled evaluator path.
  4. Workload ownership boundary — PASS. The source name must match, provider ownership must be proven, and an unproven or foreign source is retained.
  5. Destructive-action safety — PASS. Removal occurs only after the replacement generation and live identity are registered. Shared and currently reused images are retained.
  6. State consistency and race safety — PASS. Cleanup runs inside the existing sandbox mutation flow after replacement registration and before journal finalization; tests assert lifecycle ordering.
  7. Network and SSRF exposure — PASS. No URL handling, egress policy, listener, or network request behavior changes.
  8. Dependency and supply-chain risk — PASS. No dependencies, images, workflow permissions, or build inputs change.
  9. Logging and information disclosure — PASS. Messages contain only the runtime display name and image reference already present in local registry state; remediation uses the active CLI launcher and contains no secret material.

Regression evidence covers obsolete owned-image removal, replacement image reuse, shared workloads, missing replacement identity, foreign replacement generation, exact lifecycle ordering, and active-launcher remediation. npm run validate:pr and the focused 208-test set passed on the final change set.

@github-code-quality

github-code-quality Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 7ab0927 in the codex/fix-rebuild-im... branch remains at 96%, unchanged from commit 9bc461f in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 7ab0927 in the codex/fix-rebuild-im... branch remains at 81%, unchanged from commit 9bc461f in the main branch.

Show a code coverage summary of the most impacted files.
File main 9bc461f codex/fix-rebuild-im... 7ab0927 +/-
src/lib/messagi...flict-status.ts 100% 84% -16%
src/lib/onboard...age/contract.ts 95% 85% -10%
src/lib/messagi...ort-conflict.ts 92% 83% -9%
src/lib/messagi...tatus-health.ts 100% 91% -9%
src/lib/messagi.../ilink-login.ts 96% 89% -7%
src/lib/trace.ts 94% 90% -4%
src/lib/credentials/store.ts 62% 61% -1%
src/lib/onboard...lers/sandbox.ts 96% 97% +1%
src/lib/state/r...try/workload.ts 88% 98% +10%
src/lib/core/pr...mpt-activity.ts 67% 92% +25%

Updated August 01, 2026 04:58 UTC

@senthilr-nv senthilr-nv self-assigned this Aug 1, 2026
Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
@senthilr-nv senthilr-nv added bug-fix PR fixes a bug or regression area: onboarding Onboarding FSM, provider setup, sandbox launch, or first-run flow area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery security labels Aug 1, 2026
@senthilr-nv

Copy link
Copy Markdown
Collaborator Author

Exact-head sensitive-path re-review — PASS at afa28c26.

The only delta from the full nine-category PASS review at 8c6c17ac is behavior-neutral dependency wiring: the same note and cliName values are supplied through ...{ note, cliName } to keep src/lib/onboard.ts net-neutral. It adds no trust boundary, mutation, input, credential, network, dependency, logging, or cleanup behavior. The original review conclusions remain unchanged. Exact-head focused tests passed 7/7, CLI type-checking passed, and npm run validate:pr passed.

@senthilr-nv
senthilr-nv requested review from apurvvkumaria and cv August 1, 2026 03:16

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/lib/onboard/machine/handlers/sandbox.ts (1)

1283-1391: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Preserve the original source workload across resume. If createSandbox succeeds before retirement and the process exits, the next --resume reads the replacement into sourceEntry. Retirement then receives the replacement as both source and replacement, returns image-reused, and leaks the original image. Persist the source workload in the journal and add interruption/resume coverage.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/onboard/machine/handlers/sandbox.ts` around lines 1283 - 1391,
Preserve the original source workload across interruptions by storing it in the
recreate journal created by beginSandboxRecreateJournal and restoring it during
resume instead of deriving sourceEntry from the current registry. Ensure
retireSandboxRecreateSourceWorkload receives the persisted original source and
the newly created replacement, preventing image-reused from masking the leaked
original image. Add interruption/resume coverage for creation succeeding before
retirement.
🧹 Nitpick comments (2)
src/lib/onboard/sandbox-recreate-transaction.ts (1)

59-64: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Narrow the bare catch to the expected error type.

requireRuntimeProviderDestructiveCleanupAuthority throws RuntimeProviderSelectionError for known unauthorized states. The current catch {} swallows every error, including unexpected bugs (for example a TypeError from a broken provider lookup). Cleanup then silently reports "authority-unproven" with no note to the caller, unlike the "failed" path, which does warn the user. Catch only RuntimeProviderSelectionError and let unexpected errors propagate, or log the swallowed error before returning "skipped".

🛠️ Proposed narrower catch
   let authority;
   try {
     authority = requireRuntimeProviderDestructiveCleanupAuthority(sandboxName, source, providers);
-  } catch {
-    return { status: "skipped", reason: "authority-unproven" };
+  } catch (error) {
+    if (!(error instanceof RuntimeProviderSelectionError)) throw error;
+    return { status: "skipped", reason: "authority-unproven" };
   }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/onboard/sandbox-recreate-transaction.ts` around lines 59 - 64, Update
the catch around requireRuntimeProviderDestructiveCleanupAuthority in the
sandbox recreation flow to handle only RuntimeProviderSelectionError and return
the existing "skipped"/"authority-unproven" result for that expected case. Let
unexpected errors, such as TypeError, propagate instead of silently swallowing
them.
src/lib/onboard/machine/core-flow-phases.test.ts (1)

190-288: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add an explicit retireReplacedSandboxWorkload stub to this fixture.

This sandbox deps object does not set retireReplacedSandboxWorkload. Because that dependency is optional, sandbox.ts falls back to the real retireReplacedSandboxWorkloadDefault, which resolves the real Docker/Kubernetes runtime-provider bundles. Today no scenario in this file appears to reach that call, since beginSandboxRecreateJournal requires resume: true or an existing transaction. Add an explicit stub now so a future test exercising the recreate/resume path cannot silently start executing real runtime-provider cleanup logic instead of a test double.

🧪 Proposed fixture stub
       note: vi.fn(),
       cliName: () => "nemoclaw",
+      retireReplacedSandboxWorkload: vi.fn(() => ({
+        status: "skipped" as const,
+        reason: "replacement-unproven" as const,
+      })),
       updateSession: vi.fn((mutator) => mutator(createSession()) ?? createSession()),

As per coding guidelines, **/*.test.{js,ts}: "Mock external dependencies and do not call real NVIDIA APIs from unit tests."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/onboard/machine/core-flow-phases.test.ts` around lines 190 - 288, Add
an explicit test-double implementation for retireReplacedSandboxWorkload in the
sandbox deps fixture, alongside the other sandbox dependency stubs, so
recreate/resume tests cannot fall through to
retireReplacedSandboxWorkloadDefault or invoke real runtime-provider cleanup.
Keep the stub inert and consistent with the fixture’s existing vi.fn-based
mocks.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/lib/onboard/machine/handlers/sandbox.ts`:
- Around line 1283-1391: Preserve the original source workload across
interruptions by storing it in the recreate journal created by
beginSandboxRecreateJournal and restoring it during resume instead of deriving
sourceEntry from the current registry. Ensure
retireSandboxRecreateSourceWorkload receives the persisted original source and
the newly created replacement, preventing image-reused from masking the leaked
original image. Add interruption/resume coverage for creation succeeding before
retirement.

---

Nitpick comments:
In `@src/lib/onboard/machine/core-flow-phases.test.ts`:
- Around line 190-288: Add an explicit test-double implementation for
retireReplacedSandboxWorkload in the sandbox deps fixture, alongside the other
sandbox dependency stubs, so recreate/resume tests cannot fall through to
retireReplacedSandboxWorkloadDefault or invoke real runtime-provider cleanup.
Keep the stub inert and consistent with the fixture’s existing vi.fn-based
mocks.

In `@src/lib/onboard/sandbox-recreate-transaction.ts`:
- Around line 59-64: Update the catch around
requireRuntimeProviderDestructiveCleanupAuthority in the sandbox recreation flow
to handle only RuntimeProviderSelectionError and return the existing
"skipped"/"authority-unproven" result for that expected case. Let unexpected
errors, such as TypeError, propagate instead of silently swallowing them.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1999d6c3-c65b-4933-8eb9-91fa75fb76b6

📥 Commits

Reviewing files that changed from the base of the PR and between 227e8b9 and 8c6c17a.

📒 Files selected for processing (7)
  • src/lib/onboard.ts
  • src/lib/onboard/machine/core-flow-phases.test.ts
  • src/lib/onboard/machine/handlers/sandbox-recreate-journal.test.ts
  • src/lib/onboard/machine/handlers/sandbox-test-fixtures.ts
  • src/lib/onboard/machine/handlers/sandbox.ts
  • src/lib/onboard/runtime-provider/replaced-workload.test.ts
  • src/lib/onboard/sandbox-recreate-transaction.ts

@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Model comparison: normalized findings match; normalized E2E selections differ; severity counts match.
3 additional E2E selections from the second opinion

Advisory only. The primary lane did not select these E2E jobs or targets.

  • rebuild-openclaw: The completed second-opinion lane identified E2E coverage that the primary lane omitted.
  • rebuild-hermes: The completed second-opinion lane identified E2E coverage that the primary lane omitted.
  • sandbox-survival: The completed second-opinion lane identified E2E coverage that the primary lane omitted.

Second-opinion E2E selections are advisory. They do not change the primary assessment or E2E / PR Gate.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, onboard-repair, onboard-resume

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
@senthilr-nv

Copy link
Copy Markdown
Collaborator Author

Exact-head sensitive-path security review — PASS at ab9ecc51

Verdict

The review found no security findings. The new recreate-journal payload is a minimal, secret-free ownership receipt. It remains subordinate to the existing provider, generation, replacement-identity, shared-image, and image-reuse checks before any image removal.

Detailed analysis

  1. Secrets and credentials — PASS. The journal stores only bounded runtime driver, image reference, and legacy workload reference fields. Tests prove provider, model, and credential environment data are not copied.
  2. Input validation and sanitization — PASS. Persisted fields are parsed from untrusted checkpoint JSON with type, length, control-character, provider-name, and exact reference/image matching checks. No shell construction or evaluator is added.
  3. Authentication and authorization — PASS. Cleanup still requires the registered runtime provider’s destructive cleanup and workload ownership authority.
  4. Dependencies — PASS. No dependency, image, registry, or workflow permission changes.
  5. Error handling and logging — PASS. Only RuntimeProviderSelectionError becomes an authority-unproven skip; unexpected failures propagate. Messages contain no secret material.
  6. Cryptography and data protection — PASS. No cryptographic behavior changes. Existing SHA-256 identity and generation bindings remain intact.
  7. Configuration and security headers — PASS. No runtime configuration, listener, header, container privilege, or policy changes.
  8. Security testing — PASS. Tests cover malformed receipt rejection, missing legacy receipt compatibility, expected versus unexpected provider failures, shared/reused/foreign replacement protection, and interruption/resume cleanup.
  9. System security — PASS. Cleanup occurs only after the exact replacement generation and live identity are registered. The source receipt survives interruption, while shared, reused, foreign, and unproven workloads remain protected from deletion.

Files reviewed: all 10 files changed by ab9ecc51, plus the existing provider cleanup authority and Docker cleanup-plan boundaries they invoke. Exact-head focused tests passed 95/95, independent writer validation passed 70/70, and npm run validate:pr passed.

Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
@apurvvkumaria
apurvvkumaria enabled auto-merge (squash) August 1, 2026 03:49
@senthilr-nv

Copy link
Copy Markdown
Collaborator Author

Exact-head security review refresh

Verdict: PASS at 307618a10bb2b49ce5084980582fffb3a4e135fe; no security findings and safe to merge from this review's scope.

This refresh preserves the previously reviewed PR patches: the merge commit only incorporates merged main changes from #8033 and #8038, the final commit only makes the resume test fixture linear, and the exact committed tree is byte-identical to the independently reviewed rebased head. The prior full nine-category review remains applicable.

  1. Secrets and credentials — PASS: the persisted cleanup receipt is deliberately minimal and secret-free; no credentials or secret-bearing fields are introduced.
  2. Input validation and data sanitization — PASS: restored receipt fields are parsed and validated before use; no new command, path, URL, or evaluation surface is introduced.
  3. Authentication and authorization — PASS: destructive cleanup still requires provider ownership and cleanup authority; unproven authority remains fail-closed.
  4. Dependencies and third-party libraries — PASS: no dependency or registry-source changes.
  5. Error handling and logging — PASS: only the expected RuntimeProviderSelectionError maps to authority-unproven; unexpected errors propagate, and logs do not expose secrets.
  6. Cryptography and data protection — PASS: no cryptographic or sensitive-data transport/storage changes.
  7. Configuration and security headers — PASS: no service, container, port, origin, header, or privilege configuration changes.
  8. Security testing — PASS: lifecycle tests cover removal denial for reused, shared, foreign-generation, missing-identity, and unproven-ownership cases, plus interrupted-resume cleanup.
  9. System security — PASS: cleanup is least-privilege and gated on proven replacement identity and ownership; the journal preserves the original source across interruption, closing the resume/TOCTOU gap without weakening recreate recovery.

Validation at this exact tree: focused tests 95/95 passed; npm run validate:pr passed; DCO passed; both refresh commits are GitHub Verified; independent documentation review passed with no-docs-needed.

Prior full review: #8039 (comment)

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
src/lib/onboard/machine/handlers/sandbox-recreate-journal.test.ts (2)

244-262: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Assert the retirement generation against a captured value, not the mutable fixture.

replacementEntry is reassigned inside the createSandbox mock. Line 254 and line 261 then read that same mutated object to build the expectation. The generation claim becomes partly self-referential: the expected value comes from the fixture the handler consumed.

Capture the journaled targetGeneration after the first run and assert against it. The test then proves that retirement uses the journaled replacement generation.

♻️ Suggested assertion change
   await expect(handleSandboxState(options)).rejects.toThrow(
     /interrupted after replacement registration/u,
   );
   expect(session.checkpoint?.sandboxRecreate?.sourceWorkload?.imageTag).toBe(sourceEntry.imageTag);
+  const journaledGeneration = session.checkpoint?.sandboxRecreate?.targetGeneration;
+  expect(journaledGeneration).toBeTruthy();
 
   await handleSandboxState(options);
 
   expect(retireReplacedSandboxWorkload).toHaveBeenNthCalledWith(
     2,
     "saved",
-    replacementEntry.lifecycleGeneration,
+    journaledGeneration,
     expect.objectContaining({

As per path instructions: "Prefer observable outcomes through the public boundary over source-text, private-shape, or mock-call assertions."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/onboard/machine/handlers/sandbox-recreate-journal.test.ts` around
lines 244 - 262, Update the test around handleSandboxState to capture the
journaled replacement targetGeneration after the first rejected run, before the
createSandbox mock mutates replacementEntry. Use that captured value for the
subsequent retirement-generation assertion, including the expected replacement
entry where needed, so the test validates the journaled generation rather than
the mutable fixture.

Source: Path instructions


143-223: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Assert the retry enters createSandbox. Add assertions that the mock is called twice and that the second call carries the same transaction ID and target generation. The registered replacement must make createSandboxRecreateRuntime accept the target without repeating deletion or replacement registration.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/onboard/machine/handlers/sandbox-recreate-journal.test.ts` around
lines 143 - 223, The test must verify that retrying after replacement
registration re-enters createSandbox: assert createSandbox is called twice, and
inspect the second call to confirm it uses the original recreate transaction ID
and target generation. Ensure the registered replacement state allows
createSandboxRecreateRuntime to accept the target on retry without invoking
deletion or replacement registration again.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/state/onboard-checkpoint-types.ts`:
- Around line 84-87: Preserve the source workload’s sharing state across
checkpointing and sandbox recreation. In
src/lib/state/onboard-checkpoint-types.ts lines 84-87, add the shared field to
CheckpointSandboxRecreateSourceWorkload; in src/lib/state/onboard-checkpoint.ts
lines 285-296, validate and restore it from the journal; in
src/lib/onboard/sandbox-recreate-transaction.ts lines 104-127, serialize and
reconstruct the exact value instead of defaulting to false; and in
src/lib/onboard/sandbox-recreate-transaction.test.ts lines 120-125, add a
shared-workload case verifying reconstruction keeps shared: true and cleanup
retains the source image.

---

Nitpick comments:
In `@src/lib/onboard/machine/handlers/sandbox-recreate-journal.test.ts`:
- Around line 244-262: Update the test around handleSandboxState to capture the
journaled replacement targetGeneration after the first rejected run, before the
createSandbox mock mutates replacementEntry. Use that captured value for the
subsequent retirement-generation assertion, including the expected replacement
entry where needed, so the test validates the journaled generation rather than
the mutable fixture.
- Around line 143-223: The test must verify that retrying after replacement
registration re-enters createSandbox: assert createSandbox is called twice, and
inspect the second call to confirm it uses the original recreate transaction ID
and target generation. Ensure the registered replacement state allows
createSandboxRecreateRuntime to accept the target on retry without invoking
deletion or replacement registration again.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 89a42eae-8958-4e20-a78c-c553531bd061

📥 Commits

Reviewing files that changed from the base of the PR and between afa28c2 and 307618a.

📒 Files selected for processing (10)
  • src/lib/actions/sandbox/rebuild-recreate-observability.test.ts
  • src/lib/onboard/machine/core-flow-phases.test.ts
  • src/lib/onboard/machine/handlers/sandbox-recreate-journal.test.ts
  • src/lib/onboard/machine/handlers/sandbox.ts
  • src/lib/onboard/runtime-provider/replaced-workload.test.ts
  • src/lib/onboard/sandbox-recreate-transaction.test.ts
  • src/lib/onboard/sandbox-recreate-transaction.ts
  • src/lib/state/onboard-checkpoint-types.ts
  • src/lib/state/onboard-checkpoint.test.ts
  • src/lib/state/onboard-checkpoint.ts
🚧 Files skipped from review as they are similar to previous changes (2)
  • src/lib/onboard/machine/core-flow-phases.test.ts
  • src/lib/onboard/machine/handlers/sandbox.ts

Comment thread src/lib/state/onboard-checkpoint-types.ts
@senthilr-nv
senthilr-nv disabled auto-merge August 1, 2026 03:58
Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
@senthilr-nv

Copy link
Copy Markdown
Collaborator Author

Exact-head security review refresh

Verdict: PASS at d5fb2d924110b44f6ce1320c7c25c3845fae20e1; no security findings remain in the reviewed PR diff.

The new commit preserves a source workload's sharing state through interrupted rebuild resume. A proven shared source now returns shared-image before any destructive provider operation. Replacement name, generation, and live-identity checks still run first.

  1. Secrets and credentials — PASS: the checkpoint adds one boolean and remains credential-free; no credential material or secret-bearing field is persisted.
  2. Input validation and data sanitization — PASS: the parser requires an exact boolean and rejects missing or non-boolean sharing state as a corrupt checkpoint.
  3. Authentication and authorization — PASS: provider ownership and destructive-cleanup authority remain required for removal; shared sources cannot reach image removal.
  4. Dependencies and third-party libraries — PASS: no dependency or registry-source changes.
  5. Error handling and logging — PASS: malformed receipts fail closed, expected provider-selection errors retain their existing handling, and no sensitive data enters logs.
  6. Cryptography and data protection — PASS: no cryptographic or sensitive-data transport/storage changes.
  7. Configuration and security headers — PASS: no service, container, port, origin, header, or privilege configuration changes.
  8. Security testing — PASS: the regression first reproduced shared: true becoming false, then verifies round-trip preservation and confirms that image removal is not called. Parser coverage rejects a receipt without a boolean sharing state.
  9. System security — PASS: the fix preserves least-privilege cleanup across interruption and closes the state-loss path that could convert a retain decision into deletion. Retry coverage also binds the original transaction ID and target generation.

Exact-head evidence: focused tests 97/97 passed; npm run typecheck:cli passed; npm run validate:pr passed; git diff --check passed; the commit is signed, includes DCO, and GitHub reports it Verified. Independent documentation writer review passed with no-docs-needed.

Prior full review: #8039 (comment)

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/lib/onboard/sandbox-recreate-transaction.ts (1)

60-109: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Require an exact replacement identity match before deletion.

The guard only checks that replacement.lifecycleLiveIdentityFingerprint is non-empty. It does not compare that value with the journaled targetLiveIdentityFingerprint.

A stale or foreign same-name registry row can have targetGeneration and a different live identity. The function can then delete the source workload.

Pass the expected target identity to this function. Skip cleanup unless it exactly matches replacement.lifecycleLiveIdentityFingerprint. Update the handler call and add a mismatched-identity regression test.

Proposed fix
 export function retireReplacedSandboxWorkload(
   sandboxName: string,
   targetGeneration: string,
+  targetLiveIdentityFingerprint: string | null,
   source: ReplacedSandboxSourceEntry,
   replacement: SandboxEntry | null,
   deps: ReplacedSandboxWorkloadCleanupDeps = {},
 ): ReplacedSandboxWorkloadCleanupResult {
   if (
     source.name !== sandboxName ||
     replacement?.name !== sandboxName ||
     replacement.lifecycleGeneration !== targetGeneration ||
-    !replacement.lifecycleLiveIdentityFingerprint
+    !targetLiveIdentityFingerprint ||
+    replacement.lifecycleLiveIdentityFingerprint !== targetLiveIdentityFingerprint
   ) {
     return { status: "skipped", reason: "replacement-unproven" };
   }

Based on supplied checkpoint and handler context, targetLiveIdentityFingerprint is persisted but is not passed to this guard.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/onboard/sandbox-recreate-transaction.ts` around lines 60 - 109,
Require an exact live-identity match before cleanup in
retireReplacedSandboxWorkload: add the expected targetLiveIdentityFingerprint
parameter and skip with replacement-unproven unless it equals
replacement.lifecycleLiveIdentityFingerprint. Update the handler call to pass
the journaled target identity, and add a regression test covering a same-name
replacement with a matching generation but mismatched identity.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/lib/onboard/sandbox-recreate-transaction.ts`:
- Around line 60-109: Require an exact live-identity match before cleanup in
retireReplacedSandboxWorkload: add the expected targetLiveIdentityFingerprint
parameter and skip with replacement-unproven unless it equals
replacement.lifecycleLiveIdentityFingerprint. Update the handler call to pass
the journaled target identity, and add a regression test covering a same-name
replacement with a matching generation but mismatched identity.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c3dd430b-29ee-4523-be3a-21607db19b16

📥 Commits

Reviewing files that changed from the base of the PR and between 307618a and d5fb2d9.

📒 Files selected for processing (7)
  • src/lib/onboard/machine/handlers/sandbox-recreate-journal.test.ts
  • src/lib/onboard/machine/handlers/sandbox.ts
  • src/lib/onboard/sandbox-recreate-transaction.test.ts
  • src/lib/onboard/sandbox-recreate-transaction.ts
  • src/lib/state/onboard-checkpoint-types.ts
  • src/lib/state/onboard-checkpoint.test.ts
  • src/lib/state/onboard-checkpoint.ts
🚧 Files skipped from review as they are similar to previous changes (6)
  • src/lib/state/onboard-checkpoint.ts
  • src/lib/onboard/machine/handlers/sandbox-recreate-journal.test.ts
  • src/lib/state/onboard-checkpoint.test.ts
  • src/lib/onboard/sandbox-recreate-transaction.test.ts
  • src/lib/onboard/machine/handlers/sandbox.ts
  • src/lib/state/onboard-checkpoint-types.ts

Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
@senthilr-nv

Copy link
Copy Markdown
Collaborator Author

Exact-head sensitive-path review

Result: PASS at d9d2ad2a55ac033a9ed17d9cba760fecd8696737.

  1. Secrets and credentials — PASS. No credentials, tokens, private keys, or secret-bearing fields are added. The persisted cleanup receipt contains only sandbox name, lifecycle generation/identity, workload reference/kind, image tag, and sharing state.
  2. Input validation — PASS. Checkpoint parsing strictly validates the new shared boolean and rejects malformed receipts. Cleanup requires the journaled sandbox name, generation, and exact non-empty replacement live-identity fingerprint.
  3. Authentication and authorization — PASS. Destructive cleanup still requires runtime-provider ownership authority. Expected provider-selection denial fails closed; unexpected errors propagate.
  4. Dependencies and supply chain — PASS. No dependency, image, lockfile, installer, or fetched-artifact changes.
  5. Error handling and logging — PASS. Missing, stale, foreign, reused, shared, or unverifiable workloads are retained with existing bounded cleanup outcomes. No sensitive values are logged.
  6. Cryptography — PASS. No cryptographic behavior changes.
  7. Configuration and defaults — PASS. No flags, environment variables, workflow permissions, policies, or defaults change.
  8. Security regression coverage — PASS. Tests cover missing/malformed checkpoint state, shared-source resume, mismatched live identity with matching generation, transaction reload after creation, image reuse, foreign generation, and unproven authority. Focused E2E-support suite passed 98/98.
  9. System security properties — PASS. Cleanup is now bound to the exact journaled replacement identity and generation, preventing a stale or foreign same-name registry row from authorizing deletion. The handler reloads and verifies the same transaction after creation before any source retirement.

Additional exact-head evidence: npm run typecheck:cli passed; npm run validate:pr passed; commit is signed, DCO-compliant, and GitHub Verified; independent documentation-writer review passed with no-docs-needed.

@senthilr-nv

Copy link
Copy Markdown
Collaborator Author

Addressed the exact replacement-identity finding in d9d2ad2a.

  • retireReplacedSandboxWorkload now requires the journaled target live-identity fingerprint and skips cleanup unless it exactly equals the replacement registry entry, in addition to the existing name and generation checks.
  • The sandbox handler reloads the journal after createSandbox, verifies the same transaction ID, and uses the recorded target identity for cleanup and commit instead of the pre-create immutable transaction value.
  • A regression covers a same-name, same-generation replacement with a mismatched live identity and proves no workload removal occurs.

Focused E2E-support tests pass 98/98, CLI type-checking passes, and npm run validate:pr passes at the exact head.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/onboard/machine/handlers/sandbox.ts`:
- Around line 1530-1533: Wrap the post-create sequence beginning with
reloadSandboxRecreateTransaction and including
retireSandboxRecreateSourceWorkload, recordSandboxRecreateRepairSuccess, and
recordSandboxRecreateRegistryCommit in its own try/catch. On any
transaction-reload or cleanup error, call recordSandboxRecreateRepairFailure,
then rethrow the original error so provider failures still propagate. Add a
regression test covering removeOwnedWorkload throwing after replacement
registration and verify the public boundary records repair failure while
preserving the thrown error.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c5142315-9fbd-4118-bbc0-b578e5a3a377

📥 Commits

Reviewing files that changed from the base of the PR and between d5fb2d9 and d9d2ad2.

📒 Files selected for processing (5)
  • src/lib/onboard/machine/handlers/sandbox-recreate-journal.test.ts
  • src/lib/onboard/machine/handlers/sandbox.ts
  • src/lib/onboard/runtime-provider/replaced-workload.test.ts
  • src/lib/onboard/sandbox-recreate-transaction.test.ts
  • src/lib/onboard/sandbox-recreate-transaction.ts
🚧 Files skipped from review as they are similar to previous changes (3)
  • src/lib/onboard/runtime-provider/replaced-workload.test.ts
  • src/lib/onboard/sandbox-recreate-transaction.test.ts
  • src/lib/onboard/machine/handlers/sandbox-recreate-journal.test.ts

Comment thread src/lib/onboard/machine/handlers/sandbox.ts Outdated
Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
@senthilr-nv

Copy link
Copy Markdown
Collaborator Author

Exact-head sensitive-path review

Result: PASS at 7ab09275b6c34ec5619fd0c2d98afc98ef154572.

  1. Secrets and credentials — PASS. No credentials, tokens, private keys, or secret-bearing fields are added. The persisted cleanup receipt contains only sandbox name, lifecycle generation/identity, workload reference/kind, image tag, and sharing state.
  2. Input validation — PASS. Checkpoint parsing strictly validates the new shared boolean and rejects malformed receipts. Cleanup requires the journaled sandbox name, generation, and exact non-empty replacement live-identity fingerprint.
  3. Authentication and authorization — PASS. Destructive cleanup still requires runtime-provider ownership authority. Expected provider-selection denial fails closed; unexpected errors propagate.
  4. Dependencies and supply chain — PASS. No dependency, image, lockfile, installer, or fetched-artifact changes.
  5. Error handling and logging — PASS. Missing, stale, foreign, reused, shared, or unverifiable workloads are retained with existing bounded cleanup outcomes. Post-create transaction or cleanup failures record state.repair.failed and preserve the original thrown error. No sensitive values are logged.
  6. Cryptography — PASS. No cryptographic behavior changes.
  7. Configuration and defaults — PASS. No flags, environment variables, workflow permissions, policies, or defaults change.
  8. Security regression coverage — PASS. Tests cover missing/malformed checkpoint state, shared-source resume, mismatched live identity with matching generation, transaction reload after creation, post-create cleanup failure recording, image reuse, foreign generation, and unproven authority. Focused CLI suite passed 98/98.
  9. System security properties — PASS. Cleanup is now bound to the exact journaled replacement identity and generation, preventing a stale or foreign same-name registry row from authorizing deletion. The handler reloads and verifies the same transaction after creation before any source retirement.

Additional exact-head evidence: npm run typecheck:cli passed; npm run validate:pr passed; commit is signed, DCO-compliant, and GitHub Verified; independent documentation-writer review passed with no-docs-needed.

@senthilr-nv
senthilr-nv enabled auto-merge (squash) August 1, 2026 04:46
@senthilr-nv
senthilr-nv merged commit a77b44e into main Aug 1, 2026
90 of 95 checks passed
@senthilr-nv
senthilr-nv deleted the codex/fix-rebuild-image-cleanup branch August 1, 2026 05:08
apurvvkumaria pushed a commit that referenced this pull request Aug 3, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Completes the documentation follow-ups identified after the v0.0.100
tag.
The durable guides and `docs/changelog/2026-07-31.mdx` now cover final
inference-route timing, validation reuse boundaries, and
replacement-image cleanup.

## Changes

- [#8046](#8046) ->
`docs/inference/verify-inference-route.mdx` and
`docs/changelog/2026-07-31.mdx`: Documents the 2-second final
`inference.local` response budget for OpenClaw and Hermes, including the
OpenClaw client-overhead rationale.
- [#8044](#8044) ->
`docs/inference/understand-provider-validation.mdx`: Documents the exact
one-shot Chat Completions validation reuse and forced-revalidation
conditions.
- [#8039](#8039) and
[#8042](#8042) ->
`docs/manage-sandboxes/recover-rebuild-sandboxes.mdx` and
`docs/changelog/2026-07-31.mdx`: Documents obsolete owned source-image
cleanup after durable replacement proof and the `gc` recovery action.
- `docs/reference/system-readiness.mdx` and
`docs/reference/troubleshooting.mdx`: Applies title case and removes
code styling from headings while preserving literal identifiers in
prose.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [ ] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [x] Tests not applicable — justification: This documentation-only
change does not modify executable behavior.
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: `docs/changelog/2026-07-31.mdx`,
`docs/inference/understand-provider-validation.mdx`,
`docs/inference/verify-inference-route.mdx`,
`docs/manage-sandboxes/recover-rebuild-sandboxes.mdx`,
`docs/reference/system-readiness.mdx`, and
`docs/reference/troubleshooting.mdx`. The documentation-only diff was
reviewed against the writing rules and documentation style.
- Agent: Codex Desktop
<!-- docs-review-head-sha: 6d2cd17 -->
<!-- docs-review-agents-blob-sha: 3dd7c24 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit:
- Station profile/scenario:
- Result:
- Supporting evidence:

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run validate:pr` passed after refreshing `origin/main` when hooks
were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification:
Documentation-only; `npx vitest run test/changelog-docs.test.ts
test/agent-variant-docs.test.ts` passed 23 tests.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result:
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

`npm run docs` completed with 0 errors and the existing Fern warning.

---
Signed-off-by: Miyoung Choi <miyoungc@nvidia.com>

---------

Signed-off-by: Miyoung Choi <miyoungc@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: onboarding Onboarding FSM, provider setup, sandbox launch, or first-run flow area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery bug-fix PR fixes a bug or regression security

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants