fix(cli): rebuild sandboxes from locally built base images

[laitingsheng]

Summary

Rebuilding a Hermes sandbox originally created with NEMOCLAW_SANDBOX_BASE_LOCAL_BUILD=1 could reject its local base-image handoff or rebuild the base again after an upgrade. The rebuild now carries local authority only from an exact reused resolution or a fresh in-memory build lease, preserves the stable resolution metadata through recreate, and fails before sandbox mutation if that proof changes.

Related Issue

Fixes #7477

Changes

• Accept only canonical image-<sha256> refs and exact rebuild-<pid>-<nonce>-image-<sha256> handoffs. The shared parser is required by the resolver, preflight binder, and cleanup path so those consumers cannot drift; src/lib/agent/base-image-handoff.test.ts and src/lib/sandbox-base-image-resolution.test.ts protect the grammar and moved-tag cases.
• Bind a reused local image to a rebuild handoff only when the persisted resolution object is the exact object reused by the current resolver and its current schema, key, ref, image ID, OS, architecture, and build provenance all match. A mutable Docker label alone is never authority.
• When an upgraded release cannot reuse stale local metadata, perform one repository build and use only the fresh in-memory trust lease returned by that build. This fallback is required for local-build-only upgrade recovery because directly trusting newly resolved fallback metadata would allow a moved mutable tag.
• Keep operation-scoped override metadata out of the durable resolution record so the stable outer key survives recreate and the next offline rebuild can reuse it.
• Add focused coverage for copied provenance, moved source and handoff tags, malformed refs, stale metadata, bounded forced rebuild, next-rebuild reuse, metadata persistence, and cleanup.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification: Restores the documented local base-image rebuild workflow without adding a command, flag, default, configuration format, or supported workflow.
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent nine-category review found no open code findings and was refreshed on exact head d18d410b5; see fix(cli): rebuild sandboxes from locally built base images #7481 (comment). Exact-head hosted and Docker/offline E2E remain required before merge.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: no-docs-needed
• Evidence: Rebuild now preserves the trusted handoff for a locally built base image, restoring the documented rebuild and local-build contract without changing commands, flags, defaults, output, configuration, or operator steps.
• Agent: Codex Desktop

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit:
• Station profile/scenario:
• Result:
• Supporting evidence:

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: The seven-file behavior/security set passed 126 tests before the first test-only guardrail follow-up; the six-file security set then passed 120 tests. The final fixture repair passed the original failing/lease set 26/26 and all shared-harness consumers 63/63; exact-final-head CLI typecheck and npm run check:diff passed.
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result:
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Tinson Lai tinsonl@nvidia.com
Signed-off-by: Aaron Erickson aerickson@nvidia.com

Summary by CodeRabbit

• Bug Fixes
  • Improved sandbox rebuilds using temporary local base-image handoffs, including correct provenance/trust leasing and handoff binding behavior.
  • Added stricter validation for handoff refs and temporary handoff leases (rejecting malformed/untrusted refs) with safer failure handling when binding can’t be established.
  • Preserved stable outer base-image metadata across disposable-local rebuild flows, with clearer errors when staged vs outer metadata don’t match; also fixed rebuild-forcing vs metadata reuse after moved local tags.

[coderabbitai]

📝 Walkthrough

Walkthrough

Temporary rebuild image tags are recognized as content-addressed references. Base-image onboarding validates handoff provenance, while rebuild preflight derives, reuses, or recreates trusted local overrides and preserves stable outer resolution metadata.

Changes

Sandbox rebuild base-image handoff

Layer / File(s)	Summary
Recognize rebuild image references src/lib/sandbox-base-image.ts, src/lib/onboard/base-image-resolution-flow.ts, src/lib/sandbox-base-image-resolution.test.ts, src/lib/onboard/base-image-resolution-flow.test.ts	Content-addressed parsing accepts temporary rebuild references, and resolution flows preserve compatible stable metadata while rejecting mismatched disposable handoffs.
Validate handoff provenance src/lib/agent/base-image.ts, src/lib/agent/onboard.ts, src/lib/agent/base-image-handoff.test.ts, src/lib/agent/base-image.test.ts, test/helpers/base-image-test-harness.ts	Base-image onboarding validates source and handoff identity, metadata, and Docker provenance before returning trusted local overrides; operation-scoped leases and reused hints are covered by tests.
Lease trusted rebuild handoffs src/lib/actions/sandbox/rebuild-flow-helpers.ts, src/lib/actions/sandbox/*test.ts, test/helpers/rebuild-flow-harness.ts	Rebuild preflight inherits or binds trusted overrides, forces rebuilds when local tags move, fails closed without validated metadata, and updates rebuild-flow fixtures for the trusted local references.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related PRs

• NVIDIA/NemoClaw#6254: Updates rebuild preflight metadata propagation in the same helper.
• NVIDIA/NemoClaw#6884: Extends trusted local override and provenance validation behavior.
• NVIDIA/NemoClaw#7213: Covers related temporary local handoff and rebuild preflight handling.

Suggested reviewers: prekshivyas, apurvvkumaria

Sequence Diagram(s)

sequenceDiagram
  participant ensureRebuildAgentBaseImage
  participant ensureAgentBaseImage
  participant bindLocalAgentBaseImageHandoffToResolution
  participant resolveSandboxBaseImage
  ensureRebuildAgentBaseImage->>ensureAgentBaseImage: Resolve or force local rebuild
  ensureAgentBaseImage-->>ensureRebuildAgentBaseImage: Return image and metadata
  ensureRebuildAgentBaseImage->>bindLocalAgentBaseImageHandoffToResolution: Validate handoff provenance
  bindLocalAgentBaseImageHandoffToResolution-->>ensureRebuildAgentBaseImage: Return trusted local override
  ensureRebuildAgentBaseImage->>resolveSandboxBaseImage: Resolve trusted handoff ref
  resolveSandboxBaseImage-->>ensureRebuildAgentBaseImage: Return local resolution

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 21.05% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The PR implements the requested temporary rebuild ref handling, provenance binding, trust checks, and regression coverage for #7477.
Out of Scope Changes check	✅ Passed	The changes are focused on sandbox/base-image rebuild flow logic, tests, and supporting harness updates with no clear unrelated scope.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: rebuilding sandboxes from locally built base images.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/local-base-image-rebuild-trust

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit d18d410 in the fix/local-base-image... branch remains at 96%, unchanged from commit 88b3e35 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit d18d410 in the fix/local-base-image... branch remains at 80%, unchanged from commit 88b3e35 in the main branch.

Show a code coverage summary of the most impacted files.

File	main 88b3e35	fix/local-base-image... d18d410	+/-
src/lib/domain/.../connect-env.ts	97%	89%	-8%
src/lib/sandbox-base-image.ts	90%	90%	0%
src/lib/sandbox...rce-identity.ts	87%	87%	0%
src/lib/agent/onboard.ts	71%	72%	+1%
src/lib/actions...flow-helpers.ts	87%	90%	+3%
src/lib/agent/base-image.ts	85%	89%	+4%
src/lib/onboard...olution-flow.ts	67%	95%	+28%

---

_{Updated July 24, 2026 17:53 UTC}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/sandbox-base-image-resolution.test.ts`:
- Around line 334-348: Ensure both test sites clean up test doubles on success
and failure: in src/lib/sandbox-base-image-resolution.test.ts lines 334-348,
reset dockerMocks.imageInspectFormat after the assertion; in
src/lib/actions/sandbox/rebuild-flow-helpers.test.ts lines 269-281, reset
baseImageMocks.inspectLocalImageMetadata and restore every spy returned by
mockBaseImagePreflight inside a finally block.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1b30339d-b073-4b30-ae4c-3f4d13d1b823

📥 Commits

Reviewing files that changed from the base of the PR and between 99776d3 and 05b0e9d.

📒 Files selected for processing (4)

• src/lib/actions/sandbox/rebuild-flow-helpers.test.ts
• src/lib/actions/sandbox/rebuild-flow-helpers.ts
• src/lib/sandbox-base-image-resolution.test.ts
• src/lib/sandbox-base-image.ts

[github-actions]

PR Review Advisor — Informational

Advisor assessment: Informational / high confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: No actionable findings remain in the canonical review ledger.

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 1 blocker · 1 warning · 1 suggestion
• Model comparison: normalized findings differ; normalized E2E selections differ; Nemotron reported 1 more blocker, 1 more warning, 1 more suggestion.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: full-e2e, hermes-e2e, onboard-repair, onboard-resume, state-backup-restore, upgrade-stale-sandbox, cloud-onboard

1 optional E2E recommendation

• rebuild-hermes

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[ericksoa]

Maintainer hardening update for exact head 3460260127eda1ce932859ed629f443964b44039:

• Preserved the contributor commit and fast-forwarded the branch through current main; no history was rewritten. All three commits are GitHub Verified and carry DCO sign-offs.
• Independent nine-category sensitive-path review found no remaining code blocker. Local Docker labels are supporting evidence only: authority now requires either the exact reused resolution object plus current schema/key/ref/ID/OS/architecture/provenance, or a fresh in-memory lease returned by the current forced build. Copied labels, moved source/handoff tags, stale or fresh fallback metadata, and malformed temporary refs fail closed.
• A stale release hint performs at most one forced repository build. Inner override-scoped metadata is not persisted, so the stable outer resolution survives recreate and the next offline rebuild can reuse it instead of rebuilding again.
• Focused validation: 7 files / 126 tests passed; CLI build and typecheck passed; npm run check:diff passed, including pre-commit, commitlint, secret scan, source-shape/test-size, and pre-push gates.
• Documentation writer review result: no-docs-needed; this restores the documented local-base rebuild workflow without adding a command, flag, default, schema, or supported workflow.

Code is GO for exact-head CI. Merge remains gated on the new CI run and Docker/offline E2E evidence for this head.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/agent/base-image-handoff.test.ts`:
- Around line 102-125: Remove the explicit if statement from the
dockerMocks.imageInspectFormat mock implementation. Preserve the non-JSON
behavior by folding the format check into the returned expression, using the
existing ternary logic so the mock remains functionally identical and the test
body stays linear.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8a52bea9-be2a-411b-afe0-5aa1fd100aea

📥 Commits

Reviewing files that changed from the base of the PR and between 05b0e9d and 3460260.

📒 Files selected for processing (12)

• src/lib/actions/sandbox/rebuild-agent-base-image-preflight.test.ts
• src/lib/actions/sandbox/rebuild-flow-helpers.test.ts
• src/lib/actions/sandbox/rebuild-flow-helpers.ts
• src/lib/agent/base-image-handoff.test.ts
• src/lib/agent/base-image.test.ts
• src/lib/agent/base-image.ts
• src/lib/agent/onboard.ts
• src/lib/onboard/base-image-resolution-flow.test.ts
• src/lib/onboard/base-image-resolution-flow.ts
• src/lib/sandbox-base-image-resolution.test.ts
• src/lib/sandbox-base-image.ts
• test/helpers/base-image-test-harness.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• src/lib/sandbox-base-image-resolution.test.ts

[ericksoa]

Exact-head follow-up for 3c44a0044ee1ea996ab156e21fc7c14d585e1ac1: the only delta from the reviewed production head is a behavior-equivalent test-fixture rewrite from an early if return to a conditional expression, resolving the codebase-growth guardrail. No production bytes or trust invariants changed. The conditional scan now passes; 120 focused tests, CLI typecheck, Biome, diff checks, normal pre-commit, and commitlint pass. The commit is DCO-compliant and GitHub Verified. Independent security and documentation reviews were refreshed on this exact head; code remains GO for CI, with merge still gated on exact-head CI/E2E.

[ericksoa]

Exact-head CI repair is now published at d18d410b52ffadac73f3052edc7360bdb93bc950, including current main (88b3e35ed). GitHub reports every PR commit as Verified and the PR as mergeable.

The prior CLI shard failure was deterministic test-fixture drift, not a production or Docker/runtime regression: the shared rebuild harness modeled a successful local build with a mutable tag, missing display name, and incomplete trust lease. Follow-up 67c9e1a66 changes only two test files so the harness uses the real canonical content-addressed ref and operation-scoped trusted lease. All production blobs remain byte-identical to the previously reviewed 3c44a0044 head; the fail-closed trust boundary is unchanged.

Fresh validation: original failing/lease set 26/26, all shared-harness consumers 63/63, CLI typecheck passed, and exact-final-head npm run check:diff passed. Independent nine-category review is GO with no new finding; documentation review is no-docs-needed at d18d410b5 (AGENTS.md blob 9c9b36d7f). The unchanged Docker-daemon TOCTOU boundary and exact-head hosted/offline E2E remain pre-merge evidence requirements. Fresh remote CI/E2E is now running.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/helpers/rebuild-flow-harness.ts`:
- Around line 287-288: Update createRebuildFlowHarness so the content-addressed
agentBaseImageRef and valid trustedLocalOverride are supplied only for explicit
local-handoff scenarios, including the required Hermes case. Preserve the
existing "openclaw" default and omit these local fixture values for non-local
and negative-path invocations so trust-validation branches remain exercised.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 190c3881-fcb5-4fc2-8ea5-74887230b952

📥 Commits

Reviewing files that changed from the base of the PR and between 3c44a00 and d18d410.

📒 Files selected for processing (2)

• src/lib/actions/sandbox/rebuild-dcode-base-image-lease.test.ts
• test/helpers/rebuild-flow-harness.ts

[ericksoa]

Direct exact-head runtime evidence: upgrade-stale-sandbox passed on d18d410b52ffadac73f3052edc7360bdb93bc950 with expectedSha == testedSha, 1 passed / 0 failed / 0 skipped. Every phase passed, including stale detection, rebuild to the current OpenClaw runtime (80.5s), and the post-rebuild clean upgrade check.

The selected matrix also has hermes-e2e, onboard-repair, and state-backup-restore green. The only failed selected job is the unrelated systemic full-e2e cold-onboard performance budget tracked in #6660; comparative evidence is recorded at #6660 (comment).

[prekshivyas]

Reviewed PR SHA d18d410b5 against base SHA 88b3e35e8. The handoff parser rejects malformed refs. Preflight binds trust only from the exact reused local resolution and matching image, platform, and provenance data. Fresh fallback metadata forces one repository build and uses its in-memory lease. All nine security categories pass, and no code findings remain.

I accept E2E / PR Gate Coordination as a scoped exception. The selected full-e2e job completed its functional contract and exceeded only the cold-onboard performance budget tracked in #6660. Exact-head upgrade-stale-sandbox, hermes-e2e, onboard-repair, onboard-resume, and state-backup-restore pass.