fix(hermes): preserve managed BuildKit failures

[apurvvkumaria]

Summary

When an exact managed Hermes image build on a local Docker-driver gateway starts in host-side BuildKit and exits unsuccessfully, retrying the same BuildKit-only Dockerfile through the gateway builder hides the useful error behind RUN --mount requires BuildKit. This change preserves that attempted BuildKit failure while retaining the existing fallback for every path that cannot prove the same managed build provenance.

Related Issue

Part of #7140 and #7144.

This complements #7582, which fixes the stale Hermes base digest observed in #7580. It does not broaden #7253's OpenClaw compatibility behavior.

Changes

• Issue an in-memory capability only for a generated Hermes build on the local Docker-driver path.
• Bind the capability to the staged context and Dockerfile identities plus the build ID, and reject copied, malformed, or drifted provenance.
• Preserve the original failure only after that exact BuildKit attempt returns a nonzero or missing exit status.
• Retain gateway fallback when prebuild is disabled or unavailable, the build cannot start, trust validation fails, provenance changes, the image is custom, the agent is OpenClaw, or the gateway is remote.
• Cover capability issuance, provenance drift, retry behavior, exact failure preservation, and all unchanged fallback paths.
• Document the narrow managed-Hermes recovery path.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification:
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: two independent exact-diff reviews found no security or correctness blocker; the final review verified the production relocation, canonical custom-image exclusion, exact context/Dockerfile/build binding, and unchanged fallback paths.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: docs-updated
• Evidence: docs/reference/troubleshooting.mdx documents nonzero and missing BuildKit exit statuses, the recovery step, and unchanged optional/custom/OpenClaw/remote behavior.
• Agent: Codex Desktop

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit:
• Station profile/scenario:
• Result:
• Supporting evidence:

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed
• Targeted behavior tests pass for the current change set — 42/42 focused CLI tests passed across sandbox-prebuild.test.ts and sandbox-create-launch.test.ts.
• Applicable broad gate passed — npm run checks, npm run typecheck:cli, project-membership, source-shape, title-style, test-size, and Biome checks passed.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only) — Fern completed with 0 errors and 2 pre-existing hidden warnings.
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Managed Hermes onboarding now issues a scoped capability for exact local staged builds. Matching BuildKit failures propagate without gateway-builder fallback, while OpenClaw, custom Dockerfiles, remote gateways, and capability mismatches retain existing fallback behavior.

Changes

Hermes BuildKit onboarding

Layer / File(s)	Summary
Issue and propagate Hermes failure capability src/lib/onboard/sandbox-prebuild.ts, src/lib/onboard.ts, src/lib/onboard/sandbox-create-launch.ts	A capability is issued and passed through onboarding only for matching managed Hermes staged builds, with path and build identity verification.
Preserve exact local BuildKit failures src/lib/onboard/sandbox-prebuild.ts, src/lib/onboard/sandbox-prebuild.test.ts	Matching nonzero BuildKit failures are thrown instead of converted into gateway fallback; disabled, mismatched, or altered capabilities retain null-image fallback.
Validate launch routing and retry behavior src/lib/onboard/sandbox-create-launch.test.ts, src/lib/onboard/sandbox-prebuild.test.ts	Tests cover OpenClaw fallback, Hermes failure propagation, retry success, custom and remote Hermes images, and capability identity drift.
Document Hermes remediation docs/reference/troubleshooting.mdx	Troubleshooting guidance explains the failure, resume command, root-cause categories, and exceptions retaining gateway-builder behavior.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant onboard
  participant sandboxCreateLaunch
  participant prebuildSandboxImageIfEligible
  participant buildImage
  participant openshellShellCommand
  onboard->>sandboxCreateLaunch: Issue scoped Hermes capability
  sandboxCreateLaunch->>prebuildSandboxImageIfEligible: Pass capability
  prebuildSandboxImageIfEligible->>buildImage: Run local BuildKit
  alt Matching managed Hermes build fails
    prebuildSandboxImageIfEligible-->>sandboxCreateLaunch: Throw BuildKit failure
  else Fallback is allowed
    prebuildSandboxImageIfEligible->>openshellShellCommand: Build with gateway builder
  end

Loading

Possibly related PRs

• NVIDIA/NemoClaw#7253: Updates related sandbox prebuild and BuildKit fallback behavior.

Suggested labels: integration: hermes, bug-fix, area: sandbox, area: onboarding

Suggested reviewers: laitingsheng, hoyalim, ericksoa

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: preserving managed Hermes BuildKit failures instead of falling back to gateway handling.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/7140-hermes-buildkit-fail-fast

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 77856d8 in the codex/7140-hermes-bu... branch remains at 96%, unchanged from commit d4a859a in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 77856d8 in the codex/7140-hermes-bu... branch is 80%. The coverage in commit d4a859a in the main branch is 81%.

Show a code coverage summary of the most impacted files.

File	main d4a859a	codex/7140-hermes-bu... 77856d8	+/-
src/lib/actions...-add-restart.ts	19%	10%	-9%
src/lib/actions...x/mcp-bridge.ts	43%	36%	-7%
src/lib/actions...lution-probe.ts	93%	88%	-5%
src/lib/platform.ts	89%	84%	-5%
src/lib/onboard...reate-launch.ts	100%	96%	-4%
src/lib/actions...e-validation.ts	84%	81%	-3%
src/lib/actions...dbox/destroy.ts	95%	93%	-2%
src/lib/shields/index.ts	72%	71%	-1%
src/lib/onboard...eway-service.ts	82%	81%	-1%
src/lib/domain/.../connect-env.ts	89%	97%	+8%

---

_{Updated July 26, 2026 17:11 UTC}

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-7586.docs.buildwithfern.com/nemoclaw

[github-actions]

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

• GPT-5.6 Terra (primary): Completed · medium confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 1 warning · 0 suggestions
• Model comparison: normalized findings differ; normalized E2E selections differ; Nemotron reported the same number of blockers, 1 more warning, the same number of suggestions.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: onboard-repair, onboard-resume, state-backup-restore, upgrade-stale-sandbox, cloud-onboard

1 optional E2E recommendation

• rebuild-hermes

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

src/lib/onboard/sandbox-prebuild.ts (1)

164-186: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider consolidating the repeated fail-or-fallback pattern.

The if (input.gatewayFallback === "forbidden") { failRequiredBuildKit(...) } ... log(...); return { createArgs, imageRef: null, imageId: null }; shape repeats six times with only the detail/next-step/log strings varying, which is what pushed this function's complexity to high/medium per the line-range hints. A single helper (e.g. skipOrFailRequiredBuildKit(detail, nextStep, skipMessage?)) would remove the duplication and let each call site collapse to one line.

While consolidating, also thread the original caught error through as Error's cause option (Node 22 supports new Error(message, { cause })) for the two catch-based branches (staged-context inspection failure, build-start failure) so the original stack trace isn't discarded behind the synthesized message.

♻️ Sketch of a consolidated helper

-  const failRequiredBuildKit = (detail: string, nextStep: string): never => {
-    throw new Error(
-      `Local BuildKit is required for this generated sandbox image, but ${detail}. ` +
-        `${nextStep} ` +
-        "Gateway-builder fallback is disabled because this Dockerfile uses BuildKit-only instructions.",
-    );
-  };
+  const failRequiredBuildKit = (detail: string, nextStep: string, cause?: unknown): never => {
+    throw new Error(
+      `Local BuildKit is required for this generated sandbox image, but ${detail}. ` +
+        `${nextStep} ` +
+        "Gateway-builder fallback is disabled because this Dockerfile uses BuildKit-only instructions.",
+      cause === undefined ? undefined : { cause },
+    );
+  };
+  const skipOrFail = (
+    detail: string,
+    nextStep: string,
+    skipMessage?: string,
+    cause?: unknown,
+  ): SandboxPrebuildResult => {
+    if (input.gatewayFallback === "forbidden") failRequiredBuildKit(detail, nextStep, cause);
+    if (skipMessage) log(skipMessage);
+    return { createArgs, imageRef: null, imageId: null };
+  };
   if (!resolveSandboxPrebuildEnabled(env, input.dockerDriverGateway)) {
-    if (input.gatewayFallback === "forbidden") {
-      failRequiredBuildKit(
-        "the local prebuild is disabled or unavailable",
-        "Start Docker, ensure BuildKit and the local prebuild are enabled, then retry.",
-      );
-    }
-    return { createArgs, imageRef: null, imageId: null };
+    return skipOrFail(
+      "the local prebuild is disabled or unavailable",
+      "Start Docker, ensure BuildKit and the local prebuild are enabled, then retry.",
+    );
   }

Apply the same shape to the other four call sites, passing error as cause where a caught error exists.

As per path instructions, **/*.{js,ts,tsx}: "Keep function complexity low; existing complexity hotspots are tracked separately."

Also applies to: 199-229, 262-279

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/onboard/sandbox-prebuild.ts` around lines 164 - 186, Consolidate the
repeated forbidden-fallback handling in the sandbox prebuild flow into a helper
such as skipOrFailRequiredBuildKit(detail, nextStep, skipMessage?), preserving
each call site’s messages and fallback return/log behavior. Replace all six
duplicated branches with helper calls, and update failRequiredBuildKit to accept
an optional Error cause; pass the caught error as the cause in the
staged-context inspection and build-start catch branches.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/reference/troubleshooting.mdx`:
- Line 2654: Update the troubleshooting bullet around the local build guidance
so each sentence appears on its own source line: keep the existing wording and
formatting, but place the sentence ending with “root cause.” and the following
“Fix the reported…” sentence on separate lines.

---

Nitpick comments:
In `@src/lib/onboard/sandbox-prebuild.ts`:
- Around line 164-186: Consolidate the repeated forbidden-fallback handling in
the sandbox prebuild flow into a helper such as
skipOrFailRequiredBuildKit(detail, nextStep, skipMessage?), preserving each call
site’s messages and fallback return/log behavior. Replace all six duplicated
branches with helper calls, and update failRequiredBuildKit to accept an
optional Error cause; pass the caught error as the cause in the staged-context
inspection and build-start catch branches.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8ea15274-95b5-4550-ab3e-ec1ba7cd1757

📥 Commits

Reviewing files that changed from the base of the PR and between 584ed60 and 0333045.

📒 Files selected for processing (5)

• docs/reference/troubleshooting.mdx
• src/lib/onboard/sandbox-create-launch.test.ts
• src/lib/onboard/sandbox-create-launch.ts
• src/lib/onboard/sandbox-prebuild.test.ts
• src/lib/onboard/sandbox-prebuild.ts

[apurvvkumaria]

Fresh exact-head evidence reproduces the masking path this PR removes:

• In perf(messaging): stabilize image plan cache identity #7589's selected E2E run https://github.com/NVIDIA/NemoClaw/actions/runs/30203756186, channels-stop-start (hermes) ran on a healthy hosted runner and completed with artifacts plus cleanup.
• Local BuildKit first failed on the stale Hermes base's npm identity (11.13.0, expected 11.18.0).
• Current main then printed “using the gateway builder instead” and failed at the first BuildKit-only RUN --mount.

This was not runner loss, and #7589 does not touch builder selection. #7582 fixes this instance's root trigger; #7586 is still required so future local BuildKit failures retain their original evidence instead of being replaced by the deterministic classic-builder error.

[cv]

Product-scope blocker at exact head a62db35: #7140 covers E2E runner reliability, and #7144 explicitly excludes changing Hermes product behavior. Existing #6258 and #6166 define host BuildKit as an optional optimization with safe fallback, while this PR makes host-local Docker and BuildKit mandatory for generated Hermes and documents that as canonical behavior. CI cannot establish product approval for that new supported surface. The smallest safe salvage is to issue the fail-closed capability only from the exact managed, no-custom-image Hermes staging path; bind it to the exact staged context, Dockerfile, and build identity; preserve custom, remote, OpenClaw, and #7253 legacy behavior; add exact provenance plus explicit OpenClaw and injected BuildKit-failure tests; and avoid a general host-BuildKit support claim. Otherwise, please link an accepted design decision defining ownership, lifecycle, platform support, security boundary, and validation matrix, or defer until the gateway builder supports the managed Hermes Dockerfile.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/reference/troubleshooting.mdx`:
- Around line 2648-2649: Update the troubleshooting explanation for the staged
build failure path to include BuildKit returning no exit status (null) alongside
nonzero exits. State that NemoClaw preserves either failure without retrying the
managed Dockerfile with the incompatible gateway builder, while retaining the
guidance to inspect the preceding BuildKit output.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e04c308b-d133-404d-85fe-72c46c003307

📥 Commits

Reviewing files that changed from the base of the PR and between a62db35 and 6475c31.

📒 Files selected for processing (6)

• docs/reference/troubleshooting.mdx
• src/lib/onboard.ts
• src/lib/onboard/sandbox-create-launch.test.ts
• src/lib/onboard/sandbox-create-launch.ts
• src/lib/onboard/sandbox-prebuild.test.ts
• src/lib/onboard/sandbox-prebuild.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• src/lib/onboard/sandbox-create-launch.test.ts

[apurvvkumaria]

Exact-head follow-up for 77856d8ce:

• 41 checks passed and 14 were intentionally skipped.
• The credentialed E2E gate succeeded in run 30211706626.
• There are no active failures and no unresolved review threads.
• Earlier cancellations at this SHA were superseded duplicate controller runs, not test or runner failures.

The prior requested change was scoped into the production launch boundary, and the final documentation commit now covers both nonzero and missing BuildKit exit status. No further code or CI action is indicated; the remaining blocker is maintainer review of the current head.