Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
0636191
fix(rebuild): support hosts without Docker Buildx
apurvvkumaria Jul 20, 2026
0e9a222
refactor(onboard): preserve entrypoint growth budget
apurvvkumaria Jul 20, 2026
5d07406
test(rebuild): keep failure assertion linear
apurvvkumaria Jul 20, 2026
1233a9e
merge: resolve conflicts with main
github-actions[bot] Jul 26, 2026
de5a8ea
merge: resolve conflicts with main
github-actions[bot] Jul 26, 2026
dc60bb8
Merge branch 'main' into codex/7111-root-cause-investigation
prekshivyas Jul 26, 2026
c2817d0
merge: resolve conflicts with main
github-actions[bot] Jul 27, 2026
7f2f107
Merge branch 'main' into codex/7111-root-cause-investigation
cv Jul 27, 2026
28614b2
merge: sync main into Buildx fallback
apurvvkumaria Jul 28, 2026
33bcd9d
fix(rebuild): require Docker Buildx
jyaunches Jul 29, 2026
b81e779
Merge remote-tracking branch 'origin/main' into codex/pr-7253-buildx-…
jyaunches Jul 29, 2026
4c04584
chore(ci): ratchet shell quote fan-in
jyaunches Jul 29, 2026
0a077a3
test(rebuild): cover Buildx diagnostic streams
jyaunches Jul 29, 2026
ed9f04e
test(rebuild): guard Buildx mutation boundary
jyaunches Jul 29, 2026
89d5de6
fix(rebuild): support OpenClaw without Buildx
jyaunches Jul 29, 2026
ce5cf9d
refactor(onboard): keep rebuild orchestration net-neutral
jyaunches Jul 29, 2026
f5e80b7
test(rebuild): keep guardrail scenarios linear
jyaunches Jul 29, 2026
901e4a5
docs(rebuild): clarify custom Dockerfile exclusion
jyaunches Jul 29, 2026
5ca7deb
test(rebuild): bind Docker engine fixtures
jyaunches Jul 29, 2026
95124f8
test(rebuild): stop interactive fixture after backup
jyaunches Jul 29, 2026
0c559f7
test(rebuild): preserve sandbox after delete failure
jyaunches Jul 29, 2026
36a3fae
merge: resolve conflicts with main
github-actions[bot] Jul 30, 2026
5496192
merge: resolve conflicts with main
github-actions[bot] Jul 31, 2026
aff5256
merge: resolve conflicts with main
github-actions[bot] Aug 3, 2026
f52d3fd
Merge branch 'main' into codex/7111-root-cause-investigation
cv Aug 3, 2026
016ad2e
merge: resolve conflicts with main
github-actions[bot] Aug 3, 2026
8828c52
fix(rebuild): refresh retained OpenClaw rebuild
cv Aug 3, 2026
30938a5
fix(rebuild): keep onboard entrypoint net neutral
cv Aug 3, 2026
825984c
merge: resolve conflicts with main
github-actions[bot] Aug 3, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2771,6 +2771,40 @@ A rebuild preserves the recorded Deep Agents Code auto-approval capability unles
A sandbox onboarded with an explicit GPU opt-out (stored as `sandboxGpuMode: "0"`, plus legacy registry entries that only record `gpuEnabled: false`) is recreated with the same opt-out, so the inner `onboard --resume` skips the Docker CDI GPU preflight on hosts without an NVIDIA GPU.
Auto-mode sandboxes remain auto.

<AgentOnly variant="openclaw">

Before backup or deletion, `rebuild` builds and validates the replacement image with Docker's BuildKit path.
For a NemoClaw-generated OpenClaw image on a local Docker-driver gateway, NemoClaw can use one compatibility retry when Docker reports `BuildKit is enabled but the buildx component is missing or broken`.
NemoClaw runs `docker buildx version` independently and uses the compatibility retry only when that command also fails.
The retry rebuilds the same verified build context once with Docker's legacy builder.
NemoClaw then binds the retained image to the verified Docker engine and its immutable image ID.
It verifies the build context, Docker engine, mutable tag, and immutable image ID again at the final delete boundary.
Any failed build or pre-delete verification stops the rebuild without deleting the existing sandbox.
After deletion, Docker operations for replacement creation remain bound to the retained engine and immutable image ID.
If either identity changes during replacement creation, NemoClaw stops instead of operating on a different Docker target.

The legacy-builder retry does not apply in the following cases:

- The sandbox was recorded with a custom Dockerfile.
- The OpenShell gateway is not local.
- The host-side local prebuild is disabled.
- The build has an unrelated failure.
- The independent `docker buildx version` command succeeds.

When the retry does not apply and Buildx is missing or broken, NemoClaw stops and leaves the existing sandbox intact.
Install or repair Docker Buildx, then verify it with `docker buildx version`.
Rerun the rebuild after that command succeeds.

</AgentOnly>
<AgentOnly variant="hermes">

Before backup or deletion, `rebuild` builds and validates the replacement image with Docker Buildx.
If Buildx is missing or broken, NemoClaw stops and leaves the existing sandbox intact.
Install or repair Docker Buildx, then verify it with `docker buildx version`.
Rerun the rebuild after that command succeeds.

</AgentOnly>

```bash
$$nemoclaw my-assistant rebuild [--yes|-y|--force] [--verbose|-v] [--tool-disclosure <progressive|direct>] [--dcode-auto-approval <disabled|thread-opt-in>] [--observability|--no-observability]
```
Expand Down
147 changes: 147 additions & 0 deletions src/lib/actions/sandbox/rebuild-buildx-mutation-boundary.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { expectNoSandboxDelete } from "../../../../test/helpers/rebuild-delete-assertions";
import {
createRebuildFlowHarness,
resetRebuildFlowTestEnvironment,
restoreRebuildFlowTestEnvironment,
} from "../../../../test/helpers/rebuild-flow-harness";
import { fingerprintBuildContext } from "../../adapters/fs/build-context-fingerprint";
import type {
PreparedOpenClawLegacyImage,
PreparedSandboxBuildContext,
} from "../../onboard/build-context-stage";

type RetainedImageFixture = {
buildContext: PreparedSandboxBuildContext & {
contextFingerprint: string;
verifyBuildCtx(): boolean;
};
lease: PreparedOpenClawLegacyImage;
verifyImage: ReturnType<typeof vi.fn>;
retainForRecreate: ReturnType<typeof vi.fn>;
};

function createRetainedImageFixture(imageVerificationResults: boolean[]): RetainedImageFixture {
const buildCtx = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-buildx-boundary-"));
const stagedDockerfile = path.join(buildCtx, "Dockerfile");
fs.writeFileSync(stagedDockerfile, "FROM scratch\n");
const contextFingerprint = fingerprintBuildContext(buildCtx);
const verifyImage = vi.fn(() => imageVerificationResults.shift() ?? true);
const retainForRecreate = vi.fn(() => true);
const lease = Object.freeze({
dockerEnv: Object.freeze({ DOCKER_CONTEXT: "verified-builder" }),
engineId: "verified-engine",
imageRef: "nemoclaw-sandbox-local:alpha-rebuild-preflight",
imageId: `sha256:${"d".repeat(64)}`,
verify: verifyImage,
retainForRecreate,
verifyForCreate: vi.fn(() => true),
finalizeAfterCreate: vi.fn(() => ({
mutableTagVerified: true,
registryImageRef: null,
})),
abort: vi.fn(() => true),
dispose: vi.fn(() => true),
});
let buildContext: RetainedImageFixture["buildContext"];
buildContext = {
buildCtx,
stagedDockerfile,
cleanupBuildCtx: vi.fn(() => {
fs.rmSync(buildCtx, { recursive: true, force: true });
return true;
}),
origin: "generated",
buildId: "retained-build",
contextFingerprint,
preparedOpenClawLegacyImage: lease,
verifyBuildCtx(this: RetainedImageFixture["buildContext"]) {
return (
this === buildContext &&
this.preparedOpenClawLegacyImage === lease &&
fingerprintBuildContext(buildCtx) === contextFingerprint
);
},
rebuildTarget: {
agentName: null,
fromDockerfile: null,
},
};
return { buildContext, lease, verifyImage, retainForRecreate };
}

function sandboxDeleteCallOrder(runOpenshellSpy: ReturnType<typeof vi.fn>): number {
const deleteCallIndex = runOpenshellSpy.mock.calls.findIndex(
([args]) => Array.isArray(args) && args.join(" ") === "sandbox delete -g nemoclaw alpha",
);
expect(deleteCallIndex).toBeGreaterThanOrEqual(0);
return runOpenshellSpy.mock.invocationCallOrder[deleteCallIndex] ?? Number.POSITIVE_INFINITY;
}

describe("rebuildSandbox retained OpenClaw image mutation boundary", () => {
beforeEach(resetRebuildFlowTestEnvironment);
afterEach(restoreRebuildFlowTestEnvironment);

it("keeps the sandbox intact when the retained image drifts at the final delete edge (#7253)", async () => {
const fixture = createRetainedImageFixture([true, true, false]);
const harness = createRebuildFlowHarness({
rebuildImagePreflightResult: {
ok: true,
imageTag: fixture.lease.imageRef,
prepared: fixture.buildContext,
},
});

await expect(
harness.rebuildSandbox("alpha", ["--yes"], { throwOnError: true }),
).rejects.toThrow(
"The retained replacement image inputs changed before sandbox deletion. Retry the rebuild.",
);

expect(harness.preflightRebuildImageSpy).toHaveBeenCalledOnce();
expect(harness.backupSandboxStateSpy).toHaveBeenCalledOnce();
expect(fixture.verifyImage).toHaveBeenCalledTimes(3);
expect(fixture.retainForRecreate).not.toHaveBeenCalled();
expect(harness.reattachMcpProvidersAfterRebuildAbortSpy).toHaveBeenCalledOnce();
expect(harness.relockSpy).toHaveBeenCalled();
expectNoSandboxDelete(harness.runOpenshellSpy);
expect(harness.removeSandboxRegistryEntrySpy).not.toHaveBeenCalled();
expect(harness.onboardSpy).not.toHaveBeenCalled();
});

it("commits the exact retained image immediately before delete and never rebuilds it afterward (#7253)", async () => {
const fixture = createRetainedImageFixture([true, true, true]);
const harness = createRebuildFlowHarness({
rebuildImagePreflightResult: {
ok: true,
imageTag: fixture.lease.imageRef,
prepared: fixture.buildContext,
},
});

await expect(
harness.rebuildSandbox("alpha", ["--yes"], { throwOnError: true }),
).resolves.toBeUndefined();

expect(harness.preflightRebuildImageSpy).toHaveBeenCalledOnce();
expect(harness.backupSandboxStateSpy).toHaveBeenCalledOnce();
expect(fixture.verifyImage).toHaveBeenCalledTimes(3);
expect(fixture.retainForRecreate).toHaveBeenCalledOnce();
expect(fixture.verifyImage.mock.invocationCallOrder.at(-1)).toBeLessThan(
fixture.retainForRecreate.mock.invocationCallOrder[0] ?? Number.POSITIVE_INFINITY,
);
const deleteOrder = sandboxDeleteCallOrder(harness.runOpenshellSpy as never);
expect(fixture.retainForRecreate.mock.invocationCallOrder[0]).toBeLessThan(deleteOrder);
expect(deleteOrder).toBeLessThan(
harness.onboardSpy.mock.invocationCallOrder[0] ?? Number.POSITIVE_INFINITY,
);
expect(harness.removeSandboxRegistryEntrySpy).toHaveBeenCalledOnce();
expect(harness.onboardSpy).toHaveBeenCalledOnce();
});
});
Loading