fix(onboard): reconcile managed bootstrap after restart

src/lib/onboard/managed-bootstrap/README.md

@@ -91,20 +91,73 @@ sandbox identities, plan and profile
 fingerprints, exact original and replacement IDs, rollback target, and phase.
 Exact commit and cleanup receipts are durable terminal records, so adapter
 recreation does not depend on process-local transaction sets or tombstone maps.
+Rollback retains an `owner-cleanup-required` phase only after image-owned shared
+state is restored and the exact replacement is absent. That phase keeps the
+restored original quiescent and preserves the journal without a terminal
+receipt until the owning sandbox service removes the exact runtime and the
+provider proves its absence. Unknown runtime presence is a retryable durable
+cleanup failure, never evidence of absence.
 The image-owned shared-state transaction uses the same identity-bound model: a
 commit atomically moves its pending manifest and backups into a durable receipt
 namespace, compacts that state to an exact commit receipt, and rejects rollback
-when a later image-runtime invocation reads that receipt. The provider may
-retire that receipt only after it proves the external rollback backup is gone,
-so that this receipt does not block the next bootstrap attempt.
-Direct identity lookup reconstructs one known transaction record. The bounded
-[3.12b recovery slice](https://github.com/NVIDIA/NemoClaw/issues/7744) introduces
-unfinished-record enumeration together with phase reconciliation and
-cross-surface resume or rollback. The adapter reads mutable OpenShell names only
-to detect ownership reuse. Unsafe name-only deletion returns a typed retention
-error. The dormant adapter assumes the protocol's single coordinator;
-multi-process lease/arbitration remains an explicit production-activation gate.
-Activation must also inject the selected gateway's canonical state root.
+after a restart. The provider may retire that receipt only after it proves the
+external rollback backup is gone, leaving the next bootstrap attempt unblocked.
+Direct identity lookup reconstructs one known transaction record, while managed
+create-lifecycle startup uses unfinished-record enumeration to ask the selected
+provider to reconcile every identity-addressed record before a new sandbox
+create begins. The Docker provider then resumes the durable phase monotonically:
+staged work rolls back without entering cutover; cutover work follows a proven
+image-owned commit forward or durably authorizes rollback; rollback-authorized
+work completes exact restore and cleanup; and shared-state-committed work
+completes exact backup cleanup and commit. Recovery persists an identity-bound
+finalization receipt before removing the active journal, is idempotent across
+another interruption, and enumerates durable identities before loading each
+record so one unreadable transaction does not hide other results. The provider
+returns bounded `{ receipts, failures }` evidence; the coordinator validates,
+copies, freezes, and orders both arrays without routing on provider phases or
+failure codes. A failure for the requested sandbox name, or one whose sandbox
+identity cannot be proven, blocks create. An exact failure for another sandbox
+is warned and retained without blocking the requested create. The code reads
+mutable OpenShell names only to detect ownership reuse, and unsafe name-only
+deletion returns a typed retention error. Docker mutations use the previously
+journaled full container ID, whose identity cannot be rebound, then re-inspect
+that same ID after quiescence. Multi-process lease/arbitration remains an
+explicit production-activation gate. Activation must also inject the selected
+gateway's canonical state root.
+
+## Legacy journal drain (schema 1 and 2)
+
+Schema 1 and schema 2 journal bodies predate durable agent identity. They cannot
+be upgraded by guessing from a mutable sandbox name, image repository, or the
+agent selected by a later command. Recovery therefore preserves the canonical
+record and any decision sidecar, reports its exact bootstrap, provider, sandbox,
+original-runtime, and replacement-runtime identities, and fences only that
+sandbox name. A create for another sandbox may continue after warning about the
+retained record.
+
+When recovery reports one of these records:
+
+1. Stop onboarding the named sandbox. Save the complete diagnostic and back up
+   the canonical state root's
+   `managed-bootstrap/<bootstrap-identity>.json` file and any adjacent decision
+   sidecar without editing either record.
+2. Inspect the reported full runtime IDs through the owning provider. Treat
+   sandbox and container names as diagnostic text only. Never delete, rename,
+   or adopt a runtime by name, and never copy agent identity from the current
+   invocation into the old record.
+3. If either exact runtime is present, or its presence cannot be proven, leave
+   the journal in place and recover the provider-owned transaction using those
+   immutable IDs. A legacy cutover decision may be newer than the journal-body
+   phase, so the body alone never authorizes commit or rollback.
+4. If both exact runtimes are proven absent, still preserve the journal and its
+   image-owned shared-state evidence. Record the exact absence proof on
+   [epic #7744](https://github.com/NVIDIA/NemoClaw/issues/7744) for the
+   identity-checked retirement path. Until that path ships, use a different
+   sandbox name rather than deleting durable authority.
+
+Production activation must include the identity-checked retirement path and
+protected recovery qualification. This candidate remains inert, so it does not
+expose a runtime that could create these legacy records without that support.
 
 ## Architectural disposition
 
@@ -113,8 +166,8 @@ candidate Docker surface owns create routing, replacement construction,
 native-to-compatibility fallback evidence, and deferred commit or rollback.
 Central onboarding accepts that provider-neutral surface without a Docker or
 Podman selection branch. Tests register an MXC-style surface through the same
-bundle and render held launches for OpenClaw, Hermes, and LangChain Deep Agents
-Code.
+bundle, render held launches for OpenClaw, Hermes, and LangChain Deep Agents
+Code, and exercise recovery phases across all three agents.
 
 The coordinator remains the driver-neutral transaction authority: its receipt
 shapes, normalization, state transitions, and rollback proofs form one cohesive

src/lib/onboard/managed-bootstrap/adapter.test.ts

@@ -13,6 +13,7 @@ import {
 import { createManagedStartupRootApplyRequest } from "../managed-startup/root-apply";
 import {
   activateManagedBootstrapSequence,
+  enforceManagedBootstrapRecoveryForSandbox,
   finalizeManagedBootstrapSequence,
   MANAGED_BOOTSTRAP_SCHEMA_VERSION,
   type ManagedBootstrapAdapter,
@@ -23,8 +24,10 @@ import {
   type ManagedBootstrapHeldWorkloadHandle,
   type ManagedBootstrapObservedSnapshot,
   type ManagedBootstrapPreparedReplacementHandle,
+  ManagedBootstrapRecoveryBlockedError,
   type ManagedBootstrapReplacementHandle,
   prepareManagedBootstrapSequence,
+  recoverManagedBootstrapTransactions,
   renderManagedBootstrapHeldCommand,
 } from "./adapter";
 
@@ -227,6 +230,7 @@ function adapterFor(agent: ManagedStartupAgent): Fixture {
   const order: string[] = [];
   const raw: Fixture["raw"] = { handle: null, snapshot: null, prepared: null };
   const adapter: ManagedBootstrapAdapter = {
+    recoverUnfinishedTransactions: vi.fn(async () => ({ receipts: [], failures: [] })),
     createHeldWorkload: vi.fn(async (input) => {
       order.push("create");
       const receipt = await input.launch({
@@ -908,6 +912,184 @@ describe("managed bootstrap adapter contract", () => {
     expect(fixture.adapter.finalizeBootstrap).not.toHaveBeenCalled();
   });
 
+  it("normalizes, freezes, and orders provider-owned restart recovery receipts", async () => {
+    const fixture = adapterFor("openclaw");
+    const receipt = cleanupReceipt();
+    const candidate = (bootstrapIdentity: string) => ({
+      schemaVersion: MANAGED_BOOTSTRAP_SCHEMA_VERSION,
+      providerId: receipt.sandbox.driverId,
+      sourcePhase: "cutover",
+      sandbox: receipt.sandbox,
+      bootstrapIdentity,
+      outcome: "rolled-back" as const,
+      finalization: { ...receipt, bootstrapIdentity },
+    });
+    vi.mocked(fixture.adapter.recoverUnfinishedTransactions).mockResolvedValueOnce({
+      receipts: [candidate("b".repeat(64)), candidate("a".repeat(64))],
+      failures: [],
+    });
+
+    const recovered = await recoverManagedBootstrapTransactions(fixture.adapter);
+
+    expect(recovered.receipts.map(({ bootstrapIdentity }) => bootstrapIdentity)).toEqual([
+      "a".repeat(64),
+      "b".repeat(64),
+    ]);
+    expect(Object.isFrozen(recovered)).toBe(true);
+    expect(Object.isFrozen(recovered.receipts)).toBe(true);
+    expect(recovered.receipts.every((entry) => Object.isFrozen(entry.finalization))).toBe(true);
+  });
+
+  it("rejects recovery evidence whose provider does not own the durable sandbox", async () => {
+    const fixture = adapterFor("openclaw");
+    const receipt = cleanupReceipt();
+    vi.mocked(fixture.adapter.recoverUnfinishedTransactions).mockResolvedValueOnce({
+      receipts: [
+        {
+          schemaVersion: MANAGED_BOOTSTRAP_SCHEMA_VERSION,
+          providerId: "mxc",
+          sourcePhase: "cutover",
+          sandbox: receipt.sandbox,
+          bootstrapIdentity: IDENTITY,
+          outcome: "rolled-back",
+          finalization: receipt,
+        },
+      ],
+      failures: [],
+    });
+
+    await expect(recoverManagedBootstrapTransactions(fixture.adapter)).rejects.toThrow(
+      "recovery provider does not own",
+    );
+  });
+
+  it("normalizes provider-neutral failures and preserves bounded MXC-style diagnostics", async () => {
+    const fixture = adapterFor("hermes");
+    const failure = (bootstrapIdentity: string, sandboxName: string | null) => ({
+      schemaVersion: MANAGED_BOOTSTRAP_SCHEMA_VERSION,
+      providerId: "mxc",
+      sourcePhase: "provider-owned-cleanup",
+      sandbox:
+        sandboxName === null
+          ? null
+          : { sandboxName, sandboxId: `mxc-${sandboxName}`, driverId: "mxc" },
+      bootstrapIdentity,
+      code: "provider-owned-retry",
+      retryable: true,
+      detail: "opaque MXC recovery evidence",
+    });
+    vi.mocked(fixture.adapter.recoverUnfinishedTransactions).mockResolvedValueOnce({
+      receipts: [],
+      failures: [failure("b".repeat(64), "bravo"), failure("a".repeat(64), null)],
+    });
+
+    const recovered = await recoverManagedBootstrapTransactions(fixture.adapter);
+
+    expect(recovered.failures.map(({ bootstrapIdentity }) => bootstrapIdentity)).toEqual([
+      "a".repeat(64),
+      "b".repeat(64),
+    ]);
+    expect(recovered.failures[0]).toMatchObject({ sandbox: null, providerId: "mxc" });
+    expect(Object.isFrozen(recovered.failures)).toBe(true);
+    expect(recovered.failures.every(Object.isFrozen)).toBe(true);
+  });
+
+  it("rejects duplicate identities across recovered receipts and failures", async () => {
+    const fixture = adapterFor("openclaw");
+    const receipt = cleanupReceipt();
+    vi.mocked(fixture.adapter.recoverUnfinishedTransactions).mockResolvedValueOnce({
+      receipts: [
+        {
+          schemaVersion: MANAGED_BOOTSTRAP_SCHEMA_VERSION,
+          providerId: receipt.sandbox.driverId,
+          sourcePhase: "cutover",
+          sandbox: receipt.sandbox,
+          bootstrapIdentity: IDENTITY,
+          outcome: "rolled-back",
+          finalization: receipt,
+        },
+      ],
+      failures: [
+        {
+          schemaVersion: MANAGED_BOOTSTRAP_SCHEMA_VERSION,
+          providerId: receipt.sandbox.driverId,
+          sourcePhase: "cleanup",
+          sandbox: receipt.sandbox,
+          bootstrapIdentity: IDENTITY,
+          code: "retry",
+          retryable: true,
+          detail: "retained",
+        },
+      ],
+    });
+
+    await expect(recoverManagedBootstrapTransactions(fixture.adapter)).rejects.toThrow(
+      "duplicate bootstrap identities",
+    );
+  });
+
+  it("rejects an unbounded provider recovery result before normalizing records", async () => {
+    const fixture = adapterFor("hermes");
+    const candidate = {
+      schemaVersion: MANAGED_BOOTSTRAP_SCHEMA_VERSION,
+      providerId: "mxc",
+      sourcePhase: "provider-owned-cleanup",
+      sandbox: null,
+      bootstrapIdentity: IDENTITY,
+      code: "provider-owned-retry",
+      retryable: true,
+      detail: "opaque MXC recovery evidence",
+    } as const;
+    vi.mocked(fixture.adapter.recoverUnfinishedTransactions).mockResolvedValueOnce({
+      receipts: [],
+      failures: Array.from({ length: 4097 }, () => candidate),
+    });
+
+    await expect(recoverManagedBootstrapTransactions(fixture.adapter)).rejects.toThrow(
+      "provider recovery returned too many records",
+    );
+  });
+
+  it("blocks same-name and identity-unknown failures while warning for unrelated sandboxes", () => {
+    const failure = (bootstrapIdentity: string, sandboxName: string | null) =>
+      Object.freeze({
+        schemaVersion: MANAGED_BOOTSTRAP_SCHEMA_VERSION,
+        providerId: "mxc",
+        sourcePhase: "cleanup",
+        sandbox:
+          sandboxName === null
+            ? null
+            : Object.freeze({ sandboxName, sandboxId: `mxc-${sandboxName}`, driverId: "mxc" }),
+        bootstrapIdentity,
+        code: "provider-owned-retry",
+        retryable: true,
+        detail: "opaque provider detail",
+      });
+    const warn = vi.fn();
+    const unrelated = failure("a".repeat(64), "bravo");
+    const sameName = failure("b".repeat(64), "alpha");
+    const identityUnknown = failure("c".repeat(64), null);
+
+    expect(
+      enforceManagedBootstrapRecoveryForSandbox(
+        Object.freeze({ receipts: Object.freeze([]), failures: Object.freeze([unrelated]) }),
+        "alpha",
+        warn,
+      ),
+    ).toMatchObject({ failures: [unrelated] });
+    expect(warn).toHaveBeenCalledWith(expect.stringContaining("unrelated sandbox 'bravo'"));
+
+    for (const blocking of [sameName, identityUnknown]) {
+      expect(() =>
+        enforceManagedBootstrapRecoveryForSandbox(
+          Object.freeze({ receipts: Object.freeze([]), failures: Object.freeze([blocking]) }),
+          "alpha",
+          warn,
+        ),
+      ).toThrow(ManagedBootstrapRecoveryBlockedError);
+    }
+  });
+
   it.each([
     "BASHOPTS=extdebug",
     "BASH_ENV=/sandbox/attacker",