diff --git a/src/lib/onboard/managed-bootstrap/README.md b/src/lib/onboard/managed-bootstrap/README.md index b5f8434f79f..fafd40611d3 100644 --- a/src/lib/onboard/managed-bootstrap/README.md +++ b/src/lib/onboard/managed-bootstrap/README.md @@ -91,20 +91,73 @@ sandbox identities, plan and profile fingerprints, exact original and replacement IDs, rollback target, and phase. Exact commit and cleanup receipts are durable terminal records, so adapter recreation does not depend on process-local transaction sets or tombstone maps. +Rollback retains an `owner-cleanup-required` phase only after image-owned shared +state is restored and the exact replacement is absent. That phase keeps the +restored original quiescent and preserves the journal without a terminal +receipt until the owning sandbox service removes the exact runtime and the +provider proves its absence. Unknown runtime presence is a retryable durable +cleanup failure, never evidence of absence. The image-owned shared-state transaction uses the same identity-bound model: a commit atomically moves its pending manifest and backups into a durable receipt namespace, compacts that state to an exact commit receipt, and rejects rollback -when a later image-runtime invocation reads that receipt. The provider may -retire that receipt only after it proves the external rollback backup is gone, -so that this receipt does not block the next bootstrap attempt. -Direct identity lookup reconstructs one known transaction record. The bounded -[3.12b recovery slice](https://github.com/NVIDIA/NemoClaw/issues/7744) introduces -unfinished-record enumeration together with phase reconciliation and -cross-surface resume or rollback. The adapter reads mutable OpenShell names only -to detect ownership reuse. Unsafe name-only deletion returns a typed retention -error. The dormant adapter assumes the protocol's single coordinator; -multi-process lease/arbitration remains an explicit production-activation gate. -Activation must also inject the selected gateway's canonical state root. +after a restart. The provider may retire that receipt only after it proves the +external rollback backup is gone, leaving the next bootstrap attempt unblocked. +Direct identity lookup reconstructs one known transaction record, while managed +create-lifecycle startup uses unfinished-record enumeration to ask the selected +provider to reconcile every identity-addressed record before a new sandbox +create begins. The Docker provider then resumes the durable phase monotonically: +staged work rolls back without entering cutover; cutover work follows a proven +image-owned commit forward or durably authorizes rollback; rollback-authorized +work completes exact restore and cleanup; and shared-state-committed work +completes exact backup cleanup and commit. Recovery persists an identity-bound +finalization receipt before removing the active journal, is idempotent across +another interruption, and enumerates durable identities before loading each +record so one unreadable transaction does not hide other results. The provider +returns bounded `{ receipts, failures }` evidence; the coordinator validates, +copies, freezes, and orders both arrays without routing on provider phases or +failure codes. A failure for the requested sandbox name, or one whose sandbox +identity cannot be proven, blocks create. An exact failure for another sandbox +is warned and retained without blocking the requested create. The code reads +mutable OpenShell names only to detect ownership reuse, and unsafe name-only +deletion returns a typed retention error. Docker mutations use the previously +journaled full container ID, whose identity cannot be rebound, then re-inspect +that same ID after quiescence. Multi-process lease/arbitration remains an +explicit production-activation gate. Activation must also inject the selected +gateway's canonical state root. + +## Legacy journal drain (schema 1 and 2) + +Schema 1 and schema 2 journal bodies predate durable agent identity. They cannot +be upgraded by guessing from a mutable sandbox name, image repository, or the +agent selected by a later command. Recovery therefore preserves the canonical +record and any decision sidecar, reports its exact bootstrap, provider, sandbox, +original-runtime, and replacement-runtime identities, and fences only that +sandbox name. A create for another sandbox may continue after warning about the +retained record. + +When recovery reports one of these records: + +1. Stop onboarding the named sandbox. Save the complete diagnostic and back up + the canonical state root's + `managed-bootstrap/.json` file and any adjacent decision + sidecar without editing either record. +2. Inspect the reported full runtime IDs through the owning provider. Treat + sandbox and container names as diagnostic text only. Never delete, rename, + or adopt a runtime by name, and never copy agent identity from the current + invocation into the old record. +3. If either exact runtime is present, or its presence cannot be proven, leave + the journal in place and recover the provider-owned transaction using those + immutable IDs. A legacy cutover decision may be newer than the journal-body + phase, so the body alone never authorizes commit or rollback. +4. If both exact runtimes are proven absent, still preserve the journal and its + image-owned shared-state evidence. Record the exact absence proof on + [epic #7744](https://github.com/NVIDIA/NemoClaw/issues/7744) for the + identity-checked retirement path. Until that path ships, use a different + sandbox name rather than deleting durable authority. + +Production activation must include the identity-checked retirement path and +protected recovery qualification. This candidate remains inert, so it does not +expose a runtime that could create these legacy records without that support. ## Architectural disposition @@ -113,8 +166,8 @@ candidate Docker surface owns create routing, replacement construction, native-to-compatibility fallback evidence, and deferred commit or rollback. Central onboarding accepts that provider-neutral surface without a Docker or Podman selection branch. Tests register an MXC-style surface through the same -bundle and render held launches for OpenClaw, Hermes, and LangChain Deep Agents -Code. +bundle, render held launches for OpenClaw, Hermes, and LangChain Deep Agents +Code, and exercise recovery phases across all three agents. The coordinator remains the driver-neutral transaction authority: its receipt shapes, normalization, state transitions, and rollback proofs form one cohesive diff --git a/src/lib/onboard/managed-bootstrap/adapter.test.ts b/src/lib/onboard/managed-bootstrap/adapter.test.ts index 9968171289d..5d86254de2b 100644 --- a/src/lib/onboard/managed-bootstrap/adapter.test.ts +++ b/src/lib/onboard/managed-bootstrap/adapter.test.ts @@ -13,6 +13,7 @@ import { import { createManagedStartupRootApplyRequest } from "../managed-startup/root-apply"; import { activateManagedBootstrapSequence, + enforceManagedBootstrapRecoveryForSandbox, finalizeManagedBootstrapSequence, MANAGED_BOOTSTRAP_SCHEMA_VERSION, type ManagedBootstrapAdapter, @@ -23,8 +24,10 @@ import { type ManagedBootstrapHeldWorkloadHandle, type ManagedBootstrapObservedSnapshot, type ManagedBootstrapPreparedReplacementHandle, + ManagedBootstrapRecoveryBlockedError, type ManagedBootstrapReplacementHandle, prepareManagedBootstrapSequence, + recoverManagedBootstrapTransactions, renderManagedBootstrapHeldCommand, } from "./adapter"; @@ -227,6 +230,7 @@ function adapterFor(agent: ManagedStartupAgent): Fixture { const order: string[] = []; const raw: Fixture["raw"] = { handle: null, snapshot: null, prepared: null }; const adapter: ManagedBootstrapAdapter = { + recoverUnfinishedTransactions: vi.fn(async () => ({ receipts: [], failures: [] })), createHeldWorkload: vi.fn(async (input) => { order.push("create"); const receipt = await input.launch({ @@ -908,6 +912,184 @@ describe("managed bootstrap adapter contract", () => { expect(fixture.adapter.finalizeBootstrap).not.toHaveBeenCalled(); }); + it("normalizes, freezes, and orders provider-owned restart recovery receipts", async () => { + const fixture = adapterFor("openclaw"); + const receipt = cleanupReceipt(); + const candidate = (bootstrapIdentity: string) => ({ + schemaVersion: MANAGED_BOOTSTRAP_SCHEMA_VERSION, + providerId: receipt.sandbox.driverId, + sourcePhase: "cutover", + sandbox: receipt.sandbox, + bootstrapIdentity, + outcome: "rolled-back" as const, + finalization: { ...receipt, bootstrapIdentity }, + }); + vi.mocked(fixture.adapter.recoverUnfinishedTransactions).mockResolvedValueOnce({ + receipts: [candidate("b".repeat(64)), candidate("a".repeat(64))], + failures: [], + }); + + const recovered = await recoverManagedBootstrapTransactions(fixture.adapter); + + expect(recovered.receipts.map(({ bootstrapIdentity }) => bootstrapIdentity)).toEqual([ + "a".repeat(64), + "b".repeat(64), + ]); + expect(Object.isFrozen(recovered)).toBe(true); + expect(Object.isFrozen(recovered.receipts)).toBe(true); + expect(recovered.receipts.every((entry) => Object.isFrozen(entry.finalization))).toBe(true); + }); + + it("rejects recovery evidence whose provider does not own the durable sandbox", async () => { + const fixture = adapterFor("openclaw"); + const receipt = cleanupReceipt(); + vi.mocked(fixture.adapter.recoverUnfinishedTransactions).mockResolvedValueOnce({ + receipts: [ + { + schemaVersion: MANAGED_BOOTSTRAP_SCHEMA_VERSION, + providerId: "mxc", + sourcePhase: "cutover", + sandbox: receipt.sandbox, + bootstrapIdentity: IDENTITY, + outcome: "rolled-back", + finalization: receipt, + }, + ], + failures: [], + }); + + await expect(recoverManagedBootstrapTransactions(fixture.adapter)).rejects.toThrow( + "recovery provider does not own", + ); + }); + + it("normalizes provider-neutral failures and preserves bounded MXC-style diagnostics", async () => { + const fixture = adapterFor("hermes"); + const failure = (bootstrapIdentity: string, sandboxName: string | null) => ({ + schemaVersion: MANAGED_BOOTSTRAP_SCHEMA_VERSION, + providerId: "mxc", + sourcePhase: "provider-owned-cleanup", + sandbox: + sandboxName === null + ? null + : { sandboxName, sandboxId: `mxc-${sandboxName}`, driverId: "mxc" }, + bootstrapIdentity, + code: "provider-owned-retry", + retryable: true, + detail: "opaque MXC recovery evidence", + }); + vi.mocked(fixture.adapter.recoverUnfinishedTransactions).mockResolvedValueOnce({ + receipts: [], + failures: [failure("b".repeat(64), "bravo"), failure("a".repeat(64), null)], + }); + + const recovered = await recoverManagedBootstrapTransactions(fixture.adapter); + + expect(recovered.failures.map(({ bootstrapIdentity }) => bootstrapIdentity)).toEqual([ + "a".repeat(64), + "b".repeat(64), + ]); + expect(recovered.failures[0]).toMatchObject({ sandbox: null, providerId: "mxc" }); + expect(Object.isFrozen(recovered.failures)).toBe(true); + expect(recovered.failures.every(Object.isFrozen)).toBe(true); + }); + + it("rejects duplicate identities across recovered receipts and failures", async () => { + const fixture = adapterFor("openclaw"); + const receipt = cleanupReceipt(); + vi.mocked(fixture.adapter.recoverUnfinishedTransactions).mockResolvedValueOnce({ + receipts: [ + { + schemaVersion: MANAGED_BOOTSTRAP_SCHEMA_VERSION, + providerId: receipt.sandbox.driverId, + sourcePhase: "cutover", + sandbox: receipt.sandbox, + bootstrapIdentity: IDENTITY, + outcome: "rolled-back", + finalization: receipt, + }, + ], + failures: [ + { + schemaVersion: MANAGED_BOOTSTRAP_SCHEMA_VERSION, + providerId: receipt.sandbox.driverId, + sourcePhase: "cleanup", + sandbox: receipt.sandbox, + bootstrapIdentity: IDENTITY, + code: "retry", + retryable: true, + detail: "retained", + }, + ], + }); + + await expect(recoverManagedBootstrapTransactions(fixture.adapter)).rejects.toThrow( + "duplicate bootstrap identities", + ); + }); + + it("rejects an unbounded provider recovery result before normalizing records", async () => { + const fixture = adapterFor("hermes"); + const candidate = { + schemaVersion: MANAGED_BOOTSTRAP_SCHEMA_VERSION, + providerId: "mxc", + sourcePhase: "provider-owned-cleanup", + sandbox: null, + bootstrapIdentity: IDENTITY, + code: "provider-owned-retry", + retryable: true, + detail: "opaque MXC recovery evidence", + } as const; + vi.mocked(fixture.adapter.recoverUnfinishedTransactions).mockResolvedValueOnce({ + receipts: [], + failures: Array.from({ length: 4097 }, () => candidate), + }); + + await expect(recoverManagedBootstrapTransactions(fixture.adapter)).rejects.toThrow( + "provider recovery returned too many records", + ); + }); + + it("blocks same-name and identity-unknown failures while warning for unrelated sandboxes", () => { + const failure = (bootstrapIdentity: string, sandboxName: string | null) => + Object.freeze({ + schemaVersion: MANAGED_BOOTSTRAP_SCHEMA_VERSION, + providerId: "mxc", + sourcePhase: "cleanup", + sandbox: + sandboxName === null + ? null + : Object.freeze({ sandboxName, sandboxId: `mxc-${sandboxName}`, driverId: "mxc" }), + bootstrapIdentity, + code: "provider-owned-retry", + retryable: true, + detail: "opaque provider detail", + }); + const warn = vi.fn(); + const unrelated = failure("a".repeat(64), "bravo"); + const sameName = failure("b".repeat(64), "alpha"); + const identityUnknown = failure("c".repeat(64), null); + + expect( + enforceManagedBootstrapRecoveryForSandbox( + Object.freeze({ receipts: Object.freeze([]), failures: Object.freeze([unrelated]) }), + "alpha", + warn, + ), + ).toMatchObject({ failures: [unrelated] }); + expect(warn).toHaveBeenCalledWith(expect.stringContaining("unrelated sandbox 'bravo'")); + + for (const blocking of [sameName, identityUnknown]) { + expect(() => + enforceManagedBootstrapRecoveryForSandbox( + Object.freeze({ receipts: Object.freeze([]), failures: Object.freeze([blocking]) }), + "alpha", + warn, + ), + ).toThrow(ManagedBootstrapRecoveryBlockedError); + } + }); + it.each([ "BASHOPTS=extdebug", "BASH_ENV=/sandbox/attacker", diff --git a/src/lib/onboard/managed-bootstrap/adapter.ts b/src/lib/onboard/managed-bootstrap/adapter.ts index ee1a834985f..bbf8df337e0 100644 --- a/src/lib/onboard/managed-bootstrap/adapter.ts +++ b/src/lib/onboard/managed-bootstrap/adapter.ts @@ -18,6 +18,7 @@ export const MANAGED_BOOTSTRAP_IDENTITY_BYTES = 32; const SHA256_RE = /^[a-f0-9]{64}$/u; const MANIFEST_DIGEST_RE = /^sha256:[a-f0-9]{64}$/u; const ENV_ASSIGNMENT_RE = /^[A-Za-z_][A-Za-z0-9_]*=/u; +const MAX_MANAGED_BOOTSTRAP_RECOVERY_RECORDS = 4096; const PROCESS_INJECTION_ENV_KEYS = new Set([ "BASHOPTS", "BASH_ENV", @@ -239,6 +240,42 @@ export interface ManagedBootstrapFinalizationReceipt { readonly finalizedAt: string; } +/** + * Driver-neutral evidence that one durable, process-orphaned transaction was + * reconciled without reconstructing authority from mutable runtime names. + */ +export interface ManagedBootstrapRecoveryReceipt { + readonly schemaVersion: typeof MANAGED_BOOTSTRAP_SCHEMA_VERSION; + readonly providerId: string; + /** Provider-owned phase name retained for diagnostics, never central routing. */ + readonly sourcePhase: string; + readonly sandbox: ManagedBootstrapSandboxIdentity; + readonly bootstrapIdentity: string; + readonly outcome: "committed" | "rolled-back"; + readonly finalization: ManagedBootstrapFinalizationReceipt; +} + +/** Bounded provider-owned evidence that one durable transaction still needs attention. */ +export interface ManagedBootstrapRecoveryFailure { + readonly schemaVersion: typeof MANAGED_BOOTSTRAP_SCHEMA_VERSION; + readonly providerId: string; + /** Null when the durable record could not prove its provider-owned phase. */ + readonly sourcePhase: string | null; + /** Null when the durable record could not prove its sandbox identity. */ + readonly sandbox: ManagedBootstrapSandboxIdentity | null; + readonly bootstrapIdentity: string; + /** Provider-owned diagnostic code. Central orchestration must not branch on this value. */ + readonly code: string; + readonly retryable: boolean; + readonly detail: string; +} + +/** Lossless provider-neutral recovery output for one bounded enumeration pass. */ +export interface ManagedBootstrapRecoveryReport { + readonly receipts: readonly ManagedBootstrapRecoveryReceipt[]; + readonly failures: readonly ManagedBootstrapRecoveryFailure[]; +} + export class ManagedBootstrapDurableCommitCleanupPendingError extends Error { readonly bootstrapIdentity: string; readonly cleanupRuntimeId: string; @@ -297,6 +334,25 @@ export class ManagedBootstrapOwnerCleanupRequiredError extends Error { } } +export class ManagedBootstrapRecoveryBlockedError extends Error { + readonly sandboxName: string; + readonly failures: readonly ManagedBootstrapRecoveryFailure[]; + + constructor(sandboxName: string, failures: readonly ManagedBootstrapRecoveryFailure[]) { + const first = failures[0]; + super( + `Managed bootstrap recovery blocks sandbox '${sandboxName}' because ${String( + failures.length, + )} durable transaction${failures.length === 1 ? "" : "s"} still need attention.${ + first ? ` First failure ${first.bootstrapIdentity} (${first.code}): ${first.detail}` : "" + }`, + ); + this.name = "ManagedBootstrapRecoveryBlockedError"; + this.sandboxName = sandboxName; + this.failures = Object.freeze([...failures]); + } +} + export function attachManagedBootstrapRollbackError(failure: Error, rollbackError: unknown): void { ( failure as Error & { @@ -310,6 +366,12 @@ export function attachManagedBootstrapRollbackError(failure: Error, rollbackErro } export interface ManagedBootstrapAdapter { + /** + * Enumerate durable unfinished records and reconcile each through the owning + * provider. Implementations must be restart-safe and idempotent. + */ + recoverUnfinishedTransactions(): Promise; + /** Return only after one durable sandbox/driver identity reports Ready. */ createHeldWorkload( input: ManagedBootstrapCreateInput, @@ -376,6 +438,172 @@ export interface ManagedBootstrapAdapter { }): Promise; } +function normalizeRecoveryReceipt( + candidate: ManagedBootstrapRecoveryReceipt, +): ManagedBootstrapRecoveryReceipt { + if ( + typeof candidate !== "object" || + candidate === null || + Array.isArray(candidate) || + candidate.schemaVersion !== MANAGED_BOOTSTRAP_SCHEMA_VERSION || + !["committed", "rolled-back"].includes(String(candidate.outcome)) + ) { + protocolFail("recovery receipt has an invalid schema or outcome"); + } + assertOpaqueString(candidate.providerId, "recovery provider ID"); + assertOpaqueString(candidate.sourcePhase, "recovery source phase"); + assertSandboxIdentity(candidate.sandbox); + if (candidate.sandbox.driverId !== candidate.providerId) { + protocolFail("recovery provider does not own the recovered sandbox"); + } + if (!SHA256_RE.test(candidate.bootstrapIdentity)) { + protocolFail("recovery bootstrap identity must be lowercase SHA-256"); + } + const finalization = candidate.finalization; + if ( + typeof finalization !== "object" || + finalization === null || + Array.isArray(finalization) || + finalization.schemaVersion !== MANAGED_BOOTSTRAP_SCHEMA_VERSION || + finalization.outcome !== candidate.outcome || + finalization.bootstrapIdentity !== candidate.bootstrapIdentity || + !isDeepStrictEqual(finalization.sandbox, candidate.sandbox) || + typeof finalization.heldWorkloadRemoved !== "boolean" || + typeof finalization.alreadyRolledBack !== "boolean" + ) { + protocolFail("recovery finalization does not match its durable identity"); + } + if ( + (finalization.restoredRuntimeId !== null && !SHA256_RE.test(finalization.restoredRuntimeId)) || + (finalization.restoredSpecHash !== null && !SHA256_RE.test(finalization.restoredSpecHash)) || + (finalization.restoredRuntimeId === null) !== (finalization.restoredSpecHash === null) || + (candidate.outcome === "committed" && + (finalization.restoredRuntimeId !== null || + finalization.heldWorkloadRemoved || + finalization.alreadyRolledBack)) + ) { + protocolFail("recovery finalization state is inconsistent"); + } + assertTimestamp(finalization.finalizedAt, "recovery finalization timestamp"); + return Object.freeze({ + schemaVersion: MANAGED_BOOTSTRAP_SCHEMA_VERSION, + providerId: candidate.providerId, + sourcePhase: candidate.sourcePhase, + sandbox: Object.freeze({ ...candidate.sandbox }), + bootstrapIdentity: candidate.bootstrapIdentity, + outcome: candidate.outcome, + finalization: Object.freeze({ + ...finalization, + sandbox: Object.freeze({ ...candidate.sandbox }), + }), + }); +} + +function normalizeRecoveryFailure( + candidate: ManagedBootstrapRecoveryFailure, +): ManagedBootstrapRecoveryFailure { + if ( + typeof candidate !== "object" || + candidate === null || + Array.isArray(candidate) || + candidate.schemaVersion !== MANAGED_BOOTSTRAP_SCHEMA_VERSION || + typeof candidate.retryable !== "boolean" + ) { + protocolFail("recovery failure has an invalid schema"); + } + assertOpaqueString(candidate.providerId, "recovery failure provider ID", 256); + if (candidate.sourcePhase !== null) { + assertOpaqueString(candidate.sourcePhase, "recovery failure source phase", 256); + } + if (candidate.sandbox !== null) { + assertSandboxIdentity(candidate.sandbox); + if (candidate.sandbox.driverId !== candidate.providerId) { + protocolFail("recovery failure provider does not own the durable sandbox"); + } + } + if (!SHA256_RE.test(candidate.bootstrapIdentity)) { + protocolFail("recovery failure bootstrap identity must be lowercase SHA-256"); + } + assertOpaqueString(candidate.code, "recovery failure code", 256); + assertOpaqueString(candidate.detail, "recovery failure detail", 8 * 1024); + return Object.freeze({ + schemaVersion: MANAGED_BOOTSTRAP_SCHEMA_VERSION, + providerId: candidate.providerId, + sourcePhase: candidate.sourcePhase, + sandbox: candidate.sandbox === null ? null : Object.freeze({ ...candidate.sandbox }), + bootstrapIdentity: candidate.bootstrapIdentity, + code: candidate.code, + retryable: candidate.retryable, + detail: candidate.detail, + }); +} + +/** Recover process-orphaned work without relying on coordinator WeakMap state. */ +export async function recoverManagedBootstrapTransactions( + adapter: ManagedBootstrapAdapter, +): Promise { + const candidates = await adapter.recoverUnfinishedTransactions(); + if ( + typeof candidates !== "object" || + candidates === null || + Array.isArray(candidates) || + !Array.isArray(candidates.receipts) || + !Array.isArray(candidates.failures) + ) { + protocolFail("provider recovery must return bounded receipt and failure arrays"); + } + if ( + candidates.receipts.length + candidates.failures.length > + MAX_MANAGED_BOOTSTRAP_RECOVERY_RECORDS + ) { + protocolFail("provider recovery returned too many records"); + } + const receipts = candidates.receipts.map(normalizeRecoveryReceipt); + const failures = candidates.failures.map(normalizeRecoveryFailure); + const identities = [...receipts, ...failures].map(({ bootstrapIdentity }) => bootstrapIdentity); + if (new Set(identities).size !== identities.length) { + protocolFail("provider recovery returned duplicate bootstrap identities"); + } + const byBootstrapIdentity = ( + left: ManagedBootstrapRecoveryReceipt | ManagedBootstrapRecoveryFailure, + right: ManagedBootstrapRecoveryReceipt | ManagedBootstrapRecoveryFailure, + ) => left.bootstrapIdentity.localeCompare(right.bootstrapIdentity); + return Object.freeze({ + receipts: Object.freeze([...receipts].sort(byBootstrapIdentity)), + failures: Object.freeze([...failures].sort(byBootstrapIdentity)), + }); +} + +/** Block only failures that can own the requested name; warn for exact unrelated sandboxes. */ +export function enforceManagedBootstrapRecoveryForSandbox( + report: ManagedBootstrapRecoveryReport, + sandboxName: string, + warn: (message: string) => void, +): ManagedBootstrapRecoveryReport { + assertOpaqueString(sandboxName, "recovery target sandbox name"); + const blocking = report.failures.filter( + (failure) => failure.sandbox === null || failure.sandbox.sandboxName === sandboxName, + ); + for (const failure of report.failures) { + if (failure.sandbox === null || failure.sandbox.sandboxName === sandboxName) continue; + warn( + `Managed bootstrap recovery retained unrelated sandbox '${failure.sandbox.sandboxName}' ` + + `(${failure.bootstrapIdentity}, ${failure.code}).`, + ); + } + if (blocking.length > 0) { + throw new ManagedBootstrapRecoveryBlockedError( + sandboxName, + Object.freeze( + [...blocking].sort((left, right) => + left.bootstrapIdentity.localeCompare(right.bootstrapIdentity), + ), + ), + ); + } + return report; +} + export interface ManagedBootstrapPreparationInput { readonly create: ManagedBootstrapCreateInput; readonly request: ManagedStartupRootApplyRequest; @@ -404,12 +632,16 @@ function protocolFail(message: string): never { throw new Error(`Managed bootstrap protocol violation: ${message}`); } -function assertOpaqueString(value: unknown, label: string): asserts value is string { +function assertOpaqueString( + value: unknown, + label: string, + maxBytes = 64 * 1024, +): asserts value is string { if ( typeof value !== "string" || value.length === 0 || value.includes("\0") || - Buffer.byteLength(value, "utf8") > 64 * 1024 + Buffer.byteLength(value, "utf8") > maxBytes ) { protocolFail(`${label} must be one bounded non-empty string`); } diff --git a/src/lib/onboard/managed-bootstrap/docker-journal.test.ts b/src/lib/onboard/managed-bootstrap/docker-journal.test.ts index 25cd9ec0e50..457e9b4e154 100644 --- a/src/lib/onboard/managed-bootstrap/docker-journal.test.ts +++ b/src/lib/onboard/managed-bootstrap/docker-journal.test.ts @@ -12,8 +12,12 @@ import { DOCKER_MANAGED_BOOTSTRAP_FINALIZATION_SCHEMA_VERSION, DOCKER_MANAGED_BOOTSTRAP_JOURNAL_DIRECTORY, DOCKER_MANAGED_BOOTSTRAP_JOURNAL_SCHEMA_VERSION, + type DockerManagedBootstrapFinalizationContext, type DockerManagedBootstrapFinalizationRecord, type DockerManagedBootstrapJournal, + type DockerManagedBootstrapJournalStore, + DockerManagedBootstrapLegacyRecordRequiresAgentError, + normalizeDockerManagedBootstrapJournal, parseDockerManagedBootstrapFinalizationRecord, parseDockerManagedBootstrapJournal, sameDockerManagedBootstrapReceipt, @@ -23,11 +27,23 @@ import { const roots: string[] = []; const IDENTITY = "1".repeat(64); +const OTHER_IDENTITY = "0".repeat(64); + +function loadUnfinished( + store: DockerManagedBootstrapJournalStore, +): readonly DockerManagedBootstrapJournal[] { + return store.listUnfinishedIdentities().map((identity) => { + const record = store.load(identity); + expect(record, `enumerated journal ${identity} must remain loadable`).not.toBeNull(); + return record as DockerManagedBootstrapJournal; + }); +} const journal = Object.freeze({ schemaVersion: DOCKER_MANAGED_BOOTSTRAP_JOURNAL_SCHEMA_VERSION, phase: "staged", bootstrapIdentity: IDENTITY, providerId: "docker", + agent: "hermes", sandbox: { sandboxName: "alpha", sandboxId: "sandbox-alpha", @@ -65,6 +81,7 @@ const finalization = Object.freeze({ phase: "committed", bootstrapIdentity: IDENTITY, providerId: "docker", + agent: journal.agent, sandbox: journal.sandbox, planFingerprint: journal.planFingerprint, profileFingerprint: journal.profileFingerprint, @@ -98,6 +115,75 @@ const finalization = Object.freeze({ }, } satisfies DockerManagedBootstrapFinalizationRecord); +const finalizationContext = Object.freeze({ + bootstrapIdentity: finalization.bootstrapIdentity, + providerId: finalization.providerId, + agent: finalization.agent, + sandbox: finalization.sandbox, + planFingerprint: finalization.planFingerprint, + profileFingerprint: finalization.profileFingerprint, + imageReference: finalization.imageReference, +} satisfies DockerManagedBootstrapFinalizationContext); + +function legacyJournalV1() { + return Object.freeze({ + schemaVersion: 1 as const, + phase: journal.phase, + bootstrapIdentity: journal.bootstrapIdentity, + sandbox: journal.sandbox, + profileFingerprint: journal.profileFingerprint, + imageReference: journal.imageReference, + runtimeImageContentId: journal.runtimeImageContentId, + originalRuntimeId: journal.originalRuntimeId, + replacementRuntimeId: journal.replacementRuntimeId, + originalName: journal.originalName, + replacementStagingName: journal.replacementStagingName, + backupName: journal.backupName, + originalSpecHash: journal.originalSpecHash, + replacementSpecHash: journal.replacementSpecHash, + }); +} + +function legacyJournalV2() { + return Object.freeze({ + schemaVersion: 2 as const, + phase: journal.phase, + bootstrapIdentity: journal.bootstrapIdentity, + providerId: journal.providerId, + sandbox: journal.sandbox, + planFingerprint: journal.planFingerprint, + profileFingerprint: journal.profileFingerprint, + imageReference: journal.imageReference, + runtimeImageContentId: journal.runtimeImageContentId, + originalRuntimeId: journal.originalRuntimeId, + replacementRuntimeId: journal.replacementRuntimeId, + originalName: journal.originalName, + replacementStagingName: journal.replacementStagingName, + backupName: journal.backupName, + originalSpecHash: journal.originalSpecHash, + replacementSpecHash: journal.replacementSpecHash, + rollbackTargetRuntimeId: journal.rollbackTargetRuntimeId, + rollbackTargetSpecHash: journal.rollbackTargetSpecHash, + preparationReceipt: journal.preparationReceipt, + commitReceipt: journal.commitReceipt, + }); +} + +function legacyFinalizationV1() { + return Object.freeze({ + schemaVersion: 1 as const, + phase: finalization.phase, + bootstrapIdentity: finalization.bootstrapIdentity, + providerId: finalization.providerId, + sandbox: finalization.sandbox, + planFingerprint: finalization.planFingerprint, + profileFingerprint: finalization.profileFingerprint, + imageReference: finalization.imageReference, + commitReceipt: finalization.commitReceipt, + cleanupReceipt: finalization.cleanupReceipt, + }); +} + function readPinnedPrivateFile(target: string): { readonly mode: number; readonly text: string } { const descriptor = fs.openSync(target, fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW); try { @@ -120,6 +206,17 @@ afterEach(() => { }); describe("Docker managed bootstrap journal", () => { + it("rejects comma-joined keys as a different schema", () => { + const { agent: _agent, backupName: _backupName, ...withoutSeparateKeys } = journal; + + expect(() => + normalizeDockerManagedBootstrapJournal({ + ...withoutSeparateKeys, + "agent,backupName": "hermes", + }), + ).toThrow("journal schema is invalid"); + }); + it("publishes private canonical state through only monotonic phases", () => { const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-docker-journal-")); roots.push(root); @@ -147,6 +244,26 @@ describe("Docker managed bootstrap journal", () => { expect(store.load(IDENTITY)).toBeNull(); }); + it("persists owner cleanup as a restart-safe non-terminal phase", () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-docker-journal-")); + roots.push(root); + const first = createFileDockerManagedBootstrapJournalStore(root); + first.create(journal); + + const retained = first.transition(IDENTITY, "staged", "owner-cleanup-required"); + expect(retained.phase).toBe("owner-cleanup-required"); + expect(first.listUnfinishedIdentities()).toEqual([IDENTITY]); + + const restarted = createFileDockerManagedBootstrapJournalStore(root); + expect(restarted.load(IDENTITY)).toEqual(retained); + expect(restarted.listUnfinishedIdentities()).toEqual([IDENTITY]); + expect(() => + restarted.transition(IDENTITY, "owner-cleanup-required", "shared-state-committed"), + ).toThrow("unsupported"); + restarted.remove(IDENTITY, ["owner-cleanup-required"]); + expect(restarted.load(IDENTITY)).toBeNull(); + }); + it("recovers one durable cutover decision before journal replacement", () => { const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-docker-journal-")); roots.push(root); @@ -284,8 +401,12 @@ describe("Docker managed bootstrap journal", () => { roots.push(root); const first = createFileDockerManagedBootstrapJournalStore(root); first.create(journal); + expect(loadUnfinished(first)).toEqual([journal]); first.recordFinalization(finalization); + expect(loadUnfinished(first)).toEqual([journal]); + first.remove(IDENTITY, ["staged"]); + expect(loadUnfinished(first)).toEqual([]); const restarted = createFileDockerManagedBootstrapJournalStore(root); expect(restarted.loadFinalization(IDENTITY)).toEqual(finalization); expect( @@ -309,6 +430,42 @@ describe("Docker managed bootstrap journal", () => { ).toThrow("finalization record changed"); }); + it.each([ + { label: "journal", suffix: "" }, + { label: "decision", suffix: ".decision" }, + { label: "finalization", suffix: ".finalized" }, + ])("ignores an atomic $label write left by a crash during enumeration", ({ suffix }) => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-docker-journal-")); + roots.push(root); + const store = createFileDockerManagedBootstrapJournalStore(root); + store.create(journal); + const directory = path.join(root, DOCKER_MANAGED_BOOTSTRAP_JOURNAL_DIRECTORY); + fs.writeFileSync( + path.join(directory, `.${IDENTITY}.json${suffix}.1234.deadbeef.tmp`), + "partial", + { + mode: 0o600, + }, + ); + + expect(loadUnfinished(store)).toEqual([journal]); + }); + + it("rejects an unsupported journal-directory entry during enumeration", () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-docker-journal-")); + roots.push(root); + const store = createFileDockerManagedBootstrapJournalStore(root); + store.create(journal); + const directory = path.join(root, DOCKER_MANAGED_BOOTSTRAP_JOURNAL_DIRECTORY); + fs.writeFileSync(path.join(directory, `${IDENTITY}.json.unknown`), "unexpected", { + mode: 0o600, + }); + + expect(() => store.listUnfinishedIdentities()).toThrow( + "journal directory contains an unsupported entry", + ); + }); + it("reloads the exact completion receipt from a new journal store", () => { const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-docker-journal-")); roots.push(root); @@ -320,6 +477,7 @@ describe("Docker managed bootstrap journal", () => { const restarted = createFileDockerManagedBootstrapJournalStore(root); expect(restarted.recordCompletion(IDENTITY, finalization.commitReceipt)).toEqual(completed); + expect(loadUnfinished(restarted)).toEqual([completed]); expect(() => restarted.recordCompletion(IDENTITY, { ...finalization.commitReceipt, @@ -327,4 +485,210 @@ describe("Docker managed bootstrap journal", () => { }), ).toThrow("completion receipt changed"); }); + + it.each([ + [1, legacyJournalV1], + [2, legacyJournalV2], + ] as const)("fails typed and closed for exact legacy journal schema %i", (schemaVersion, legacy) => { + const record = legacy(); + const serialized = `${JSON.stringify(record)}\n`; + let failure: unknown; + try { + parseDockerManagedBootstrapJournal(serialized); + } catch (error) { + failure = error; + } + expect(failure).toBeInstanceOf(DockerManagedBootstrapLegacyRecordRequiresAgentError); + expect(failure).toMatchObject({ + bootstrapIdentity: IDENTITY, + journalContext: { + schemaVersion, + phase: record.phase, + bootstrapIdentity: IDENTITY, + providerId: record.sandbox.driverId, + sandbox: record.sandbox, + originalRuntimeId: record.originalRuntimeId, + replacementRuntimeId: record.replacementRuntimeId, + }, + recordKind: "journal", + schemaVersion, + }); + const legacyFailure = failure as DockerManagedBootstrapLegacyRecordRequiresAgentError; + expect(Object.isFrozen(legacyFailure.journalContext)).toBe(true); + expect(Object.isFrozen(legacyFailure.journalContext?.sandbox)).toBe(true); + + const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-docker-journal-")); + roots.push(root); + const store = createFileDockerManagedBootstrapJournalStore(root); + expect(loadUnfinished(store)).toEqual([]); + const target = path.join(root, DOCKER_MANAGED_BOOTSTRAP_JOURNAL_DIRECTORY, `${IDENTITY}.json`); + fs.writeFileSync(target, serialized, { mode: 0o600 }); + expect(store.listUnfinishedIdentities()).toEqual([IDENTITY]); + expect(() => store.load(IDENTITY)).toThrowError( + DockerManagedBootstrapLegacyRecordRequiresAgentError, + ); + expect(readPinnedPrivateFile(target).text).toBe(serialized); + }); + + it("does not classify a malformed legacy journal as upgradeable authority", () => { + const malformed = { ...legacyJournalV2(), agent: "hermes" }; + expect(() => parseDockerManagedBootstrapJournal(`${JSON.stringify(malformed)}\n`)).toThrow( + "legacy journal schema is invalid", + ); + try { + parseDockerManagedBootstrapJournal(`${JSON.stringify(malformed)}\n`); + } catch (error) { + expect(error).not.toBeInstanceOf(DockerManagedBootstrapLegacyRecordRequiresAgentError); + } + }); + + it("rejects owner cleanup as authority invented by a legacy journal", () => { + expect(() => + parseDockerManagedBootstrapJournal( + `${JSON.stringify({ ...legacyJournalV2(), phase: "owner-cleanup-required" })}\n`, + ), + ).toThrow("legacy phase is unsupported"); + }); + + it("upgrades legacy finalization only with exact immutable transaction context", () => { + const serialized = `${JSON.stringify(legacyFinalizationV1())}\n`; + let missingContextFailure: unknown; + try { + parseDockerManagedBootstrapFinalizationRecord(serialized); + } catch (error) { + missingContextFailure = error; + } + expect(missingContextFailure).toBeInstanceOf( + DockerManagedBootstrapLegacyRecordRequiresAgentError, + ); + expect(missingContextFailure).toMatchObject({ reason: "missing-context" }); + + let contextMismatchFailure: unknown; + try { + parseDockerManagedBootstrapFinalizationRecord(serialized, { + ...finalizationContext, + planFingerprint: "0".repeat(64), + }); + } catch (error) { + contextMismatchFailure = error; + } + expect(contextMismatchFailure).toBeInstanceOf( + DockerManagedBootstrapLegacyRecordRequiresAgentError, + ); + expect(contextMismatchFailure).toMatchObject({ + message: expect.stringContaining("supplied durable context does not match this record"), + reason: "context-mismatch", + }); + expect(parseDockerManagedBootstrapFinalizationRecord(serialized, finalizationContext)).toEqual( + finalization, + ); + + const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-docker-journal-")); + roots.push(root); + const store = createFileDockerManagedBootstrapJournalStore(root); + expect(loadUnfinished(store)).toEqual([]); + const target = path.join( + root, + DOCKER_MANAGED_BOOTSTRAP_JOURNAL_DIRECTORY, + `${IDENTITY}.json.finalized`, + ); + fs.writeFileSync(target, serialized, { mode: 0o600 }); + expect(() => store.loadFinalization(IDENTITY)).toThrowError( + DockerManagedBootstrapLegacyRecordRequiresAgentError, + ); + expect(() => store.recordFinalization(finalization)).toThrowError( + DockerManagedBootstrapLegacyRecordRequiresAgentError, + ); + expect(readPinnedPrivateFile(target).text).toBe(serialized); + + expect(() => + store.recordFinalization(finalization, { + ...finalizationContext, + agent: "openclaw", + }), + ).toThrow("does not match supplied durable context"); + expect(readPinnedPrivateFile(target).text).toBe(serialized); + + store.recordFinalization(finalization, finalizationContext); + expect(store.loadFinalization(IDENTITY)).toEqual(finalization); + expect(readPinnedPrivateFile(target).text).toBe( + serializeDockerManagedBootstrapFinalizationRecord(finalization), + ); + }); + + it("rejects a current finalization that contradicts supplied durable context", () => { + const wrongAgent = Object.freeze({ ...finalization, agent: "openclaw" as const }); + expect(() => + parseDockerManagedBootstrapFinalizationRecord( + serializeDockerManagedBootstrapFinalizationRecord(wrongAgent), + finalizationContext, + ), + ).toThrow("does not match supplied durable context"); + }); + + it("does not create a finalization before validating durable context", () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-docker-journal-")); + roots.push(root); + const store = createFileDockerManagedBootstrapJournalStore(root); + expect(loadUnfinished(store)).toEqual([]); + const target = path.join( + root, + DOCKER_MANAGED_BOOTSTRAP_JOURNAL_DIRECTORY, + `${IDENTITY}.json.finalized`, + ); + + expect(() => + store.recordFinalization(finalization, { + ...finalizationContext, + agent: "openclaw", + }), + ).toThrow("does not match supplied durable context"); + expect(fs.existsSync(target)).toBe(false); + }); + + it("rejects current journal and finalization records stored under another identity", () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-docker-journal-")); + roots.push(root); + const store = createFileDockerManagedBootstrapJournalStore(root); + expect(loadUnfinished(store)).toEqual([]); + const directory = path.join(root, DOCKER_MANAGED_BOOTSTRAP_JOURNAL_DIRECTORY); + const misplacedJournal = path.join(directory, `${OTHER_IDENTITY}.json`); + const misplacedFinalization = `${misplacedJournal}.finalized`; + fs.writeFileSync(misplacedJournal, serializeDockerManagedBootstrapJournal(journal), { + mode: 0o600, + }); + fs.writeFileSync( + misplacedFinalization, + serializeDockerManagedBootstrapFinalizationRecord(finalization), + { mode: 0o600 }, + ); + + expect(() => store.load(OTHER_IDENTITY)).toThrow( + "journal bootstrap identity does not match its file name", + ); + expect(() => store.loadFinalization(OTHER_IDENTITY)).toThrow( + "finalization bootstrap identity does not match its file name", + ); + fs.writeFileSync(misplacedJournal, `${JSON.stringify(legacyJournalV2())}\n`, { + mode: 0o600, + }); + expect(() => store.load(OTHER_IDENTITY)).toThrow( + "journal bootstrap identity does not match its file name", + ); + expect(fs.existsSync(misplacedJournal)).toBe(true); + expect(fs.existsSync(misplacedFinalization)).toBe(true); + }); + + it("fails closed when enumeration encounters an unsupported state entry", () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-docker-journal-")); + roots.push(root); + const store = createFileDockerManagedBootstrapJournalStore(root); + store.create(journal); + fs.writeFileSync( + path.join(root, DOCKER_MANAGED_BOOTSTRAP_JOURNAL_DIRECTORY, "unexpected.json"), + "{}\n", + { mode: 0o600 }, + ); + expect(() => store.listUnfinishedIdentities()).toThrow("unsupported entry"); + }); }); diff --git a/src/lib/onboard/managed-bootstrap/docker-journal.ts b/src/lib/onboard/managed-bootstrap/docker-journal.ts index 015bda1d12d..47f1022e884 100644 --- a/src/lib/onboard/managed-bootstrap/docker-journal.ts +++ b/src/lib/onboard/managed-bootstrap/docker-journal.ts @@ -3,7 +3,7 @@ import fs from "node:fs"; import path from "node:path"; - +import { MANAGED_STARTUP_AGENTS, type ManagedStartupAgent } from "../managed-startup/profile"; import type { ManagedBootstrapCompletionReceipt, ManagedBootstrapDurablePreparationReceipt, @@ -11,9 +11,9 @@ import type { ManagedBootstrapSandboxIdentity, } from "./adapter"; -export const DOCKER_MANAGED_BOOTSTRAP_JOURNAL_SCHEMA_VERSION = 2 as const; +export const DOCKER_MANAGED_BOOTSTRAP_JOURNAL_SCHEMA_VERSION = 3 as const; export const DOCKER_MANAGED_BOOTSTRAP_JOURNAL_DIRECTORY = "managed-bootstrap"; -export const DOCKER_MANAGED_BOOTSTRAP_FINALIZATION_SCHEMA_VERSION = 1 as const; +export const DOCKER_MANAGED_BOOTSTRAP_FINALIZATION_SCHEMA_VERSION = 2 as const; const SHA256_RE = /^[a-f0-9]{64}$/u; const MANIFEST_DIGEST_RE = /^sha256:[a-f0-9]{64}$/u; @@ -29,6 +29,7 @@ export type DockerManagedBootstrapJournalPhase = | "staged" | "cutover" | "rollback-authorized" + | "owner-cleanup-required" | "shared-state-committed"; export interface DockerManagedBootstrapJournal { @@ -36,6 +37,7 @@ export interface DockerManagedBootstrapJournal { readonly phase: DockerManagedBootstrapJournalPhase; readonly bootstrapIdentity: string; readonly providerId: string; + readonly agent: ManagedStartupAgent; readonly sandbox: ManagedBootstrapSandboxIdentity; readonly planFingerprint: string; readonly profileFingerprint: string; @@ -59,6 +61,7 @@ export interface DockerManagedBootstrapFinalizationRecord { readonly phase: "committed" | "rolled-back"; readonly bootstrapIdentity: string; readonly providerId: string; + readonly agent: ManagedStartupAgent; readonly sandbox: ManagedBootstrapSandboxIdentity; readonly planFingerprint: string; readonly profileFingerprint: string; @@ -67,9 +70,21 @@ export interface DockerManagedBootstrapFinalizationRecord { readonly cleanupReceipt: ManagedBootstrapFinalizationReceipt; } +export type DockerManagedBootstrapFinalizationContext = Pick< + DockerManagedBootstrapFinalizationRecord, + | "agent" + | "bootstrapIdentity" + | "imageReference" + | "planFingerprint" + | "profileFingerprint" + | "providerId" + | "sandbox" +>; + export interface DockerManagedBootstrapJournalStore { create(journal: DockerManagedBootstrapJournal): void; load(bootstrapIdentity: string): DockerManagedBootstrapJournal | null; + listUnfinishedIdentities(): readonly string[]; transition( bootstrapIdentity: string, expected: DockerManagedBootstrapJournalPhase, @@ -80,8 +95,24 @@ export interface DockerManagedBootstrapJournalStore { receipt: ManagedBootstrapCompletionReceipt, ): DockerManagedBootstrapJournal; remove(bootstrapIdentity: string, expected: readonly DockerManagedBootstrapJournalPhase[]): void; - recordFinalization(record: DockerManagedBootstrapFinalizationRecord): void; - loadFinalization(bootstrapIdentity: string): DockerManagedBootstrapFinalizationRecord | null; + recordFinalization( + record: DockerManagedBootstrapFinalizationRecord, + context?: DockerManagedBootstrapFinalizationContext, + ): void; + loadFinalization( + bootstrapIdentity: string, + context?: DockerManagedBootstrapFinalizationContext, + ): DockerManagedBootstrapFinalizationRecord | null; +} + +export interface DockerManagedBootstrapLegacyJournalContext { + readonly schemaVersion: 1 | 2; + readonly phase: Exclude; + readonly bootstrapIdentity: string; + readonly providerId: string; + readonly sandbox: ManagedBootstrapSandboxIdentity; + readonly originalRuntimeId: string; + readonly replacementRuntimeId: string; } /** @@ -96,6 +127,52 @@ export class DockerManagedBootstrapJournalAcknowledgementLostError extends Error } } +export class DockerManagedBootstrapLegacyRecordRequiresAgentError extends Error { + readonly bootstrapIdentity: string; + readonly journalContext: DockerManagedBootstrapLegacyJournalContext | null; + readonly recordKind: "finalization" | "journal"; + readonly reason: "context-mismatch" | "missing-context" | undefined; + readonly schemaVersion: number; + + constructor(input: { + readonly bootstrapIdentity: string; + readonly journalContext?: DockerManagedBootstrapLegacyJournalContext; + readonly recordKind: "finalization" | "journal"; + readonly reason?: "context-mismatch" | "missing-context"; + readonly schemaVersion: number; + }) { + const reason = input.reason; + const journalContext = input.journalContext + ? Object.freeze({ + ...input.journalContext, + sandbox: Object.freeze({ ...input.journalContext.sandbox }), + }) + : undefined; + const journalGuidance = journalContext + ? `; recovery is fenced to sandbox '${journalContext.sandbox.sandboxName}' ` + + `(ID ${journalContext.sandbox.sandboxId}, provider ${journalContext.providerId}, ` + + `journal-body phase ${journalContext.phase}) with exact original runtime ` + + `${journalContext.originalRuntimeId} and replacement runtime ` + + `${journalContext.replacementRuntimeId}; preserve the journal and follow ` + + "https://github.com/NVIDIA/NemoClaw/blob/main/src/lib/onboard/managed-bootstrap/README.md#legacy-journal-drain-schema-1-and-2" + : ""; + super( + `Managed bootstrap Docker ${input.recordKind} schema ${input.schemaVersion} for ` + + `${input.bootstrapIdentity} lacks durable agent identity` + + (reason === "context-mismatch" + ? "; supplied durable context does not match this record" + : "") + + journalGuidance, + ); + this.name = "DockerManagedBootstrapLegacyRecordRequiresAgentError"; + this.bootstrapIdentity = input.bootstrapIdentity; + this.journalContext = journalContext ?? null; + this.recordKind = input.recordKind; + this.reason = reason; + this.schemaVersion = input.schemaVersion; + } +} + class DockerManagedBootstrapJournalExistsError extends Error { constructor() { super( @@ -107,14 +184,25 @@ class DockerManagedBootstrapJournalExistsError extends Error { const ALLOWED_TRANSITIONS = new Set([ "staged->cutover", + "staged->owner-cleanup-required", "cutover->rollback-authorized", "cutover->shared-state-committed", + "rollback-authorized->owner-cleanup-required", ]); function fail(message: string): never { throw new Error(`Managed bootstrap Docker journal is invalid: ${message}`); } +function hasExactKeys(record: Readonly>, expected: readonly string[]) { + const actualKeys = Object.keys(record).sort(); + const expectedKeys = [...expected].sort(); + return ( + actualKeys.length === expectedKeys.length && + actualKeys.every((key, index) => key === expectedKeys[index]) + ); +} + function exactString(value: unknown, label: string, maxBytes = 4096): string { if ( typeof value !== "string" || @@ -137,13 +225,37 @@ function exactSha256(value: unknown, label: string): string { function exactPhase(value: unknown): DockerManagedBootstrapJournalPhase { if ( - !["staged", "cutover", "rollback-authorized", "shared-state-committed"].includes(String(value)) + ![ + "staged", + "cutover", + "rollback-authorized", + "owner-cleanup-required", + "shared-state-committed", + ].includes(String(value)) ) { fail("phase is unsupported"); } return value as DockerManagedBootstrapJournalPhase; } +function exactLegacyPhase( + value: unknown, +): Exclude { + if ( + !["staged", "cutover", "rollback-authorized", "shared-state-committed"].includes(String(value)) + ) { + fail("legacy phase is unsupported"); + } + return value as Exclude; +} + +function exactAgent(value: unknown): ManagedStartupAgent { + if (!MANAGED_STARTUP_AGENTS.includes(value as ManagedStartupAgent)) { + fail("agent is unsupported"); + } + return value as ManagedStartupAgent; +} + function exactSandbox(value: unknown): ManagedBootstrapSandboxIdentity { if (typeof value !== "object" || value === null || Array.isArray(value)) { fail("sandbox identity must be an object"); @@ -159,6 +271,196 @@ function exactSandbox(value: unknown): ManagedBootstrapSandboxIdentity { }); } +function sameSandboxIdentity( + left: ManagedBootstrapSandboxIdentity, + right: ManagedBootstrapSandboxIdentity, +): boolean { + return ( + left.sandboxName === right.sandboxName && + left.sandboxId === right.sandboxId && + left.driverId === right.driverId + ); +} + +// Frozen historical schemas: these branches must reproduce the exact canonical +// bytes written by schema 1 and schema 2. Do not share their implementation with +// the current normalizer or update them when the current schema changes. +function normalizeLegacyDockerManagedBootstrapJournal( + journal: Readonly>, + schemaVersion: 1 | 2, +): { + readonly bootstrapIdentity: string; + readonly canonical: string; + readonly journalContext: DockerManagedBootstrapLegacyJournalContext; +} { + if (schemaVersion === 1) { + const expectedKeys = [ + "backupName", + "bootstrapIdentity", + "imageReference", + "originalName", + "originalRuntimeId", + "originalSpecHash", + "phase", + "profileFingerprint", + "replacementRuntimeId", + "replacementSpecHash", + "replacementStagingName", + "runtimeImageContentId", + "sandbox", + "schemaVersion", + ]; + if (!hasExactKeys(journal, expectedKeys)) fail("legacy journal schema is invalid"); + const normalized = Object.freeze({ + schemaVersion: 1 as const, + phase: exactLegacyPhase(journal.phase), + bootstrapIdentity: exactSha256(journal.bootstrapIdentity, "bootstrap identity"), + sandbox: exactSandbox(journal.sandbox), + profileFingerprint: exactSha256(journal.profileFingerprint, "profile fingerprint"), + imageReference: exactString(journal.imageReference, "image reference"), + runtimeImageContentId: exactString(journal.runtimeImageContentId, "runtime image content ID"), + originalRuntimeId: exactSha256(journal.originalRuntimeId, "original runtime ID"), + replacementRuntimeId: exactSha256(journal.replacementRuntimeId, "replacement runtime ID"), + originalName: exactString(journal.originalName, "original name", 253), + replacementStagingName: exactString( + journal.replacementStagingName, + "replacement staging name", + 253, + ), + backupName: exactString(journal.backupName, "backup name", 253), + originalSpecHash: exactSha256(journal.originalSpecHash, "original spec hash"), + replacementSpecHash: exactSha256(journal.replacementSpecHash, "replacement spec hash"), + }); + if (normalized.originalRuntimeId === normalized.replacementRuntimeId) { + fail("original and replacement runtime IDs must differ"); + } + if ( + new Set([normalized.originalName, normalized.replacementStagingName, normalized.backupName]) + .size !== 3 + ) { + fail("original, staging, and backup names must be distinct"); + } + return { + bootstrapIdentity: normalized.bootstrapIdentity, + canonical: `${JSON.stringify(normalized)}\n`, + journalContext: Object.freeze({ + schemaVersion, + phase: normalized.phase, + bootstrapIdentity: normalized.bootstrapIdentity, + providerId: normalized.sandbox.driverId, + sandbox: normalized.sandbox, + originalRuntimeId: normalized.originalRuntimeId, + replacementRuntimeId: normalized.replacementRuntimeId, + }), + }; + } + + const expectedKeys = [ + "backupName", + "bootstrapIdentity", + "commitReceipt", + "imageReference", + "originalName", + "originalRuntimeId", + "originalSpecHash", + "phase", + "planFingerprint", + "preparationReceipt", + "profileFingerprint", + "providerId", + "replacementRuntimeId", + "replacementSpecHash", + "replacementStagingName", + "rollbackTargetRuntimeId", + "rollbackTargetSpecHash", + "runtimeImageContentId", + "sandbox", + "schemaVersion", + ]; + if (!hasExactKeys(journal, expectedKeys)) fail("legacy journal schema is invalid"); + const normalized = Object.freeze({ + schemaVersion: 2 as const, + phase: exactLegacyPhase(journal.phase), + bootstrapIdentity: exactSha256(journal.bootstrapIdentity, "bootstrap identity"), + providerId: exactString(journal.providerId, "provider ID"), + sandbox: exactSandbox(journal.sandbox), + planFingerprint: exactSha256(journal.planFingerprint, "plan fingerprint"), + profileFingerprint: exactSha256(journal.profileFingerprint, "profile fingerprint"), + imageReference: exactString(journal.imageReference, "image reference"), + runtimeImageContentId: exactString(journal.runtimeImageContentId, "runtime image content ID"), + originalRuntimeId: exactSha256(journal.originalRuntimeId, "original runtime ID"), + replacementRuntimeId: exactSha256(journal.replacementRuntimeId, "replacement runtime ID"), + originalName: exactString(journal.originalName, "original name", 253), + replacementStagingName: exactString( + journal.replacementStagingName, + "replacement staging name", + 253, + ), + backupName: exactString(journal.backupName, "backup name", 253), + originalSpecHash: exactSha256(journal.originalSpecHash, "original spec hash"), + replacementSpecHash: exactSha256(journal.replacementSpecHash, "replacement spec hash"), + rollbackTargetRuntimeId: exactSha256( + journal.rollbackTargetRuntimeId, + "rollback target runtime ID", + ), + rollbackTargetSpecHash: exactSha256( + journal.rollbackTargetSpecHash, + "rollback target spec hash", + ), + preparationReceipt: + journal.preparationReceipt === null + ? null + : exactPreparationReceipt(journal.preparationReceipt), + commitReceipt: + journal.commitReceipt === null ? null : exactCompletionReceipt(journal.commitReceipt), + }); + if (normalized.originalRuntimeId === normalized.replacementRuntimeId) { + fail("original and replacement runtime IDs must differ"); + } + if ( + new Set([normalized.originalName, normalized.replacementStagingName, normalized.backupName]) + .size !== 3 + ) { + fail("original, staging, and backup names must be distinct"); + } + if ( + normalized.providerId !== normalized.sandbox.driverId || + normalized.rollbackTargetRuntimeId !== normalized.originalRuntimeId || + normalized.rollbackTargetSpecHash !== normalized.originalSpecHash + ) { + fail("provider or rollback authority does not match the transaction identity"); + } + if ( + (normalized.preparationReceipt !== null && + (normalized.preparationReceipt.bootstrapIdentity !== normalized.bootstrapIdentity || + !sameSandboxIdentity(normalized.preparationReceipt.sandbox, normalized.sandbox))) || + (normalized.commitReceipt !== null && + (normalized.commitReceipt.bootstrapIdentity !== normalized.bootstrapIdentity || + !sameSandboxIdentity(normalized.commitReceipt.sandbox, normalized.sandbox) || + normalized.commitReceipt.runtimeId !== normalized.replacementRuntimeId || + normalized.commitReceipt.profileFingerprint !== normalized.profileFingerprint || + normalized.commitReceipt.originalSpecHash !== normalized.originalSpecHash || + normalized.commitReceipt.replacementSpecHash !== normalized.replacementSpecHash || + `${normalized.commitReceipt.image.repository}@${normalized.commitReceipt.image.manifestDigest}` !== + normalized.imageReference)) + ) { + fail("durable preparation or commit receipt does not match the transaction identity"); + } + return { + bootstrapIdentity: normalized.bootstrapIdentity, + canonical: `${JSON.stringify(normalized)}\n`, + journalContext: Object.freeze({ + schemaVersion, + phase: normalized.phase, + bootstrapIdentity: normalized.bootstrapIdentity, + providerId: normalized.providerId, + sandbox: normalized.sandbox, + originalRuntimeId: normalized.originalRuntimeId, + replacementRuntimeId: normalized.replacementRuntimeId, + }), + }; +} + export function normalizeDockerManagedBootstrapJournal( value: unknown, ): DockerManagedBootstrapJournal { @@ -166,7 +468,17 @@ export function normalizeDockerManagedBootstrapJournal( fail("journal must be an object"); } const journal = value as Record; + if (journal.schemaVersion === 1 || journal.schemaVersion === 2) { + const legacy = normalizeLegacyDockerManagedBootstrapJournal(journal, journal.schemaVersion); + throw new DockerManagedBootstrapLegacyRecordRequiresAgentError({ + bootstrapIdentity: legacy.bootstrapIdentity, + journalContext: legacy.journalContext, + recordKind: "journal", + schemaVersion: journal.schemaVersion, + }); + } const expectedKeys = [ + "agent", "backupName", "bootstrapIdentity", "commitReceipt", @@ -189,7 +501,7 @@ export function normalizeDockerManagedBootstrapJournal( "schemaVersion", ]; if ( - Object.keys(journal).sort().join(",") !== expectedKeys.sort().join(",") || + !hasExactKeys(journal, expectedKeys) || journal.schemaVersion !== DOCKER_MANAGED_BOOTSTRAP_JOURNAL_SCHEMA_VERSION ) { fail("journal schema is invalid"); @@ -199,6 +511,7 @@ export function normalizeDockerManagedBootstrapJournal( phase: exactPhase(journal.phase), bootstrapIdentity: exactSha256(journal.bootstrapIdentity, "bootstrap identity"), providerId: exactString(journal.providerId, "provider ID"), + agent: exactAgent(journal.agent), sandbox: exactSandbox(journal.sandbox), planFingerprint: exactSha256(journal.planFingerprint, "plan fingerprint"), profileFingerprint: exactSha256(journal.profileFingerprint, "profile fingerprint"), @@ -249,14 +562,10 @@ export function normalizeDockerManagedBootstrapJournal( if ( (normalized.preparationReceipt !== null && (normalized.preparationReceipt.bootstrapIdentity !== normalized.bootstrapIdentity || - normalized.preparationReceipt.sandbox.sandboxName !== normalized.sandbox.sandboxName || - normalized.preparationReceipt.sandbox.sandboxId !== normalized.sandbox.sandboxId || - normalized.preparationReceipt.sandbox.driverId !== normalized.sandbox.driverId)) || + !sameSandboxIdentity(normalized.preparationReceipt.sandbox, normalized.sandbox))) || (normalized.commitReceipt !== null && (normalized.commitReceipt.bootstrapIdentity !== normalized.bootstrapIdentity || - normalized.commitReceipt.sandbox.sandboxName !== normalized.sandbox.sandboxName || - normalized.commitReceipt.sandbox.sandboxId !== normalized.sandbox.sandboxId || - normalized.commitReceipt.sandbox.driverId !== normalized.sandbox.driverId || + !sameSandboxIdentity(normalized.commitReceipt.sandbox, normalized.sandbox) || normalized.commitReceipt.runtimeId !== normalized.replacementRuntimeId || normalized.commitReceipt.profileFingerprint !== normalized.profileFingerprint || normalized.commitReceipt.originalSpecHash !== normalized.originalSpecHash || @@ -294,6 +603,23 @@ export function parseDockerManagedBootstrapJournal(text: string): DockerManagedB } catch { fail("serialized journal is not valid JSON"); } + if ( + typeof parsed === "object" && + parsed !== null && + !Array.isArray(parsed) && + ((parsed as Record).schemaVersion === 1 || + (parsed as Record).schemaVersion === 2) + ) { + const record = parsed as Record & { readonly schemaVersion: 1 | 2 }; + const legacy = normalizeLegacyDockerManagedBootstrapJournal(record, record.schemaVersion); + if (legacy.canonical !== text) fail("serialized legacy journal is not canonical"); + throw new DockerManagedBootstrapLegacyRecordRequiresAgentError({ + bootstrapIdentity: legacy.bootstrapIdentity, + journalContext: legacy.journalContext, + recordKind: "journal", + schemaVersion: record.schemaVersion, + }); + } const journal = normalizeDockerManagedBootstrapJournal(parsed); if (serializeDockerManagedBootstrapJournal(journal) !== text) { fail("serialized journal is not canonical"); @@ -476,31 +802,16 @@ function exactCleanupReceipt(value: unknown): ManagedBootstrapFinalizationReceip }); } -export function normalizeDockerManagedBootstrapFinalizationRecord( - value: unknown, -): DockerManagedBootstrapFinalizationRecord { - if (typeof value !== "object" || value === null || Array.isArray(value)) { - fail("finalization record must be an object"); - } - const record = value as Record; - const expectedKeys = [ - "bootstrapIdentity", - "cleanupReceipt", - "commitReceipt", - "imageReference", - "phase", - "planFingerprint", - "profileFingerprint", - "providerId", - "sandbox", - "schemaVersion", - ]; - if ( - Object.keys(record).sort().join(",") !== expectedKeys.sort().join(",") || - record.schemaVersion !== DOCKER_MANAGED_BOOTSTRAP_FINALIZATION_SCHEMA_VERSION || - !["committed", "rolled-back"].includes(String(record.phase)) - ) { - fail("finalization record schema is invalid"); +type DockerManagedBootstrapFinalizationWithoutAgent = Omit< + DockerManagedBootstrapFinalizationRecord, + "agent" | "schemaVersion" +>; + +function normalizeFinalizationWithoutAgent( + record: Readonly>, +): DockerManagedBootstrapFinalizationWithoutAgent { + if (!["committed", "rolled-back"].includes(String(record.phase))) { + fail("finalization phase is invalid"); } const phase = record.phase as "committed" | "rolled-back"; const sandbox = exactSandbox(record.sandbox); @@ -508,7 +819,6 @@ export function normalizeDockerManagedBootstrapFinalizationRecord( record.commitReceipt === null ? null : exactCompletionReceipt(record.commitReceipt); const cleanupReceipt = exactCleanupReceipt(record.cleanupReceipt); const normalized = Object.freeze({ - schemaVersion: DOCKER_MANAGED_BOOTSTRAP_FINALIZATION_SCHEMA_VERSION, phase, bootstrapIdentity: exactSha256(record.bootstrapIdentity, "finalization bootstrap identity"), providerId: exactString(record.providerId, "finalization provider ID"), @@ -518,17 +828,17 @@ export function normalizeDockerManagedBootstrapFinalizationRecord( imageReference: exactString(record.imageReference, "finalization image reference"), commitReceipt, cleanupReceipt, - } satisfies DockerManagedBootstrapFinalizationRecord); + } satisfies DockerManagedBootstrapFinalizationWithoutAgent); if ( normalized.providerId !== sandbox.driverId || normalized.bootstrapIdentity !== cleanupReceipt.bootstrapIdentity || normalized.phase !== cleanupReceipt.outcome || - JSON.stringify(normalized.sandbox) !== JSON.stringify(cleanupReceipt.sandbox) || + !sameSandboxIdentity(normalized.sandbox, cleanupReceipt.sandbox) || (phase === "committed") !== (commitReceipt !== null) || (commitReceipt !== null && (commitReceipt.bootstrapIdentity !== normalized.bootstrapIdentity || commitReceipt.profileFingerprint !== normalized.profileFingerprint || - JSON.stringify(commitReceipt.sandbox) !== JSON.stringify(normalized.sandbox) || + !sameSandboxIdentity(commitReceipt.sandbox, normalized.sandbox) || `${commitReceipt.image.repository}@${commitReceipt.image.manifestDigest}` !== normalized.imageReference)) ) { @@ -537,6 +847,176 @@ export function normalizeDockerManagedBootstrapFinalizationRecord( return normalized; } +function normalizeLegacyFinalizationShape(value: unknown): { + readonly canonical: string; + readonly record: DockerManagedBootstrapFinalizationWithoutAgent; +} { + if (typeof value !== "object" || value === null || Array.isArray(value)) { + fail("finalization record must be an object"); + } + const record = value as Record; + const expectedKeys = [ + "bootstrapIdentity", + "cleanupReceipt", + "commitReceipt", + "imageReference", + "phase", + "planFingerprint", + "profileFingerprint", + "providerId", + "sandbox", + "schemaVersion", + ]; + if (!hasExactKeys(record, expectedKeys) || record.schemaVersion !== 1) { + fail("legacy finalization record schema is invalid"); + } + const normalized = normalizeFinalizationWithoutAgent(record); + const legacy = Object.freeze({ + schemaVersion: 1 as const, + phase: normalized.phase, + bootstrapIdentity: normalized.bootstrapIdentity, + providerId: normalized.providerId, + sandbox: normalized.sandbox, + planFingerprint: normalized.planFingerprint, + profileFingerprint: normalized.profileFingerprint, + imageReference: normalized.imageReference, + commitReceipt: normalized.commitReceipt, + cleanupReceipt: normalized.cleanupReceipt, + }); + return { canonical: `${JSON.stringify(legacy)}\n`, record: normalized }; +} + +function normalizeFinalizationContext( + context: DockerManagedBootstrapFinalizationContext, +): DockerManagedBootstrapFinalizationContext { + return Object.freeze({ + bootstrapIdentity: exactSha256( + context.bootstrapIdentity, + "finalization context bootstrap identity", + ), + providerId: exactString(context.providerId, "finalization context provider ID"), + agent: exactAgent(context.agent), + sandbox: exactSandbox(context.sandbox), + planFingerprint: exactSha256(context.planFingerprint, "finalization context plan fingerprint"), + profileFingerprint: exactSha256( + context.profileFingerprint, + "finalization context profile fingerprint", + ), + imageReference: exactString(context.imageReference, "finalization context image reference"), + }); +} + +function matchesFinalizationContext( + record: DockerManagedBootstrapFinalizationWithoutAgent, + context: DockerManagedBootstrapFinalizationContext, +): boolean { + return ( + context.bootstrapIdentity === record.bootstrapIdentity && + context.providerId === record.providerId && + sameSandboxIdentity(context.sandbox, record.sandbox) && + context.planFingerprint === record.planFingerprint && + context.profileFingerprint === record.profileFingerprint && + context.imageReference === record.imageReference + ); +} + +function assertFinalizationMatchesContext( + record: DockerManagedBootstrapFinalizationRecord, + context: DockerManagedBootstrapFinalizationContext, +): void { + const normalizedContext = normalizeFinalizationContext(context); + if ( + record.agent !== normalizedContext.agent || + !matchesFinalizationContext(record, normalizedContext) + ) { + fail("finalization record does not match supplied durable context"); + } +} + +function upgradeLegacyFinalization( + legacy: DockerManagedBootstrapFinalizationWithoutAgent, + context: DockerManagedBootstrapFinalizationContext | undefined, +): DockerManagedBootstrapFinalizationRecord { + const missingAgent = (reason: "context-mismatch" | "missing-context" = "missing-context") => + new DockerManagedBootstrapLegacyRecordRequiresAgentError({ + bootstrapIdentity: legacy.bootstrapIdentity, + recordKind: "finalization", + reason, + schemaVersion: 1, + }); + // Runtime names and image repositories are mutable descriptions, never + // agent authority. Only an exact live handle or current journal may supply + // the field omitted by schema v1. + if (!context) throw missingAgent(); + const normalizedContext = normalizeFinalizationContext(context); + if (!matchesFinalizationContext(legacy, normalizedContext)) { + throw missingAgent("context-mismatch"); + } + return Object.freeze({ + schemaVersion: DOCKER_MANAGED_BOOTSTRAP_FINALIZATION_SCHEMA_VERSION, + phase: legacy.phase, + bootstrapIdentity: legacy.bootstrapIdentity, + providerId: legacy.providerId, + agent: normalizedContext.agent, + sandbox: legacy.sandbox, + planFingerprint: legacy.planFingerprint, + profileFingerprint: legacy.profileFingerprint, + imageReference: legacy.imageReference, + commitReceipt: legacy.commitReceipt, + cleanupReceipt: legacy.cleanupReceipt, + }); +} + +export function normalizeDockerManagedBootstrapFinalizationRecord( + value: unknown, +): DockerManagedBootstrapFinalizationRecord { + if (typeof value !== "object" || value === null || Array.isArray(value)) { + fail("finalization record must be an object"); + } + const record = value as Record; + if (record.schemaVersion === 1) { + const legacy = normalizeLegacyFinalizationShape(record).record; + throw new DockerManagedBootstrapLegacyRecordRequiresAgentError({ + bootstrapIdentity: legacy.bootstrapIdentity, + recordKind: "finalization", + schemaVersion: 1, + }); + } + const expectedKeys = [ + "agent", + "bootstrapIdentity", + "cleanupReceipt", + "commitReceipt", + "imageReference", + "phase", + "planFingerprint", + "profileFingerprint", + "providerId", + "sandbox", + "schemaVersion", + ]; + if ( + !hasExactKeys(record, expectedKeys) || + record.schemaVersion !== DOCKER_MANAGED_BOOTSTRAP_FINALIZATION_SCHEMA_VERSION + ) { + fail("finalization record schema is invalid"); + } + const normalized = normalizeFinalizationWithoutAgent(record); + return Object.freeze({ + schemaVersion: DOCKER_MANAGED_BOOTSTRAP_FINALIZATION_SCHEMA_VERSION, + phase: normalized.phase, + bootstrapIdentity: normalized.bootstrapIdentity, + providerId: normalized.providerId, + agent: exactAgent(record.agent), + sandbox: normalized.sandbox, + planFingerprint: normalized.planFingerprint, + profileFingerprint: normalized.profileFingerprint, + imageReference: normalized.imageReference, + commitReceipt: normalized.commitReceipt, + cleanupReceipt: normalized.cleanupReceipt, + }); +} + export function serializeDockerManagedBootstrapFinalizationRecord( record: DockerManagedBootstrapFinalizationRecord, ): string { @@ -547,9 +1027,10 @@ export function serializeDockerManagedBootstrapFinalizationRecord( return serialized; } -export function parseDockerManagedBootstrapFinalizationRecord( +function parseDockerManagedBootstrapFinalizationRecordWithContext( text: string, -): DockerManagedBootstrapFinalizationRecord { + context?: DockerManagedBootstrapFinalizationContext, +): { readonly record: DockerManagedBootstrapFinalizationRecord; readonly upgradedLegacy: boolean } { if ( text.length === 0 || text.includes("\0") || @@ -563,11 +1044,29 @@ export function parseDockerManagedBootstrapFinalizationRecord( } catch { fail("serialized finalization record is not valid JSON"); } + if ( + typeof parsed === "object" && + parsed !== null && + !Array.isArray(parsed) && + (parsed as Record).schemaVersion === 1 + ) { + const legacy = normalizeLegacyFinalizationShape(parsed); + if (legacy.canonical !== text) fail("serialized legacy finalization record is not canonical"); + return { record: upgradeLegacyFinalization(legacy.record, context), upgradedLegacy: true }; + } const record = normalizeDockerManagedBootstrapFinalizationRecord(parsed); if (serializeDockerManagedBootstrapFinalizationRecord(record) !== text) { fail("serialized finalization record is not canonical"); } - return record; + if (context) assertFinalizationMatchesContext(record, context); + return { record, upgradedLegacy: false }; +} + +export function parseDockerManagedBootstrapFinalizationRecord( + text: string, + context?: DockerManagedBootstrapFinalizationContext, +): DockerManagedBootstrapFinalizationRecord { + return parseDockerManagedBootstrapFinalizationRecordWithContext(text, context).record; } function assertDirectory(directory: string): void { @@ -733,7 +1232,22 @@ export function createFileDockerManagedBootstrapJournalStore( const target = journalPath(directory, bootstrapIdentity); const contents = readPrivateFile(target, "journal"); if (contents === null) return null; - const journal = parseDockerManagedBootstrapJournal(contents); + let journal: DockerManagedBootstrapJournal; + try { + journal = parseDockerManagedBootstrapJournal(contents); + } catch (error) { + if ( + error instanceof DockerManagedBootstrapLegacyRecordRequiresAgentError && + error.recordKind === "journal" && + error.bootstrapIdentity !== bootstrapIdentity + ) { + fail("journal bootstrap identity does not match its file name"); + } + throw error; + } + if (journal.bootstrapIdentity !== bootstrapIdentity) { + fail("journal bootstrap identity does not match its file name"); + } const decision = readPrivateFile(decisionPath(target), "decision"); if (decision === null) return journal; const phase = decision.endsWith("\n") ? decision.slice(0, -1) : ""; @@ -751,13 +1265,24 @@ export function createFileDockerManagedBootstrapJournalStore( }; const loadFinalization = ( bootstrapIdentity: string, + context?: DockerManagedBootstrapFinalizationContext, ): DockerManagedBootstrapFinalizationRecord | null => { assertDirectory(directory); - const contents = readPrivateFile( - finalizationPath(journalPath(directory, bootstrapIdentity)), - "finalization", - ); - return contents === null ? null : parseDockerManagedBootstrapFinalizationRecord(contents); + const target = finalizationPath(journalPath(directory, bootstrapIdentity)); + const contents = readPrivateFile(target, "finalization"); + if (contents === null) return null; + const parsed = parseDockerManagedBootstrapFinalizationRecordWithContext(contents, context); + if (parsed.record.bootstrapIdentity !== bootstrapIdentity) { + fail("finalization bootstrap identity does not match its file name"); + } + if (parsed.upgradedLegacy) { + const serialized = serializeDockerManagedBootstrapFinalizationRecord(parsed.record); + atomicWrite(directory, target, serialized, false); + if (readPrivateFile(target, "finalization") !== serialized) { + fail("upgraded finalization record was not durably re-readable"); + } + } + return parsed.record; }; return Object.freeze({ create(journal: DockerManagedBootstrapJournal) { @@ -777,6 +1302,25 @@ export function createFileDockerManagedBootstrapJournalStore( atomicWrite(directory, target, serializeDockerManagedBootstrapJournal(normalized), true); }, load, + listUnfinishedIdentities() { + assertDirectory(directory); + const identities: string[] = []; + for (const name of fs.readdirSync(directory)) { + const match = name.match(/^([a-f0-9]{64})\.json$/u); + if (match) { + identities.push(match[1]); + continue; + } + if ( + /^\.[a-f0-9]{64}\.json(?:\.decision|\.finalized)?\.[0-9]+\.[a-f0-9]+\.tmp$/u.test(name) || + /^[a-f0-9]{64}\.json\.(?:decision|finalized)$/u.test(name) + ) { + continue; + } + fail(`journal directory contains an unsupported entry: ${name}`); + } + return Object.freeze(identities.sort()); + }, transition( bootstrapIdentity: string, expected: DockerManagedBootstrapJournalPhase, @@ -851,23 +1395,38 @@ export function createFileDockerManagedBootstrapJournalStore( fs.unlinkSync(target); fsyncDirectory(directory); }, - recordFinalization(record: DockerManagedBootstrapFinalizationRecord) { + recordFinalization( + record: DockerManagedBootstrapFinalizationRecord, + context?: DockerManagedBootstrapFinalizationContext, + ) { const normalized = normalizeDockerManagedBootstrapFinalizationRecord(record); + if (context) assertFinalizationMatchesContext(normalized, context); assertDirectory(directory); const target = finalizationPath(journalPath(directory, normalized.bootstrapIdentity)); const serialized = serializeDockerManagedBootstrapFinalizationRecord(normalized); - const existing = readPrivateFile(target, "finalization"); + const existing = loadFinalization(normalized.bootstrapIdentity, context); if (existing !== null) { - if (existing !== serialized) + if (serializeDockerManagedBootstrapFinalizationRecord(existing) !== serialized) { fail("finalization record changed for this bootstrap identity"); + } return; } try { atomicWrite(directory, target, serialized, true); } catch (error) { - if (readPrivateFile(target, "finalization") !== serialized) throw error; + const recovered = loadFinalization(normalized.bootstrapIdentity, context); + if ( + !recovered || + serializeDockerManagedBootstrapFinalizationRecord(recovered) !== serialized + ) { + throw error; + } } - if (readPrivateFile(target, "finalization") !== serialized) { + const persisted = loadFinalization(normalized.bootstrapIdentity, context); + if ( + !persisted || + serializeDockerManagedBootstrapFinalizationRecord(persisted) !== serialized + ) { fail("finalization record was not durably re-readable"); } }, diff --git a/src/lib/onboard/managed-bootstrap/docker-recovery.test.ts b/src/lib/onboard/managed-bootstrap/docker-recovery.test.ts new file mode 100644 index 00000000000..2d6fb3c1110 --- /dev/null +++ b/src/lib/onboard/managed-bootstrap/docker-recovery.test.ts @@ -0,0 +1,546 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { describe, expect, it, vi } from "vitest"; + +import { + enforceManagedBootstrapRecoveryForSandbox, + ManagedBootstrapRecoveryBlockedError, + recoverManagedBootstrapTransactions, +} from "./adapter"; +import { createDockerManagedBootstrapAdapter } from "./docker"; +import { + type DockerManagedBootstrapJournalStore, + DockerManagedBootstrapLegacyRecordRequiresAgentError, +} from "./docker-journal"; +import { + authority, + type DockerFixtureOptions, + durablePreparation, + fixture, + IDENTITY, + NEW_ID, + OLD_ID, +} from "./docker-test-fixture"; + +async function prepareTransaction( + fake: ReturnType, + agent: Parameters[0] = "hermes", +) { + const adapter = createDockerManagedBootstrapAdapter(fake.deps); + const { handle, request, snapshot } = authority(agent); + const prepared = await adapter.prepareBootstrapReplacement({ + handle, + snapshot, + request, + replacementOptions: { values: {} }, + }); + return { + adapter, + handle, + prepared, + snapshot, + durable: durablePreparation(handle, snapshot, prepared), + }; +} + +function expectEventBefore(events: readonly string[], before: string, after: string): void { + expect(events).toContain(before); + expect(events).toContain(after); + expect(events.indexOf(before)).toBeLessThan(events.indexOf(after)); +} + +function dockerMutationEvents(events: readonly string[]): readonly string[] { + return events.filter((event) => /^(?:create:|rename:|rm:|start:|stop:)/u.test(event)); +} + +describe("Docker managed bootstrap restart recovery", () => { + it("scopes an exact legacy journal to its durable sandbox without inventing agent authority", async () => { + const fake = fixture(); + const delegate = fake.deps.journalStore as DockerManagedBootstrapJournalStore; + const legacyStore: DockerManagedBootstrapJournalStore = { + ...delegate, + listUnfinishedIdentities: () => [IDENTITY], + load() { + throw new DockerManagedBootstrapLegacyRecordRequiresAgentError({ + bootstrapIdentity: IDENTITY, + journalContext: { + schemaVersion: 2, + phase: "cutover", + bootstrapIdentity: IDENTITY, + providerId: "docker", + sandbox: authority().handle.sandbox, + originalRuntimeId: OLD_ID, + replacementRuntimeId: NEW_ID, + }, + recordKind: "journal", + schemaVersion: 2, + }); + }, + }; + const adapter = createDockerManagedBootstrapAdapter({ + ...fake.deps, + journalStore: legacyStore, + }); + + const report = await recoverManagedBootstrapTransactions(adapter); + expect(report).toMatchObject({ + receipts: [], + failures: [ + { + bootstrapIdentity: IDENTITY, + providerId: "docker", + sourcePhase: null, + sandbox: authority().handle.sandbox, + code: "legacy-agent-required", + retryable: true, + detail: expect.stringContaining(OLD_ID), + }, + ], + }); + const warn = vi.fn(); + expect(enforceManagedBootstrapRecoveryForSandbox(report, "bravo", warn)).toBe(report); + expect(warn).toHaveBeenCalledWith(expect.stringContaining("unrelated sandbox 'alpha'")); + expect(() => enforceManagedBootstrapRecoveryForSandbox(report, "alpha", warn)).toThrow( + ManagedBootstrapRecoveryBlockedError, + ); + expect(dockerMutationEvents(fake.events)).toEqual([]); + expect(fake.events).toEqual([]); + expect(fake.journal).toBeNull(); + }); + + it.each([ + { + label: "staged", + options: { + journalCreateFailures: [new Error("injected crash after durable staged fence")], + }, + phase: "staged", + }, + { + label: "cutover", + options: { + journalTransitionFailures: { + cutover: new Error("injected crash after durable cutover fence"), + }, + }, + phase: "cutover", + }, + ] satisfies readonly { + readonly label: string; + readonly options: DockerFixtureOptions; + readonly phase: "cutover" | "staged"; + }[])("retains owner cleanup after a process restart from the durable $label phase", async ({ + options, + phase, + }) => { + const fake = fixture(options); + const transaction = await prepareTransaction(fake); + + await expect( + transaction.adapter.activateBootstrapReplacement({ + handle: transaction.handle, + snapshot: transaction.snapshot, + prepared: transaction.prepared, + durablePreparation: transaction.durable, + }), + ).rejects.toThrow(`crash after durable ${phase} fence`); + expect(fake.journal?.phase).toBe(phase); + + const restarted = createDockerManagedBootstrapAdapter(fake.deps); + await expect(restarted.recoverUnfinishedTransactions()).resolves.toMatchObject({ + receipts: [], + failures: [ + { + bootstrapIdentity: IDENTITY, + sourcePhase: "owner-cleanup-required", + code: "owner-cleanup-required", + retryable: true, + }, + ], + }); + expect(fake.journal?.phase).toBe("owner-cleanup-required"); + expect(fake.finalization).toBeNull(); + expect(fake.replacement).toBeNull(); + expect(fake.original?.State?.Running).toBe(false); + expectEventBefore(fake.events, `rm:${NEW_ID}`, "journal:owner-cleanup-required"); + + const ownerTransitions = fake.events.filter( + (event) => event === "journal:owner-cleanup-required", + ).length; + const mutationsAfterFirstRecovery = dockerMutationEvents(fake.events); + await expect(restarted.recoverUnfinishedTransactions()).resolves.toMatchObject({ + receipts: [], + failures: [{ code: "owner-cleanup-required" }], + }); + expect(fake.events.filter((event) => event === "journal:owner-cleanup-required")).toHaveLength( + ownerTransitions, + ); + expect(fake.journal?.phase).toBe("owner-cleanup-required"); + expect(fake.finalization).toBeNull(); + expect(dockerMutationEvents(fake.events)).toEqual(mutationsAfterFirstRecovery); + + fake.removeOriginalExternally(); + await expect(restarted.recoverUnfinishedTransactions()).resolves.toMatchObject({ + receipts: [ + { + bootstrapIdentity: IDENTITY, + sourcePhase: "owner-cleanup-required", + outcome: "rolled-back", + }, + ], + failures: [], + }); + expect(fake.journal).toBeNull(); + expect(fake.finalization?.phase).toBe("rolled-back"); + expect(fake.replacement).toBeNull(); + await expect(restarted.recoverUnfinishedTransactions()).resolves.toEqual({ + receipts: [], + failures: [], + }); + }); + + it("publishes owner cleanup only after shared rollback, replacement cleanup, and restoration", async () => { + const fake = fixture({ + agent: "openclaw", + journalTransitionFailures: { + "rollback-authorized": new Error("injected crash after durable rollback fence"), + }, + sharedState: "pending", + }); + const transaction = await prepareTransaction(fake, "openclaw"); + const replacement = await transaction.adapter.activateBootstrapReplacement({ + handle: transaction.handle, + snapshot: transaction.snapshot, + prepared: transaction.prepared, + durablePreparation: transaction.durable, + }); + + await expect( + transaction.adapter.finalizeBootstrap({ + outcome: "rollback", + handle: transaction.handle, + snapshot: transaction.snapshot, + prepared: transaction.prepared, + durablePreparation: transaction.durable, + replacement, + completion: null, + }), + ).rejects.toThrow("crash after durable rollback fence"); + expect(fake.journal?.phase).toBe("rollback-authorized"); + expect(fake.sharedState).toBe("pending"); + + const restarted = createDockerManagedBootstrapAdapter(fake.deps); + await expect(restarted.recoverUnfinishedTransactions()).resolves.toMatchObject({ + receipts: [], + failures: [{ sourcePhase: "owner-cleanup-required", code: "owner-cleanup-required" }], + }); + expect(fake.sharedState).toBe("none"); + expect(fake.replacement).toBeNull(); + expect(fake.original?.State?.Running).toBe(false); + expect(fake.journal?.phase).toBe("owner-cleanup-required"); + expect(fake.finalization).toBeNull(); + expectEventBefore(fake.events, "shared:rollback", `rm:${NEW_ID}`); + expectEventBefore(fake.events, `rm:${NEW_ID}`, "journal:owner-cleanup-required"); + + const mutationsAfterFirstRecovery = dockerMutationEvents(fake.events); + await expect(restarted.recoverUnfinishedTransactions()).resolves.toMatchObject({ + receipts: [], + failures: [{ sourcePhase: "owner-cleanup-required", code: "owner-cleanup-required" }], + }); + expect(dockerMutationEvents(fake.events)).toEqual(mutationsAfterFirstRecovery); + + fake.removeOriginalExternally(); + await expect(restarted.recoverUnfinishedTransactions()).resolves.toMatchObject({ + receipts: [{ outcome: "rolled-back", sourcePhase: "owner-cleanup-required" }], + failures: [], + }); + }); + + it("retains owner authority while exact runtime presence is unknown", async () => { + const fake = fixture({ + journalCreateFailures: [new Error("injected crash after durable staged fence")], + }); + const transaction = await prepareTransaction(fake); + await expect( + transaction.adapter.activateBootstrapReplacement({ + handle: transaction.handle, + snapshot: transaction.snapshot, + prepared: transaction.prepared, + durablePreparation: transaction.durable, + }), + ).rejects.toThrow("crash after durable staged fence"); + + const restarted = createDockerManagedBootstrapAdapter(fake.deps); + await restarted.recoverUnfinishedTransactions(); + expect(fake.journal?.phase).toBe("owner-cleanup-required"); + fake.setDockerInspectUnknown(OLD_ID, true); + + await expect(restarted.recoverUnfinishedTransactions()).resolves.toMatchObject({ + receipts: [], + failures: [ + { + sourcePhase: "owner-cleanup-required", + code: "commit-state-indeterminate", + retryable: true, + }, + ], + }); + expect(fake.journal?.phase).toBe("owner-cleanup-required"); + expect(fake.finalization).toBeNull(); + + fake.setDockerInspectUnknown(OLD_ID, false); + fake.removeOriginalExternally(); + await expect(restarted.recoverUnfinishedTransactions()).resolves.toMatchObject({ + receipts: [{ outcome: "rolled-back" }], + failures: [], + }); + }); + + it("retains durable commit authority when image receipt retirement fails", async () => { + const fake = fixture({ + agent: "langchain-deepagents-code", + dockerRemoveFailures: [new Error("injected crash before exact Docker removal")], + sharedReceiptClearFailures: [new Error("injected image receipt cleanup failure")], + sharedState: "pending", + }); + const transaction = await prepareTransaction(fake, "langchain-deepagents-code"); + const replacement = await transaction.adapter.activateBootstrapReplacement({ + handle: transaction.handle, + snapshot: transaction.snapshot, + prepared: transaction.prepared, + durablePreparation: transaction.durable, + }); + const completion = await transaction.adapter.awaitBootstrap({ + handle: transaction.handle, + snapshot: transaction.snapshot, + replacement, + timeoutSecs: 1, + }); + + await expect( + transaction.adapter.finalizeBootstrap({ + outcome: "commit", + handle: transaction.handle, + snapshot: transaction.snapshot, + prepared: transaction.prepared, + durablePreparation: transaction.durable, + replacement, + completion, + }), + ).rejects.toThrow("crash before exact Docker removal"); + expect(fake.journal?.phase).toBe("shared-state-committed"); + expect(fake.sharedState).toBe("committed"); + + const restarted = createDockerManagedBootstrapAdapter(fake.deps); + await expect(restarted.recoverUnfinishedTransactions()).resolves.toMatchObject({ + receipts: [], + failures: [ + { + sourcePhase: "shared-state-committed", + code: "durable-cleanup-pending", + retryable: true, + }, + ], + }); + expect(fake.journal?.phase).toBe("shared-state-committed"); + expect(fake.finalization).toBeNull(); + expect(fake.sharedState).toBe("committed"); + + const mutationsAfterFailedRetirement = dockerMutationEvents(fake.events); + await expect(restarted.recoverUnfinishedTransactions()).resolves.toMatchObject({ + receipts: [{ outcome: "committed", sourcePhase: "shared-state-committed" }], + failures: [], + }); + expect(fake.journal).toBeNull(); + expect(fake.finalization?.phase).toBe("committed"); + expect(fake.sharedState).toBe("none"); + expect(dockerMutationEvents(fake.events)).toEqual(mutationsAfterFailedRetirement); + }); + + it("retains durable commit authority after a non-zero Docker removal result", async () => { + const fake = fixture({ + dockerRemoveFailures: [new Error("injected crash before exact Docker removal")], + dockerRemoveResults: [{ status: 1, stderr: "injected non-zero Docker removal" }], + sharedState: "pending", + }); + const transaction = await prepareTransaction(fake); + const replacement = await transaction.adapter.activateBootstrapReplacement({ + handle: transaction.handle, + snapshot: transaction.snapshot, + prepared: transaction.prepared, + durablePreparation: transaction.durable, + }); + const completion = await transaction.adapter.awaitBootstrap({ + handle: transaction.handle, + snapshot: transaction.snapshot, + replacement, + timeoutSecs: 1, + }); + + await expect( + transaction.adapter.finalizeBootstrap({ + outcome: "commit", + handle: transaction.handle, + snapshot: transaction.snapshot, + prepared: transaction.prepared, + durablePreparation: transaction.durable, + replacement, + completion, + }), + ).rejects.toThrow("crash before exact Docker removal"); + + const restarted = createDockerManagedBootstrapAdapter(fake.deps); + await expect(restarted.recoverUnfinishedTransactions()).resolves.toMatchObject({ + receipts: [], + failures: [ + { + sourcePhase: "shared-state-committed", + code: "durable-cleanup-pending", + detail: expect.stringContaining("injected non-zero Docker removal"), + }, + ], + }); + expect(fake.journal?.phase).toBe("shared-state-committed"); + expect(fake.finalization).toBeNull(); + expect(fake.original).not.toBeNull(); + + await expect(restarted.recoverUnfinishedTransactions()).resolves.toMatchObject({ + receipts: [{ sourcePhase: "shared-state-committed", outcome: "committed" }], + failures: [], + }); + expect(fake.journal).toBeNull(); + expect(fake.finalization?.phase).toBe("committed"); + }); + + it("compacts a terminal commit journal after another restart interruption", async () => { + const fake = fixture({ + agent: "langchain-deepagents-code", + dockerRemoveFailures: [new Error("injected crash before exact Docker removal")], + journalRemoveFailures: [new Error("injected crash before terminal journal removal")], + sharedState: "pending", + }); + const transaction = await prepareTransaction(fake, "langchain-deepagents-code"); + const replacement = await transaction.adapter.activateBootstrapReplacement({ + handle: transaction.handle, + snapshot: transaction.snapshot, + prepared: transaction.prepared, + durablePreparation: transaction.durable, + }); + const completion = await transaction.adapter.awaitBootstrap({ + handle: transaction.handle, + snapshot: transaction.snapshot, + replacement, + timeoutSecs: 1, + }); + + await expect( + transaction.adapter.finalizeBootstrap({ + outcome: "commit", + handle: transaction.handle, + snapshot: transaction.snapshot, + prepared: transaction.prepared, + durablePreparation: transaction.durable, + replacement, + completion, + }), + ).rejects.toThrow("crash before exact Docker removal"); + + const restarted = createDockerManagedBootstrapAdapter(fake.deps); + await expect(restarted.recoverUnfinishedTransactions()).resolves.toMatchObject({ + receipts: [], + failures: [ + { + sourcePhase: "shared-state-committed", + code: "provider-recovery-failed", + detail: "injected crash before terminal journal removal", + }, + ], + }); + expect(fake.journal?.phase).toBe("shared-state-committed"); + expect(fake.finalization?.phase).toBe("committed"); + + const resumed = createDockerManagedBootstrapAdapter(fake.deps); + await expect(resumed.recoverUnfinishedTransactions()).resolves.toMatchObject({ + receipts: [{ sourcePhase: "shared-state-committed", outcome: "committed" }], + failures: [], + }); + expect(fake.journal).toBeNull(); + expect(fake.sharedState).toBe("none"); + expect(fake.replacement?.State?.Running).toBe(true); + }); + + it("isolates identity-first failures and returns bounded lossless mixed evidence", async () => { + const fake = fixture({ + dockerRemoveFailures: [new Error("injected crash before exact Docker removal")], + journalRemoveFailures: [new Error("injected crash before terminal journal removal")], + sharedState: "pending", + }); + const transaction = await prepareTransaction(fake); + const replacement = await transaction.adapter.activateBootstrapReplacement({ + handle: transaction.handle, + snapshot: transaction.snapshot, + prepared: transaction.prepared, + durablePreparation: transaction.durable, + }); + const completion = await transaction.adapter.awaitBootstrap({ + handle: transaction.handle, + snapshot: transaction.snapshot, + replacement, + timeoutSecs: 1, + }); + await expect( + transaction.adapter.finalizeBootstrap({ + outcome: "commit", + handle: transaction.handle, + snapshot: transaction.snapshot, + prepared: transaction.prepared, + durablePreparation: transaction.durable, + replacement, + completion, + }), + ).rejects.toThrow("crash before exact Docker removal"); + await createDockerManagedBootstrapAdapter(fake.deps).recoverUnfinishedTransactions(); + expect(fake.finalization?.phase).toBe("committed"); + expect(fake.journal).not.toBeNull(); + + const badIdentity = "0".repeat(64); + const delegate = fake.deps.journalStore as DockerManagedBootstrapJournalStore; + const failUnreadableRecord = (): never => { + throw new Error(`${"💥".repeat(3000)}\0tail`); + }; + const mixedStore: DockerManagedBootstrapJournalStore = { + ...delegate, + listUnfinishedIdentities: () => [badIdentity, IDENTITY], + load(bootstrapIdentity) { + return bootstrapIdentity === badIdentity + ? failUnreadableRecord() + : delegate.load(bootstrapIdentity); + }, + }; + const adapter = createDockerManagedBootstrapAdapter({ ...fake.deps, journalStore: mixedStore }); + + const report = await recoverManagedBootstrapTransactions(adapter); + + expect(report.receipts).toMatchObject([ + { bootstrapIdentity: IDENTITY, sourcePhase: "shared-state-committed", outcome: "committed" }, + ]); + expect(report.failures).toMatchObject([ + { + bootstrapIdentity: badIdentity, + providerId: "docker", + sourcePhase: null, + sandbox: null, + code: "provider-recovery-failed", + }, + ]); + expect(report.failures[0]?.detail).not.toContain("\0"); + expect(Buffer.byteLength(report.failures[0]?.detail ?? "", "utf8")).toBeLessThanOrEqual(8192); + expect(Object.isFrozen(report)).toBe(true); + expect(Object.isFrozen(report.receipts)).toBe(true); + expect(Object.isFrozen(report.failures)).toBe(true); + expect(report.receipts.every(Object.isFrozen)).toBe(true); + expect(report.failures.every(Object.isFrozen)).toBe(true); + expect(fake.journal).toBeNull(); + }); +}); diff --git a/src/lib/onboard/managed-bootstrap/docker-runtime.ts b/src/lib/onboard/managed-bootstrap/docker-runtime.ts index 6da0f754f1b..dba07739d18 100644 --- a/src/lib/onboard/managed-bootstrap/docker-runtime.ts +++ b/src/lib/onboard/managed-bootstrap/docker-runtime.ts @@ -21,6 +21,7 @@ import { finalizeManagedBootstrapSequence, MANAGED_BOOTSTRAP_SCHEMA_VERSION, prepareManagedBootstrapSequence, + recoverManagedBootstrapTransactions, } from "./adapter"; import { createDockerManagedBootstrapAdapter } from "./docker"; import type { @@ -137,6 +138,9 @@ function createDockerLifecycle( return { launchArgv: input.launchArgv, patch, + async recoverUnfinished() { + return recoverManagedBootstrapTransactions(adapter); + }, async prepareNetwork() { if (input.route !== "compatibility") return; const { enforceDockerGpuPatchPreserveNetwork } = await import( diff --git a/src/lib/onboard/managed-bootstrap/docker-test-fixture.ts b/src/lib/onboard/managed-bootstrap/docker-test-fixture.ts index 088a3ed7a05..dd1a56a0f9c 100644 --- a/src/lib/onboard/managed-bootstrap/docker-test-fixture.ts +++ b/src/lib/onboard/managed-bootstrap/docker-test-fixture.ts @@ -26,6 +26,8 @@ import { DockerManagedBootstrapJournalAcknowledgementLostError, type DockerManagedBootstrapJournalPhase, type DockerManagedBootstrapJournalStore, + sameDockerManagedBootstrapReceipt, + serializeDockerManagedBootstrapFinalizationRecord, } from "./docker-journal"; import { normalizeDockerManagedBootstrapLaunchSpec } from "./docker-spec"; import { @@ -58,6 +60,8 @@ export type DockerFixtureAcknowledgement = | "container:stop" | "journal:create" | "journal:cutover" + | "journal:completion" + | "journal:owner-cleanup-required" | "journal:remove" | "journal:rollback-authorized" | "journal:staged" @@ -65,7 +69,12 @@ export type DockerFixtureAcknowledgement = export type DockerFixtureOptions = { readonly agent?: ManagedStartupAgent; + readonly dockerRemoveFailures?: readonly Error[]; + readonly dockerRemoveResults?: readonly FixtureCommandResult[]; + readonly dockerInspectUnknownIds?: readonly string[]; readonly dockerStartResults?: Readonly>; + readonly journalCreateFailures?: readonly Error[]; + readonly journalRemoveFailures?: readonly Error[]; readonly journalTransitionFailures?: Partial< Readonly> >; @@ -73,6 +82,7 @@ export type DockerFixtureOptions = { readonly ownerId?: string; readonly sharedState?: "committed" | "none" | "pending"; readonly sharedStateCommitResult?: FixtureCommandResult; + readonly sharedReceiptClearFailures?: readonly Error[]; }; function agentInputs(agent: ManagedStartupAgent = "hermes") { @@ -214,6 +224,12 @@ export function fixture(options: DockerFixtureOptions = {}) { let finalization: DockerManagedBootstrapFinalizationRecord | null = null; let sharedState: "committed" | "none" | "pending" = options.sharedState ?? "none"; const events: string[] = []; + const dockerRemoveFailures = [...(options.dockerRemoveFailures ?? [])]; + const dockerRemoveResults = [...(options.dockerRemoveResults ?? [])]; + const journalCreateFailures = [...(options.journalCreateFailures ?? [])]; + const journalRemoveFailures = [...(options.journalRemoveFailures ?? [])]; + const sharedReceiptClearFailures = [...(options.sharedReceiptClearFailures ?? [])]; + const dockerInspectUnknownIds = new Set(options.dockerInspectUnknownIds ?? []); const lostAcknowledgements = new Set(options.lostAcknowledgements ?? []); const losesAcknowledgement = (operation: DockerFixtureAcknowledgement) => lostAcknowledgements.has(operation); @@ -223,6 +239,13 @@ export function fixture(options: DockerFixtureOptions = {}) { create(value) { journal = structuredClone(value); events.push("journal:staged"); + const injectedFailure = journalCreateFailures.shift(); + switch (injectedFailure) { + case undefined: + break; + default: + throw injectedFailure; + } if (losesAcknowledgement("journal:create")) { throw new DockerManagedBootstrapJournalAcknowledgementLostError( "lost journal create acknowledgement", @@ -230,6 +253,7 @@ export function fixture(options: DockerFixtureOptions = {}) { } }, load: () => copyJournal(), + listUnfinishedIdentities: () => (journal ? [journal.bootstrapIdentity] : []), transition(_identity, expected, next) { const current = journal !== null && journal.phase === expected @@ -252,12 +276,17 @@ export function fixture(options: DockerFixtureOptions = {}) { } if ( journal.commitReceipt !== null && - JSON.stringify(journal.commitReceipt) !== JSON.stringify(receipt) + !sameDockerManagedBootstrapReceipt("completion", journal.commitReceipt, receipt) ) { throw new Error("completion changed"); } journal = { ...journal, commitReceipt: structuredClone(receipt) }; events.push("journal:completion"); + if (losesAcknowledgement("journal:completion")) { + throw new DockerManagedBootstrapJournalAcknowledgementLostError( + "lost journal completion acknowledgement", + ); + } return structuredClone(journal); }, remove(_identity, expected) { @@ -265,6 +294,13 @@ export function fixture(options: DockerFixtureOptions = {}) { void (current !== null && expected.includes(current.phase) ? current : failFixture("stale journal remove")); + const injectedFailure = journalRemoveFailures.shift(); + switch (injectedFailure) { + case undefined: + break; + default: + throw injectedFailure; + } journal = null; events.push("journal:removed"); if (losesAcknowledgement("journal:remove")) { @@ -274,7 +310,11 @@ export function fixture(options: DockerFixtureOptions = {}) { } }, recordFinalization(value) { - if (finalization && JSON.stringify(finalization) !== JSON.stringify(value)) { + if ( + finalization && + serializeDockerManagedBootstrapFinalizationRecord(finalization) !== + serializeDockerManagedBootstrapFinalizationRecord(value) + ) { throw new Error("finalization changed"); } finalization = structuredClone(value); @@ -334,6 +374,9 @@ export function fixture(options: DockerFixtureOptions = {}) { return ok(original ? OLD_ID : ""); case "inspect": { const id = String(args[3] ?? ""); + if (dockerInspectUnknownIds.has(id)) { + return { status: 1, stderr: `injected unknown inspect state for ${id}` }; + } try { inspect(id); return ok(`[{"Id":"${id}"}]`); @@ -398,6 +441,10 @@ export function fixture(options: DockerFixtureOptions = {}) { return result; } case args.includes("--clear-shared-state-commit-receipt"): + { + const injectedFailure = sharedReceiptClearFailures.shift(); + if (injectedFailure) throw injectedFailure; + } sharedState = "none"; events.push("shared:clear"); return ok(); @@ -453,17 +500,28 @@ export function fixture(options: DockerFixtureOptions = {}) { }), dockerRm: vi.fn((id) => { events.push(`rm:${id}`); - switch (id) { - case OLD_ID: - original = null; - break; - case NEW_ID: - replacement = null; + const injectedFailure = dockerRemoveFailures.shift(); + switch (injectedFailure) { + case undefined: break; + default: + throw injectedFailure; + } + const result = dockerRemoveResults.shift() ?? ok(); + switch (result.status) { + case 0: + switch (id) { + case OLD_ID: + original = null; + break; + case NEW_ID: + replacement = null; + break; + } } return losesAcknowledgement("container:remove") ? { status: 1, stderr: "lost rm acknowledgement" } - : ok(); + : result; }), runCaptureOpenshell: vi.fn(() => `Name: alpha\nID: ${options.ownerId ?? "sandbox-alpha"}\n`), runOpenshell: vi.fn(() => ok()), @@ -487,6 +545,14 @@ export function fixture(options: DockerFixtureOptions = {}) { get sharedState() { return sharedState; }, + removeOriginalExternally() { + original = null as unknown as DockerContainerInspect; + events.push(`external-rm:${OLD_ID}`); + }, + setDockerInspectUnknown(runtimeId: string, indeterminate: boolean) { + if (indeterminate) dockerInspectUnknownIds.add(runtimeId); + else dockerInspectUnknownIds.delete(runtimeId); + }, }; } diff --git a/src/lib/onboard/managed-bootstrap/docker.test.ts b/src/lib/onboard/managed-bootstrap/docker.test.ts index 8e94b573589..cf1e1e63fdd 100644 --- a/src/lib/onboard/managed-bootstrap/docker.test.ts +++ b/src/lib/onboard/managed-bootstrap/docker.test.ts @@ -24,6 +24,12 @@ import { SUPPORTED_AGENTS, } from "./docker-test-fixture"; +function expectEventBefore(events: readonly string[], before: string, after: string): void { + expect(events).toContain(before); + expect(events).toContain(after); + expect(events.indexOf(before)).toBeLessThan(events.indexOf(after)); +} + describe("Docker managed bootstrap adapter", () => { it("publishes durable commit authority before deleting the rollback backup after lost acknowledgements", async () => { const fake = fixture({ @@ -35,6 +41,7 @@ describe("Docker managed bootstrap adapter", () => { "container:stop", "journal:create", "journal:cutover", + "journal:completion", "journal:remove", "journal:shared-state-committed", ], @@ -71,12 +78,8 @@ describe("Docker managed bootstrap adapter", () => { durablePreparation: reorderedDurable, }); const order = fake.events; - expect(order).toContain("authority:recorded"); - expect(order).toContain("journal:staged"); - expect(order.indexOf("journal:staged")).toBeGreaterThan(order.indexOf("authority:recorded")); - expect(order).toContain("journal:cutover"); - expect(order).toContain(`stop:${OLD_ID}`); - expect(order.indexOf("journal:cutover")).toBeLessThan(order.indexOf(`stop:${OLD_ID}`)); + expectEventBefore(order, "authority:recorded", "journal:staged"); + expectEventBefore(order, "journal:cutover", `stop:${OLD_ID}`); expect(fake.journal).toMatchObject({ phase: "cutover", originalRuntimeId: OLD_ID, @@ -124,11 +127,8 @@ describe("Docker managed bootstrap adapter", () => { completion: reorderedCommitReceipt, }); expect(finalized).toMatchObject({ outcome: "committed" }); - expect(fake.events).toContain("journal:shared-state-committed"); - expect(fake.events).toContain(`rm:${OLD_ID}`); - expect(fake.events.indexOf("journal:shared-state-committed")).toBeLessThan( - fake.events.indexOf(`rm:${OLD_ID}`), - ); + expectEventBefore(fake.events, "journal:shared-state-committed", `rm:${OLD_ID}`); + expectEventBefore(fake.events, "finalization:committed", "journal:removed"); expect(fake.journal).toBeNull(); expect(fake.finalization).toMatchObject({ phase: "committed", commitReceipt }); expect(fake.sharedState).toBe("none"); @@ -255,12 +255,10 @@ describe("Docker managed bootstrap adapter", () => { completion: null, }), ).rejects.toBeInstanceOf(ManagedBootstrapOwnerCleanupRequiredError); - expect(fake.events).toContain("journal:rollback-authorized"); - expect(fake.events).toContain(`rm:${NEW_ID}`); - expect(fake.events.indexOf("journal:rollback-authorized")).toBeLessThan( - fake.events.indexOf(`rm:${NEW_ID}`), - ); - expect(fake.journal).toBeNull(); + expectEventBefore(fake.events, "journal:rollback-authorized", `rm:${NEW_ID}`); + expectEventBefore(fake.events, `rm:${NEW_ID}`, "journal:owner-cleanup-required"); + expect(fake.finalization).toBeNull(); + expect(fake.journal?.phase).toBe("owner-cleanup-required"); expect(fake.replacement).toBeNull(); expect(fake.original).not.toBeNull(); expect(fake.original?.Name).toBe("/openshell-alpha"); @@ -302,12 +300,10 @@ describe("Docker managed bootstrap adapter", () => { completion: null, }), ).rejects.toBeInstanceOf(ManagedBootstrapOwnerCleanupRequiredError); - expect(fake.events).toContain("journal:rollback-authorized"); - expect(fake.events).toContain(`rm:${NEW_ID}`); - expect(fake.events.indexOf("journal:rollback-authorized")).toBeLessThan( - fake.events.indexOf(`rm:${NEW_ID}`), - ); - expect(fake.journal).toBeNull(); + expectEventBefore(fake.events, "journal:rollback-authorized", `rm:${NEW_ID}`); + expectEventBefore(fake.events, `rm:${NEW_ID}`, "journal:owner-cleanup-required"); + expect(fake.finalization).toBeNull(); + expect(fake.journal?.phase).toBe("owner-cleanup-required"); }); it("fences rollback when image-owned shared state is already committed", async () => { @@ -411,8 +407,12 @@ describe("Docker managed bootstrap adapter", () => { completion: null, }), ).rejects.toBeInstanceOf(ManagedBootstrapOwnerCleanupRequiredError); - expect(fake.journal).toBeNull(); + expect(fake.journal?.phase).toBe("owner-cleanup-required"); + expect(fake.finalization).toBeNull(); expect(fake.replacement).toBeNull(); + expect(fake.original?.State?.Running).toBe(false); + expectEventBefore(fake.events, "shared:rollback", `rm:${NEW_ID}`); + expectEventBefore(fake.events, `rm:${NEW_ID}`, "journal:owner-cleanup-required"); expect( vi.mocked(fake.deps.dockerRun!).mock.calls.some(([args]) => { const agentIndex = args.indexOf("--agent"); diff --git a/src/lib/onboard/managed-bootstrap/docker.ts b/src/lib/onboard/managed-bootstrap/docker.ts index e633ee9f2ed..8dc146f1df7 100644 --- a/src/lib/onboard/managed-bootstrap/docker.ts +++ b/src/lib/onboard/managed-bootstrap/docker.ts @@ -58,6 +58,9 @@ import { type ManagedBootstrapObservedSnapshot, ManagedBootstrapOwnerCleanupRequiredError, type ManagedBootstrapPreparedReplacementHandle, + type ManagedBootstrapRecoveryFailure, + type ManagedBootstrapRecoveryReceipt, + type ManagedBootstrapRecoveryReport, type ManagedBootstrapReplacementHandle, type ManagedBootstrapReplacementOptions, type ManagedBootstrapSandboxIdentity, @@ -67,10 +70,12 @@ import { createFileDockerManagedBootstrapJournalStore, DOCKER_MANAGED_BOOTSTRAP_FINALIZATION_SCHEMA_VERSION, DOCKER_MANAGED_BOOTSTRAP_JOURNAL_SCHEMA_VERSION, + type DockerManagedBootstrapFinalizationContext, type DockerManagedBootstrapFinalizationRecord, type DockerManagedBootstrapJournal, DockerManagedBootstrapJournalAcknowledgementLostError, type DockerManagedBootstrapJournalStore, + DockerManagedBootstrapLegacyRecordRequiresAgentError, parseDockerManagedBootstrapJournal, sameDockerManagedBootstrapReceipt, serializeDockerManagedBootstrapFinalizationRecord, @@ -102,9 +107,56 @@ const REQUEST_TEMP_PREFIX = "nemoclaw-managed-bootstrap-request"; const COMPLETION_TEMP_PREFIX = "nemoclaw-managed-bootstrap-completion"; const COMPLETION_MAX_BYTES = 4096; const DOCKER_DRIVER_ID = "docker"; +const MAX_RECOVERY_FAILURE_DETAIL_BYTES = 8 * 1024; export const MANAGED_BOOTSTRAP_TRAMPOLINE_EXECUTABLE = "/usr/local/bin/nemoclaw-managed-bootstrap"; +function boundedRecoveryFailureDetail(error: unknown): string { + const raw = (error instanceof Error ? error.message : String(error)).replaceAll("\0", "�"); + const detail = raw.length > 0 ? raw : "Docker recovery failed without diagnostic detail"; + const bytes = Buffer.from(detail, "utf8"); + if (bytes.length <= MAX_RECOVERY_FAILURE_DETAIL_BYTES) return detail; + let bounded = bytes.subarray(0, MAX_RECOVERY_FAILURE_DETAIL_BYTES).toString("utf8"); + while (Buffer.byteLength(bounded, "utf8") > MAX_RECOVERY_FAILURE_DETAIL_BYTES) { + bounded = [...bounded].slice(0, -1).join(""); + } + return bounded; +} + +function dockerManagedBootstrapRecoveryFailure( + bootstrapIdentity: string, + journal: DockerBootstrapTransaction | null, + error: unknown, +): ManagedBootstrapRecoveryFailure { + const legacyJournalContext = + error instanceof DockerManagedBootstrapLegacyRecordRequiresAgentError + ? error.journalContext + : null; + const classified = + error instanceof ManagedBootstrapOwnerCleanupRequiredError + ? { code: "owner-cleanup-required", retryable: true } + : error instanceof ManagedBootstrapDurableCommitCleanupPendingError + ? { code: "durable-cleanup-pending", retryable: true } + : error instanceof ManagedBootstrapCommitStateIndeterminateError + ? { code: "commit-state-indeterminate", retryable: true } + : error instanceof DockerManagedBootstrapLegacyRecordRequiresAgentError + ? { code: "legacy-agent-required", retryable: true } + : { code: "provider-recovery-failed", retryable: true }; + return Object.freeze({ + schemaVersion: MANAGED_BOOTSTRAP_SCHEMA_VERSION, + providerId: journal?.providerId ?? legacyJournalContext?.providerId ?? DOCKER_DRIVER_ID, + // A legacy cutover decision sidecar may have advanced beyond the phase in + // the journal body. Keep the provider phase unknown until agent-bound + // recovery can validate both records, while retaining its exact sandbox. + sourcePhase: journal?.phase ?? null, + sandbox: journal?.sandbox ?? legacyJournalContext?.sandbox ?? null, + bootstrapIdentity, + code: classified.code, + retryable: classified.retryable, + detail: boundedRecoveryFailureDetail(error), + }); +} + type DockerCommandResult = { readonly status?: number | null; readonly stdout?: string | Buffer | null; @@ -1271,24 +1323,26 @@ function retainOwnedWorkloadForOwnerCleanup( `Managed bootstrap owner cleanup refused retention after exact runtime ${runtimeId} ownership changed for sandbox ${sandbox.sandboxId}.`, ); } - let stopped: DockerCommandResult; - try { - stopped = deps.dockerStop(runtimeId, { - ignoreError: true, - suppressOutput: true, - timeout: DOCKER_GPU_PATCH_STOP_TIMEOUT_MS, - }); - } catch (error) { - throw new Error( - `Managed bootstrap could not quiesce retained sandbox ${sandbox.sandboxId} exact runtime ${runtimeId}: ${ - error instanceof Error ? error.message : String(error) - }`, + if (!isExplicitlyStopped(inspect)) { + let stopped: DockerCommandResult; + try { + stopped = deps.dockerStop(runtimeId, { + ignoreError: true, + suppressOutput: true, + timeout: DOCKER_GPU_PATCH_STOP_TIMEOUT_MS, + }); + } catch (error) { + throw new Error( + `Managed bootstrap could not quiesce retained sandbox ${sandbox.sandboxId} exact runtime ${runtimeId}: ${ + error instanceof Error ? error.message : String(error) + }`, + ); + } + assertZero( + stopped, + `Managed bootstrap could not quiesce retained sandbox ${sandbox.sandboxId} exact runtime ${runtimeId}`, ); } - assertZero( - stopped, - `Managed bootstrap could not quiesce retained sandbox ${sandbox.sandboxId} exact runtime ${runtimeId}`, - ); let retained: DockerContainerInspect; try { retained = inspectExact(runtimeId, deps); @@ -1416,6 +1470,16 @@ function managedSharedStateTransaction( } as const; } +function recoveredManagedSharedStateTransaction(journal: DockerBootstrapTransaction) { + return { + agent: journal.agent, + bootstrapIdentity: journal.bootstrapIdentity, + containerId: journal.replacementRuntimeId, + image: journal.runtimeImageContentId, + profileFingerprint: journal.profileFingerprint, + } as const; +} + function sameDockerBootstrapJournal( left: DockerBootstrapTransaction, right: DockerBootstrapTransaction, @@ -1466,7 +1530,7 @@ function createDockerBootstrapJournalDurably( function transitionDockerBootstrapJournalDurably( journal: DockerBootstrapTransaction, - next: "cutover" | "rollback-authorized" | "shared-state-committed", + next: "cutover" | "rollback-authorized" | "owner-cleanup-required" | "shared-state-committed", deps: ResolvedDeps, ): DockerBootstrapTransaction { try { @@ -1736,51 +1800,49 @@ export function createDockerManagedBootstrapAdapter( dependencies: DockerManagedBootstrapDeps = {}, ): DockerManagedBootstrapAdapter { const deps = resolveDeps(dependencies); + const finalizationContext = (handle: ManagedBootstrapHeldWorkloadHandle) => + Object.freeze({ + bootstrapIdentity: handle.bootstrapIdentity, + providerId: handle.sandbox.driverId, + agent: handle.plan.profile.agent, + sandbox: handle.sandbox, + planFingerprint: createManagedBootstrapPlanFingerprint(handle.plan), + profileFingerprint: handle.plan.profile.fingerprint, + imageReference: expectedImageReference( + handle.plan.image.repository, + handle.plan.image.manifestDigest, + ), + }); const finalizationRecord = ( handle: ManagedBootstrapHeldWorkloadHandle, ): DockerManagedBootstrapFinalizationRecord | null => { - const record = deps.journalStore.loadFinalization(handle.bootstrapIdentity); + const context = finalizationContext(handle); + const record = deps.journalStore.loadFinalization(handle.bootstrapIdentity, context); if (!record) return null; if ( - record.providerId !== handle.sandbox.driverId || - record.sandbox.sandboxName !== handle.sandbox.sandboxName || - record.sandbox.sandboxId !== handle.sandbox.sandboxId || - record.sandbox.driverId !== handle.sandbox.driverId || - record.planFingerprint !== createManagedBootstrapPlanFingerprint(handle.plan) || - record.profileFingerprint !== handle.plan.profile.fingerprint || - record.imageReference !== - expectedImageReference(handle.plan.image.repository, handle.plan.image.manifestDigest) + record.bootstrapIdentity !== context.bootstrapIdentity || + record.providerId !== context.providerId || + record.agent !== context.agent || + record.sandbox.sandboxName !== context.sandbox.sandboxName || + record.sandbox.sandboxId !== context.sandbox.sandboxId || + record.sandbox.driverId !== context.sandbox.driverId || + record.planFingerprint !== context.planFingerprint || + record.profileFingerprint !== context.profileFingerprint || + record.imageReference !== context.imageReference ) { throw new Error("Managed bootstrap finalization record does not match its durable identity."); } return record; }; - const persistFinalization = ( - handle: ManagedBootstrapHeldWorkloadHandle, - phase: "committed" | "rolled-back", - commitReceipt: ManagedBootstrapCompletionReceipt | null, - cleanupReceipt: ManagedBootstrapFinalizationReceipt, + const persistFinalizationRecord = ( + record: DockerManagedBootstrapFinalizationRecord, + context: DockerManagedBootstrapFinalizationContext, ): ManagedBootstrapFinalizationReceipt => { - const record = Object.freeze({ - schemaVersion: DOCKER_MANAGED_BOOTSTRAP_FINALIZATION_SCHEMA_VERSION, - phase, - bootstrapIdentity: handle.bootstrapIdentity, - providerId: handle.sandbox.driverId, - sandbox: handle.sandbox, - planFingerprint: createManagedBootstrapPlanFingerprint(handle.plan), - profileFingerprint: handle.plan.profile.fingerprint, - imageReference: expectedImageReference( - handle.plan.image.repository, - handle.plan.image.manifestDigest, - ), - commitReceipt, - cleanupReceipt, - } satisfies DockerManagedBootstrapFinalizationRecord); const serialized = serializeDockerManagedBootstrapFinalizationRecord(record); try { - deps.journalStore.recordFinalization(record); + deps.journalStore.recordFinalization(record, context); } catch (error) { - const recovered = deps.journalStore.loadFinalization(handle.bootstrapIdentity); + const recovered = deps.journalStore.loadFinalization(record.bootstrapIdentity, context); if ( !recovered || serializeDockerManagedBootstrapFinalizationRecord(recovered) !== serialized @@ -1788,12 +1850,34 @@ export function createDockerManagedBootstrapAdapter( throw error; } } - const persisted = deps.journalStore.loadFinalization(handle.bootstrapIdentity); + const persisted = deps.journalStore.loadFinalization(record.bootstrapIdentity, context); if (!persisted || serializeDockerManagedBootstrapFinalizationRecord(persisted) !== serialized) { throw new Error("Managed bootstrap finalization receipt was not durably re-readable."); } return persisted.cleanupReceipt; }; + const persistFinalization = ( + handle: ManagedBootstrapHeldWorkloadHandle, + phase: "committed" | "rolled-back", + commitReceipt: ManagedBootstrapCompletionReceipt | null, + cleanupReceipt: ManagedBootstrapFinalizationReceipt, + ): ManagedBootstrapFinalizationReceipt => { + const context = finalizationContext(handle); + const record = Object.freeze({ + schemaVersion: DOCKER_MANAGED_BOOTSTRAP_FINALIZATION_SCHEMA_VERSION, + phase, + bootstrapIdentity: context.bootstrapIdentity, + providerId: context.providerId, + agent: context.agent, + sandbox: context.sandbox, + planFingerprint: context.planFingerprint, + profileFingerprint: context.profileFingerprint, + imageReference: context.imageReference, + commitReceipt, + cleanupReceipt, + } satisfies DockerManagedBootstrapFinalizationRecord); + return persistFinalizationRecord(record, context); + }; const completedRollback = ( handle: ManagedBootstrapHeldWorkloadHandle, alreadyRolledBack: boolean, @@ -1811,6 +1895,73 @@ export function createDockerManagedBootstrapAdapter( } satisfies ManagedBootstrapFinalizationReceipt); return persistFinalization(handle, "rolled-back", null, receipt); }; + const requireExactOwnerCleanup = (journal: DockerBootstrapTransaction): void => { + const presence = probeExactDockerContainerAbsence(journal.originalRuntimeId, deps); + if (presence === "unknown") { + throw new ManagedBootstrapCommitStateIndeterminateError({ + bootstrapIdentity: journal.bootstrapIdentity, + runtimeId: journal.originalRuntimeId, + detail: "exact owner-cleanup runtime presence is unknown", + }); + } + if (presence === "absent") return; + const original = inspectExact(journal.originalRuntimeId, deps); + assertTransactionOriginal(journal, original); + if ( + dockerContainerName(original) !== journal.originalName || + normalizeDockerManagedBootstrapLaunchSpec(original).hash !== journal.originalSpecHash || + (journal.phase === "owner-cleanup-required" && !isExplicitlyStopped(original)) || + (journal.phase !== "owner-cleanup-required" && + !isExplicitlyStopped(original) && + !isStableRunning(original)) + ) { + throw new ManagedBootstrapCommitStateIndeterminateError({ + bootstrapIdentity: journal.bootstrapIdentity, + runtimeId: journal.originalRuntimeId, + detail: "owner-cleanup runtime does not match its exact restored durable authority", + }); + } + try { + retainOwnedWorkloadForOwnerCleanup(journal.sandbox, deps, journal.originalRuntimeId); + } catch (error) { + if (!(error instanceof ManagedBootstrapOwnerCleanupRequiredError)) throw error; + if (error.runtimeId !== journal.originalRuntimeId) { + throw new ManagedBootstrapCommitStateIndeterminateError({ + bootstrapIdentity: journal.bootstrapIdentity, + runtimeId: journal.originalRuntimeId, + detail: "owner cleanup retained a runtime other than the durable original", + }); + } + if (journal.phase !== "owner-cleanup-required") { + if (journal.phase !== "staged" && journal.phase !== "rollback-authorized") { + throw new ManagedBootstrapCommitStateIndeterminateError({ + bootstrapIdentity: journal.bootstrapIdentity, + runtimeId: journal.originalRuntimeId, + detail: `owner cleanup cannot be retained from durable phase ${journal.phase}`, + }); + } + const current = deps.journalStore.load(journal.bootstrapIdentity); + if (!current || !sameDockerBootstrapJournal(current, journal)) { + throw new ManagedBootstrapCommitStateIndeterminateError({ + bootstrapIdentity: journal.bootstrapIdentity, + runtimeId: journal.originalRuntimeId, + detail: "durable authority changed before owner cleanup was retained", + }); + } + transitionDockerBootstrapJournalDurably(journal, "owner-cleanup-required", deps); + } + throw error; + } + }; + const completeRollbackTransaction = ( + handle: ManagedBootstrapHeldWorkloadHandle, + journal: DockerBootstrapTransaction, + ): ManagedBootstrapFinalizationReceipt => { + requireExactOwnerCleanup(journal); + const finalization = completedRollback(handle, false); + removeDockerBootstrapJournalDurably(journal, deps); + return finalization; + }; const completedCommit = ( handle: ManagedBootstrapHeldWorkloadHandle, commitReceipt: ManagedBootstrapCompletionReceipt, @@ -1845,6 +1996,333 @@ export function createDockerManagedBootstrapAdapter( alreadyRolledBack: true, }); }; + const persistRecoveredFinalization = ( + journal: DockerBootstrapTransaction, + phase: "committed" | "rolled-back", + commitReceipt: ManagedBootstrapCompletionReceipt | null, + cleanupReceipt: ManagedBootstrapFinalizationReceipt, + ): ManagedBootstrapFinalizationReceipt => { + const record = Object.freeze({ + schemaVersion: DOCKER_MANAGED_BOOTSTRAP_FINALIZATION_SCHEMA_VERSION, + phase, + bootstrapIdentity: journal.bootstrapIdentity, + providerId: journal.providerId, + agent: journal.agent, + sandbox: journal.sandbox, + planFingerprint: journal.planFingerprint, + profileFingerprint: journal.profileFingerprint, + imageReference: journal.imageReference, + commitReceipt, + cleanupReceipt, + } satisfies DockerManagedBootstrapFinalizationRecord); + return persistFinalizationRecord(record, journal); + }; + const recoveredReceipt = ( + journal: DockerBootstrapTransaction, + sourcePhase: DockerBootstrapTransaction["phase"], + finalization: ManagedBootstrapFinalizationReceipt, + ): ManagedBootstrapRecoveryReceipt => + Object.freeze({ + schemaVersion: MANAGED_BOOTSTRAP_SCHEMA_VERSION, + providerId: journal.providerId, + sourcePhase, + sandbox: journal.sandbox, + bootstrapIdentity: journal.bootstrapIdentity, + outcome: finalization.outcome, + finalization, + }); + const compactRecoveredFinalization = ( + journal: DockerBootstrapTransaction, + sourcePhase: DockerBootstrapTransaction["phase"], + ): ManagedBootstrapRecoveryReceipt | null => { + const finalization = deps.journalStore.loadFinalization(journal.bootstrapIdentity, journal); + if (!finalization) return null; + const phaseMatches = + (finalization.phase === "committed" && + journal.phase === "shared-state-committed" && + finalization.commitReceipt !== null && + journal.commitReceipt !== null && + sameDockerManagedBootstrapReceipt( + "completion", + finalization.commitReceipt, + journal.commitReceipt, + )) || + (finalization.phase === "rolled-back" && + (journal.phase === "staged" || + journal.phase === "rollback-authorized" || + journal.phase === "owner-cleanup-required") && + finalization.commitReceipt === null); + if ( + !phaseMatches || + finalization.bootstrapIdentity !== journal.bootstrapIdentity || + finalization.providerId !== journal.providerId || + finalization.agent !== journal.agent || + finalization.sandbox.sandboxName !== journal.sandbox.sandboxName || + finalization.sandbox.sandboxId !== journal.sandbox.sandboxId || + finalization.sandbox.driverId !== journal.sandbox.driverId || + finalization.planFingerprint !== journal.planFingerprint || + finalization.profileFingerprint !== journal.profileFingerprint || + finalization.imageReference !== journal.imageReference + ) { + throw new ManagedBootstrapCommitStateIndeterminateError({ + bootstrapIdentity: journal.bootstrapIdentity, + runtimeId: + journal.phase === "shared-state-committed" + ? journal.replacementRuntimeId + : journal.originalRuntimeId, + detail: "terminal finalization does not match its retained durable journal", + }); + } + if (finalization.phase === "rolled-back") requireExactOwnerCleanup(journal); + removeDockerBootstrapJournalDurably(journal, deps); + return recoveredReceipt(journal, sourcePhase, finalization.cleanupReceipt); + }; + const finishRecoveredRollback = ( + journal: DockerBootstrapTransaction, + sourcePhase: DockerBootstrapTransaction["phase"], + ): ManagedBootstrapRecoveryReceipt => { + requireExactOwnerCleanup(journal); + const cleanupReceipt = Object.freeze({ + schemaVersion: MANAGED_BOOTSTRAP_SCHEMA_VERSION, + sandbox: journal.sandbox, + bootstrapIdentity: journal.bootstrapIdentity, + outcome: "rolled-back", + restoredRuntimeId: null, + restoredSpecHash: null, + heldWorkloadRemoved: true, + alreadyRolledBack: false, + finalizedAt: deps.now().toISOString(), + } satisfies ManagedBootstrapFinalizationReceipt); + const finalization = persistRecoveredFinalization(journal, "rolled-back", null, cleanupReceipt); + removeDockerBootstrapJournalDurably(journal, deps); + return recoveredReceipt(journal, sourcePhase, finalization); + }; + const finishRecoveredCommit = ( + journal: DockerBootstrapTransaction, + sourcePhase: DockerBootstrapTransaction["phase"], + ): ManagedBootstrapRecoveryReceipt => { + if (journal.phase !== "shared-state-committed" || journal.commitReceipt === null) { + throw new ManagedBootstrapCommitStateIndeterminateError({ + bootstrapIdentity: journal.bootstrapIdentity, + runtimeId: journal.replacementRuntimeId, + detail: "durable commit recovery requires its exact completion receipt and commit fence", + }); + } + const replacement = inspectTransactionRuntime(journal, journal.replacementRuntimeId, deps); + if (!replacement) { + throw new ManagedBootstrapCommitStateIndeterminateError({ + bootstrapIdentity: journal.bootstrapIdentity, + runtimeId: journal.replacementRuntimeId, + detail: "the exact committed replacement is absent during restart recovery", + }); + } + assertTransactionReplacement(journal, replacement); + if ( + dockerContainerName(replacement) !== journal.originalName || + !isStableRunning(replacement) || + normalizeDockerManagedBootstrapLaunchSpec(replacement).hash !== journal.replacementSpecHash + ) { + throw new ManagedBootstrapCommitStateIndeterminateError({ + bootstrapIdentity: journal.bootstrapIdentity, + runtimeId: journal.replacementRuntimeId, + detail: "the committed replacement does not match its durable runtime authority", + }); + } + const sharedTransaction = recoveredManagedSharedStateTransaction(journal); + const sharedStatus = probeDockerManagedStartupSharedState( + { transaction: sharedTransaction, profileFingerprint: journal.profileFingerprint }, + deps, + ); + if (sharedStatus === "pending") { + throw new ManagedBootstrapCommitStateIndeterminateError({ + bootstrapIdentity: journal.bootstrapIdentity, + runtimeId: journal.replacementRuntimeId, + detail: "shared state is pending after the durable commit fence", + }); + } + const original = inspectTransactionRuntime(journal, journal.originalRuntimeId, deps); + if (original) { + assertTransactionOriginal(journal, original); + if ( + dockerContainerName(original) !== journal.backupName || + !isExplicitlyStopped(original) || + normalizeDockerManagedBootstrapLaunchSpec({ + ...original, + Name: `/${journal.originalName}`, + }).hash !== journal.originalSpecHash + ) { + throw new ManagedBootstrapCommitStateIndeterminateError({ + bootstrapIdentity: journal.bootstrapIdentity, + runtimeId: journal.originalRuntimeId, + detail: "the exact rollback backup changed before recovered commit cleanup", + }); + } + if (sharedStatus === "none") { + throw new ManagedBootstrapCommitStateIndeterminateError({ + bootstrapIdentity: journal.bootstrapIdentity, + runtimeId: journal.originalRuntimeId, + detail: "the shared commit receipt was retired before exact backup absence was proven", + }); + } + const removed = deps.dockerRm(journal.originalRuntimeId, { + ignoreError: true, + suppressOutput: true, + timeout: DOCKER_GPU_PATCH_TIMEOUT_MS, + }); + if ( + !hasZeroDockerExitStatus(removed) && + probeExactDockerContainerAbsence(journal.originalRuntimeId, deps) !== "absent" + ) { + throw new ManagedBootstrapDurableCommitCleanupPendingError({ + bootstrapIdentity: journal.bootstrapIdentity, + cleanupRuntimeId: journal.originalRuntimeId, + detail: `${commandDetail(removed) || "Docker removal failed"}; exact backup absence was not proven`, + }); + } + } + if (sharedStatus === "committed") { + try { + clearDockerManagedStartupSharedStateCommitReceipt(sharedTransaction, deps); + } catch (error) { + throw new ManagedBootstrapDurableCommitCleanupPendingError({ + bootstrapIdentity: journal.bootstrapIdentity, + cleanupRuntimeId: journal.replacementRuntimeId, + detail: `the image-owned commit receipt could not be retired during restart recovery: ${ + error instanceof Error ? error.message : String(error) + }`, + }); + } + } + if (probeExactDockerContainerAbsence(journal.originalRuntimeId, deps) !== "absent") { + throw new ManagedBootstrapDurableCommitCleanupPendingError({ + bootstrapIdentity: journal.bootstrapIdentity, + cleanupRuntimeId: journal.originalRuntimeId, + detail: "exact rollback-backup absence was not durable after restart recovery", + }); + } + const cleanupReceipt = Object.freeze({ + schemaVersion: MANAGED_BOOTSTRAP_SCHEMA_VERSION, + sandbox: journal.sandbox, + bootstrapIdentity: journal.bootstrapIdentity, + outcome: "committed", + restoredRuntimeId: null, + restoredSpecHash: null, + heldWorkloadRemoved: false, + alreadyRolledBack: false, + finalizedAt: deps.now().toISOString(), + } satisfies ManagedBootstrapFinalizationReceipt); + const finalization = persistRecoveredFinalization( + journal, + "committed", + journal.commitReceipt, + cleanupReceipt, + ); + removeDockerBootstrapJournalDurably(journal, deps); + return recoveredReceipt(journal, sourcePhase, finalization); + }; + const finishRecoveredRollbackPhase = ( + journal: DockerBootstrapTransaction, + sourcePhase: DockerBootstrapTransaction["phase"], + ): ManagedBootstrapRecoveryReceipt => { + if (journal.phase === "owner-cleanup-required") { + return finishRecoveredRollback(journal, sourcePhase); + } + const original = inspectTransactionRuntime(journal, journal.originalRuntimeId, deps); + if (!original) { + throw new ManagedBootstrapCommitStateIndeterminateError({ + bootstrapIdentity: journal.bootstrapIdentity, + runtimeId: journal.originalRuntimeId, + detail: "the exact rollback original is absent during restart recovery", + }); + } + assertTransactionOriginal(journal, original); + const replacement = inspectTransactionRuntime(journal, journal.replacementRuntimeId, deps); + if (replacement) assertTransactionReplacement(journal, replacement); + if (journal.phase === "staged") { + if ( + dockerContainerName(original) !== journal.originalName || + !isStableRunning(original) || + normalizeDockerManagedBootstrapLaunchSpec(original).hash !== journal.originalSpecHash || + (replacement !== null && + (dockerContainerName(replacement) !== journal.replacementStagingName || + !isExplicitlyStopped(replacement))) + ) { + throw new ManagedBootstrapCommitStateIndeterminateError({ + bootstrapIdentity: journal.bootstrapIdentity, + runtimeId: journal.originalRuntimeId, + detail: "staged restart recovery does not match its pre-cutover fence", + }); + } + if (replacement) removeExactReplacement(journal, replacement, deps); + return finishRecoveredRollback(journal, sourcePhase); + } + if (journal.phase !== "cutover" && journal.phase !== "rollback-authorized") { + return finishRecoveredCommit(journal, sourcePhase); + } + let activeJournal = journal; + if (!replacement && dockerContainerName(original) !== journal.originalName) { + throw new ManagedBootstrapCommitStateIndeterminateError({ + bootstrapIdentity: journal.bootstrapIdentity, + runtimeId: journal.replacementRuntimeId, + detail: "the replacement disappeared before exact rollback restoration was proven", + }); + } + if (replacement) { + const sharedTransaction = recoveredManagedSharedStateTransaction(journal); + const sharedStatus = probeDockerManagedStartupSharedState( + { transaction: sharedTransaction, profileFingerprint: journal.profileFingerprint }, + deps, + ); + if (sharedStatus === "committed") { + if (journal.phase === "rollback-authorized") { + throw new ManagedBootstrapCommitStateIndeterminateError({ + bootstrapIdentity: journal.bootstrapIdentity, + runtimeId: journal.replacementRuntimeId, + detail: "shared state committed after durable rollback authorization", + }); + } + activeJournal = transitionDockerBootstrapJournalDurably( + journal, + "shared-state-committed", + deps, + ); + return finishRecoveredCommit(activeJournal, sourcePhase); + } + if (journal.phase === "cutover") { + activeJournal = transitionDockerBootstrapJournalDurably( + journal, + "rollback-authorized", + deps, + ); + } + if (sharedStatus === "pending") { + finalizeDockerManagedStartupSharedState( + { + transaction: sharedTransaction, + supervisorReady: false, + retainContainerAfterRollback: true, + }, + deps, + ); + } + } else if (journal.phase === "cutover") { + activeJournal = transitionDockerBootstrapJournalDurably(journal, "rollback-authorized", deps); + } + restoreOriginal(activeJournal, deps); + const restored = inspectExact(activeJournal.originalRuntimeId, deps); + if ( + !isStableRunning(restored) || + dockerContainerName(restored) !== activeJournal.originalName || + normalizeDockerManagedBootstrapLaunchSpec(restored).hash !== activeJournal.originalSpecHash + ) { + throw new ManagedBootstrapCommitStateIndeterminateError({ + bootstrapIdentity: activeJournal.bootstrapIdentity, + runtimeId: activeJournal.originalRuntimeId, + detail: "restart recovery did not restore the exact original runtime and launch spec", + }); + } + return finishRecoveredRollback(activeJournal, sourcePhase); + }; const rollbackBootstrapNow = ({ handle, snapshot, @@ -1986,6 +2464,9 @@ export function createDockerManagedBootstrapAdapter( replacement, durablePreparation, ); + if (journal.phase === "owner-cleanup-required") { + return completeRollbackTransaction(handle, journal); + } const original = inspectTransactionRuntime(journal, journal.originalRuntimeId, deps); if (!original) { throw new ManagedBootstrapCommitStateIndeterminateError({ @@ -2002,6 +2483,7 @@ export function createDockerManagedBootstrapAdapter( ); if (journal.phase === "staged") { + const stagedJournal: DockerBootstrapTransaction = journal; assertStableRunning(original, "staged original"); if (observedReplacement) { assertExplicitlyStopped(observedReplacement, "staged replacement"); @@ -2020,9 +2502,7 @@ export function createDockerManagedBootstrapAdapter( if (observedReplacement) { removeExactReplacement(journal, observedReplacement, deps); } - removeDockerBootstrapJournalDurably(journal, deps); - retainOwnedWorkloadForOwnerCleanup(handle.sandbox, deps, journal.originalRuntimeId); - return completedRollback(handle, false); + return completeRollbackTransaction(handle, stagedJournal); } if (journal.phase !== "cutover" && journal.phase !== "rollback-authorized") { @@ -2200,9 +2680,7 @@ export function createDockerManagedBootstrapAdapter( ) { throw new Error("Managed bootstrap Docker rollback did not restore its exact original."); } - removeDockerBootstrapJournalDurably(activeJournal, deps); - retainOwnedWorkloadForOwnerCleanup(handle.sandbox, deps, activeJournal.originalRuntimeId); - return completedRollback(handle, false); + return completeRollbackTransaction(handle, activeJournal); }; const commitBootstrapNow = ( handle: ManagedBootstrapHeldWorkloadHandle, @@ -2303,8 +2781,9 @@ export function createDockerManagedBootstrapAdapter( }); } } + const finalization = completedCommit(handle, receipt); removeDockerBootstrapJournalDurably(transaction, deps); - return completedCommit(handle, receipt); + return finalization; }; const finalizeBootstrap = async ( input: Parameters[0], @@ -2493,6 +2972,43 @@ export function createDockerManagedBootstrapAdapter( }); }; return { + async recoverUnfinishedTransactions() { + const receipts: ManagedBootstrapRecoveryReceipt[] = []; + const failures: ManagedBootstrapRecoveryFailure[] = []; + for (const bootstrapIdentity of deps.journalStore.listUnfinishedIdentities()) { + let journal: DockerBootstrapTransaction | null = null; + try { + journal = deps.journalStore.load(bootstrapIdentity); + if (!journal) { + throw new Error("durable journal disappeared after identity enumeration"); + } + const sourcePhase = journal.phase; + const finalized = compactRecoveredFinalization(journal, sourcePhase); + receipts.push( + finalized ?? + (journal.phase === "shared-state-committed" + ? finishRecoveredCommit(journal, sourcePhase) + : finishRecoveredRollbackPhase(journal, sourcePhase)), + ); + } catch (error) { + try { + journal = deps.journalStore.load(bootstrapIdentity) ?? journal; + } catch { + // Preserve the first per-record failure when the durable re-read also fails. + } + failures.push(dockerManagedBootstrapRecoveryFailure(bootstrapIdentity, journal, error)); + } + } + const byBootstrapIdentity = ( + left: ManagedBootstrapRecoveryReceipt | ManagedBootstrapRecoveryFailure, + right: ManagedBootstrapRecoveryReceipt | ManagedBootstrapRecoveryFailure, + ) => left.bootstrapIdentity.localeCompare(right.bootstrapIdentity); + return Object.freeze({ + receipts: Object.freeze(receipts.sort(byBootstrapIdentity)), + failures: Object.freeze(failures.sort(byBootstrapIdentity)), + } satisfies ManagedBootstrapRecoveryReport); + }, + async createHeldWorkload(input) { if ( input.plan.schemaVersion !== MANAGED_BOOTSTRAP_SCHEMA_VERSION || @@ -2747,6 +3263,7 @@ export function createDockerManagedBootstrapAdapter( phase: "staged", bootstrapIdentity: handle.bootstrapIdentity, providerId: handle.sandbox.driverId, + agent: handle.plan.profile.agent, sandbox: Object.freeze({ ...handle.sandbox }), planFingerprint: createManagedBootstrapPlanFingerprint(handle.plan), profileFingerprint: handle.plan.profile.fingerprint, diff --git a/src/lib/onboard/managed-bootstrap/index.ts b/src/lib/onboard/managed-bootstrap/index.ts index c55768afe02..78873418467 100644 --- a/src/lib/onboard/managed-bootstrap/index.ts +++ b/src/lib/onboard/managed-bootstrap/index.ts @@ -3,6 +3,7 @@ export { activateManagedBootstrapSequence, + enforceManagedBootstrapRecoveryForSandbox, finalizeManagedBootstrapSequence, MANAGED_BOOTSTRAP_SCHEMA_VERSION, type ManagedBootstrapActivatedTransaction, @@ -10,7 +11,12 @@ export { type ManagedBootstrapAuthorityStore, type ManagedBootstrapExpectedPlan, type ManagedBootstrapPreparedTransaction, + ManagedBootstrapRecoveryBlockedError, + type ManagedBootstrapRecoveryFailure, + type ManagedBootstrapRecoveryReceipt, + type ManagedBootstrapRecoveryReport, prepareManagedBootstrapSequence, + recoverManagedBootstrapTransactions, } from "./adapter"; export { MANAGED_BOOTSTRAP_COMPLETION_FILE, diff --git a/src/lib/onboard/managed-bootstrap/runtime-create.ts b/src/lib/onboard/managed-bootstrap/runtime-create.ts index 198d91710f0..6ffcc028966 100644 --- a/src/lib/onboard/managed-bootstrap/runtime-create.ts +++ b/src/lib/onboard/managed-bootstrap/runtime-create.ts @@ -10,6 +10,7 @@ import type { ManagedBootstrapAuthorityStore, ManagedBootstrapCreateReceipt, ManagedBootstrapImageIdentity, + ManagedBootstrapRecoveryReport, } from "./adapter"; export interface ManagedBootstrapRuntimeCommandResult { @@ -92,6 +93,7 @@ export interface ManagedBootstrapRuntimeCreateLaunchResult { export interface ManagedBootstrapRuntimeCreateLifecycle { readonly launchArgv: readonly string[]; readonly patch: ManagedBootstrapRuntimePatch; + recoverUnfinished(): Promise; prepareNetwork(): Promise; runCreate( launch: (input: { diff --git a/src/lib/onboard/runtime-provider/runtime-provider-contract.test.ts b/src/lib/onboard/runtime-provider/runtime-provider-contract.test.ts index b5628f99b78..818ae30eb30 100644 --- a/src/lib/onboard/runtime-provider/runtime-provider-contract.test.ts +++ b/src/lib/onboard/runtime-provider/runtime-provider-contract.test.ts @@ -209,6 +209,7 @@ describe("RuntimeProviderBundle registry contract", () => { printReadinessFailureIfEnabled: vi.fn(), verifyGpuOrExit: vi.fn(async (verify) => verify("alpha")), }, + recoverUnfinished: vi.fn(async () => ({ receipts: [], failures: [] })), prepareNetwork: vi.fn(async () => undefined), runCreate: vi.fn(), })); diff --git a/src/lib/onboard/sandbox-gpu-create-flow.test.ts b/src/lib/onboard/sandbox-gpu-create-flow.test.ts index 8602f2e51f5..f60f0a59abe 100644 --- a/src/lib/onboard/sandbox-gpu-create-flow.test.ts +++ b/src/lib/onboard/sandbox-gpu-create-flow.test.ts @@ -62,6 +62,10 @@ import { setupGpuFlowMocks, VERIFIED_GPU_PROOF as VERIFIED_PROOF, } from "./__test-helpers__/sandbox-gpu-create-flow"; +import { + MANAGED_BOOTSTRAP_SCHEMA_VERSION, + type ManagedBootstrapRecoveryReport, +} from "./managed-bootstrap/adapter"; import type { ManagedBootstrapRuntimeCreateLifecycleInput, ManagedBootstrapRuntimePatch, @@ -166,7 +170,7 @@ beforeEach(() => setupGpuFlowMocks(mocks)); afterEach(resetGpuFlowMocks); describe("runSandboxGpuCreateFlow provider-owned managed create", () => { - it("runs an MXC-style bundle without a Docker branch in central orchestration", async () => { + it("recovers before an MXC-style create without a Docker branch in central orchestration", async () => { const input = createInput(); input.sandboxGpuConfig = { mode: "0", @@ -201,11 +205,40 @@ describe("runSandboxGpuCreateFlow provider-owned managed create", () => { input.sandboxEnv = launch.sandboxEnv; input.sandboxStartupCommand = launch.sandboxStartupCommand; const patch = createPatch() as unknown as ManagedBootstrapRuntimePatch; + const recoveryReport = ( + sandboxName: string | null, + detail = "opaque MXC recovery detail", + ): ManagedBootstrapRecoveryReport => + Object.freeze({ + receipts: Object.freeze([]), + failures: Object.freeze([ + Object.freeze({ + schemaVersion: MANAGED_BOOTSTRAP_SCHEMA_VERSION, + providerId: "mxc", + sourcePhase: "provider-owned-cleanup", + sandbox: + sandboxName === null + ? null + : Object.freeze({ + sandboxName, + sandboxId: `mxc-${sandboxName}`, + driverId: "mxc", + }), + bootstrapIdentity: "e".repeat(64), + code: "mxc-recovery-retry", + retryable: true, + detail, + }), + ]), + }); + const recoverUnfinished = vi.fn(async () => recoveryReport("bravo")); + const prepareNetwork = vi.fn(async () => undefined); const createLifecycle = vi.fn( (lifecycleInput: ManagedBootstrapRuntimeCreateLifecycleInput) => ({ launchArgv: ["mxc-launch", ...lifecycleInput.launchArgv.slice(1)], patch, - prepareNetwork: vi.fn(async () => undefined), + recoverUnfinished, + prepareNetwork, runCreate: async ( start: (held: { readonly heldWorkloadArgv: readonly string[]; @@ -284,6 +317,15 @@ describe("runSandboxGpuCreateFlow provider-owned managed create", () => { vi.mocked(deps.runCaptureOpenshell).mockImplementation((args) => args[1] === "get" ? "ID: mxc-alpha\n" : "alpha Ready", ); + recoverUnfinished.mockRejectedValueOnce(new Error("unfinished recovery failed")); + + await expect(runSandboxGpuCreateFlow(input, deps)).rejects.toThrow( + "unfinished recovery failed", + ); + expect(prepareNetwork).not.toHaveBeenCalled(); + expect(mocks.streamSandboxCreate).not.toHaveBeenCalled(); + recoverUnfinished.mockClear(); + createLifecycle.mockClear(); const result = await runSandboxGpuCreateFlow(input, deps); @@ -297,10 +339,38 @@ describe("runSandboxGpuCreateFlow provider-owned managed create", () => { input.sandboxEnv, expect.anything(), ); + expect(recoverUnfinished.mock.invocationCallOrder[0]).toBeLessThan( + prepareNetwork.mock.invocationCallOrder[0], + ); + expect(prepareNetwork.mock.invocationCallOrder[0]).toBeLessThan( + mocks.streamSandboxCreate.mock.invocationCallOrder[0], + ); expect(mocks.createDockerGpuSandboxCreatePatch).not.toHaveBeenCalled(); expect(mocks.queryOpenShellDockerSandboxContainers).not.toHaveBeenCalled(); expect(mocks.queryOpenShellDockerSandboxRuntimeSnapshot).not.toHaveBeenCalled(); expect(mocks.enforceDockerGpuPatchPreserveNetwork).not.toHaveBeenCalled(); + + expect(vi.mocked(console.warn).mock.calls.flat().join("\n")).toContain( + "unrelated sandbox 'bravo'", + ); + const recoverySecret = "opaque-recovery-token"; + recoverUnfinished.mockResolvedValueOnce( + recoveryReport("alpha", `Authorization: Bearer ${recoverySecret}`), + ); + prepareNetwork.mockClear(); + mocks.streamSandboxCreate.mockClear(); + mockExit(); + + await expect(runSandboxGpuCreateFlow(input, deps)).rejects.toThrow("process.exit:1"); + expect(prepareNetwork).not.toHaveBeenCalled(); + expect(mocks.streamSandboxCreate).not.toHaveBeenCalled(); + expect(errorOutput()).toContain("recovery stopped before sandbox 'alpha' was created"); + expect(errorOutput()).toContain("Transaction"); + expect(errorOutput()).toContain("durable sandbox ID mxc-alpha"); + expect(errorOutput()).toContain("OpenShell's sandbox get command"); + expect(errorOutput()).toContain("never delete a runtime by mutable sandbox name"); + expect(errorOutput()).toContain("Authorization: Bearer "); + expect(errorOutput()).not.toContain(recoverySecret); }); }); diff --git a/src/lib/onboard/sandbox-gpu-create-flow.ts b/src/lib/onboard/sandbox-gpu-create-flow.ts index 48acddb658d..46f34359a2e 100644 --- a/src/lib/onboard/sandbox-gpu-create-flow.ts +++ b/src/lib/onboard/sandbox-gpu-create-flow.ts @@ -10,11 +10,12 @@ import type { DockerGpuPatchDeps, DockerUlimit } from "./docker-gpu-patch-types" import type { SelectedDockerGpuRoute } from "./docker-gpu-route"; import { renderCompatibilityFallbackCreateArgs } from "./docker-gpu-route"; import { adaptDockerGpuRouteForPatch } from "./docker-gpu-route-patch-adapter"; -import type { - ManagedBootstrapAdapter, - ManagedBootstrapAgentIdentity, - ManagedBootstrapAuthorityStore, - ManagedBootstrapImageIdentity, +import { + type ManagedBootstrapAdapter, + type ManagedBootstrapAgentIdentity, + type ManagedBootstrapAuthorityStore, + type ManagedBootstrapImageIdentity, + ManagedBootstrapRecoveryBlockedError, } from "./managed-bootstrap/adapter"; import type { ManagedBootstrapRuntimePatch } from "./managed-bootstrap/runtime-create"; import type { ManagedStartupRootApplyRequest } from "./managed-startup/root-apply"; @@ -31,6 +32,36 @@ import { addTraceEvent } from "./tracing"; export { resolveDockerStartupCommandPatch } from "./docker-startup-command-agent"; +/* + * Keep recovery rendering at this public command boundary. Providers own the + * detail and remediation; central orchestration only renders their bounded, + * identity-bound evidence and never branches on provider IDs or error codes. + */ +function exitForManagedBootstrapRecovery(error: ManagedBootstrapRecoveryBlockedError): never { + console.error(""); + console.error( + ` Managed bootstrap recovery stopped before sandbox '${error.sandboxName}' was created.`, + ); + for (const failure of error.failures) { + const scope = failure.sandbox + ? `sandbox '${failure.sandbox.sandboxName}' (durable sandbox ID ${failure.sandbox.sandboxId}, provider ${failure.providerId})` + : "a sandbox whose durable identity could not be recovered"; + console.error( + ` Transaction ${failure.bootstrapIdentity} requires ${failure.retryable ? "a provider retry after its recovery condition is resolved" : "operator recovery"} for ${scope}.`, + ); + console.error(` ${redactFull(failure.detail)}`); + if (failure.sandbox) { + console.error( + ` Before any provider action, query that exact name with OpenShell's sandbox get command and verify it returns durable sandbox ID ${failure.sandbox.sandboxId}.`, + ); + } + } + console.error( + " Preserve every durable recovery record. Act only on exact provider, sandbox, and runtime IDs from the provider guidance; never delete a runtime by mutable sandbox name.", + ); + process.exit(1); +} + type RunOpenshell = NonNullable; type RunCaptureOpenshell = NonNullable; type Sleep = NonNullable; @@ -104,9 +135,8 @@ export async function runSandboxGpuCreateFlow( ): Promise { let registryImageRef: string | null = input.prebuild.imageRef; const attemptRunner = createSandboxGpuCreateAttemptRunner(input, deps); - const gpuCreateOutcome = await sandboxGpuCreateAttempt.executeSandboxGpuCreatePlan( - input.gpuRoutePlan, - { + const gpuCreateOutcome = await sandboxGpuCreateAttempt + .executeSandboxGpuCreatePlan(input.gpuRoutePlan, { runAttempt: attemptRunner.runAttempt, captureNativeFailure: (failure) => { const routeAdapter = adaptDockerGpuRouteForPatch(failure.route); @@ -192,8 +222,13 @@ export async function runSandboxGpuCreateFlow( input.sandboxGpuConfig.sandboxGpuProof = null; }, traceEvent: addTraceEvent, - }, - ); + }) + .catch((error: unknown) => { + if (error instanceof ManagedBootstrapRecoveryBlockedError) { + exitForManagedBootstrapRecovery(error); + } + throw error; + }); if (!gpuCreateOutcome.ok) { console.error(""); console.error(" Operator-authorized GPU fallback stopped before compatibility retry."); diff --git a/src/lib/onboard/sandbox-gpu-create-run-attempt.ts b/src/lib/onboard/sandbox-gpu-create-run-attempt.ts index 8eeb4a8aac0..49c3271ddd6 100644 --- a/src/lib/onboard/sandbox-gpu-create-run-attempt.ts +++ b/src/lib/onboard/sandbox-gpu-create-run-attempt.ts @@ -14,6 +14,7 @@ import { reportSandboxCreateFailure } from "./created-sandbox-failure"; import * as dockerGpuLocalInference from "./docker-gpu-local-inference"; import type { SelectedDockerGpuRoute } from "./docker-gpu-route"; import { createDockerGpuSandboxCreatePatch } from "./docker-gpu-sandbox-create"; +import { enforceManagedBootstrapRecoveryForSandbox } from "./managed-bootstrap/adapter"; import type { ManagedBootstrapRuntimeSnapshot } from "./managed-bootstrap/runtime-create"; import { queryOpenShellDockerSandboxContainers, @@ -143,6 +144,12 @@ export function createSandboxGpuCreateAttemptRunner( backend: input.sandboxGpuConfig.hostGpuPlatform === "jetson" ? "jetson" : "generic", deps, }); + const recovery = await managedLifecycle?.recoverUnfinished(); + if (recovery) { + enforceManagedBootstrapRecoveryForSandbox(recovery, input.sandboxName, (message) => + console.warn(` ⚠ ${message}`), + ); + } await managedLifecycle?.prepareNetwork(); const [createExecutable, ...createExecutableArgs] = managedLifecycle?.launchArgv ?? attemptArgv; if (!createExecutable) throw new Error("Sandbox create executable is missing.");