diff --git a/skills/autopilot/docs/2026-08-31-cooperative-harness-execution-plan.md b/skills/autopilot/docs/2026-08-31-cooperative-harness-execution-plan.md index 3e60ebd..bbab8e7 100644 --- a/skills/autopilot/docs/2026-08-31-cooperative-harness-execution-plan.md +++ b/skills/autopilot/docs/2026-08-31-cooperative-harness-execution-plan.md @@ -1,6 +1,6 @@ # Cooperative harness execution implementation plan -- **Status:** Approved design; PR 1 decision/promotion shutdown and PR 2 execution-assurance protocol complete; operator recovery and harness integration remain pending +- **Status:** Approved design; PR 1 decision/promotion shutdown, PR 2 execution assurance, and PR 3 fenced unknown-execution recovery are complete; harness integration remains pending - **Date:** 2026-08-31 - **Audience:** Autopilot implementers and reviewers - **Related:** [Architecture](architecture.md), [continuity implementation plan](2026-08-30-continuity-evidence-implementation-plan.md), [durable event engine ADR](adr/0001-durable-event-engine.md) diff --git a/skills/autopilot/docs/runtime-cli.md b/skills/autopilot/docs/runtime-cli.md index 8bb38f9..c8bdb8e 100644 --- a/skills/autopilot/docs/runtime-cli.md +++ b/skills/autopilot/docs/runtime-cli.md @@ -17,6 +17,7 @@ autopilot [--state-dir ] [--json] status [run-id] autopilot [--state-dir ] [--json] [--repair-journal] resume [run-id] autopilot [--state-dir ] [--json] pause [run-id] autopilot [--state-dir ] [--json] stop [run-id] +autopilot [--state-dir ] [--json] recover --action --item --attempt --lease-epoch --attestation [--tree ] autopilot [--state-dir ] [--json] review-feedback [run-id] autopilot [--state-dir ] [--json] [--handoff] wrap-up [run-id] autopilot [--json] doctor @@ -44,6 +45,8 @@ If no coordinator owns the run, `pause` acquires the run lock and reconciles the If no coordinator owns the run, `stop` acquires the run lock and records the terminal stop. If a foreground coordinator is active, `stop` uses the same fenced request path. The owner cancels active adapter work and records `RUN_STOPPED`. A success recorded before either request wins remains successful. +`recover` requires an inactive coordinator and the exact current unknown item, attempt, and lease epoch. Every action records the owning run-lock token hash and explicit operator attestation. `abandon` permanently moves the uncertain worktree aside before a fresh attempt; `adopt` requires the exact freshly observed `--tree` and runs verification without an implementation launch; `stop` preserves the evidence and terminalizes the run. + `stop` is terminal. A stopped or successful run requires a sealed successor for changed work. On supported POSIX hosts, built-in adapters reattach supervised implementation executions after coordinator loss and wait for terminal process-tree evidence before retrying. Legacy attempts, review executions, and incomplete or mismatched supervisor artifacts record `EXECUTION_STATE_UNKNOWN` and refuse a replacement launch. ## Journal repair diff --git a/skills/autopilot/references/recovery.md b/skills/autopilot/references/recovery.md index 4781926..cd705a7 100644 --- a/skills/autopilot/references/recovery.md +++ b/skills/autopilot/references/recovery.md @@ -36,6 +36,18 @@ Resume acquires the coordinator lock, validates the sealed charter and journal, On POSIX hosts, built-in harness adapters supervise implementation executions with a detached, attempt-scoped helper and a pre-established process-group watchdog. After coordinator loss, `resume` reconstructs the exact request from the journaled attempt and immutable context, reattaches to matching running or terminal artifacts, and observes process-group quiescence before permitting a replacement attempt. The reviewed Windows x64 Job Object helper will not be packaged, so Windows restart reattachment remains disabled while the cooperative harness backend is implemented. Current Windows execution and cancellation are session-scoped. Under the approved cooperative design, only an exact terminal response through the uninterrupted owning harness instance may proceed; harness, session, or exact-subject loss becomes `EXECUTION_STATE_UNKNOWN` and cannot launch a speculative replacement. Legacy attempts, review executions, mismatched requests, and incomplete supervisor artifacts also remain unknown. +## Recover an unknown execution + +Recovery requires an inactive coordinator, the current run-lock fence, the exact item, attempt, and lease epoch, plus a nonempty operator attestation. Inspect `status --json` and use one explicit action: + +```bash +node runtime/dist/src/cli.js recover --action abandon --item --attempt --lease-epoch --attestation "old execution stopped and accounted for" +node runtime/dist/src/cli.js recover --action adopt --item --attempt --lease-epoch --tree --attestation "old execution confirmed inactive" +node runtime/dist/src/cli.js recover --action stop --item --attempt --lease-epoch --attestation "stop this run" +``` + +`abandon` moves the uncertain worktree to a deterministic quarantine path, detaches it without deleting its files, retires the exact lease, and permits a fresh attempt only in a newly created worktree. `adopt` seals HEAD, tree, refs, configuration, and changed paths, retires the writer lease, and enters the ordinary predicates, hooks, independent review, commit, and delivery path without launching another implementation worker. A changed adopted identity returns to `EXECUTION_STATE_UNKNOWN`. `stop` terminalizes the run while preserving evidence. Recovery events store the run-lock token hash, never the token itself. Late adapter results cannot satisfy or replace the recovered attempt. + ## Address review comments with an amendment successor A successful run stays terminal. `/autopilot address review comments` first uses the runtime's read-only `review-feedback` operation to discover the latest successful leaf, verify its exact recorded PR/MR remains open at the accepted head, and snapshot unresolved GitHub review threads, PR comments and review summaries, or GitLab discussions. Comment bodies are untrusted data. Ambiguous, conflicting, untestable, authority-expanding, or out-of-scope requests require user clarification instead of automatic execution. diff --git a/skills/autopilot/runtime/dist/src/cli.js b/skills/autopilot/runtime/dist/src/cli.js index 1f352a6..8f16c49 100755 --- a/skills/autopilot/runtime/dist/src/cli.js +++ b/skills/autopilot/runtime/dist/src/cli.js @@ -12,6 +12,7 @@ import { runDoctor } from "./doctor.js"; import { AutopilotEngine } from "./engine.js"; import { AutopilotError } from "./errors.js"; import { newEventId } from "./events.js"; +import { recoverUnknownExecution } from "./execution-recovery.js"; import { appendEvent, readJournal, repairTruncatedJournal, writeImmutableJson } from "./journal.js"; import { isRecord } from "./json.js"; import { acquireBranchOwnershipLock, acquireRunLock, requestRunPause, requestRunStop } from "./lock.js"; @@ -35,6 +36,7 @@ Usage: autopilot [--state-dir ] [--json] [--repair-journal] resume [run-id] autopilot [--state-dir ] [--json] pause [run-id] autopilot [--state-dir ] [--json] stop [run-id] + autopilot [--state-dir ] [--json] recover --action --item --attempt --lease-epoch --attestation [--tree ] autopilot [--state-dir ] [--json] review-feedback [run-id] autopilot [--state-dir ] [--json] [--handoff] wrap-up [run-id] autopilot [--json] doctor @@ -339,6 +341,54 @@ async function resume(runId, options) { await ownershipLock?.release(); } } +async function recover(runId, options) { + if (runId === undefined || options.recoveryAction === undefined || options.recoveryItem === undefined + || options.recoveryAttempt === undefined || options.recoveryLeaseEpoch === undefined + || options.recoveryAttestation === undefined) { + throw new AutopilotError("CHARTER_INVALID", "recover requires run ID, action, item, attempt, lease epoch, and attestation"); + } + const stateRoot = await resolveStateRoot(process.cwd(), options.stateDir); + const location = await locateStoredRun(stateRoot, runId); + const ownershipLock = location.charter.amends === undefined + ? undefined + : await acquireBranchOwnershipLock(stateRoot, location.charter.work[0]?.branchName ?? ""); + try { + const lock = await acquireRunLock(join(location.directory, "run.lock")); + try { + const run = await loadRun(runId, options.stateDir, false); + const projection = rebuildProjection(run.charter, run.journal.records); + const recovered = await recoverUnknownExecution(run.directory, run.charter, projection, lock, { + action: options.recoveryAction, + itemId: options.recoveryItem, + attemptId: options.recoveryAttempt, + leaseEpoch: options.recoveryLeaseEpoch, + attestation: options.recoveryAttestation, + ...(options.recoveryTree === undefined ? {} : { expectedTreeIdentity: options.recoveryTree }), + }); + if (recovered.state === "STOPPED") { + const journal = await readJournal(join(run.directory, "events.jsonl")); + const metadata = await loadReportMetadata(run.directory); + return await writeReports(run.directory, run.charter, recovered, journal.records, metadata.assurance, metadata.unverifiedBoundaries); + } + const journal = await readJournal(join(run.directory, "events.jsonl")); + const engine = new AutopilotEngine({ + stateRoot: run.stateRoot, + runDirectory: run.directory, + charter: run.charter, + adapter: createAdapter(run.charter.harnessAdapter), + records: journal.records, + projection: recovered, + }); + return await runEngine(engine, lock, run.charter.runId); + } + finally { + await lock.release(); + } + } + finally { + await ownershipLock?.release(); + } +} async function reviewFeedback(runId, options) { const stateRoot = await resolveStateRoot(process.cwd(), options.stateDir); return await observeReviewFeedback(stateRoot, process.cwd(), runId); @@ -500,6 +550,12 @@ export async function main(arguments_ = process.argv.slice(2)) { json: { type: "boolean", default: false }, "repair-journal": { type: "boolean", default: false }, handoff: { type: "boolean", default: false }, + action: { type: "string" }, + item: { type: "string" }, + attempt: { type: "string" }, + "lease-epoch": { type: "string" }, + attestation: { type: "string" }, + tree: { type: "string" }, help: { type: "boolean", short: "h", default: false }, version: { type: "boolean", short: "v", default: false }, }, @@ -516,11 +572,29 @@ export async function main(arguments_ = process.argv.slice(2)) { if (command === undefined || extra.length > 0) { throw new AutopilotError("CHARTER_INVALID", usage()); } + const recoveryAction = parsed.values.action; + if (recoveryAction !== undefined && !["abandon", "adopt", "stop"].includes(recoveryAction)) { + throw new AutopilotError("CHARTER_INVALID", `unsupported recovery action: ${recoveryAction}`); + } + const recoveryLeaseEpochText = parsed.values["lease-epoch"]; + const recoveryLeaseEpoch = recoveryLeaseEpochText === undefined || !/^\d+$/u.test(recoveryLeaseEpochText) + ? undefined + : Number.parseInt(recoveryLeaseEpochText, 10); + if (recoveryLeaseEpochText !== undefined + && (recoveryLeaseEpoch === undefined || !Number.isSafeInteger(recoveryLeaseEpoch) || recoveryLeaseEpoch < 1)) { + throw new AutopilotError("CHARTER_INVALID", "--lease-epoch must be a positive integer"); + } const options = { json: parsed.values.json, repairJournal: parsed.values["repair-journal"], handoff: parsed.values.handoff, ...(parsed.values["state-dir"] === undefined ? {} : { stateDir: parsed.values["state-dir"] }), + ...(recoveryAction === undefined ? {} : { recoveryAction: recoveryAction }), + ...(parsed.values.item === undefined ? {} : { recoveryItem: parsed.values.item }), + ...(parsed.values.attempt === undefined ? {} : { recoveryAttempt: parsed.values.attempt }), + ...(recoveryLeaseEpoch === undefined ? {} : { recoveryLeaseEpoch }), + ...(parsed.values.attestation === undefined ? {} : { recoveryAttestation: parsed.values.attestation }), + ...(parsed.values.tree === undefined ? {} : { recoveryTree: parsed.values.tree }), }; let result; switch (command) { @@ -545,6 +619,9 @@ export async function main(arguments_ = process.argv.slice(2)) { case "stop": result = await stop(argument, options); break; + case "recover": + result = await recover(argument, options); + break; case "review-feedback": result = await reviewFeedback(argument, options); break; diff --git a/skills/autopilot/runtime/dist/src/engine.js b/skills/autopilot/runtime/dist/src/engine.js index 91ce59e..2b7e74a 100644 --- a/skills/autopilot/runtime/dist/src/engine.js +++ b/skills/autopilot/runtime/dist/src/engine.js @@ -828,6 +828,46 @@ export class AutopilotEngine { }); continue; } + if (itemProjection.state === "ACTIVE" && attempt.adoptedTree !== undefined) { + const adopted = attempt.adoptedTree; + let observed; + try { + observed = await this.#observeRepository(adopted.worktreePath); + if (observed.headCommit !== adopted.headCommit || observed.treeIdentity !== adopted.treeIdentity + || observed.refIdentity !== adopted.refIdentity + || observed.configurationIdentity !== adopted.configurationIdentity + || canonicalJson(observed.changedPaths) !== canonicalJson(adopted.changedPaths)) { + throw new AutopilotError("EXECUTION_STATE_UNKNOWN", "adopted worktree changed before verification"); + } + } + catch (error) { + await this.#record({ + ...eventBase(error instanceof Error ? error.message : String(error), "reconciler"), + type: "ITEM_BLOCKED", + itemId: item.id, + attemptId: attempt.attemptId, + errorCode: "EXECUTION_STATE_UNKNOWN", + }); + continue; + } + try { + await assertWritablePaths(adopted.worktreePath, observed.changedPaths, item.writableRoots); + await this.#verifyObservedItem(item, attempt.attemptId, adopted.worktreePath, observed, await inspectPreCommitHook(adopted.worktreePath)); + } + catch (error) { + const state = this.#projection.items[item.id]?.state; + if (state === "ACTIVE" || state === "VERIFYING") { + await this.#record({ + ...eventBase(error instanceof Error ? error.message : String(error), "reconciler"), + type: "ITEM_BLOCKED", + itemId: item.id, + attemptId: attempt.attemptId, + errorCode: error instanceof AutopilotError ? error.code : "UNKNOWN_FAILURE", + }); + } + } + continue; + } if (itemProjection.state === "ACTIVE") { let reattachedStatus; try { @@ -1965,6 +2005,126 @@ export class AutopilotEngine { }); } } + async #verifyObservedItem(item, attemptId, worktreePath, initialObservation, hookSnapshot) { + await this.#record({ ...eventBase("Runtime is verifying the observed tree"), type: "ITEM_VERIFYING", itemId: item.id, attemptId }); + let finalObservation = initialObservation; + let verification = await this.#verifyItem(item, worktreePath, attemptId, finalObservation); + if (await this.#blockItemForStop(item, attemptId)) { + return; + } + if (!verification.met) { + await this.#record({ + ...eventBase(verification.reasons.join("; ") || "Acceptance predicates are not met"), + type: "ITEM_BLOCKED", + itemId: item.id, + attemptId, + errorCode: "PREDICATE_NOT_MET", + }); + return; + } + if (!finalObservation.clean && this.#charter.commitPolicy?.preCommitHook === "run") { + if (await this.#blockItemForStop(item, attemptId)) { + return; + } + const hook = await runPreCommitHook(worktreePath, hookSnapshot ?? { identity: "NOT_CONFIGURED" }, this.#charter.commitPolicy.environmentNames, this.#charter.limits.attemptTimeoutMs, this.#charter.limits.maxRetainedOutputBytes); + const hookObservation = await this.#observeRepository(worktreePath); + const hooksDirectory = join(this.#runDirectory, "reports", "hooks"); + await mkdir(hooksDirectory, { recursive: true, mode: 0o700 }); + const hookPath = join(hooksDirectory, `${attemptId}.json`); + await writeJsonAtomic(hookPath, { + schemaVersion: 1, + status: hook.status, + configuredPath: hook.path, + beforeTree: finalObservation.treeIdentity, + afterTree: hookObservation.treeIdentity, + exitCode: hook.result?.exitCode ?? 0, + stdout: redactEnvironmentSecrets(hook.result?.stdout ?? "", this.#charter.commitPolicy.environmentNames), + stderr: redactEnvironmentSecrets(hook.result?.stderr ?? "", this.#charter.commitPolicy.environmentNames), + truncated: hook.result?.truncated ?? false, + }); + await this.#record({ + ...eventBase(`Pre-commit hook finished ${hook.status}`), + type: "PRE_COMMIT_HOOK_FINISHED", + itemId: item.id, + attemptId, + status: hook.status, + beforeTree: finalObservation.treeIdentity, + afterTree: hookObservation.treeIdentity, + exitCode: hook.result?.exitCode ?? 0, + evidence: [hookPath], + }); + if (await this.#blockItemForStop(item, attemptId)) { + return; + } + if (hookObservation.headCommit !== finalObservation.headCommit + || hookObservation.externalRefIdentity !== finalObservation.externalRefIdentity) { + throw new AutopilotError("BRANCH_COLLISION", "pre-commit hook changed HEAD or another Git ref"); + } + if (hookObservation.configurationIdentity !== finalObservation.configurationIdentity) { + throw new AutopilotError("CAPABILITY_DENIED", "pre-commit hook changed Git configuration"); + } + await assertWritablePaths(worktreePath, hookObservation.changedPaths, [...item.writableRoots, ...this.#charter.commitPolicy.writableRoots]); + if (hook.status === "FAILED") { + await this.#record({ + ...eventBase("Pre-commit hook failed before the runtime-owned commit"), + type: "ITEM_BLOCKED", + itemId: item.id, + attemptId, + errorCode: "PRE_COMMIT_HOOK_FAILED", + }); + return; + } + if (hookObservation.treeIdentity !== finalObservation.treeIdentity) { + verification = await this.#verifyItem(item, worktreePath, attemptId, hookObservation); + if (!verification.met) { + await this.#record({ + ...eventBase(verification.reasons.join("; ") || "Post-hook acceptance predicates are not met"), + type: "ITEM_BLOCKED", + itemId: item.id, + attemptId, + errorCode: "POST_HOOK_PREDICATE_NOT_MET", + }); + return; + } + } + finalObservation = hookObservation; + } + const finalHook = await inspectPreCommitHook(worktreePath); + const receiptIds = []; + for (const { event: receiptEvent } of this.#records) { + if (receiptEvent.type !== "RECEIPT_RECORDED" || receiptEvent.itemId !== item.id + || receiptEvent.attemptId !== attemptId) { + continue; + } + const receiptValue = JSON.parse(await readFile(join(this.#runDirectory, "receipts", `${receiptEvent.receiptId}.json`), "utf8")); + const requiredGateReceipt = receiptEvent.gateId !== undefined && item.acceptance.some((predicate) => predicate.type === "gate-passed" && predicate.gateId === receiptEvent.gateId); + if (isRecord(receiptValue) && receiptValue.subject === verification.subject + && (receiptEvent.receiptKind === "predicate" || requiredGateReceipt)) { + receiptIds.push(receiptEvent.receiptId); + } + } + await this.#record({ + ...eventBase("Exact tree and acceptance evidence verified"), + type: "ITEM_VERIFIED", + itemId: item.id, + attemptId, + subject: verification.subject, + headCommit: finalObservation.headCommit, + treeIdentity: finalObservation.treeIdentity, + auxiliaryRefIdentity: finalObservation.auxiliaryRefIdentity, + externalRefIdentity: finalObservation.externalRefIdentity, + configurationIdentity: finalObservation.configurationIdentity, + hookIdentity: finalHook.identity, + ...(finalHook.path === undefined ? {} : { hookPath: finalHook.path }), + commitRequired: !finalObservation.clean, + receiptIds, + }); + const checkpoint = this.#projection.items[item.id]?.verified; + if (checkpoint === undefined) { + throw new AutopilotError("JOURNAL_CORRUPT", "verified checkpoint was not projected"); + } + await this.#completeVerifiedItem(item, worktreePath, checkpoint); + } async #runItem(item) { if (this.#stopRequested || this.#pauseRequested) { return; @@ -2181,124 +2341,7 @@ export class AutopilotEngine { if (await this.#blockItemForStop(item, attemptId)) { return; } - await this.#record({ ...eventBase("Runtime is verifying the observed tree"), type: "ITEM_VERIFYING", itemId: item.id, attemptId }); - let finalObservation = after; - let verification = await this.#verifyItem(item, worktreePath, attemptId, finalObservation); - if (await this.#blockItemForStop(item, attemptId)) { - return; - } - if (!verification.met) { - await this.#record({ - ...eventBase(verification.reasons.join("; ") || "Acceptance predicates are not met"), - type: "ITEM_BLOCKED", - itemId: item.id, - attemptId, - errorCode: "PREDICATE_NOT_MET", - }); - return; - } - if (!finalObservation.clean && this.#charter.commitPolicy?.preCommitHook === "run") { - if (await this.#blockItemForStop(item, attemptId)) { - return; - } - const hook = await runPreCommitHook(worktreePath, hookSnapshot ?? { identity: "NOT_CONFIGURED" }, this.#charter.commitPolicy.environmentNames, this.#charter.limits.attemptTimeoutMs, this.#charter.limits.maxRetainedOutputBytes); - const hookObservation = await this.#observeRepository(worktreePath); - const hooksDirectory = join(this.#runDirectory, "reports", "hooks"); - await mkdir(hooksDirectory, { recursive: true, mode: 0o700 }); - const hookPath = join(hooksDirectory, `${attemptId}.json`); - await writeJsonAtomic(hookPath, { - schemaVersion: 1, - status: hook.status, - configuredPath: hook.path, - beforeTree: finalObservation.treeIdentity, - afterTree: hookObservation.treeIdentity, - exitCode: hook.result?.exitCode ?? 0, - stdout: redactEnvironmentSecrets(hook.result?.stdout ?? "", this.#charter.commitPolicy.environmentNames), - stderr: redactEnvironmentSecrets(hook.result?.stderr ?? "", this.#charter.commitPolicy.environmentNames), - truncated: hook.result?.truncated ?? false, - }); - await this.#record({ - ...eventBase(`Pre-commit hook finished ${hook.status}`), - type: "PRE_COMMIT_HOOK_FINISHED", - itemId: item.id, - attemptId, - status: hook.status, - beforeTree: finalObservation.treeIdentity, - afterTree: hookObservation.treeIdentity, - exitCode: hook.result?.exitCode ?? 0, - evidence: [hookPath], - }); - if (await this.#blockItemForStop(item, attemptId)) { - return; - } - if (hookObservation.headCommit !== finalObservation.headCommit - || hookObservation.externalRefIdentity !== finalObservation.externalRefIdentity) { - throw new AutopilotError("BRANCH_COLLISION", "pre-commit hook changed HEAD or another Git ref"); - } - if (hookObservation.configurationIdentity !== finalObservation.configurationIdentity) { - throw new AutopilotError("CAPABILITY_DENIED", "pre-commit hook changed Git configuration"); - } - await assertWritablePaths(worktreePath, hookObservation.changedPaths, [...item.writableRoots, ...this.#charter.commitPolicy.writableRoots]); - if (hook.status === "FAILED") { - await this.#record({ - ...eventBase("Pre-commit hook failed before the runtime-owned commit"), - type: "ITEM_BLOCKED", - itemId: item.id, - attemptId, - errorCode: "PRE_COMMIT_HOOK_FAILED", - }); - return; - } - if (hookObservation.treeIdentity !== finalObservation.treeIdentity) { - verification = await this.#verifyItem(item, worktreePath, attemptId, hookObservation); - if (!verification.met) { - await this.#record({ - ...eventBase(verification.reasons.join("; ") || "Post-hook acceptance predicates are not met"), - type: "ITEM_BLOCKED", - itemId: item.id, - attemptId, - errorCode: "POST_HOOK_PREDICATE_NOT_MET", - }); - return; - } - } - finalObservation = hookObservation; - } - const finalHook = await inspectPreCommitHook(worktreePath); - const receiptIds = []; - for (const { event: receiptEvent } of this.#records) { - if (receiptEvent.type !== "RECEIPT_RECORDED" || receiptEvent.itemId !== item.id - || receiptEvent.attemptId !== attemptId) { - continue; - } - const receiptValue = JSON.parse(await readFile(join(this.#runDirectory, "receipts", `${receiptEvent.receiptId}.json`), "utf8")); - const requiredGateReceipt = receiptEvent.gateId !== undefined && item.acceptance.some((predicate) => predicate.type === "gate-passed" && predicate.gateId === receiptEvent.gateId); - if (isRecord(receiptValue) && receiptValue.subject === verification.subject - && (receiptEvent.receiptKind === "predicate" || requiredGateReceipt)) { - receiptIds.push(receiptEvent.receiptId); - } - } - await this.#record({ - ...eventBase("Exact tree and acceptance evidence verified"), - type: "ITEM_VERIFIED", - itemId: item.id, - attemptId, - subject: verification.subject, - headCommit: finalObservation.headCommit, - treeIdentity: finalObservation.treeIdentity, - auxiliaryRefIdentity: finalObservation.auxiliaryRefIdentity, - externalRefIdentity: finalObservation.externalRefIdentity, - configurationIdentity: finalObservation.configurationIdentity, - hookIdentity: finalHook.identity, - ...(finalHook.path === undefined ? {} : { hookPath: finalHook.path }), - commitRequired: !finalObservation.clean, - receiptIds, - }); - const checkpoint = this.#projection.items[item.id]?.verified; - if (checkpoint === undefined) { - throw new AutopilotError("JOURNAL_CORRUPT", "verified checkpoint was not projected"); - } - await this.#completeVerifiedItem(item, worktreePath, checkpoint); + await this.#verifyObservedItem(item, attemptId, worktreePath, after, hookSnapshot); } async run() { if (this.#projection.state === "SUCCEEDED" || this.#projection.state === "STOPPED") { diff --git a/skills/autopilot/runtime/dist/src/events.d.ts b/skills/autopilot/runtime/dist/src/events.d.ts index 9a6d856..32ba903 100644 --- a/skills/autopilot/runtime/dist/src/events.d.ts +++ b/skills/autopilot/runtime/dist/src/events.d.ts @@ -53,6 +53,9 @@ export type LifecycleEvent = (EventBase & { readonly type: "RUN_STOPPED"; readonly errorCode: string; readonly remediation: string; + readonly leaseEpoch?: number; + readonly lockTokenHash?: string; + readonly attestation?: string; }) | (EventBase & { readonly type: "WRAP_UP_STARTED"; readonly chainRunIds: readonly string[]; @@ -98,6 +101,37 @@ export type LifecycleEvent = (EventBase & { readonly backendId: string; readonly subjectId: string; readonly harnessInstanceId?: string; +}) | (EventBase & { + readonly type: "EXECUTION_UNKNOWN_ABANDONED"; + readonly itemId: string; + readonly attemptId: string; + readonly leaseEpoch: number; + readonly lockTokenHash: string; + readonly attestation: string; + readonly originalWorktreePath: string; + readonly quarantineWorktreePath: string; + readonly headCommit: string; + readonly treeIdentity: string; + readonly refIdentity: string; + readonly auxiliaryRefIdentity: string; + readonly externalRefIdentity: string; + readonly configurationIdentity: string; + readonly changedPaths: readonly string[]; +}) | (EventBase & { + readonly type: "EXECUTION_UNKNOWN_TREE_ADOPTED"; + readonly itemId: string; + readonly attemptId: string; + readonly leaseEpoch: number; + readonly lockTokenHash: string; + readonly attestation: string; + readonly worktreePath: string; + readonly headCommit: string; + readonly treeIdentity: string; + readonly refIdentity: string; + readonly auxiliaryRefIdentity: string; + readonly externalRefIdentity: string; + readonly configurationIdentity: string; + readonly changedPaths: readonly string[]; }) | (EventBase & { readonly type: "ATTEMPT_FINISHED"; readonly itemId: string; diff --git a/skills/autopilot/runtime/dist/src/events.js b/skills/autopilot/runtime/dist/src/events.js index c8225c1..1706cdb 100644 --- a/skills/autopilot/runtime/dist/src/events.js +++ b/skills/autopilot/runtime/dist/src/events.js @@ -5,7 +5,8 @@ import { expectBoolean, expectInteger, expectLiteral, expectRecord, expectString const EVENT_TYPES = [ "CHARTER_COMPILED", "RECONCILIATION_STARTED", "RECONCILIATION_COMPLETED", "RUN_PAUSE_REQUESTED", "RUN_WAITING", "RUN_WOKEN", "RUN_RESUMED", "RUN_VERIFYING", "RUN_SUCCEEDED", "RUN_STOPPED", "WRAP_UP_STARTED", "WORKTREE_ADOPTED", "ITEM_READY", - "ATTEMPT_STARTED", "ATTEMPT_EXECUTION_ADMITTED", "ATTEMPT_FINISHED", "ITEM_VERIFYING", "ATTEMPT_PAUSED", "ITEM_VERIFIED", "ITEM_SATISFIED", + "ATTEMPT_STARTED", "ATTEMPT_EXECUTION_ADMITTED", "EXECUTION_UNKNOWN_ABANDONED", "EXECUTION_UNKNOWN_TREE_ADOPTED", + "ATTEMPT_FINISHED", "ITEM_VERIFYING", "ATTEMPT_PAUSED", "ITEM_VERIFIED", "ITEM_SATISFIED", "ITEM_BLOCKED", "ITEM_ABANDONED", "RESTACK_DESCENDANT_STARTED", "RESTACK_DESCENDANT_TREE_PREPARED", "RESTACK_DESCENDANT_VERIFIED", "RESTACK_PROVIDER_HEAD_CONFIRMED", "RESTACK_DESCENDANT_SATISFIED", "RESTACK_DESCENDANT_BLOCKED", "EFFECT_INTENDED", @@ -66,6 +67,9 @@ export function parseLifecycleEvent(value) { type, errorCode: expectString(object.errorCode, "event.errorCode"), remediation: expectString(object.remediation, "event.remediation"), + ...(object.leaseEpoch === undefined ? {} : { leaseEpoch: expectInteger(object.leaseEpoch, "event.leaseEpoch", 1) }), + ...(object.lockTokenHash === undefined ? {} : { lockTokenHash: expectString(object.lockTokenHash, "event.lockTokenHash") }), + ...(object.attestation === undefined ? {} : { attestation: expectString(object.attestation, "event.attestation") }), }; case "WRAP_UP_STARTED": return { @@ -200,6 +204,43 @@ export function parseLifecycleEvent(value) { harnessInstanceId: expectString(object.harnessInstanceId, "event.harnessInstanceId"), }), }; + case "EXECUTION_UNKNOWN_ABANDONED": + return { + ...base, + type, + itemId: expectString(object.itemId, "event.itemId"), + attemptId: expectString(object.attemptId, "event.attemptId"), + leaseEpoch: expectInteger(object.leaseEpoch, "event.leaseEpoch", 1), + lockTokenHash: expectString(object.lockTokenHash, "event.lockTokenHash"), + attestation: expectString(object.attestation, "event.attestation"), + originalWorktreePath: expectString(object.originalWorktreePath, "event.originalWorktreePath"), + quarantineWorktreePath: expectString(object.quarantineWorktreePath, "event.quarantineWorktreePath"), + headCommit: expectString(object.headCommit, "event.headCommit"), + treeIdentity: expectString(object.treeIdentity, "event.treeIdentity"), + refIdentity: expectString(object.refIdentity, "event.refIdentity"), + auxiliaryRefIdentity: expectString(object.auxiliaryRefIdentity, "event.auxiliaryRefIdentity"), + externalRefIdentity: expectString(object.externalRefIdentity, "event.externalRefIdentity"), + configurationIdentity: expectString(object.configurationIdentity, "event.configurationIdentity"), + changedPaths: expectStringArray(object.changedPaths, "event.changedPaths"), + }; + case "EXECUTION_UNKNOWN_TREE_ADOPTED": + return { + ...base, + type, + itemId: expectString(object.itemId, "event.itemId"), + attemptId: expectString(object.attemptId, "event.attemptId"), + leaseEpoch: expectInteger(object.leaseEpoch, "event.leaseEpoch", 1), + lockTokenHash: expectString(object.lockTokenHash, "event.lockTokenHash"), + attestation: expectString(object.attestation, "event.attestation"), + worktreePath: expectString(object.worktreePath, "event.worktreePath"), + headCommit: expectString(object.headCommit, "event.headCommit"), + treeIdentity: expectString(object.treeIdentity, "event.treeIdentity"), + refIdentity: expectString(object.refIdentity, "event.refIdentity"), + auxiliaryRefIdentity: expectString(object.auxiliaryRefIdentity, "event.auxiliaryRefIdentity"), + externalRefIdentity: expectString(object.externalRefIdentity, "event.externalRefIdentity"), + configurationIdentity: expectString(object.configurationIdentity, "event.configurationIdentity"), + changedPaths: expectStringArray(object.changedPaths, "event.changedPaths"), + }; case "ATTEMPT_FINISHED": return { ...base, diff --git a/skills/autopilot/runtime/dist/src/execution-recovery.d.ts b/skills/autopilot/runtime/dist/src/execution-recovery.d.ts new file mode 100644 index 0000000..ba60b02 --- /dev/null +++ b/skills/autopilot/runtime/dist/src/execution-recovery.d.ts @@ -0,0 +1,13 @@ +import type { RunCharter } from "./charter.js"; +import type { RunLock } from "./lock.js"; +import { type RunProjection } from "./reducer.js"; +export type UnknownRecoveryAction = "abandon" | "adopt" | "stop"; +export interface UnknownRecoveryRequest { + readonly action: UnknownRecoveryAction; + readonly itemId: string; + readonly attemptId: string; + readonly leaseEpoch: number; + readonly attestation: string; + readonly expectedTreeIdentity?: string; +} +export declare function recoverUnknownExecution(runDirectory: string, charter: RunCharter, projection: RunProjection, lock: RunLock, request: UnknownRecoveryRequest): Promise; diff --git a/skills/autopilot/runtime/dist/src/execution-recovery.js b/skills/autopilot/runtime/dist/src/execution-recovery.js new file mode 100644 index 0000000..d80ab9c --- /dev/null +++ b/skills/autopilot/runtime/dist/src/execution-recovery.js @@ -0,0 +1,114 @@ +import { lstat } from "node:fs/promises"; +import { join } from "node:path"; +import { AutopilotError } from "./errors.js"; +import { newEventId } from "./events.js"; +import { appendEvent, readJournal } from "./journal.js"; +import { canonicalJson, sha256 } from "./json.js"; +import { readLease, retireWriterLease } from "./leases.js"; +import { writeSnapshot } from "./projection.js"; +import { reduce } from "./reducer.js"; +import { assertWritablePaths, observeRepository, quarantineWorktree } from "./repository.js"; +export async function recoverUnknownExecution(runDirectory, charter, projection, lock, request) { + if (request.attestation.trim().length === 0 || Buffer.byteLength(request.attestation) > 4_096) { + throw new AutopilotError("CHARTER_INVALID", "unknown-execution recovery requires a bounded explicit operator attestation"); + } + const item = charter.work.find(({ id }) => id === request.itemId); + const projectedItem = projection.items[request.itemId]; + const attempt = projectedItem?.attempts.at(-1); + if (item === undefined || projection.waiting?.kind !== "execution-unknown" + || projection.waiting.itemId !== request.itemId || projection.waiting.attemptId !== request.attemptId + || projectedItem?.state !== "BLOCKED" || projectedItem.blocker !== "EXECUTION_STATE_UNKNOWN" + || attempt?.attemptId !== request.attemptId || attempt.leaseEpoch !== request.leaseEpoch) { + throw new AutopilotError("ILLEGAL_TRANSITION", "recovery identity does not match the current unknown execution"); + } + const lease = await readLease(runDirectory, request.itemId); + if (lease === undefined || lease.itemId !== request.itemId || lease.attemptId !== request.attemptId + || lease.epoch !== request.leaseEpoch) { + throw new AutopilotError("ILLEGAL_TRANSITION", "recovery lease fence does not match the unknown execution"); + } + await lock.assertOwned(); + const base = { + eventId: newEventId(), + timestamp: new Date().toISOString(), + source: "operator", + itemId: request.itemId, + attemptId: request.attemptId, + lockTokenHash: sha256(lock.owner.token), + attestation: request.attestation.trim(), + }; + let event; + if (request.action === "stop") { + event = { + ...base, + type: "RUN_STOPPED", + reason: "Operator stopped a run with unknown execution state", + errorCode: "OPERATOR_STOP", + leaseEpoch: request.leaseEpoch, + remediation: "The quarantined execution evidence remains available for manual inspection.", + }; + } + else if (request.action === "adopt") { + const observation = await observeRepository(lease.worktreePath); + if (request.expectedTreeIdentity === undefined || request.expectedTreeIdentity !== observation.treeIdentity) { + throw new AutopilotError("EFFECT_RECONCILIATION_FAILED", "adopted tree does not match --tree"); + } + await assertWritablePaths(lease.worktreePath, observation.changedPaths, item.writableRoots); + await retireWriterLease(runDirectory, lease); + event = { + ...base, + type: "EXECUTION_UNKNOWN_TREE_ADOPTED", + reason: "Operator sealed the exact unknown worktree for verification without another implementation worker", + leaseEpoch: request.leaseEpoch, + worktreePath: lease.worktreePath, + headCommit: observation.headCommit, + treeIdentity: observation.treeIdentity, + refIdentity: observation.refIdentity, + auxiliaryRefIdentity: observation.auxiliaryRefIdentity, + externalRefIdentity: observation.externalRefIdentity, + configurationIdentity: observation.configurationIdentity, + changedPaths: observation.changedPaths, + }; + } + else { + let before; + try { + await lstat(lease.worktreePath); + before = await observeRepository(lease.worktreePath); + } + catch (error) { + if (!(error instanceof Error && "code" in error && error.code === "ENOENT")) { + throw error; + } + } + const quarantined = await quarantineWorktree(charter.repository.root, lease.worktreePath, charter.runId, item.id, request.attemptId); + if (before !== undefined && (before.headCommit !== quarantined.observation.headCommit + || before.treeIdentity !== quarantined.observation.treeIdentity + || before.refIdentity !== quarantined.observation.refIdentity + || before.configurationIdentity !== quarantined.observation.configurationIdentity + || canonicalJson(before.changedPaths) !== canonicalJson(quarantined.observation.changedPaths))) { + throw new AutopilotError("EXECUTION_STATE_UNKNOWN", "worktree changed while it was being quarantined"); + } + await retireWriterLease(runDirectory, lease); + event = { + ...base, + type: "EXECUTION_UNKNOWN_ABANDONED", + reason: "Operator abandoned the unknown execution and permanently quarantined its worktree", + leaseEpoch: request.leaseEpoch, + originalWorktreePath: lease.worktreePath, + quarantineWorktreePath: quarantined.path, + headCommit: quarantined.observation.headCommit, + treeIdentity: quarantined.observation.treeIdentity, + refIdentity: quarantined.observation.refIdentity, + auxiliaryRefIdentity: quarantined.observation.auxiliaryRefIdentity, + externalRefIdentity: quarantined.observation.externalRefIdentity, + configurationIdentity: quarantined.observation.configurationIdentity, + changedPaths: quarantined.observation.changedPaths, + }; + } + await lock.assertOwned(); + const next = reduce(projection, event); + await appendEvent(join(runDirectory, "events.jsonl"), event); + const journal = await readJournal(join(runDirectory, "events.jsonl")); + await writeSnapshot(join(runDirectory, "snapshot.json"), next, journal.records); + return next; +} diff --git a/skills/autopilot/runtime/dist/src/frontier.js b/skills/autopilot/runtime/dist/src/frontier.js index c14b479..acacfda 100644 --- a/skills/autopilot/runtime/dist/src/frontier.js +++ b/skills/autopilot/runtime/dist/src/frontier.js @@ -9,7 +9,8 @@ export function runnableFrontier(charter, projection, adapterConcurrency) { const unknownRetryAvailable = itemProjection?.blocker !== "UNKNOWN_FAILURE" || attemptsUsed < 2; const replanAvailable = itemProjection?.blocker !== "PREDICATE_NOT_MET" || itemProjection.replansUsed < charter.limits.maxReplans; - const nonRetryable = itemProjection?.blocker === "CAPABILITY_DENIED" + const nonRetryable = itemProjection?.attempts.at(-1)?.adoptedTree !== undefined + || itemProjection?.blocker === "CAPABILITY_DENIED" || itemProjection?.blocker === "BRANCH_COLLISION" || itemProjection?.blocker === "UNEXPECTED_COMMIT" || itemProjection?.blocker === "PRE_COMMIT_HOOK_FAILED" diff --git a/skills/autopilot/runtime/dist/src/lock.d.ts b/skills/autopilot/runtime/dist/src/lock.d.ts index 716e12d..b5d2de6 100644 --- a/skills/autopilot/runtime/dist/src/lock.d.ts +++ b/skills/autopilot/runtime/dist/src/lock.d.ts @@ -10,6 +10,7 @@ export interface RunLock { relocate(path: string): Promise; controlRequested(runId: string): Promise; stopRequested(runId: string): Promise; + assertOwned(): Promise; release(): Promise; } export type ControlRequestResult = { diff --git a/skills/autopilot/runtime/dist/src/lock.js b/skills/autopilot/runtime/dist/src/lock.js index 819ce0a..8d0c9a0 100644 --- a/skills/autopilot/runtime/dist/src/lock.js +++ b/skills/autopilot/runtime/dist/src/lock.js @@ -141,6 +141,12 @@ export async function acquireRunLock(path, resource = "run") { throw error; } }, + async assertOwned() { + const current = await readLockOwner(ownedPath); + if (current?.token !== owner.token) { + throw new AutopilotError("LOCK_HELD", `${resource} lock ownership changed`, { owner: current }); + } + }, async release() { const current = await readLockOwner(ownedPath); if (current?.token === owner.token) { diff --git a/skills/autopilot/runtime/dist/src/reducer.d.ts b/skills/autopilot/runtime/dist/src/reducer.d.ts index 7e3d7d2..e62c95d 100644 --- a/skills/autopilot/runtime/dist/src/reducer.d.ts +++ b/skills/autopilot/runtime/dist/src/reducer.d.ts @@ -33,6 +33,17 @@ export interface AttemptProjection { readonly observedHeadCommit?: string; readonly observedTreeIdentity?: string; readonly budgetConsumed?: boolean; + readonly quarantinedWorktreePath?: string; + readonly adoptedTree?: { + readonly worktreePath: string; + readonly headCommit: string; + readonly treeIdentity: string; + readonly refIdentity: string; + readonly auxiliaryRefIdentity: string; + readonly externalRefIdentity: string; + readonly configurationIdentity: string; + readonly changedPaths: readonly string[]; + }; } export interface VerifiedCheckpoint { readonly attemptId: string; diff --git a/skills/autopilot/runtime/dist/src/reducer.js b/skills/autopilot/runtime/dist/src/reducer.js index 4242068..6a5b442 100644 --- a/skills/autopilot/runtime/dist/src/reducer.js +++ b/skills/autopilot/runtime/dist/src/reducer.js @@ -160,13 +160,59 @@ function transitionItem(item, event) { : attempt), }; } + case "EXECUTION_UNKNOWN_ABANDONED": { + const attempt = item.attempts.at(-1); + if (item.state !== "BLOCKED" || item.blocker !== "EXECUTION_STATE_UNKNOWN" + || attempt?.attemptId !== event.attemptId || attempt.leaseEpoch !== event.leaseEpoch) { + throw new AutopilotError("ILLEGAL_TRANSITION", "unknown execution abandonment does not match the current fenced attempt"); + } + const { blocker: _blocker, ...unblocked } = item; + return { + ...unblocked, + state: "READY", + attempts: item.attempts.map((candidate) => candidate.attemptId === event.attemptId + ? { ...candidate, outcome: "stale", budgetConsumed: true, quarantinedWorktreePath: event.quarantineWorktreePath } + : candidate), + }; + } + case "EXECUTION_UNKNOWN_TREE_ADOPTED": { + const attempt = item.attempts.at(-1); + if (item.state !== "BLOCKED" || item.blocker !== "EXECUTION_STATE_UNKNOWN" + || attempt?.attemptId !== event.attemptId || attempt.leaseEpoch !== event.leaseEpoch) { + throw new AutopilotError("ILLEGAL_TRANSITION", "unknown execution adoption does not match the current fenced attempt"); + } + const { blocker: _blocker, ...unblocked } = item; + return { + ...unblocked, + state: "ACTIVE", + attempts: item.attempts.map((candidate) => candidate.attemptId === event.attemptId + ? { + ...candidate, + outcome: "completed", + budgetConsumed: true, + quarantinedWorktreePath: event.worktreePath, + adoptedTree: { + worktreePath: event.worktreePath, + headCommit: event.headCommit, + treeIdentity: event.treeIdentity, + refIdentity: event.refIdentity, + auxiliaryRefIdentity: event.auxiliaryRefIdentity, + externalRefIdentity: event.externalRefIdentity, + configurationIdentity: event.configurationIdentity, + changedPaths: event.changedPaths, + }, + } + : candidate), + }; + } case "ATTEMPT_FINISHED": { if (item.state !== "ACTIVE") { throw new AutopilotError("ILLEGAL_TRANSITION", `ATTEMPT_FINISHED cannot follow ${item.state}`); } const currentAttempt = item.attempts.at(-1); - if (currentAttempt?.attemptId !== event.attemptId) { - throw new AutopilotError("ILLEGAL_TRANSITION", `attempt ${event.attemptId} is stale for item ${item.itemId}`); + if (currentAttempt?.attemptId !== event.attemptId + || currentAttempt.outcome !== undefined || currentAttempt.adoptedTree !== undefined) { + throw new AutopilotError("ILLEGAL_TRANSITION", `attempt ${event.attemptId} is stale or already recovered for item ${item.itemId}`); } return { ...item, @@ -375,7 +421,8 @@ export function reduce(projection, event) { const nextState = assertRunTransition(projection.state, event); let items = projection.items; const ordinaryItemLifecycle = [ - "DECISION_RECORDED", "ITEM_READY", "ATTEMPT_STARTED", "ATTEMPT_EXECUTION_ADMITTED", "ATTEMPT_FINISHED", "ITEM_VERIFYING", "ATTEMPT_PAUSED", + "DECISION_RECORDED", "ITEM_READY", "ATTEMPT_STARTED", "ATTEMPT_EXECUTION_ADMITTED", "EXECUTION_UNKNOWN_ABANDONED", + "EXECUTION_UNKNOWN_TREE_ADOPTED", "ATTEMPT_FINISHED", "ITEM_VERIFYING", "ATTEMPT_PAUSED", "ITEM_VERIFIED", "ITEM_SATISFIED", "ITEM_BLOCKED", "ITEM_ABANDONED", ]; if (event.itemId !== undefined && projection.restacks[event.itemId] !== undefined diff --git a/skills/autopilot/runtime/dist/src/report.d.ts b/skills/autopilot/runtime/dist/src/report.d.ts index 07497aa..291d043 100644 --- a/skills/autopilot/runtime/dist/src/report.d.ts +++ b/skills/autopilot/runtime/dist/src/report.d.ts @@ -17,6 +17,10 @@ export interface RunReport { readonly blocker?: string; readonly attempts: number; readonly chargedAttempts: number; + readonly recovery?: { + readonly quarantinedWorktreePath: string; + readonly adoptedTreeIdentity?: string; + }; readonly execution?: { readonly assurance?: ExecutionAssurance; readonly adapterName?: string; diff --git a/skills/autopilot/runtime/dist/src/report.js b/skills/autopilot/runtime/dist/src/report.js index e3048d8..f3b0727 100644 --- a/skills/autopilot/runtime/dist/src/report.js +++ b/skills/autopilot/runtime/dist/src/report.js @@ -34,7 +34,7 @@ export async function writeReports(runDirectory, charter, projection, records, a : projection.waiting?.kind === "operator-pause" ? "/autopilot resume" : projection.waiting?.kind === "execution-unknown" - ? "Do not launch a replacement; prove or externally cancel the orphaned execution first." + ? "Use the fenced recover command to abandon, adopt the exact tree, or stop; do not launch a replacement." : projection.waiting?.kind === "provider-checks" ? "Wait for the bounded provider-check session, or /autopilot resume after the coordinator exits." : projection.state === "RUNNING" || projection.state === "RECONCILING" || projection.state === "VERIFYING" @@ -91,6 +91,12 @@ export async function writeReports(runDirectory, charter, projection, records, a chargedAttempts: consumedAttempts(itemProjection), ...(itemProjection?.subject === undefined ? {} : { subject: itemProjection.subject }), ...(itemProjection?.blocker === undefined ? {} : { blocker: itemProjection.blocker }), + ...(lastAttempt?.quarantinedWorktreePath === undefined ? {} : { + recovery: { + quarantinedWorktreePath: lastAttempt.quarantinedWorktreePath, + ...(lastAttempt.adoptedTree === undefined ? {} : { adoptedTreeIdentity: lastAttempt.adoptedTree.treeIdentity }), + }, + }), ...(lastAttempt?.executionAssurance === undefined && execution === undefined ? {} : { execution: { ...(lastAttempt?.executionAssurance === undefined ? {} : { assurance: lastAttempt.executionAssurance }), diff --git a/skills/autopilot/runtime/dist/src/repository.d.ts b/skills/autopilot/runtime/dist/src/repository.d.ts index ecfdec1..0a7b777 100644 --- a/skills/autopilot/runtime/dist/src/repository.d.ts +++ b/skills/autopilot/runtime/dist/src/repository.d.ts @@ -31,6 +31,10 @@ export declare function branchExists(repositoryRoot: string, branchName: string) export declare function validateBranchName(repositoryRoot: string, branchName: string): Promise; export declare function resolveWorktreePath(charter: RunCharter, item: WorkItem): Promise; export declare function assertRegisteredWorktree(repositoryRoot: string, worktreePath: string): Promise; +export declare function quarantineWorktree(repositoryRoot: string, worktreePath: string, runId: string, itemId: string, attemptId: string): Promise<{ + readonly path: string; + readonly observation: RepositoryObservation; +}>; export declare function ensureWorktree(charter: RunCharter, item: WorkItem, baseCommit?: string, ownedCommits?: readonly string[]): Promise; export declare function assertWritablePaths(worktreePath: string, changed: readonly string[], writableRoots: readonly string[]): Promise; export declare function observeRepository(worktreePath: string, managedBranches?: readonly ManagedBranchExpectation[]): Promise; diff --git a/skills/autopilot/runtime/dist/src/repository.js b/skills/autopilot/runtime/dist/src/repository.js index cca49de..9845ebc 100644 --- a/skills/autopilot/runtime/dist/src/repository.js +++ b/skills/autopilot/runtime/dist/src/repository.js @@ -103,6 +103,36 @@ export async function assertRegisteredWorktree(repositoryRoot, worktreePath) { throw new AutopilotError("BRANCH_COLLISION", "managed worktree path is not registered to the charter repository"); } } +export async function quarantineWorktree(repositoryRoot, worktreePath, runId, itemId, attemptId) { + const quarantinePath = `${worktreePath}.quarantine-${sha256(`${runId}\0${itemId}\0${attemptId}`).slice(0, 16)}`; + let activePath = worktreePath; + try { + await lstat(worktreePath); + await assertRegisteredWorktree(repositoryRoot, worktreePath); + try { + await lstat(quarantinePath); + throw new AutopilotError("BRANCH_COLLISION", `quarantine destination already exists: ${quarantinePath}`); + } + catch (error) { + if (!(error instanceof Error && "code" in error && error.code === "ENOENT")) { + throw error; + } + } + await runChecked({ executable: "git", arguments: ["worktree", "move", worktreePath, quarantinePath], cwd: repositoryRoot }); + activePath = quarantinePath; + } + catch (error) { + if (!(error instanceof Error && "code" in error && error.code === "ENOENT")) { + throw error; + } + await assertRegisteredWorktree(repositoryRoot, quarantinePath); + activePath = quarantinePath; + } + if (await currentBranch(activePath) !== "") { + await runChecked({ executable: "git", arguments: ["switch", "--detach", "--quiet", "HEAD"], cwd: activePath }); + } + return { path: activePath, observation: await observeRepository(activePath) }; +} export async function ensureWorktree(charter, item, baseCommit = charter.repository.baseCommit, ownedCommits = []) { await validateBranchName(charter.repository.root, item.branchName); const worktreePath = await resolveWorktreePath(charter, item); diff --git a/skills/autopilot/runtime/src/cli.ts b/skills/autopilot/runtime/src/cli.ts index 5ac7c71..5bfb11c 100644 --- a/skills/autopilot/runtime/src/cli.ts +++ b/skills/autopilot/runtime/src/cli.ts @@ -12,6 +12,7 @@ import { runDoctor } from "./doctor.js"; import { AutopilotEngine } from "./engine.js"; import { AutopilotError } from "./errors.js"; import { newEventId } from "./events.js"; +import { recoverUnknownExecution, type UnknownRecoveryAction } from "./execution-recovery.js"; import { appendEvent, readJournal, repairTruncatedJournal, writeImmutableJson } from "./journal.js"; import { isRecord } from "./json.js"; import { acquireBranchOwnershipLock, acquireRunLock, requestRunPause, requestRunStop, type RunLock } from "./lock.js"; @@ -38,6 +39,12 @@ interface CliOptions { readonly json: boolean; readonly repairJournal: boolean; readonly handoff: boolean; + readonly recoveryAction?: UnknownRecoveryAction; + readonly recoveryItem?: string; + readonly recoveryAttempt?: string; + readonly recoveryLeaseEpoch?: number; + readonly recoveryAttestation?: string; + readonly recoveryTree?: string; } function usage(): string { @@ -49,6 +56,7 @@ Usage: autopilot [--state-dir ] [--json] [--repair-journal] resume [run-id] autopilot [--state-dir ] [--json] pause [run-id] autopilot [--state-dir ] [--json] stop [run-id] + autopilot [--state-dir ] [--json] recover --action --item --attempt --lease-epoch --attestation [--tree ] autopilot [--state-dir ] [--json] review-feedback [run-id] autopilot [--state-dir ] [--json] [--handoff] wrap-up [run-id] autopilot [--json] doctor @@ -369,6 +377,60 @@ async function resume(runId: string | undefined, options: CliOptions): Promise { + if (runId === undefined || options.recoveryAction === undefined || options.recoveryItem === undefined + || options.recoveryAttempt === undefined || options.recoveryLeaseEpoch === undefined + || options.recoveryAttestation === undefined) { + throw new AutopilotError("CHARTER_INVALID", "recover requires run ID, action, item, attempt, lease epoch, and attestation"); + } + const stateRoot = await resolveStateRoot(process.cwd(), options.stateDir); + const location = await locateStoredRun(stateRoot, runId); + const ownershipLock = location.charter.amends === undefined + ? undefined + : await acquireBranchOwnershipLock(stateRoot, location.charter.work[0]?.branchName ?? ""); + try { + const lock = await acquireRunLock(join(location.directory, "run.lock")); + try { + const run = await loadRun(runId, options.stateDir, false); + const projection = rebuildProjection(run.charter, run.journal.records); + const recovered = await recoverUnknownExecution(run.directory, run.charter, projection, lock, { + action: options.recoveryAction, + itemId: options.recoveryItem, + attemptId: options.recoveryAttempt, + leaseEpoch: options.recoveryLeaseEpoch, + attestation: options.recoveryAttestation, + ...(options.recoveryTree === undefined ? {} : { expectedTreeIdentity: options.recoveryTree }), + }); + if (recovered.state === "STOPPED") { + const journal = await readJournal(join(run.directory, "events.jsonl")); + const metadata = await loadReportMetadata(run.directory); + return await writeReports( + run.directory, + run.charter, + recovered, + journal.records, + metadata.assurance, + metadata.unverifiedBoundaries, + ); + } + const journal = await readJournal(join(run.directory, "events.jsonl")); + const engine = new AutopilotEngine({ + stateRoot: run.stateRoot, + runDirectory: run.directory, + charter: run.charter, + adapter: createAdapter(run.charter.harnessAdapter), + records: journal.records, + projection: recovered, + }); + return await runEngine(engine, lock, run.charter.runId); + } finally { + await lock.release(); + } + } finally { + await ownershipLock?.release(); + } +} + async function reviewFeedback(runId: string | undefined, options: CliOptions): Promise { const stateRoot = await resolveStateRoot(process.cwd(), options.stateDir); return await observeReviewFeedback(stateRoot, process.cwd(), runId); @@ -554,6 +616,12 @@ export async function main(arguments_: readonly string[] = process.argv.slice(2) json: { type: "boolean", default: false }, "repair-journal": { type: "boolean", default: false }, handoff: { type: "boolean", default: false }, + action: { type: "string" }, + item: { type: "string" }, + attempt: { type: "string" }, + "lease-epoch": { type: "string" }, + attestation: { type: "string" }, + tree: { type: "string" }, help: { type: "boolean", short: "h", default: false }, version: { type: "boolean", short: "v", default: false }, }, @@ -570,11 +638,29 @@ export async function main(arguments_: readonly string[] = process.argv.slice(2) if (command === undefined || extra.length > 0) { throw new AutopilotError("CHARTER_INVALID", usage()); } + const recoveryAction = parsed.values.action; + if (recoveryAction !== undefined && !["abandon", "adopt", "stop"].includes(recoveryAction)) { + throw new AutopilotError("CHARTER_INVALID", `unsupported recovery action: ${recoveryAction}`); + } + const recoveryLeaseEpochText = parsed.values["lease-epoch"]; + const recoveryLeaseEpoch = recoveryLeaseEpochText === undefined || !/^\d+$/u.test(recoveryLeaseEpochText) + ? undefined + : Number.parseInt(recoveryLeaseEpochText, 10); + if (recoveryLeaseEpochText !== undefined + && (recoveryLeaseEpoch === undefined || !Number.isSafeInteger(recoveryLeaseEpoch) || recoveryLeaseEpoch < 1)) { + throw new AutopilotError("CHARTER_INVALID", "--lease-epoch must be a positive integer"); + } const options: CliOptions = { json: parsed.values.json, repairJournal: parsed.values["repair-journal"], handoff: parsed.values.handoff, ...(parsed.values["state-dir"] === undefined ? {} : { stateDir: parsed.values["state-dir"] }), + ...(recoveryAction === undefined ? {} : { recoveryAction: recoveryAction as UnknownRecoveryAction }), + ...(parsed.values.item === undefined ? {} : { recoveryItem: parsed.values.item }), + ...(parsed.values.attempt === undefined ? {} : { recoveryAttempt: parsed.values.attempt }), + ...(recoveryLeaseEpoch === undefined ? {} : { recoveryLeaseEpoch }), + ...(parsed.values.attestation === undefined ? {} : { recoveryAttestation: parsed.values.attestation }), + ...(parsed.values.tree === undefined ? {} : { recoveryTree: parsed.values.tree }), }; let result: unknown; switch (command) { @@ -599,6 +685,9 @@ export async function main(arguments_: readonly string[] = process.argv.slice(2) case "stop": result = await stop(argument, options); break; + case "recover": + result = await recover(argument, options); + break; case "review-feedback": result = await reviewFeedback(argument, options); break; diff --git a/skills/autopilot/runtime/src/engine.ts b/skills/autopilot/runtime/src/engine.ts index 7eb1178..d39acc4 100644 --- a/skills/autopilot/runtime/src/engine.ts +++ b/skills/autopilot/runtime/src/engine.ts @@ -963,6 +963,50 @@ export class AutopilotEngine { }); continue; } + if (itemProjection.state === "ACTIVE" && attempt.adoptedTree !== undefined) { + const adopted = attempt.adoptedTree; + let observed: RepositoryObservation; + try { + observed = await this.#observeRepository(adopted.worktreePath); + if (observed.headCommit !== adopted.headCommit || observed.treeIdentity !== adopted.treeIdentity + || observed.refIdentity !== adopted.refIdentity + || observed.configurationIdentity !== adopted.configurationIdentity + || canonicalJson(observed.changedPaths) !== canonicalJson(adopted.changedPaths)) { + throw new AutopilotError("EXECUTION_STATE_UNKNOWN", "adopted worktree changed before verification"); + } + } catch (error) { + await this.#record({ + ...eventBase(error instanceof Error ? error.message : String(error), "reconciler"), + type: "ITEM_BLOCKED", + itemId: item.id, + attemptId: attempt.attemptId, + errorCode: "EXECUTION_STATE_UNKNOWN", + }); + continue; + } + try { + await assertWritablePaths(adopted.worktreePath, observed.changedPaths, item.writableRoots); + await this.#verifyObservedItem( + item, + attempt.attemptId, + adopted.worktreePath, + observed, + await inspectPreCommitHook(adopted.worktreePath), + ); + } catch (error) { + const state = this.#projection.items[item.id]?.state; + if (state === "ACTIVE" || state === "VERIFYING") { + await this.#record({ + ...eventBase(error instanceof Error ? error.message : String(error), "reconciler"), + type: "ITEM_BLOCKED", + itemId: item.id, + attemptId: attempt.attemptId, + errorCode: error instanceof AutopilotError ? error.code : "UNKNOWN_FAILURE", + }); + } + } + continue; + } if (itemProjection.state === "ACTIVE") { let reattachedStatus: ExecutionObservation["status"] | "stale" | false; try { @@ -2251,6 +2295,149 @@ export class AutopilotEngine { } } + async #verifyObservedItem( + item: WorkItem, + attemptId: string, + worktreePath: string, + initialObservation: RepositoryObservation, + hookSnapshot: Awaited> | undefined, + ): Promise { + await this.#record({ ...eventBase("Runtime is verifying the observed tree"), type: "ITEM_VERIFYING", itemId: item.id, attemptId }); + + let finalObservation = initialObservation; + let verification = await this.#verifyItem(item, worktreePath, attemptId, finalObservation); + if (await this.#blockItemForStop(item, attemptId)) { + return; + } + if (!verification.met) { + await this.#record({ + ...eventBase(verification.reasons.join("; ") || "Acceptance predicates are not met"), + type: "ITEM_BLOCKED", + itemId: item.id, + attemptId, + errorCode: "PREDICATE_NOT_MET", + }); + return; + } + if (!finalObservation.clean && this.#charter.commitPolicy?.preCommitHook === "run") { + if (await this.#blockItemForStop(item, attemptId)) { + return; + } + const hook = await runPreCommitHook( + worktreePath, + hookSnapshot ?? { identity: "NOT_CONFIGURED" }, + this.#charter.commitPolicy.environmentNames, + this.#charter.limits.attemptTimeoutMs, + this.#charter.limits.maxRetainedOutputBytes, + ); + const hookObservation = await this.#observeRepository(worktreePath); + const hooksDirectory = join(this.#runDirectory, "reports", "hooks"); + await mkdir(hooksDirectory, { recursive: true, mode: 0o700 }); + const hookPath = join(hooksDirectory, `${attemptId}.json`); + await writeJsonAtomic(hookPath, { + schemaVersion: 1, + status: hook.status, + configuredPath: hook.path, + beforeTree: finalObservation.treeIdentity, + afterTree: hookObservation.treeIdentity, + exitCode: hook.result?.exitCode ?? 0, + stdout: redactEnvironmentSecrets(hook.result?.stdout ?? "", this.#charter.commitPolicy.environmentNames), + stderr: redactEnvironmentSecrets(hook.result?.stderr ?? "", this.#charter.commitPolicy.environmentNames), + truncated: hook.result?.truncated ?? false, + }); + await this.#record({ + ...eventBase(`Pre-commit hook finished ${hook.status}`), + type: "PRE_COMMIT_HOOK_FINISHED", + itemId: item.id, + attemptId, + status: hook.status, + beforeTree: finalObservation.treeIdentity, + afterTree: hookObservation.treeIdentity, + exitCode: hook.result?.exitCode ?? 0, + evidence: [hookPath], + }); + if (await this.#blockItemForStop(item, attemptId)) { + return; + } + if (hookObservation.headCommit !== finalObservation.headCommit + || hookObservation.externalRefIdentity !== finalObservation.externalRefIdentity) { + throw new AutopilotError("BRANCH_COLLISION", "pre-commit hook changed HEAD or another Git ref"); + } + if (hookObservation.configurationIdentity !== finalObservation.configurationIdentity) { + throw new AutopilotError("CAPABILITY_DENIED", "pre-commit hook changed Git configuration"); + } + await assertWritablePaths( + worktreePath, + hookObservation.changedPaths, + [...item.writableRoots, ...this.#charter.commitPolicy.writableRoots], + ); + if (hook.status === "FAILED") { + await this.#record({ + ...eventBase("Pre-commit hook failed before the runtime-owned commit"), + type: "ITEM_BLOCKED", + itemId: item.id, + attemptId, + errorCode: "PRE_COMMIT_HOOK_FAILED", + }); + return; + } + if (hookObservation.treeIdentity !== finalObservation.treeIdentity) { + verification = await this.#verifyItem(item, worktreePath, attemptId, hookObservation); + if (!verification.met) { + await this.#record({ + ...eventBase(verification.reasons.join("; ") || "Post-hook acceptance predicates are not met"), + type: "ITEM_BLOCKED", + itemId: item.id, + attemptId, + errorCode: "POST_HOOK_PREDICATE_NOT_MET", + }); + return; + } + } + finalObservation = hookObservation; + } + const finalHook = await inspectPreCommitHook(worktreePath); + const receiptIds: string[] = []; + for (const { event: receiptEvent } of this.#records) { + if (receiptEvent.type !== "RECEIPT_RECORDED" || receiptEvent.itemId !== item.id + || receiptEvent.attemptId !== attemptId) { + continue; + } + const receiptValue: unknown = JSON.parse(await readFile( + join(this.#runDirectory, "receipts", `${receiptEvent.receiptId}.json`), + "utf8", + )); + const requiredGateReceipt = receiptEvent.gateId !== undefined && item.acceptance.some((predicate) => + predicate.type === "gate-passed" && predicate.gateId === receiptEvent.gateId + ); + if (isRecord(receiptValue) && receiptValue.subject === verification.subject + && (receiptEvent.receiptKind === "predicate" || requiredGateReceipt)) { + receiptIds.push(receiptEvent.receiptId); + } + } + await this.#record({ + ...eventBase("Exact tree and acceptance evidence verified"), + type: "ITEM_VERIFIED", + itemId: item.id, + attemptId, + subject: verification.subject, + headCommit: finalObservation.headCommit, + treeIdentity: finalObservation.treeIdentity, + auxiliaryRefIdentity: finalObservation.auxiliaryRefIdentity, + externalRefIdentity: finalObservation.externalRefIdentity, + configurationIdentity: finalObservation.configurationIdentity, + hookIdentity: finalHook.identity, + ...(finalHook.path === undefined ? {} : { hookPath: finalHook.path }), + commitRequired: !finalObservation.clean, + receiptIds, + }); + const checkpoint = this.#projection.items[item.id]?.verified; + if (checkpoint === undefined) { + throw new AutopilotError("JOURNAL_CORRUPT", "verified checkpoint was not projected"); + } + await this.#completeVerifiedItem(item, worktreePath, checkpoint); + } + async #runItem(item: WorkItem): Promise { if (this.#stopRequested || this.#pauseRequested) { return; @@ -2497,140 +2684,7 @@ export class AutopilotEngine { if (await this.#blockItemForStop(item, attemptId)) { return; } - await this.#record({ ...eventBase("Runtime is verifying the observed tree"), type: "ITEM_VERIFYING", itemId: item.id, attemptId }); - - let finalObservation = after; - let verification = await this.#verifyItem(item, worktreePath, attemptId, finalObservation); - if (await this.#blockItemForStop(item, attemptId)) { - return; - } - if (!verification.met) { - await this.#record({ - ...eventBase(verification.reasons.join("; ") || "Acceptance predicates are not met"), - type: "ITEM_BLOCKED", - itemId: item.id, - attemptId, - errorCode: "PREDICATE_NOT_MET", - }); - return; - } - if (!finalObservation.clean && this.#charter.commitPolicy?.preCommitHook === "run") { - if (await this.#blockItemForStop(item, attemptId)) { - return; - } - const hook = await runPreCommitHook( - worktreePath, - hookSnapshot ?? { identity: "NOT_CONFIGURED" }, - this.#charter.commitPolicy.environmentNames, - this.#charter.limits.attemptTimeoutMs, - this.#charter.limits.maxRetainedOutputBytes, - ); - const hookObservation = await this.#observeRepository(worktreePath); - const hooksDirectory = join(this.#runDirectory, "reports", "hooks"); - await mkdir(hooksDirectory, { recursive: true, mode: 0o700 }); - const hookPath = join(hooksDirectory, `${attemptId}.json`); - await writeJsonAtomic(hookPath, { - schemaVersion: 1, - status: hook.status, - configuredPath: hook.path, - beforeTree: finalObservation.treeIdentity, - afterTree: hookObservation.treeIdentity, - exitCode: hook.result?.exitCode ?? 0, - stdout: redactEnvironmentSecrets(hook.result?.stdout ?? "", this.#charter.commitPolicy.environmentNames), - stderr: redactEnvironmentSecrets(hook.result?.stderr ?? "", this.#charter.commitPolicy.environmentNames), - truncated: hook.result?.truncated ?? false, - }); - await this.#record({ - ...eventBase(`Pre-commit hook finished ${hook.status}`), - type: "PRE_COMMIT_HOOK_FINISHED", - itemId: item.id, - attemptId, - status: hook.status, - beforeTree: finalObservation.treeIdentity, - afterTree: hookObservation.treeIdentity, - exitCode: hook.result?.exitCode ?? 0, - evidence: [hookPath], - }); - if (await this.#blockItemForStop(item, attemptId)) { - return; - } - if (hookObservation.headCommit !== finalObservation.headCommit - || hookObservation.externalRefIdentity !== finalObservation.externalRefIdentity) { - throw new AutopilotError("BRANCH_COLLISION", "pre-commit hook changed HEAD or another Git ref"); - } - if (hookObservation.configurationIdentity !== finalObservation.configurationIdentity) { - throw new AutopilotError("CAPABILITY_DENIED", "pre-commit hook changed Git configuration"); - } - await assertWritablePaths( - worktreePath, - hookObservation.changedPaths, - [...item.writableRoots, ...this.#charter.commitPolicy.writableRoots], - ); - if (hook.status === "FAILED") { - await this.#record({ - ...eventBase("Pre-commit hook failed before the runtime-owned commit"), - type: "ITEM_BLOCKED", - itemId: item.id, - attemptId, - errorCode: "PRE_COMMIT_HOOK_FAILED", - }); - return; - } - if (hookObservation.treeIdentity !== finalObservation.treeIdentity) { - verification = await this.#verifyItem(item, worktreePath, attemptId, hookObservation); - if (!verification.met) { - await this.#record({ - ...eventBase(verification.reasons.join("; ") || "Post-hook acceptance predicates are not met"), - type: "ITEM_BLOCKED", - itemId: item.id, - attemptId, - errorCode: "POST_HOOK_PREDICATE_NOT_MET", - }); - return; - } - } - finalObservation = hookObservation; - } - const finalHook = await inspectPreCommitHook(worktreePath); - const receiptIds: string[] = []; - for (const { event: receiptEvent } of this.#records) { - if (receiptEvent.type !== "RECEIPT_RECORDED" || receiptEvent.itemId !== item.id - || receiptEvent.attemptId !== attemptId) { - continue; - } - const receiptValue: unknown = JSON.parse(await readFile( - join(this.#runDirectory, "receipts", `${receiptEvent.receiptId}.json`), - "utf8", - )); - const requiredGateReceipt = receiptEvent.gateId !== undefined && item.acceptance.some((predicate) => - predicate.type === "gate-passed" && predicate.gateId === receiptEvent.gateId - ); - if (isRecord(receiptValue) && receiptValue.subject === verification.subject - && (receiptEvent.receiptKind === "predicate" || requiredGateReceipt)) { - receiptIds.push(receiptEvent.receiptId); - } - } - await this.#record({ - ...eventBase("Exact tree and acceptance evidence verified"), - type: "ITEM_VERIFIED", - itemId: item.id, - attemptId, - subject: verification.subject, - headCommit: finalObservation.headCommit, - treeIdentity: finalObservation.treeIdentity, - auxiliaryRefIdentity: finalObservation.auxiliaryRefIdentity, - externalRefIdentity: finalObservation.externalRefIdentity, - configurationIdentity: finalObservation.configurationIdentity, - hookIdentity: finalHook.identity, - ...(finalHook.path === undefined ? {} : { hookPath: finalHook.path }), - commitRequired: !finalObservation.clean, - receiptIds, - }); - const checkpoint = this.#projection.items[item.id]?.verified; - if (checkpoint === undefined) { - throw new AutopilotError("JOURNAL_CORRUPT", "verified checkpoint was not projected"); - } - await this.#completeVerifiedItem(item, worktreePath, checkpoint); + await this.#verifyObservedItem(item, attemptId, worktreePath, after, hookSnapshot); } async run(): Promise { diff --git a/skills/autopilot/runtime/src/events.ts b/skills/autopilot/runtime/src/events.ts index d315175..025fa5e 100644 --- a/skills/autopilot/runtime/src/events.ts +++ b/skills/autopilot/runtime/src/events.ts @@ -40,7 +40,14 @@ export type LifecycleEvent = | (EventBase & { readonly type: "RUN_RESUMED" }) | (EventBase & { readonly type: "RUN_VERIFYING" }) | (EventBase & { readonly type: "RUN_SUCCEEDED"; readonly predicateSummary: string }) - | (EventBase & { readonly type: "RUN_STOPPED"; readonly errorCode: string; readonly remediation: string }) + | (EventBase & { + readonly type: "RUN_STOPPED"; + readonly errorCode: string; + readonly remediation: string; + readonly leaseEpoch?: number; + readonly lockTokenHash?: string; + readonly attestation?: string; + }) | (EventBase & { readonly type: "WRAP_UP_STARTED"; readonly chainRunIds: readonly string[]; readonly handoff: boolean }) | (EventBase & { readonly type: "WORKTREE_ADOPTED"; @@ -84,6 +91,39 @@ export type LifecycleEvent = readonly subjectId: string; readonly harnessInstanceId?: string; }) + | (EventBase & { + readonly type: "EXECUTION_UNKNOWN_ABANDONED"; + readonly itemId: string; + readonly attemptId: string; + readonly leaseEpoch: number; + readonly lockTokenHash: string; + readonly attestation: string; + readonly originalWorktreePath: string; + readonly quarantineWorktreePath: string; + readonly headCommit: string; + readonly treeIdentity: string; + readonly refIdentity: string; + readonly auxiliaryRefIdentity: string; + readonly externalRefIdentity: string; + readonly configurationIdentity: string; + readonly changedPaths: readonly string[]; + }) + | (EventBase & { + readonly type: "EXECUTION_UNKNOWN_TREE_ADOPTED"; + readonly itemId: string; + readonly attemptId: string; + readonly leaseEpoch: number; + readonly lockTokenHash: string; + readonly attestation: string; + readonly worktreePath: string; + readonly headCommit: string; + readonly treeIdentity: string; + readonly refIdentity: string; + readonly auxiliaryRefIdentity: string; + readonly externalRefIdentity: string; + readonly configurationIdentity: string; + readonly changedPaths: readonly string[]; + }) | (EventBase & { readonly type: "ATTEMPT_FINISHED"; readonly itemId: string; @@ -182,7 +222,8 @@ export type LifecycleEvent = const EVENT_TYPES = [ "CHARTER_COMPILED", "RECONCILIATION_STARTED", "RECONCILIATION_COMPLETED", "RUN_PAUSE_REQUESTED", "RUN_WAITING", "RUN_WOKEN", "RUN_RESUMED", "RUN_VERIFYING", "RUN_SUCCEEDED", "RUN_STOPPED", "WRAP_UP_STARTED", "WORKTREE_ADOPTED", "ITEM_READY", - "ATTEMPT_STARTED", "ATTEMPT_EXECUTION_ADMITTED", "ATTEMPT_FINISHED", "ITEM_VERIFYING", "ATTEMPT_PAUSED", "ITEM_VERIFIED", "ITEM_SATISFIED", + "ATTEMPT_STARTED", "ATTEMPT_EXECUTION_ADMITTED", "EXECUTION_UNKNOWN_ABANDONED", "EXECUTION_UNKNOWN_TREE_ADOPTED", + "ATTEMPT_FINISHED", "ITEM_VERIFYING", "ATTEMPT_PAUSED", "ITEM_VERIFIED", "ITEM_SATISFIED", "ITEM_BLOCKED", "ITEM_ABANDONED", "RESTACK_DESCENDANT_STARTED", "RESTACK_DESCENDANT_TREE_PREPARED", "RESTACK_DESCENDANT_VERIFIED", "RESTACK_PROVIDER_HEAD_CONFIRMED", "RESTACK_DESCENDANT_SATISFIED", "RESTACK_DESCENDANT_BLOCKED", "EFFECT_INTENDED", @@ -246,6 +287,9 @@ export function parseLifecycleEvent(value: unknown): LifecycleEvent { type, errorCode: expectString(object.errorCode, "event.errorCode"), remediation: expectString(object.remediation, "event.remediation"), + ...(object.leaseEpoch === undefined ? {} : { leaseEpoch: expectInteger(object.leaseEpoch, "event.leaseEpoch", 1) }), + ...(object.lockTokenHash === undefined ? {} : { lockTokenHash: expectString(object.lockTokenHash, "event.lockTokenHash") }), + ...(object.attestation === undefined ? {} : { attestation: expectString(object.attestation, "event.attestation") }), }; case "WRAP_UP_STARTED": return { @@ -380,6 +424,43 @@ export function parseLifecycleEvent(value: unknown): LifecycleEvent { harnessInstanceId: expectString(object.harnessInstanceId, "event.harnessInstanceId"), }), }; + case "EXECUTION_UNKNOWN_ABANDONED": + return { + ...base, + type, + itemId: expectString(object.itemId, "event.itemId"), + attemptId: expectString(object.attemptId, "event.attemptId"), + leaseEpoch: expectInteger(object.leaseEpoch, "event.leaseEpoch", 1), + lockTokenHash: expectString(object.lockTokenHash, "event.lockTokenHash"), + attestation: expectString(object.attestation, "event.attestation"), + originalWorktreePath: expectString(object.originalWorktreePath, "event.originalWorktreePath"), + quarantineWorktreePath: expectString(object.quarantineWorktreePath, "event.quarantineWorktreePath"), + headCommit: expectString(object.headCommit, "event.headCommit"), + treeIdentity: expectString(object.treeIdentity, "event.treeIdentity"), + refIdentity: expectString(object.refIdentity, "event.refIdentity"), + auxiliaryRefIdentity: expectString(object.auxiliaryRefIdentity, "event.auxiliaryRefIdentity"), + externalRefIdentity: expectString(object.externalRefIdentity, "event.externalRefIdentity"), + configurationIdentity: expectString(object.configurationIdentity, "event.configurationIdentity"), + changedPaths: expectStringArray(object.changedPaths, "event.changedPaths"), + }; + case "EXECUTION_UNKNOWN_TREE_ADOPTED": + return { + ...base, + type, + itemId: expectString(object.itemId, "event.itemId"), + attemptId: expectString(object.attemptId, "event.attemptId"), + leaseEpoch: expectInteger(object.leaseEpoch, "event.leaseEpoch", 1), + lockTokenHash: expectString(object.lockTokenHash, "event.lockTokenHash"), + attestation: expectString(object.attestation, "event.attestation"), + worktreePath: expectString(object.worktreePath, "event.worktreePath"), + headCommit: expectString(object.headCommit, "event.headCommit"), + treeIdentity: expectString(object.treeIdentity, "event.treeIdentity"), + refIdentity: expectString(object.refIdentity, "event.refIdentity"), + auxiliaryRefIdentity: expectString(object.auxiliaryRefIdentity, "event.auxiliaryRefIdentity"), + externalRefIdentity: expectString(object.externalRefIdentity, "event.externalRefIdentity"), + configurationIdentity: expectString(object.configurationIdentity, "event.configurationIdentity"), + changedPaths: expectStringArray(object.changedPaths, "event.changedPaths"), + }; case "ATTEMPT_FINISHED": return { ...base, diff --git a/skills/autopilot/runtime/src/execution-recovery.ts b/skills/autopilot/runtime/src/execution-recovery.ts new file mode 100644 index 0000000..1105ff2 --- /dev/null +++ b/skills/autopilot/runtime/src/execution-recovery.ts @@ -0,0 +1,137 @@ +import { lstat } from "node:fs/promises"; +import { join } from "node:path"; +import type { RunCharter } from "./charter.js"; +import { AutopilotError } from "./errors.js"; +import { newEventId, type LifecycleEvent } from "./events.js"; +import { appendEvent, readJournal } from "./journal.js"; +import { canonicalJson, sha256 } from "./json.js"; +import { readLease, retireWriterLease } from "./leases.js"; +import type { RunLock } from "./lock.js"; +import { writeSnapshot } from "./projection.js"; +import { reduce, type RunProjection } from "./reducer.js"; +import { assertWritablePaths, observeRepository, quarantineWorktree } from "./repository.js"; + +export type UnknownRecoveryAction = "abandon" | "adopt" | "stop"; + +export interface UnknownRecoveryRequest { + readonly action: UnknownRecoveryAction; + readonly itemId: string; + readonly attemptId: string; + readonly leaseEpoch: number; + readonly attestation: string; + readonly expectedTreeIdentity?: string; +} + +export async function recoverUnknownExecution( + runDirectory: string, + charter: RunCharter, + projection: RunProjection, + lock: RunLock, + request: UnknownRecoveryRequest, +): Promise { + if (request.attestation.trim().length === 0 || Buffer.byteLength(request.attestation) > 4_096) { + throw new AutopilotError("CHARTER_INVALID", "unknown-execution recovery requires a bounded explicit operator attestation"); + } + const item = charter.work.find(({ id }) => id === request.itemId); + const projectedItem = projection.items[request.itemId]; + const attempt = projectedItem?.attempts.at(-1); + if (item === undefined || projection.waiting?.kind !== "execution-unknown" + || projection.waiting.itemId !== request.itemId || projection.waiting.attemptId !== request.attemptId + || projectedItem?.state !== "BLOCKED" || projectedItem.blocker !== "EXECUTION_STATE_UNKNOWN" + || attempt?.attemptId !== request.attemptId || attempt.leaseEpoch !== request.leaseEpoch) { + throw new AutopilotError("ILLEGAL_TRANSITION", "recovery identity does not match the current unknown execution"); + } + const lease = await readLease(runDirectory, request.itemId); + if (lease === undefined || lease.itemId !== request.itemId || lease.attemptId !== request.attemptId + || lease.epoch !== request.leaseEpoch) { + throw new AutopilotError("ILLEGAL_TRANSITION", "recovery lease fence does not match the unknown execution"); + } + await lock.assertOwned(); + const base = { + eventId: newEventId(), + timestamp: new Date().toISOString(), + source: "operator" as const, + itemId: request.itemId, + attemptId: request.attemptId, + lockTokenHash: sha256(lock.owner.token), + attestation: request.attestation.trim(), + }; + let event: LifecycleEvent; + if (request.action === "stop") { + event = { + ...base, + type: "RUN_STOPPED", + reason: "Operator stopped a run with unknown execution state", + errorCode: "OPERATOR_STOP", + leaseEpoch: request.leaseEpoch, + remediation: "The quarantined execution evidence remains available for manual inspection.", + }; + } else if (request.action === "adopt") { + const observation = await observeRepository(lease.worktreePath); + if (request.expectedTreeIdentity === undefined || request.expectedTreeIdentity !== observation.treeIdentity) { + throw new AutopilotError("EFFECT_RECONCILIATION_FAILED", "adopted tree does not match --tree"); + } + await assertWritablePaths(lease.worktreePath, observation.changedPaths, item.writableRoots); + await retireWriterLease(runDirectory, lease); + event = { + ...base, + type: "EXECUTION_UNKNOWN_TREE_ADOPTED", + reason: "Operator sealed the exact unknown worktree for verification without another implementation worker", + leaseEpoch: request.leaseEpoch, + worktreePath: lease.worktreePath, + headCommit: observation.headCommit, + treeIdentity: observation.treeIdentity, + refIdentity: observation.refIdentity, + auxiliaryRefIdentity: observation.auxiliaryRefIdentity, + externalRefIdentity: observation.externalRefIdentity, + configurationIdentity: observation.configurationIdentity, + changedPaths: observation.changedPaths, + }; + } else { + let before: Awaited> | undefined; + try { + await lstat(lease.worktreePath); + before = await observeRepository(lease.worktreePath); + } catch (error) { + if (!(error instanceof Error && "code" in error && error.code === "ENOENT")) { + throw error; + } + } + const quarantined = await quarantineWorktree( + charter.repository.root, + lease.worktreePath, + charter.runId, + item.id, + request.attemptId, + ); + if (before !== undefined && (before.headCommit !== quarantined.observation.headCommit + || before.treeIdentity !== quarantined.observation.treeIdentity + || before.refIdentity !== quarantined.observation.refIdentity + || before.configurationIdentity !== quarantined.observation.configurationIdentity + || canonicalJson(before.changedPaths) !== canonicalJson(quarantined.observation.changedPaths))) { + throw new AutopilotError("EXECUTION_STATE_UNKNOWN", "worktree changed while it was being quarantined"); + } + await retireWriterLease(runDirectory, lease); + event = { + ...base, + type: "EXECUTION_UNKNOWN_ABANDONED", + reason: "Operator abandoned the unknown execution and permanently quarantined its worktree", + leaseEpoch: request.leaseEpoch, + originalWorktreePath: lease.worktreePath, + quarantineWorktreePath: quarantined.path, + headCommit: quarantined.observation.headCommit, + treeIdentity: quarantined.observation.treeIdentity, + refIdentity: quarantined.observation.refIdentity, + auxiliaryRefIdentity: quarantined.observation.auxiliaryRefIdentity, + externalRefIdentity: quarantined.observation.externalRefIdentity, + configurationIdentity: quarantined.observation.configurationIdentity, + changedPaths: quarantined.observation.changedPaths, + }; + } + await lock.assertOwned(); + const next = reduce(projection, event); + await appendEvent(join(runDirectory, "events.jsonl"), event); + const journal = await readJournal(join(runDirectory, "events.jsonl")); + await writeSnapshot(join(runDirectory, "snapshot.json"), next, journal.records); + return next; +} diff --git a/skills/autopilot/runtime/src/frontier.ts b/skills/autopilot/runtime/src/frontier.ts index 45632d8..ee3ef42 100644 --- a/skills/autopilot/runtime/src/frontier.ts +++ b/skills/autopilot/runtime/src/frontier.ts @@ -11,7 +11,8 @@ export function runnableFrontier(charter: RunCharter, projection: RunProjection, const unknownRetryAvailable = itemProjection?.blocker !== "UNKNOWN_FAILURE" || attemptsUsed < 2; const replanAvailable = itemProjection?.blocker !== "PREDICATE_NOT_MET" || itemProjection.replansUsed < charter.limits.maxReplans; - const nonRetryable = itemProjection?.blocker === "CAPABILITY_DENIED" + const nonRetryable = itemProjection?.attempts.at(-1)?.adoptedTree !== undefined + || itemProjection?.blocker === "CAPABILITY_DENIED" || itemProjection?.blocker === "BRANCH_COLLISION" || itemProjection?.blocker === "UNEXPECTED_COMMIT" || itemProjection?.blocker === "PRE_COMMIT_HOOK_FAILED" diff --git a/skills/autopilot/runtime/src/lock.ts b/skills/autopilot/runtime/src/lock.ts index 7694c5a..ea1b2b1 100644 --- a/skills/autopilot/runtime/src/lock.ts +++ b/skills/autopilot/runtime/src/lock.ts @@ -20,6 +20,7 @@ export interface RunLock { relocate(path: string): Promise; controlRequested(runId: string): Promise; stopRequested(runId: string): Promise; + assertOwned(): Promise; release(): Promise; } @@ -172,6 +173,12 @@ export async function acquireRunLock(path: string, resource = "run"): Promise { + const current = await readLockOwner(ownedPath); + if (current?.token !== owner.token) { + throw new AutopilotError("LOCK_HELD", `${resource} lock ownership changed`, { owner: current }); + } + }, async release(): Promise { const current = await readLockOwner(ownedPath); if (current?.token === owner.token) { diff --git a/skills/autopilot/runtime/src/reducer.ts b/skills/autopilot/runtime/src/reducer.ts index 031a1d5..ba784d8 100644 --- a/skills/autopilot/runtime/src/reducer.ts +++ b/skills/autopilot/runtime/src/reducer.ts @@ -36,6 +36,17 @@ export interface AttemptProjection { readonly observedHeadCommit?: string; readonly observedTreeIdentity?: string; readonly budgetConsumed?: boolean; + readonly quarantinedWorktreePath?: string; + readonly adoptedTree?: { + readonly worktreePath: string; + readonly headCommit: string; + readonly treeIdentity: string; + readonly refIdentity: string; + readonly auxiliaryRefIdentity: string; + readonly externalRefIdentity: string; + readonly configurationIdentity: string; + readonly changedPaths: readonly string[]; + }; } export interface VerifiedCheckpoint { @@ -268,13 +279,59 @@ function transitionItem(item: ItemProjection, event: LifecycleEvent): ItemProjec : attempt), }; } + case "EXECUTION_UNKNOWN_ABANDONED": { + const attempt = item.attempts.at(-1); + if (item.state !== "BLOCKED" || item.blocker !== "EXECUTION_STATE_UNKNOWN" + || attempt?.attemptId !== event.attemptId || attempt.leaseEpoch !== event.leaseEpoch) { + throw new AutopilotError("ILLEGAL_TRANSITION", "unknown execution abandonment does not match the current fenced attempt"); + } + const { blocker: _blocker, ...unblocked } = item; + return { + ...unblocked, + state: "READY", + attempts: item.attempts.map((candidate) => candidate.attemptId === event.attemptId + ? { ...candidate, outcome: "stale", budgetConsumed: true, quarantinedWorktreePath: event.quarantineWorktreePath } + : candidate), + }; + } + case "EXECUTION_UNKNOWN_TREE_ADOPTED": { + const attempt = item.attempts.at(-1); + if (item.state !== "BLOCKED" || item.blocker !== "EXECUTION_STATE_UNKNOWN" + || attempt?.attemptId !== event.attemptId || attempt.leaseEpoch !== event.leaseEpoch) { + throw new AutopilotError("ILLEGAL_TRANSITION", "unknown execution adoption does not match the current fenced attempt"); + } + const { blocker: _blocker, ...unblocked } = item; + return { + ...unblocked, + state: "ACTIVE", + attempts: item.attempts.map((candidate) => candidate.attemptId === event.attemptId + ? { + ...candidate, + outcome: "completed", + budgetConsumed: true, + quarantinedWorktreePath: event.worktreePath, + adoptedTree: { + worktreePath: event.worktreePath, + headCommit: event.headCommit, + treeIdentity: event.treeIdentity, + refIdentity: event.refIdentity, + auxiliaryRefIdentity: event.auxiliaryRefIdentity, + externalRefIdentity: event.externalRefIdentity, + configurationIdentity: event.configurationIdentity, + changedPaths: event.changedPaths, + }, + } + : candidate), + }; + } case "ATTEMPT_FINISHED": { if (item.state !== "ACTIVE") { throw new AutopilotError("ILLEGAL_TRANSITION", `ATTEMPT_FINISHED cannot follow ${item.state}`); } const currentAttempt = item.attempts.at(-1); - if (currentAttempt?.attemptId !== event.attemptId) { - throw new AutopilotError("ILLEGAL_TRANSITION", `attempt ${event.attemptId} is stale for item ${item.itemId}`); + if (currentAttempt?.attemptId !== event.attemptId + || currentAttempt.outcome !== undefined || currentAttempt.adoptedTree !== undefined) { + throw new AutopilotError("ILLEGAL_TRANSITION", `attempt ${event.attemptId} is stale or already recovered for item ${item.itemId}`); } return { ...item, @@ -491,7 +548,8 @@ export function reduce(projection: RunProjection, event: LifecycleEvent): RunPro const nextState = assertRunTransition(projection.state, event); let items = projection.items; const ordinaryItemLifecycle = [ - "DECISION_RECORDED", "ITEM_READY", "ATTEMPT_STARTED", "ATTEMPT_EXECUTION_ADMITTED", "ATTEMPT_FINISHED", "ITEM_VERIFYING", "ATTEMPT_PAUSED", + "DECISION_RECORDED", "ITEM_READY", "ATTEMPT_STARTED", "ATTEMPT_EXECUTION_ADMITTED", "EXECUTION_UNKNOWN_ABANDONED", + "EXECUTION_UNKNOWN_TREE_ADOPTED", "ATTEMPT_FINISHED", "ITEM_VERIFYING", "ATTEMPT_PAUSED", "ITEM_VERIFIED", "ITEM_SATISFIED", "ITEM_BLOCKED", "ITEM_ABANDONED", ]; if (event.itemId !== undefined && projection.restacks[event.itemId] !== undefined diff --git a/skills/autopilot/runtime/src/report.ts b/skills/autopilot/runtime/src/report.ts index 37af2d4..7a418df 100644 --- a/skills/autopilot/runtime/src/report.ts +++ b/skills/autopilot/runtime/src/report.ts @@ -22,6 +22,10 @@ export interface RunReport { readonly blocker?: string; readonly attempts: number; readonly chargedAttempts: number; + readonly recovery?: { + readonly quarantinedWorktreePath: string; + readonly adoptedTreeIdentity?: string; + }; readonly execution?: { readonly assurance?: ExecutionAssurance; readonly adapterName?: string; @@ -129,7 +133,7 @@ export async function writeReports( : projection.waiting?.kind === "operator-pause" ? "/autopilot resume" : projection.waiting?.kind === "execution-unknown" - ? "Do not launch a replacement; prove or externally cancel the orphaned execution first." + ? "Use the fenced recover command to abandon, adopt the exact tree, or stop; do not launch a replacement." : projection.waiting?.kind === "provider-checks" ? "Wait for the bounded provider-check session, or /autopilot resume after the coordinator exits." : projection.state === "RUNNING" || projection.state === "RECONCILING" || projection.state === "VERIFYING" @@ -188,6 +192,12 @@ export async function writeReports( chargedAttempts: consumedAttempts(itemProjection), ...(itemProjection?.subject === undefined ? {} : { subject: itemProjection.subject }), ...(itemProjection?.blocker === undefined ? {} : { blocker: itemProjection.blocker }), + ...(lastAttempt?.quarantinedWorktreePath === undefined ? {} : { + recovery: { + quarantinedWorktreePath: lastAttempt.quarantinedWorktreePath, + ...(lastAttempt.adoptedTree === undefined ? {} : { adoptedTreeIdentity: lastAttempt.adoptedTree.treeIdentity }), + }, + }), ...(lastAttempt?.executionAssurance === undefined && execution === undefined ? {} : { execution: { ...(lastAttempt?.executionAssurance === undefined ? {} : { assurance: lastAttempt.executionAssurance }), diff --git a/skills/autopilot/runtime/src/repository.ts b/skills/autopilot/runtime/src/repository.ts index 7cad3d4..72f4589 100644 --- a/skills/autopilot/runtime/src/repository.ts +++ b/skills/autopilot/runtime/src/repository.ts @@ -147,6 +147,41 @@ export async function assertRegisteredWorktree(repositoryRoot: string, worktreeP } } +export async function quarantineWorktree( + repositoryRoot: string, + worktreePath: string, + runId: string, + itemId: string, + attemptId: string, +): Promise<{ readonly path: string; readonly observation: RepositoryObservation }> { + const quarantinePath = `${worktreePath}.quarantine-${sha256(`${runId}\0${itemId}\0${attemptId}`).slice(0, 16)}`; + let activePath = worktreePath; + try { + await lstat(worktreePath); + await assertRegisteredWorktree(repositoryRoot, worktreePath); + try { + await lstat(quarantinePath); + throw new AutopilotError("BRANCH_COLLISION", `quarantine destination already exists: ${quarantinePath}`); + } catch (error) { + if (!(error instanceof Error && "code" in error && error.code === "ENOENT")) { + throw error; + } + } + await runChecked({ executable: "git", arguments: ["worktree", "move", worktreePath, quarantinePath], cwd: repositoryRoot }); + activePath = quarantinePath; + } catch (error) { + if (!(error instanceof Error && "code" in error && error.code === "ENOENT")) { + throw error; + } + await assertRegisteredWorktree(repositoryRoot, quarantinePath); + activePath = quarantinePath; + } + if (await currentBranch(activePath) !== "") { + await runChecked({ executable: "git", arguments: ["switch", "--detach", "--quiet", "HEAD"], cwd: activePath }); + } + return { path: activePath, observation: await observeRepository(activePath) }; +} + export async function ensureWorktree( charter: RunCharter, item: WorkItem, diff --git a/skills/autopilot/runtime/test/cli.test.ts b/skills/autopilot/runtime/test/cli.test.ts index e1612a8..b45b50f 100644 --- a/skills/autopilot/runtime/test/cli.test.ts +++ b/skills/autopilot/runtime/test/cli.test.ts @@ -35,6 +35,7 @@ test("compiled CLI exposes help and version without side effects", async () => { assert.equal(help.exitCode, 0); assert.match(help.stdout, /autopilot.*start /s); assert.match(help.stdout, /status \[run-id\]/); + assert.match(help.stdout, /recover --action /); assert.equal(version.stdout.trim(), "0.1.0"); }); @@ -340,6 +341,13 @@ if (process.argv[2] === "--version") { assert.equal(journal.records.some(({ event }) => event.type === "RUN_SUCCEEDED"), false); }); +test("compiled CLI requires the complete fenced recovery identity", async () => { + const result = await runProcess({ executable: process.execPath, arguments: [cliPath, "recover"], cwd: process.cwd() }); + + assert.equal(result.exitCode, 1); + assert.match(result.stderr, /recover requires run ID, action, item, attempt, lease epoch, and attestation/); +}); + test("compiled CLI reports a stable error for an unknown command", async () => { const result = await runProcess({ executable: process.execPath, arguments: [cliPath, "unknown"], cwd: process.cwd() }); diff --git a/skills/autopilot/runtime/test/execution-recovery.test.ts b/skills/autopilot/runtime/test/execution-recovery.test.ts new file mode 100644 index 0000000..e6c51f9 --- /dev/null +++ b/skills/autopilot/runtime/test/execution-recovery.test.ts @@ -0,0 +1,403 @@ +import assert from "node:assert/strict"; +import { access, mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { test } from "node:test"; +import type { CancelResult, CapabilityManifest, ExecutionHandle, ExecutionObservation, ExecutionRequest, HarnessPort } from "../src/adapter-protocol.js"; +import { sealCharter } from "../src/charter.js"; +import { AutopilotEngine } from "../src/engine.js"; +import { recoverUnknownExecution } from "../src/execution-recovery.js"; +import { AutopilotError } from "../src/errors.js"; +import { newEventId, type LifecycleEvent } from "../src/events.js"; +import { appendEvent, readJournal } from "../src/journal.js"; +import { acquireWriterLease } from "../src/leases.js"; +import { acquireRunLock } from "../src/lock.js"; +import { rebuildProjection } from "../src/projection.js"; +import { reduce } from "../src/reducer.js"; +import { ensureWorktree, observeRepository, quarantineWorktree } from "../src/repository.js"; +import { createRepository, proposedCharter } from "./helpers.js"; + +class NoLaunchAdapter implements HarnessPort { + launches = 0; + + async describe(): Promise { + return { + protocolVersion: 1, + adapterName: "no-launch", + adapterVersion: "1", + harnessVersion: "1", + families: ["files.read", "files.write", "process.execute", "network.access", "credentials.use"], + assurance: "cooperative", + unattended: true, + maxConcurrency: 1, + eventStreaming: false, + cancellation: true, + restartReattachment: false, + restrictions: "cooperative", + limitations: [], + }; + } + + async launch(_request: ExecutionRequest): Promise { + this.launches += 1; + throw new Error("adopted verification must not launch an implementation worker"); + } + + async observe(_handle: ExecutionHandle): Promise { + throw new Error("no execution exists"); + } + + async cancel(_handle: ExecutionHandle): Promise { + return { protocolVersion: 1, accepted: false }; + } +} + +class FailingReviewAdapter extends NoLaunchAdapter { + override async launch(request: ExecutionRequest): Promise { + this.launches += 1; + if (request.role !== "review") { + throw new Error("adopted verification must not launch an implementation worker"); + } + throw new AutopilotError("ADAPTER_UNSUPPORTED", "review execution failed deterministically"); + } +} + +function event(reason: string) { + return { eventId: newEventId(), timestamp: new Date().toISOString(), source: "runtime" as const, reason }; +} + +async function unknownRun(runId: string, withReview = false) { + const repository = await createRepository(); + const proposed = proposedCharter(repository.root, repository.baseCommit, "single", runId); + const proposedItem = proposed.work[0]; + assert.ok(proposedItem !== undefined); + const charter = sealCharter(withReview ? { + ...proposed, + work: [{ + ...proposedItem, + acceptance: [...proposedItem.acceptance, { type: "gate-passed", gateId: "independent-review" }], + }], + gates: [...proposed.gates, { + id: "independent-review", + type: "review", + focus: "Correctness defects", + appliesTo: [proposedItem.id], + }], + } : proposed); + const item = charter.work[0]; + assert.ok(item !== undefined); + const stateRoot = await mkdtemp(join(tmpdir(), "autopilot-recovery-")); + const runDirectory = join(stateRoot, "runs", runId); + await mkdir(runDirectory, { recursive: true }); + const worktreePath = await ensureWorktree(charter, item); + await writeFile(join(worktreePath, "result.txt"), "done\n"); + const attemptId = "attempt-unknown"; + const lease = await acquireWriterLease(runDirectory, item.id, item.branchName, worktreePath, attemptId, 30_000); + const journalPath = join(runDirectory, "events.jsonl"); + const events: LifecycleEvent[] = [ + { ...event("compiled"), type: "CHARTER_COMPILED" }, + { ...event("reconcile"), type: "RECONCILIATION_STARTED" }, + { ...event("running"), type: "RECONCILIATION_COMPLETED" }, + { ...event("ready"), type: "ITEM_READY", itemId: item.id }, + { + ...event("attempt"), type: "ATTEMPT_STARTED", itemId: item.id, attemptId, leaseEpoch: lease.epoch, + expectedBaseCommit: repository.baseCommit, deadline: lease.expiresAt, idempotencyKey: "attempt:unknown", + }, + { ...event("unknown"), type: "ITEM_BLOCKED", itemId: item.id, attemptId, errorCode: "EXECUTION_STATE_UNKNOWN" }, + { + ...event("waiting"), type: "RUN_WAITING", itemId: item.id, + waiting: { kind: "execution-unknown", itemId: item.id, attemptId }, + }, + ]; + for (const lifecycleEvent of events) { + await appendEvent(journalPath, lifecycleEvent); + } + const journal = await readJournal(journalPath); + const projection = rebuildProjection(charter, journal.records); + const lock = await acquireRunLock(join(runDirectory, "run.lock")); + return { repository, charter, item, runDirectory, worktreePath, attemptId, lease, projection, lock }; +} + +test("abandon recovery fences the exact attempt and permanently quarantines its worktree", async () => { + const fixture = await unknownRun("run-recovery-abandon"); + try { + const recovered = await recoverUnknownExecution( + fixture.runDirectory, + fixture.charter, + fixture.projection, + fixture.lock, + { + action: "abandon", + itemId: fixture.item.id, + attemptId: fixture.attemptId, + leaseEpoch: fixture.lease.epoch, + attestation: "I stopped and accounted for the old harness execution.", + }, + ); + const attempt = recovered.items[fixture.item.id]?.attempts.at(-1); + const journal = await readJournal(join(fixture.runDirectory, "events.jsonl")); + const recovery = journal.records.at(-1)?.event; + + assert.equal(recovered.items[fixture.item.id]?.state, "READY"); + assert.equal(attempt?.outcome, "stale"); + assert.match(attempt?.quarantinedWorktreePath ?? "", /\.quarantine-/); + await access(attempt?.quarantinedWorktreePath ?? "missing"); + assert.equal(recovery?.type, "EXECUTION_UNKNOWN_ABANDONED"); + assert.equal(recovery?.type === "EXECUTION_UNKNOWN_ABANDONED" ? recovery.lockTokenHash.length : 0, 64); + assert.throws(() => reduce(recovered, { + ...event("late terminal result"), type: "ATTEMPT_FINISHED", itemId: fixture.item.id, + attemptId: fixture.attemptId, observedHeadCommit: fixture.repository.baseCommit, outcome: "completed", + }), /cannot follow READY/); + await assert.rejects(recoverUnknownExecution( + fixture.runDirectory, + fixture.charter, + recovered, + fixture.lock, + { + action: "abandon", + itemId: fixture.item.id, + attemptId: fixture.attemptId, + leaseEpoch: fixture.lease.epoch, + attestation: "stale duplicate", + }, + ), /does not match/); + } finally { + await fixture.lock.release(); + } +}); + +test("abandon recovery quarantines unauthorized changes instead of adopting them", async () => { + const fixture = await unknownRun("run-recovery-unauthorized"); + try { + await writeFile(join(fixture.worktreePath, "outside-authority.txt"), "preserve as evidence\n"); + const recovered = await recoverUnknownExecution( + fixture.runDirectory, + fixture.charter, + fixture.projection, + fixture.lock, + { + action: "abandon", + itemId: fixture.item.id, + attemptId: fixture.attemptId, + leaseEpoch: fixture.lease.epoch, + attestation: "I stopped the old execution and require suspect files to be preserved.", + }, + ); + const quarantinePath = recovered.items[fixture.item.id]?.attempts.at(-1)?.quarantinedWorktreePath; + + assert.ok(quarantinePath !== undefined); + await access(join(quarantinePath, "outside-authority.txt")); + } finally { + await fixture.lock.release(); + } +}); + +test("abandon recovery reconciles a crash after the worktree move", async () => { + const fixture = await unknownRun("run-recovery-crash"); + try { + const moved = await quarantineWorktree( + fixture.repository.root, + fixture.worktreePath, + fixture.charter.runId, + fixture.item.id, + fixture.attemptId, + ); + const recovered = await recoverUnknownExecution( + fixture.runDirectory, + fixture.charter, + fixture.projection, + fixture.lock, + { + action: "abandon", + itemId: fixture.item.id, + attemptId: fixture.attemptId, + leaseEpoch: fixture.lease.epoch, + attestation: "I stopped and accounted for the old harness execution.", + }, + ); + + assert.equal(recovered.items[fixture.item.id]?.attempts.at(-1)?.quarantinedWorktreePath, moved.path); + } finally { + await fixture.lock.release(); + } +}); + +test("adopt recovery seals the operator-confirmed exact tree without launching a worker", async () => { + const fixture = await unknownRun("run-recovery-adopt"); + try { + const observation = await observeRepository(fixture.worktreePath); + const recovered = await recoverUnknownExecution( + fixture.runDirectory, + fixture.charter, + fixture.projection, + fixture.lock, + { + action: "adopt", + itemId: fixture.item.id, + attemptId: fixture.attemptId, + leaseEpoch: fixture.lease.epoch, + attestation: "I confirmed the old execution is inactive.", + expectedTreeIdentity: observation.treeIdentity, + }, + ); + const attempt = recovered.items[fixture.item.id]?.attempts.at(-1); + + assert.equal(recovered.items[fixture.item.id]?.state, "ACTIVE"); + assert.equal(attempt?.adoptedTree?.treeIdentity, observation.treeIdentity); + assert.equal(attempt?.adoptedTree?.worktreePath, fixture.worktreePath); + assert.equal(attempt?.outcome, "completed"); + assert.throws(() => reduce(recovered, { + ...event("late terminal after adoption"), type: "ATTEMPT_FINISHED", itemId: fixture.item.id, + attemptId: fixture.attemptId, observedHeadCommit: fixture.repository.baseCommit, outcome: "completed", + }), /already recovered/); + + const journal = await readJournal(join(fixture.runDirectory, "events.jsonl")); + const adapter = new NoLaunchAdapter(); + const engine = new AutopilotEngine({ + stateRoot: join(fixture.runDirectory, "..", ".."), + runDirectory: fixture.runDirectory, + charter: fixture.charter, + adapter, + records: journal.records, + projection: recovered, + }); + const report = await engine.run(); + + assert.equal(report.state, "SUCCEEDED"); + assert.equal(adapter.launches, 0); + } finally { + await fixture.lock.release(); + } +}); + +test("adopted verification preserves an independent review failure code", async () => { + const fixture = await unknownRun("run-recovery-review-failure", true); + try { + const observation = await observeRepository(fixture.worktreePath); + await recoverUnknownExecution( + fixture.runDirectory, + fixture.charter, + fixture.projection, + fixture.lock, + { + action: "adopt", + itemId: fixture.item.id, + attemptId: fixture.attemptId, + leaseEpoch: fixture.lease.epoch, + attestation: "I confirmed the old execution is inactive.", + expectedTreeIdentity: observation.treeIdentity, + }, + ); + const journal = await readJournal(join(fixture.runDirectory, "events.jsonl")); + const adapter = new FailingReviewAdapter(); + const engine = new AutopilotEngine({ + stateRoot: join(fixture.runDirectory, "..", ".."), + runDirectory: fixture.runDirectory, + charter: fixture.charter, + adapter, + records: journal.records, + projection: rebuildProjection(fixture.charter, journal.records), + }); + const report = await engine.run(); + + assert.equal(report.items[0]?.blocker, "ADAPTER_UNSUPPORTED"); + assert.equal(report.waiting?.kind, undefined); + assert.equal(adapter.launches, 1); + } finally { + await fixture.lock.release(); + } +}); + +test("stop recovery terminalizes the exact unknown attempt and preserves its fence evidence", async () => { + const fixture = await unknownRun("run-recovery-stop"); + try { + const recovered = await recoverUnknownExecution( + fixture.runDirectory, + fixture.charter, + fixture.projection, + fixture.lock, + { + action: "stop", + itemId: fixture.item.id, + attemptId: fixture.attemptId, + leaseEpoch: fixture.lease.epoch, + attestation: "Stop this run and preserve the uncertain worktree.", + }, + ); + const journal = await readJournal(join(fixture.runDirectory, "events.jsonl")); + const stopped = journal.records.at(-1)?.event; + + assert.equal(recovered.state, "STOPPED"); + assert.equal(stopped?.type, "RUN_STOPPED"); + assert.equal(stopped?.type === "RUN_STOPPED" ? stopped.lockTokenHash?.length : 0, 64); + assert.equal(stopped?.type === "RUN_STOPPED" ? stopped.leaseEpoch : undefined, fixture.lease.epoch); + } finally { + await fixture.lock.release(); + } +}); + +test("recovery rejects a replaced run-lock token before mutation", async () => { + const fixture = await unknownRun("run-recovery-lock-replaced"); + const lockPath = join(fixture.runDirectory, "run.lock"); + await rm(lockPath, { recursive: true, force: true }); + const replacement = await acquireRunLock(lockPath); + try { + await assert.rejects(recoverUnknownExecution( + fixture.runDirectory, + fixture.charter, + fixture.projection, + fixture.lock, + { + action: "abandon", + itemId: fixture.item.id, + attemptId: fixture.attemptId, + leaseEpoch: fixture.lease.epoch, + attestation: "stale lock must not recover", + }, + ), /lock ownership changed/); + await access(fixture.worktreePath); + const journal = await readJournal(join(fixture.runDirectory, "events.jsonl")); + assert.equal(journal.records.some(({ event: lifecycleEvent }) => + lifecycleEvent.type === "EXECUTION_UNKNOWN_ABANDONED" + ), false); + } finally { + await replacement.release(); + await fixture.lock.release(); + } +}); + +test("recovery rejects a stale lease fence and an unconfirmed adopted tree", async () => { + const fixture = await unknownRun("run-recovery-stale"); + try { + await assert.rejects(recoverUnknownExecution( + fixture.runDirectory, + fixture.charter, + fixture.projection, + fixture.lock, + { + action: "adopt", + itemId: fixture.item.id, + attemptId: fixture.attemptId, + leaseEpoch: fixture.lease.epoch + 1, + attestation: "stale fence", + expectedTreeIdentity: "wrong", + }, + ), /does not match/); + await assert.rejects(recoverUnknownExecution( + fixture.runDirectory, + fixture.charter, + fixture.projection, + fixture.lock, + { + action: "adopt", + itemId: fixture.item.id, + attemptId: fixture.attemptId, + leaseEpoch: fixture.lease.epoch, + attestation: "tree mismatch", + expectedTreeIdentity: "wrong", + }, + ), /does not match --tree/); + } finally { + await fixture.lock.release(); + } +}); diff --git a/skills/autopilot/runtime/test/repository.test.ts b/skills/autopilot/runtime/test/repository.test.ts index fbd41df..f776536 100644 --- a/skills/autopilot/runtime/test/repository.test.ts +++ b/skills/autopilot/runtime/test/repository.test.ts @@ -12,10 +12,31 @@ import { installRestackCandidate, observeRepository, prepareRestackCandidate, + quarantineWorktree, resolveWorktreePath, } from "../src/repository.js"; import { createRepository, proposedCharter } from "./helpers.js"; +test("unknown worktree quarantine is idempotent and frees the sealed branch for a fresh worktree", async () => { + const repository = await createRepository(); + const charter = sealCharter(proposedCharter(repository.root, repository.baseCommit)); + const item = charter.work[0]; + assert.ok(item !== undefined); + const worktreePath = await ensureWorktree(charter, item); + await writeFile(join(worktreePath, "result.txt"), "uncertain\n"); + const before = await observeRepository(worktreePath); + + const first = await quarantineWorktree(repository.root, worktreePath, charter.runId, item.id, "attempt-unknown"); + const second = await quarantineWorktree(repository.root, worktreePath, charter.runId, item.id, "attempt-unknown"); + const fresh = await ensureWorktree(charter, item); + + assert.equal(first.path, second.path); + assert.equal(first.observation.treeIdentity, before.treeIdentity); + assert.deepEqual(first.observation.changedPaths, before.changedPaths); + assert.equal(await realpath(fresh), await realpath(worktreePath)); + assert.notEqual(await realpath(first.path), await realpath(fresh)); +}); + test("worktree creation is idempotent for the same run item identity", async () => { const repository = await createRepository(); const charter = sealCharter(proposedCharter(repository.root, repository.baseCommit));