feat(ship): recover original units without replacement work

[justinhelmer]

Switchboard can resume a proven terminal ship unit without inventing a replacement plan, budget, branch, or owner. It preserves the original durable identity while a separate Workflow transports the bounded recovery.

Why: #2265 exposed that reissuing a checkpoint silently creates nearby work instead of continuing the retained unit. The accepted owner record requires one owner and clock; this targets main independently and does not stack on draft #1625.

Where to look

1. Explicit recovery entry recognizes only a named original unit and cannot fall through to generated-plan handoff. ⚠ A malformed request must never mint work.
2. Evidence reconstruction attributes one completed findings child before advancing the publication head. ⚠ Ambiguous or external heads must refuse.
3. Recovery Workflow driver reclaims the original row and validates that the physical checkpoint still owns it.
4. Periodic ownership fence returns a named transient instead of running a rebase while ownership is rebuilding. ⚠ Unknown ownership must stay fail-closed.
5. Active-recovery index filters terminal history in SQLite while retaining malformed candidates as hard failures.
6. #405-shaped regression captures the completed findings push followed by terminal findings/pr-check failure.
7. Design boundary explains why transport may be new while identity, ownership, namespace, and limits remain original.

Feedback wanted: Check the claim/rollback/CAS concurrency boundary and whether periodic ownership rebuilding remains retryable without weakening sole-owner enforcement.

Risk: 3,868 changed lines across recovery admission, runtime wiring, proofs, and docs. Splitting admission from runtime was rejected because either half alone breaks the atomic recovery contract. Roll back this commit to disable the command.

Verified: Current-head bot suites: 819 passed. TypeScript, ESLint, Prettier, specs, coverage, hygiene, vocabulary, user-message, and diff checks passed; Linux CI is green: 28 passed, 1 skipped.

Decisions (6)

• New Workflow is transport only. A terminal Workflow cannot reopen, so a stable recovery-<run> checkpoint drives the walk while every child and durable mutation retains the original instance/unit identity.
• Evidence, never thread prose. Admission reconstructs requester, stage, target, budgets, and publication owner from durable rows, run records, and fresh GitHub facts; missing or ambiguous facts refuse.
• Completed findings are consumed once. One attributable completed child moves directly to read-only re-review even when its changes did not move the head; a moved head additionally requires the exact push receipt.
• Absolute limits survive restart. The durable coordinator tag carries the original deadline and target through resume/restart; checkpoint-local counters are labeled as such and never presented as lifetime history.
• Ownership rebuilds stay closed. Claims and reservations still throw during every complete periodic rebuild; callers translate that boundary to publication_ownership_unknown rather than treating stale state as unowned.
• Terminal means no replacement. Human/draft holds and consumed receipts fence later replies from generic reissue; failed reply persistence notifies the sender instead of claiming retention.

Validation (10 criteria)

Criterion	Proof
request_changes resumes findings and re-review in the original namespace	src/core/coordinator/driver.test.ts::runOriginalUnitRecovery
no_verdict starts one read-only original-unit review	src/core/coordinator/driver.test.ts::runOriginalUnitRecovery::continues no_verdict…
#405-shaped completed findings advance only their proven head	src/channels/adminCoordinator.test.ts::…::reconstructs a failed original findings pr-check…
Stale settlement, lost CAS response, successor ownership, and typed holds fail safely	src/channels/adminCoordinator.test.ts::POST /admin/coordinator/recover-unit — unchanged-head original-unit recovery::*
Rebase during periodic ownership recovery returns publication_ownership_unknown and invokes no resolver	src/channels/adminCoordinator.test.ts::…::the runner's rebase returns the named transient while periodic ownership recovery is in flight
Legacy continuation during the same boundary starts no replacement work	src/core/dispatcher.test.ts::…::a legacy continuation returns publication_ownership_unknown when periodic ownership recovery starts before reservation
Active recovery listing excludes terminal rows and treats malformed candidates as fail-closed	deploy/cloudflare-memory/runLedger.test.ts::…::lists only active recovery rows in SQL, ignores terminal history, and fails closed on malformed candidate JSON — passed in the Linux memory-worker gate
Resume/restart retains transport, deadline, and exact-target authority	src/core/dispatch/reattach.test.ts::carriedCoordinatorTag…; src/core/dispatcher.test.ts::executor provisioning by agent resources::*
Current-head affected bot suites stay green	npx vitest run src/channels/adminCoordinator.test.ts src/core/dispatcher.test.ts src/core/runnerOwnership.test.ts src/core/refusal.test.ts — 819/819 passed
Static/spec gates stay green	Root TypeScript, changed-file ESLint/Prettier, specs:check, specs:coverage --test-guard, hygiene, vocabulary, user-message, and git diff --check passed

For agents

The physical recovery Workflow is not the durable owner: use parentInstanceId + unit for identity and transportWorkflowId only for lifecycle delivery. Recovery reports intentionally describe checkpoint-local elapsed categories because legacy terminal rows do not retain a complete lifetime split. The memory-worker regression and full clean gate passed in Linux CI; no install mutated the shared local dependency state. The production #405 compatibility claim is based on captured durable-shape fixtures and fresh read-only facts; no live recovery was run before merge/deploy.

[coreplane-switchboard]

Changes requested: Recovery flow is carefully fenced, but making RunnerOwnershipFence.claim/reserve throw mid-rebuild breaks existing callers that don't catch it during every periodic sweep.

Warning

Changes requested · head e1f12f7 · 2 findings: 1 minor, 1 nit

Severity	Finding	Where
minor	F1 claim()/reserve() now throw during every sweep's ownership rebuild; existing uncaught callers (rebaseStep, legacy continuation reserve) turn that into 500s / uncaught refusals	src/core/runnerOwnership.ts:74
nit	F2 recoveryTransport queried twice per resumed signal; listActiveRecoveries is an unindexed full-table scan run on every reclaim sweep	deploy/cloudflare-memory/worker.ts:3055

Full review

F1. RunnerOwnershipFence.claim and reserve now throw whenever recoveryComplete is false. Before this PR only the read-side owns/owner threw. The problem is timing: recover() sets recoveryComplete = false on every complete reclaim pass, not just at boot, and keeps it false across the awaits that rebuild ownership. Those passes come from guardCards inside startReclaimSweep, every LEASE_MS. The code this PR adds catches the throw, but at least two existing callers don't:

• rebaseStep (adminCoordinator.ts:3803) calls deps.runnerOwnership?.claim(...) bare. It escapes to the handler's catch-all and returns a generic 500 internal error, which the driver can't recognise as a transient refusal.
• reserveLegacyOwnership (dispatcher.ts:1440) calls reserve outside any try. A legacy continuation that lands mid-sweep becomes an uncaught/system refusal instead of the named "ownership could not be verified" reply.

Rarely, a rebase round or a continuation fails for no reason the user can see. Two possible fixes:

• Limit the throw to the initial boot rebuild.
• Wrap every existing claim/reserve caller the way owner() is wrapped already. Each should answer publication_ownership_unknown, which this PR already added to the driver's TRANSIENT set.

Also add a test for a rebase and a legacy continuation arriving while a sweep's recover() is in flight.

F2. In worker.ts (around line 3055), recoveryTransport runs its lookup twice: once in the condition and again in the spread. Compute it once, the way the finish path does. Separately, listActiveRecoveries scans and parses every coordinator_units row, and the reclaim sweep now calls it on each pass. Filtering in SQL on json_extract(json,'$.recovery') IS NOT NULL would keep that cost bounded as the table grows.

[justinhelmer]

Current-head fix round at a96d9cc59a729dea367a3c20da2e56b3f64dd98d addresses every actionable finding from the prior review. F1: all affected claim/reservation boundaries now fail closed with publication_ownership_unknown; red-first rebase and legacy-continuation regressions prove no resolver or replacement work starts during an in-flight rebuild. F2: resumed-signal transport is computed once, and active recovery listing filters terminal rows in SQLite while retaining malformed candidates as fail-closed errors. Both CodeQL cleanup branches now use the flow-narrowed exact reservation token without changing CAS or cleanup-failure behavior. No re-review requested here.

[justinhelmer]

Re-review requested at a96d9cc — ownership-rebuild claim/reservation failures now return the named retryable refusal, repeated recovery lookups and historical-row scans are reduced, and both redundant CodeQL comparisons are removed while preserving cleanup. Focused regressions and Linux CI pass; all four prior findings have fix dispositions. Please verify the full ownership/CAS/cleanup boundary and original identity/budget guarantees at this exact head.

[coreplane-switchboard]

LGTM: Both prior findings are fixed: every fence claim and reserve call now returns the retryable publication_ownership_unknown refusal during a rebuild, and recovery lookups are indexed and done once.

Note

Approved · head a96d9cc · no findings

Full review

I found no new issues at a96d9cc, and both earlier findings are fixed:

• F1: while the ownership fence is being rebuilt, every call that claims or reserves a pull request now returns the retryable publication_ownership_unknown refusal instead of throwing. That includes rebaseStep (503) and the legacy-continuation reservation, which now gets a named refusal.
• F2: recoveryTransport is looked up once per signal. listActiveRecoveries now filters in SQL on json_type(json,'$.recovery') instead of reading the whole table.

The test guard's only flag is the renamed test in deploy/cloudflare/coordinator.test.ts, and http-ingress.md allows it.

[github-actions]

Auto-approved: coreplane-switchboard[bot] reviewed this PR and posted an LGTM verdict (see its review). This repository opted in through its REVIEW_BOT_LOGIN and REVIEW_BOT_ID variables.