fix(deploy): keep resident drain closed until reconciliation

[justinhelmer]

A resident deploy now leaves the fleet drained when an upload may have landed but the new Worker or reconciliation cannot be verified. Proven preflight refusals still lift the drain.

Why: After #2429, the deploy's finally block could call /undrain after a failed upload command, an old Worker health read, or a failed /reconcile. That could reopen admissions against an unverified image.

Where to look

1. Drain lift gate Tracks whether upload might have occurred and permits the lift only after a proven refusal or successful reconciliation.
2. Ambiguous command result A failed resident command becomes a partial deployment; a refusal with upload evidence cannot count as a safe preflight stop.
3. Failure cases Covers failed and thrown commands, an old Worker build, a missing reconcile answer, and an ambiguous refusal.
4. Release contract Explains the conditional lift and uncertain-upload recovery; item 27 also corrects the live-table description.
5. Executable receipt test budget Allows real shell and jq processes up to 30 seconds under CI load, with a 35-second case limit to surface a hung child.

Feedback wanted: Check that every path after a possible resident upload keeps the drain closed until reconciliation confirms the affected fleet.

Risk: A failed or uncertain upload can leave admissions closed until a verified operator lift or the drain backstop. The deployment reports partial failure and gives the recovery path.

Verified: Red-first drain tests; 201 focused cases and scoped checks passed on #2440 base. After #2445 rebase and timeout update, 44 focused cases, TypeScript and formatting pass. CI/review pending.

Decisions (2)

• Treat command failures as uncertain uploads. The command can fail after the platform accepted the Worker image; its exit code alone cannot prove that no upload landed.
• Lift only after proof. A structured preflight refusal with no upload evidence proves no new image; successful reconciliation proves the new image was accounted for.

Validation (4 criteria)

Criterion	Proof
A failed command, thrown command, old Worker build, or missing reconcile answer leaves an attempted drain closed	src/deploy/residentDrainRun.test.ts::deployStep (resident) drains the fleet::; ::resident deployment readiness::
A proven preflight refusal still lifts the drain and successful reconciliation still allows readiness	src/deploy/residentDrainRun.test.ts::deployStep (resident) drains the fleet::; ::resident deployment readiness::
A containing release reports exact Worker commits, an undrained registry, current image reports, and ordinary admission	[gap] Human-gated production release and ordinary admission receipt after merge
The executable release receipt has enough time for real shell processes on a loaded CI shard	src/deploy/liveReceipt.test.ts::what is live deployment receipt::*; CI at the new head still pending

For agents

Follow-up to merged #2429. Prior F1 was fixed in item 31 and approved at 36b7aba. This head rebases on main after #2440 and #2445. Release PR #2438 showed the receipt shell hit its 10-second child cap under CI load, so this PR raises it to 30 seconds and gives Vitest 35 seconds. No production deploy or admission probe was run.

🤖 Generated with Claude Code

[coreplane-switchboard]

Changes requested: The drain guard is sound, but the touched behavioral spec still promises the opposite cleanup behavior.

Warning

Changes requested · head ff7c1a6 · 1 finding: 1 minor

Severity	Finding	Where
minor	F1 Spec contradiction — release-and-deploy.md item 31: failed or unreconciled uploads no longer always lift the drain	docs/reference/specs/release-and-deploy.md:49

Full review

F1 (minor, high confidence): Item 31’s prose still says the runner posts /undrain in finally whenever /drain was posted, including after a failed step, and says a failed step has deployed nothing. The new guard intentionally leaves a possibly landed drain closed after an uncertain upload or failed reconciliation. Update the prose to match the corrected validation row and behavior.

[coreplane-switchboard]

LGTM: F1 is resolved at the exact head; no remaining review findings (two CI checks still running).

Note

Approved · head 36b7aba · no findings

Full review

F1 is resolved at the requested head: item 31, the code, and the PR description agree that /undrain follows a proven preflight refusal or successful reconciliation, while an uncertain upload leaves the fleet drained. Current-head CI has no failures; two checks were still running at the last read.

[github-actions]

Auto-approved: coreplane-switchboard[bot] reviewed this PR and posted an LGTM verdict (see its review). This repository opted in through its REVIEW_BOT_LOGIN and REVIEW_BOT_ID variables.

[coreplane-switchboard]

Changes requested: Code and description remain sound after the rebase, but exact-head bot CI fails on test shard 2 of 4.

Warning

Changes requested · head 643b746 · 1 finding: 1 minor

Severity	Finding	Where
minor	F1 Exact-head bot test shard 2 of 4 fails	.depot/workflows/ci.yml:98

Full review

F1 (minor, high confidence): The bot test shard 2 of 4 and its parent bot check fail at 8e3d216. The Depot log requires sign-in, so I cannot attribute the failure to this PR. Diagnose it and get an exact-head green run before merge; the rebased code and description otherwise preserve the previously approved change.

Reviewed at 8e3d216; the head moved to 643b746 during the review — a rebase of the same 1 commit — so this review is posted against 643b746.

[coreplane-switchboard]

LGTM: Prior F1 is resolved; code and description agree, and exact-head CI is green (28 passed, 2 skipped).

Note

Approved · head 643b746 · no findings

Full review

F1 is resolved at 643b746: the drain spec, code, tests, and PR description agree on when the fleet may reopen. Exact-head CI has 28 successful and 2 skipped checks, with no failures.

[github-actions]

Auto-approved: coreplane-switchboard[bot] reviewed this PR and posted an LGTM verdict (see its review). This repository opted in through its REVIEW_BOT_LOGIN and REVIEW_BOT_ID variables.