Skip to content

fix(recovery): retry infrastructure failures on the retained candidate without product repair #243

Open

Description

@ebarti

Implementation progress - 1 November 2026

Partial implementation; acceptance remains open. Merged #248, #261, #263 and #265 provide existing merge, registry, readiness and custody foundations. Merged #276 (merge commit 55dd68398a60594d7577915d8e943fa4402d0678) adds a bounded same-candidate readiness retry that preserves the original attempt, selectors, ownership, effects and product allowance. Its two retries use 30- and 60-second backoff from authenticated predecessor closure. It also distinguishes supported early feature-review preparation failures from genuine findings using retained identity and custody evidence.

Verification: Public readiness, worker-handoff, gate and lifecycle fixtures exercised the integrated route, including local Temporal and controlled-clock zero-product-debit checks. These are bounded fixtures; they do not qualify native readiness retry or every infrastructure failure class.

Configured hosted qualification on the reviewed pre-merge head/base authenticated Check 41338611024, all five jobs, CodeQL, ZIP digests and four complete disjoint reports/JUnit: 1,442 runtime passes and 188 source passes. Four existing runtime and three existing source opt-in/capability bodies remain skipped and unexecuted.

Remaining: Native retry requires #242 descendant settlement. Feature reservation/GO and debit reconciliation, generic early-prelaunch resource observation, other infrastructure classifications/backoff, and public controller-transport and real product-repair/independent-verification coverage remain open. This progress report does not close the issue; the required behavior and acceptance below remain authoritative.
Earlier text below describes the historical #238 candidate. PR #238 and foundation stack #253 have merged; statements calling them open are historical.

Problem and evidence

Environment/prelaunch failures and lost check monitoring can route a retained candidate into a product repair path, or end with “recorded gate failure has no repair findings.” The run then spends time recovering orchestration instead of executing its still-required validation.

Devflow already has some infrastructure classifications and preparation retry paths (#150, #187, #226 and #230). Extend and unify their coverage; do not claim those features are absent or replace authentic product findings with infrastructure labels.

Required behavior

  • Emit a structured failure class and provenance at the failing layer: environment/prelaunch, transport/controller, product assertion/review finding, cancelled, or unknown.
  • When source is unchanged and authorized environment remediation is verified, retry the required preparation/check/assessment stage on that candidate without a fresh implementation role, publication, plan or product repair debit.
  • Preserve original unsuccessful/unknown attempts, exact candidate and test selectors, cumulative product allowance and pending legitimate findings.
  • Use bounded infrastructure retry accounting and backoff separately from product repair cycles. Repeated deterministic failure must stop with an actionable blocker.
  • Reconcile process and external-effect uncertainty before retry. Unknown results never become passed merely because resources are now gone.
  • A real product failure still follows the normal product repair and independent verification path.

Acceptance

Through public controls, cover missing dependencies, unavailable browser/runtime binding, failure before native launch, a lost monitor with independently observed teardown, transient controller transport and an actual failing product assertion. Prove which cases reuse unchanged implementation, which must stop and which require product repair.

Measure launches and budget/effect identities, and inspect original and retried artifacts. Passing a narrower control must not replace the mandatory full gate.

Existing implementation to reuse

PR #238 is open at bfc918a545ce820f47c814e99a6fe5e92b1c3fc1, matching the inspected installed runtime. Reuse its prerequisite-only gate retry and frozen-worker diagnostic. This issue closes failure-classification and public-route coverage while preserving authentic failed/unknown outcomes and product accounting.

Dependencies

Blocked by:

These are required contracts for this issue's acceptance. Related work without a blocking relationship can proceed in parallel.

Blocks:

Activity

  1. ebarti commented on Oct 26, 2026

    @ebarti
    OwnerAuthor

    Reassessment finding, 2026-10-26 — P2: authentic early prelaunch failures remain incompatible with feature-revision recovery and completed-resource observation. Reusing #243's infrastructure/prelaunch recovery scope, related #240/#246. This is an additional consumer of the producer shape missed by the first audit, beyond the ordinary-merge path under PR #248.

    Paired public recovery sequence. Run actual DeliverySupervisor.run for the original revision review with a controlled preparation-verifier refusal. The real supervisor returns blocked/prelaunch/cleanup confirmed, with no session or usage and a genuinely empty attempt directory. Capacity/provider tripwires do not fire. No request file is deleted to manufacture the shape.

    Using the fixture's declared sealed stopped-coordinator closure, CSRF-authenticated POST /api/runs/{id}/revise-feature-plan returns 200 and one successor. Derived preparation and begin_revision succeed. Real record_proposal rejects the same valid correction:

    nonprogressing repeated proposal is already rejected
    

    A paired late-prelaunch control, where the request had legitimately been persisted before a no-provider refusal, accepts the identical correction. Both use normal finite planning accounting (5 → 6); no duplicate-debit or budget-reset claim is made.

    Cause: supervisor.py:628 verifies preparation before request persistence at 675. The revision unreviewed-rejection reader nevertheless opens request.json before its typed prelaunch branch; absence becomes no proof, so the substantive repeated-proposal veto remains. The existing regression prewrites request.json and covers only late refusal.

    A second owning-reader probe confirms the same contract mismatch: actual delivery_role → supervisor early refusal creates no NativeProcess and no request. RunResources.finalize reports confirmed, then observe_completed_resources via delivery_prelaunch_cleanup.py:106 raises FileNotFoundError for request.json. This sibling was not exercised through complete public cleanup recovery.

    Expected / smallest coherent correction: authenticate the original sealed prelaunch intent/job/owner/candidate/revision and proven absent-provider state when refusal predates the later launch envelope. Route it through existing unreviewed-predecessor and completed-resource contracts. A substantive veto must require actual review; no-launch infrastructure failure cannot supply one. Missing files alone must never authorize recovery. Preserve genuine adverse findings, live/unknown actor refusal, immutable historical results and finite budgets; callers must not fabricate request files.

    Acceptance: use the real early producer, public revision continuation and unchanged valid correction; preserve the late-prelaunch control and substantive adverse-review refusal. Exercise restart/response loss and forged/changed custody/candidate/owner negatives. Qualify generic completed-resource observation and the separately source-inspected stricter reconsideration reader against the early shape as distinct cases.

    Seams: real Git/SQLite, trusted-local preparation, supervisor early result, HTTP admission, successor preparation/start and proposal validation. Original proposal/provider/Temporal closure use existing synthetic historical fixtures; the verifier refusal is injected and returned proposal represented by the canonical retained plan. No complete Temporal successor-role/provider or hosted GitHub execution is claimed.

    Open/closed issue and all-state PR searches for prelaunch/proposal rejection/request.json/reconsideration identify #243/#240. PR #137 is merged and supplies the useful same-proposal exception; preserve it. PRs #139/#163/#169 address other safeguards. Frozen open PR #248 changes ordinary merge consumers, not these revision/resource readers. No additional blocking prerequisite.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions