Skip to content

fix(recovery): preserve resumable work independently of disposable worktrees #239

Open

Description

@ebarti

Implementation progress - 1 November 2026

Partial implementation; acceptance remains open. Merged #266 journals restoration and uncertain custody; merged #267 verifies retained workspaces and supports historical restoration. Merged #275 (merge commit 62f6d3877dfa8e22dc23cf78958f77443e6771e0) adds current checkpoint readback checks for changed run/budget, owner, candidate, workspace, index, evidence and session custody. Missing reconstructible files remain eligible for restore; changed or unverified existing state blocks rather than being overwritten. Restore retains its independent admission checks.

Verification: Public checkpoint, retained-state and stopped-recovery scenarios exercised the changes with owned fixtures, including a real local Temporal workflow and controlled provider/remote boundaries. This does not establish real-provider continuation or native writer closure.

Configured hosted qualification on the reviewed pre-merge head/base authenticated Check 41338611603, all five jobs, CodeQL, ZIP digests and four complete disjoint reports/JUnit: 1,456 runtime passes and 188 source passes. Four existing runtime and three existing source opt-in/capability bodies remain skipped and unexecuted.

Remaining: Complete native descendant settlement is blocked by #242. Full public restore/restart/cleanup composition, feature and publication custody, and integrated running pause/resume remain to be qualified. This progress report does not close the issue; the required behavior and acceptance below remain authoritative.
Earlier text below describes the historical #238 candidate. PR #238 and foundation stack #253 have merged; statements calling them open are historical.

Problem and evidence

A persisted run ID is insufficient if its only dirty candidate, unpushed commit, native session or recovery evidence lives in a removable checkout. Existing cleanup preservation and cleaned-checkout recovery in #211 and #231 are useful, but protection inside Devflow's own cleanup does not establish survival of external managed-worktree cleanup.

The installed controller was relocated after an actual cleanup incident. Separately establish what happens to active, blocked and eventually paused delivery work when its associated chat checkout disappears. Do not assume the desktop's Git snapshot includes every ignored file, native session, database or evidence artifact needed to resume.

Required behavior

  • Store authoritative execution state, effect journals, accepted evidence, necessary native session material and recoverable source outside disposable managed chat worktrees.
  • Maintain an atomic, versioned recovery manifest binding original run/issue/owner, authorized plan, candidate and dirty/untracked changes, publication identity, sessions, completed checks, pending effects and cumulative budget.
  • Own delivery workspaces independently for active work; before reporting a blocked or paused checkpoint as resumable, prove its needed contents have durable custody. Do not claim checkpoint preservation while a writer is still changing it.
  • Preserve dirty and unpublished work without automatically committing or publishing it. Persist only the necessary owned artifacts, privately; do not copy unrelated profiles, credentials or foreign files.
  • Restore into a new owned workspace and authenticate the same content/lineage even when old absolute paths or Git worktree metadata are gone. Missing or corrupt evidence produces a precise recoverable refusal, never a fabricated success.
  • Coordinate retention and pruning with ownership and unresolved effects. Expiration must be explicit and visible; active, blocked or paused recoverable work cannot be silently discarded.

Acceptance

Through public recovery, delete only owned disposable fixture checkouts after passed implementation, dirty unpublished progress, and a stopped gate. Rehydrate the original candidate and pending work, including required session/evidence references, without resetting budget or creating a second owner. Test crash during checkpoint creation, corrupted/incomplete snapshots and unknown external effects.

The pause feature must consume this preservation contract. Cleanup remains enabled.

Reference: worktree cleanup and restore.

Existing implementation to reuse

PR #238 is open at bfc918a545ce820f47c814e99a6fe5e92b1c3fc1, matching the inspected installed runtime. Reuse its retained source, native session, resource and closed-history custody. This issue proves and closes remaining survival/rehydration gaps when external cleanup removes a disposable checkout.

Dependencies

Native resumable checkpoint acceptance is blocked by #242's complete descendant-settlement contract. Recorded native identities and monitor completion do not establish complete writer closure. Non-native and independently proven no-launch preservation can proceed in parallel. Integrated qualification is tracked in #246.

Blocks:

Activity

  1. ebarti commented on Oct 27, 2026

    @ebarti
    OwnerAuthor

    Implementation finding, 2026-10-27 — P1: observed native closure is insufficient to acknowledge a resumable checkpoint. This reuses #239 and the confirmed descendant-accounting defect in #242; no duplicate issue is being created.

    Trigger: the unpublished checkpoint candidate's default public create route runs after an actual native role and exact original SDK execution close, while an owned descendant omitted from the monitor's recorded identity set is still alive. The diagnostic candidate ac2d69e observed observed-native-confirmed and monitoring_complete; the CSRF-authenticated POST returned200 with resumable=true, and the same descendant identity remained live after acknowledgement. The final test failed in cleanup with ProcessLookupError after those assertions, so this is evidence of the unsafe acknowledgement, not a wholly passing or fully cleaned qualification. No additional process reproduction is needed to establish the failure.

    Owning code: delivery_checkpoints._closed delegates to RunResources._closed_journals, whose closure check examines only the recorded owned identities, monitor and declared ports. The v2 independent writer lease is useful for its actual holder lifetime; it does not prove completeness of the descendant set. A closed SDK workflow also does not prove that native descendants are gone.

    Expected: unknown native custody must prevent acknowledgement and restore; a historical acknowledgement must remain inspectable while current resume eligibility reports unknown. Do not infer safe settlement from repeated polling, disappeared processes, or a caller-provided marker.

    Smallest containment is implemented in local guard89f13ed atop36069f8: a registered native journal produces typed native_checkpoint_custody_unknown before create/restore effects, while supported non-native and no-launch custody continue. Six focused fixture cases and three final refusal/readback cases passed; no new native reproduction was run. This guard is not yet published or composed into the current stack and does not complete native checkpoint support.

    Acceptance: keep the actual unsafe public sequence as a regression; refuse with no checkpoint acknowledgement or restore effect until a genuine complete custody capability is integrated; preserve original budgets/owner/effect uncertainty; retain non-native/no-launch create, response-loss replay, source-deletion restore and historical readback positives. Qualify the final exact integrated candidate under #246.

    Genuine dependency: native resumable checkpoints in #239, and therefore native pause/resume in #245, are blocked by #242's complete descendant-settlement contract. Non-native preservation work can proceed independently. PR #265 repairs the separate pre-ready interruption sequence, but does not supply this broader post-launch capability. No installed runtime, settings or active delivery was modified.

  2. ebarti commented on Oct 27, 2026

    @ebarti
    OwnerAuthor

    Partial checkpoint foundation is proposed in #266. It journals restoration, preserves read-only settlement, and refuses resumability when effect or current registry evidence is missing. Its focused nonnative checks pass; real Temporal and complete native descendant custody remain unqualified. Native checkpoint admission continues to depend on #242, and this issue remains open.

  3. ebarti commented on Oct 27, 2026

    @ebarti
    OwnerAuthor

    A second partial repair is proposed in #267: public verification of an intact stopped ordinary workspace under existing custody, plus compatibility for restoring and replaying older v1 checkpoints without inventing unrecorded index flags. It does not start work or qualify pause, native descendant, feature, or publication custody. Those dependencies remain open, including #242.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions