Skip to content

test(runtime): qualify recovery, pause and cleanup through public interfaces #246

Open

Description

@ebarti

Implementation progress — 27 October 2026

Integrated qualification remains incomplete. Reliability stack #253 currently contains 18 open PRs, #248 through #269 in the published dependency chain.

There is no completed integrated scenario matrix or installed-runtime acceptance. The known unfinished #237/#239/#242/#243/#245 contracts remain material, and current-head hosted checks are still being repaired/qualified. Selected public and controlled-provider fixture results remain scoped evidence.

Live inventory: 15 open reliability issues (excluding unrelated #54): nine candidate-led issues still needing acceptance qualification (#235, #236, #240, #241, #244, #247, #249, #250, #251), five with known substantive implementation gaps (#237, #239, #242, #243, #245), and this integrated qualification issue. Eighteen PRs are implementation layers, not eighteen issues or eighteen completed fixes.

Issue Candidate PRs / current boundary
#235 #257 preserves control lineage and publication custody; #260 binds original execution authority; #261 adds registry observation epochs.
#236 #259 adds durable operations, exact receipt reconciliation and API/CLI/MCP/dashboard state.
#237 #264 supplies the durable runtime/offline-recovery foundation; #268 adds retained managed generations. #248 also contains the narrow uv 0.12.17 registration compatibility repair.
#239 #266 journals checkpoint restoration and uncertain custody; #267 verifies intact retained ordinary workspaces and supports historical v1 restoration.
#240 #262 adds shared typed lifecycle assessment and custody checks; #248 repairs the ordinary native merge-preparation recovery consumer.
#241 #263 prepares and verifies the actual execution environment and carries candidate-bound readiness receipts.
#242 #265 preserves unknown custody across the pre-ready launch window.
#243 #248 repairs ordinary merge prelaunch consumers; #261 provides registry epochs; #263 provides readiness; #265 preserves pre-ready unknown custody.
#244 #256 preserves complete CI evidence, duplicate-name ambiguity and explicit execution/readback uncertainty.
#245 #269 adds durable pause/resume only for queued deliveries before workflow start. #265 is a narrow pre-ready custody foundation.
#246 Integrated matrix outstanding
#247 #255 preserves cancellation intent during uncertain responses; #259 composes durable operation/reload handling.
#249 #254 retains service custody when process observation is unknown.
#250 #252 prevents routine item mirroring from replacing the Project Status schema.
#251 #258 refuses fresh native-stack merge submission when no atomic complete-member/head condition is supported.

The referenced candidates are open and unmerged in stack #253. This progress map is non-closing; issue acceptance remains authoritative. Historical text below calling #238 open is superseded: #238 merged as c43cf0b.

Problem

The incident crossed combinations of boundaries that isolated passing tests did not cover: preparation refusal before a plan, retained implementation without publication, cleanup uncertainty, runtime deletion/relocation, long-running control timeouts, tracker conflicts and CI execution outside the PR rollup.

A large passing aggregate does not prove that a user can recover through the installed public interface. Qualify realistic failure sequences on the integrated runtime and verify that the active installation contains the reviewed fixes.

Required qualification

  • Use owned disposable repositories, real Temporal/service/worker processes, real Git operations and the public CLI/API/MCP entry points. Keep fixtures private and independent of real product data and active runs.
  • Inject interruption before and after durable writes/acknowledgements, lost HTTP responses after success, worker/API restart, tracker transport failures, stale owner projections and duplicate commands.
  • Exercise supported pre-plan, unpublished, failed-gate, later-cleanup and uncertain-publication recovery while preserving candidate, evidence, owner and product allowance.
  • Pause at every stage, restart while pausing/paused, and remove only owned disposable chat/check worktrees. Verify explicit resume and retained dirty/unpublished source through the public path.
  • Qualify the installed runtime with the original setup checkout gone; include legacy admitted histories and a failed upgrade/rollback.
  • Include resource-pressure admission and authentic CI inventory/PR eligibility separation. Controlled external integration requires its own explicit authorization and owned resources.
  • Record per-scenario commands, state transitions, candidate/effect/budget identities, actual artifacts and pass/fail/unknown result. A scenario succeeds only when the claimed behavior is observed.
  • Add bounded diagnostic capture and progress on failure. Do not silently skip mandatory scenarios, weaken gates, replace them with mocked snapshots or count repeated tests as additional confidence.

Delivery acceptance

Publish a compact scenario matrix tied to the reviewed source revision and actual installed revision. First reuse and qualify existing repairs; add tests only for meaningful missing boundary coverage. Read back readiness, migration compatibility and public recovery after the authorized installation.

Feature-specific tests belong to each prerequisite issue and can be developed in parallel. This issue is blocked for final integrated qualification until their contracts are delivered; it must not prevent earlier fixes from being reviewed or released independently.

Existing implementation to reuse

PR #238 is open at bfc918a545ce820f47c814e99a6fe5e92b1c3fc1, matching the inspected installed runtime. The PR carries 1,137 distinct selected passing cases and explicit external-service seams, but does not claim every downstream delivery has completed. Reuse that evidence and add only the missing real public-boundary combinations, including the historical cleanup/CI-only recovery case.

Dependencies

Blocked by:

These are required contracts for this issue's acceptance. Related work without a blocking relationship can proceed in parallel.

Activity

  1. ebarti commented on Oct 26, 2026

    @ebarti
    OwnerAuthor

    Independent audit readback and qualification additions, 2026-10-26.

    Current state verified, superseding the older “PR #238 is open” references: #238 merged at c43cf0bec931f4651fdeefdfc82a9e7d523c36b2. Public main and this checkout match it. The running service reports clean bfc918a545ce820f47c814e99a6fe5e92b1c3fc1; both commits have tree 7bc096ef3c1ce21bd6f24156c54e748a0a3206af. Installed Python/editable API imports resolve inside the permanent installation, and the installed CLI identifies API/worker/Temporal as running. This proves source/registration/readiness observations, not every recovery scenario's effectiveness.

    All existing reliability issues #235–237 and #239–246 were individually read and remain open. Their useful already-landed safeguards were inspected before probing additional combinations. The absence of a general public pause/resume operation remains tracked by #245; it was not treated as an implemented feature. #244's known CI-only historical-cleanup follow-up was not relabeled fixed merely because #238 merged.

    The independent audit produced four confirmed gaps:

    Scenario Observation Owning issue
    Slow aggregate publication, production 180-second activity timeout, retry while thread lives Two real owned mutation subprocesses overlap for 53.637 seconds; one effect row becomes complete while the second subprocess remains running #235 evidence
    Public timer disable, transient unpublished stop, authentic closure/finalization, automatic successor The successor retains the old deadline but loses the accepted disable receipt and immediately cancels before preparation #235 evidence
    Ordinary installer from disposable setup checkout, then source removal Installer returns 0; service skill/helper/reference registrations become unreadable #237 evidence
    Genuine stale cancellation revision, refresh, retry with unchanged reason UI retains rejected ID but changes payload revision: 409 stale followed by 409 digest conflict; a fresh-ID control succeeds #247

    Each owning issue/comment records severity, concrete trigger, code, observed/expected behavior, reproduction seams, smallest owning-layer correction and acceptance. Existing scopes were reused; #247 is a distinct frontend request-lifetime defect. It needs no backend redesign prerequisite, and no new blocking edge is asserted. Existing blocking dependencies remain intact.

    Useful positive control: a real Temporal cancellation accepted before local receipt-write failure was recovered after workflow closure and API reconstruction by replaying the exact original ID/body. It returned 200 with one completed mutation. Preserve that working backend idempotency; neither stale refused cancellation was accepted.

    Add these combinations to this issue's qualification matrix. Prove at most one live publication mutator, authenticated timer-control lineage across automatic successors, setup-checkout-independent registration/readiness, and rendered cancellation retry/unknown-outcome request preservation.

    Coverage limits: only owned disposable fixture Git/state/Temporal resources were mutated. Publication used a minimal workflow entering production scheduler/activity/broker and local gh/Git latency adapters; no actual GitHub PR duplication is claimed. Timer and cancellation used public authenticated HTTP controls and real workflow execution, with declared provider/tracker/base-observation seams. Cancellation's browser rendering was not run; exact source lifecycle plus public API replay established the protocol failure. Installation reproduced registration loss, not a full restart failure of the migrated live installation. Live provider/CI, full pause/resume, external worktree rehydration, host reboot/power loss, failed upgrade/rollback and host resource-pressure admission remain unqualified by this audit.

    No production code, installed runtime, settings or active deliveries were changed. No fixes or installations were requested by this audit.

  2. ebarti commented on Oct 26, 2026

    @ebarti
    OwnerAuthor

    Independent reassessment, 2026-10-26: the earlier audit was materially incomplete. The ordinary merge-resolver failure now under open PR #248 was not one of its four reproduced findings. The reassessment explicitly mapped separate launch/preparation/effect/recovery families instead of using service health as proof of those paths.

    Frozen state: audited main c43cf0bec931f4651fdeefdfc82a9e7d523c36b2; installed clean bfc918a545ce820f47c814e99a6fe5e92b1c3fc1, with identical tracked trees. PR #238 is merged. PR #248 was independently compared at 1ebbc23338d8a22cc4d051ef023451e7846d84b5, still open/uninstalled; no claim that its ordinary-merge repair fixes the sibling consumers below.

    Eight additional confirmed gaps

    Severity Failure sequence Owning evidence
    P1 Primary process identity observation fails; public stop clears custody of a still-live recorded member #249
    P1 Same-name required CI failure/running entry is overwritten by a later success, so ordinary observer returns passed #244
    P1 Missing canonical Status makes routine item mirroring send a stale whole-schema replacement omitting a foreign concurrent option #250
    P1 Feature stack submission omits the qualified selected-head condition; drift is checked only after execution #251
    P2 Expired product timer cancels terminal-only tracker reconciliation before preflight, losing its frozen intended outcome #235
    P2 Historical same-iteration completion makes a newly failed public local gate appear completed #235
    P2 One transient diagnostic metadata read failure clears its only scheduling hint and leaves accepted work queued until manual exact ACK/restart #236
    P2 Genuine early prelaunch empty-request shape breaks revision same-proposal recovery and completed-resource observation #243

    Each owning issue has trigger, code, actual versus expected behavior, reproduction/seams, smallest owning-layer fix and acceptance. All-state overlap searches preceded new issues. New narrow scopes are #249/#250/#251; matching existing scopes were reused for the rest.

    Additional blocking dependencies for final integrated qualification

    These are prerequisites for this issue's final expanded integration acceptance, not prerequisites between the three independent repairs and not reasons to prevent earlier fixes from shipping. Existing dependencies on #242–245 remain. Reused #235/#236/#243/#244 already feed this issue's state/recovery contracts; no artificial ordering between their bounded corrections is introduced.

    Coverage and limits

    The structural inventory maps 24 runtime families and 13 installer/helper families, with public roots, preparation owner, durable intent, executor, settlement and recovery. Those counts are an index, not passing tests or a completeness claim. Per-family evidence distinguishes actual fixture execution, controlled transport/history, source tracing and untested stages.

    Real fixture boundaries included public lifecycle CLI over an owned process; public tracker recovery with authentic closed Temporal history; actual required-check subprocess failure and public detail projection; production CI activity through Temporal; actual supervisor early refusal and public revision admission; public diagnostic ACK/SQLite queue; selected feature replay/fencing controls; and the registered disposable CLI's duplicate/conflicting start and cancellation at a decision boundary.

    Project-schema and feature-stack findings combine production adapter/control flow with documented or pinned primary-source remote semantics. No hosted Project loss or unapproved GitHub merge was observed. Stack optional sha protects the selected head; the whole lower-member vector needs its own qualification. Diagnostic scheduling uses a no-effect tripwire, not an executed diagnostic body. Revision history/closure and phase-gate recovery admission retain explicitly controlled seams.

    Still unqualified: complete parent/child late native completion across restart, every full native/provider chain, host reboot/power loss, crash-at-GO capture, exact detached-spawn/manifest interruption, full three-process cold restart, empty/offline dependency/browser matrix, external worktree rehydration, full upgrade/rollback, live hosted CI and real remote merge/schema races. These cannot be substituted with aggregate case counts.

    Useful exclusions: failed callback no-resend matches its documented conservative contract; a permission-failed admitted revision replays with one debit; legacy direct-helper work locks do fence its canonical handoff; missing/corrupt custody correctly refuses rather than fabricating recovery. The existing strict merge CI duplicate guard and exact-ID cancellation replay remain useful safeguards.

    Production source, installed runtime, configuration and active deliveries received no audit mutations. Owned processes were settled and fixtures/evidence retained.

    Final remote readback: PR #248 advanced to bfad5a1bbee2ec1a2bb39199f90d2dcc17f64adc and remains open/unmerged/uninstalled. I fetched that head into the disposable audit clone; every source file owning the eight reassessment findings is unchanged from audited main. Its changed files remain confined to ordinary merge preparation/follow-up/resolution and their docs/tests. This is a changed-file comparison, not an independent full verification of the latest repair candidate.

  3. ebarti commented on Oct 26, 2026

    @ebarti
    OwnerAuthor

    Current qualification boundary: stack #253 has ten published, unmerged PRs through #261. Focused real SDK/native/public tests support individual layers, but the combined reliability matrix is not passed. All fifteen reliability issues remain open.

    Hosted runtime checks at the current #248 and #259 heads are failing. Work is removing the private historical fixture tests as directly requested, provisioning retained public fixture dependencies correctly, correcting operation discovery/read-only contract expectations, and investigating a public browser-amendment workflow timeout without declaring it solved or merely raising the timeout.

    End-to-end pause/restart/cleanup qualification is blocked by the confirmed resource-settlement gap in #242 and the unfinished #239/#243/#245 composition. Final evidence must identify one exact integrated SHA and distinguish actual public execution from controlled provider, remote GitHub, history or custody facts. A source review or test count is not a completed scenario. Active installation/upgrade remains outside the currently authorized unmerged-PR delivery endpoint.

  4. ebarti commented on Oct 27, 2026

    @ebarti
    OwnerAuthor

    Qualification checkpoint on exact stack candidate 501b320d0120334556bf0f2436abb046dadb5d27: owned non-native public tests exercised feature preflight/execute agreement, stale revision/owner/config refusal, an unknown closure with unchanged database and budget, takeover during observation, bounded operation deduplication, and readiness refusal before product debit.

    A separate cross-process probe submitted the same command twice over the actual API HTTP interface and observed one run; actual candidate CLI and MCP stdio processes then independently read that run over HTTP. Temporal was intentionally unavailable in that probe, so it proves admission and readback, not worker execution. Provider and hosted GitHub effects were controlled where recorded; the run used the shared Python3.13.11/Temporal SDK1.33.0 dependencies with candidate import paths verified, not a separate fully frozen environment.

    The earlier fe71a3a evidence is retained but does not qualify later layers. #239 checkpoint composition, #242 native descendant settlement, #243 public retry, #245 full pause/resume, actual native worker/restart combinations and hosted CI remain required. This issue stays open; no installed runtime or active delivery was modified.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions