Skip to content

feat(runtime): make long-running control operations durable and idempotent #236

Closed

Description

@ebarti

Implementation progress — 27 October 2026

Implementation candidate; acceptance not yet qualified. #259 adds durable operations, exact receipt reconciliation and API/CLI/MCP/dashboard state.

Current-head hosted checks and the full public response-loss/restart acceptance remain to be qualified. Later lifecycle and pause endpoints must compose their own operation kinds without treating unknown effects as completed.

The referenced candidates are open and unmerged in stack #253. This progress map is non-closing; issue acceptance remains authoritative. Historical text below calling #238 open is superseded: #238 merged as c43cf0b.

Problem and evidence

A diagnostic/recovery preflight can exceed its caller's timeout. The caller then cannot distinguish rejection, still running, successful completion or a lost response. Retrying can collide with a one-use acknowledgement, reservation or approval even though the user has already authorized that same operation.

Existing command IDs and typed durable CI inventory commands in the locally installed recovery branch are useful foundations. Generalize the supported contract rather than adding another special retry command.

Required behavior

  • Persist an operation ID and stable request digest before starting long-running work; acknowledge promptly and expose queryable pending/running/succeeded/refused/failed/unknown states with timestamps and progress.
  • Repeating the identical command returns the same operation and result. Reusing its ID with different contents refuses. A client timeout is not cancellation or permission to launch a second operation.
  • Persist result and acknowledgement state across API/worker restart and lost responses. Reconcile uncertain external effects before any retry.
  • Keep observational preflight results, reservations, one-use execution grants and actual execution as distinct recorded stages. Pure observation must not accidentally consume execution authorization.
  • Expose the same behavior in API, CLI, MCP and dashboard, with bounded status reads and an explicit next action.
  • Preserve the existing user's scoped authorization while checking current revision/ownership before effectful execution; changed scope still requires a new decision.

Acceptance

Use real public operations with response loss both before and after durable completion, concurrent duplicate requests, service restart and an expired client timeout. Verify one operation/effect, no duplicate reservation or repair debit, recoverable results and deterministic conflict handling. Include a slow diagnostic/preflight that outlives the client's connection.

Do not implement timeout recovery by automatically reissuing an effectful command with a new ID.

Existing implementation to reuse

PR #238 is open at bfc918a545ce820f47c814e99a6fe5e92b1c3fc1, matching the inspected installed runtime. Reuse its typed diagnostic and CI preflight/acknowledgement operations. This issue adds a general durable operation/status contract so caller timeout does not strand an otherwise authorized operation.

Dependencies

No blocking prerequisite in this issue set. This work can proceed in parallel with the other independent foundations; integrated qualification will verify their composition.

Blocks:

Activity

  1. ebarti commented on Oct 26, 2026

    @ebarti
    OwnerAuthor

    Reassessment finding, 2026-10-26 — P2: a transient diagnostic dispatch read failure consumes its only wake hint and strands accepted queued work. Reusing #236's durable accepted-operation progress scope.

    Public reproduction: owned HTTP preflight and authenticated ACK return 200, state=queued, wake hint true. Inject one local OSError during the first dispatch's effective_spec read, before any body is scheduled. That pass leaves the hint false. Three healthy subsequent scheduler passes perform no further spec read, schedule zero tasks, and public GET still reports queued.

    Working manual control: replaying the exact same ACK returns 200 and restores the hint. The next pump schedules one no-effect tripwire. This is not lost authorization or a claim that manual recovery is unavailable; it is a durable queue becoming invisible to normal pumping after a temporary observation failure.

    Code/cause: delivery_native_diagnostic.py:720–758 clears the optimization hint before reading/validating queued rows. An initial queue-query exception restores it, but a per-row spec/ownership exception simply continues. The next pump returns before reading durable queue membership. Exact ACK replay already reawakens the identity correctly.

    Invariant / fix owner: persisted queue membership owns accepted work; an in-memory performance hint cannot silently remove its scheduling visibility. Retain bounded retry/backoff or equivalent durable reconsideration for unresolved queued rows. Preserve empty-queue idle optimization, ownership validation and refusal to relaunch running/unknown/retired identities.

    Acceptance: public ACK → one metadata/ownership-observation failure → restored observation must schedule exactly one same-identity diagnostic through normal pumping without another ACK/restart. Include first startup pass with queued work, idle empty queue, foreign/current owner changes and unknown-body no-relaunch controls. Exact ACK replay must remain idempotent.

    Seams/limits: historical provider/GitHub/closed Temporal records and actor observations use the existing controlled diagnostic fixture. Public proof readers, HTTP ACK/status, SQLite and dispatcher run unchanged. The transient read failure is injected; final scheduling uses a no-effect tripwire, not a diagnostic/native/provider body. No real gate, external effect or production/runtime/settings/active-delivery mutation ran.

    All-state issue/PR searches diagnostic/queued/wake/dispatch found #236 as owner, no exact repair. PR #157's ordinary-outbox retry does not cover this separate API-owned task scheduler; PR #248 leaves it unchanged. #243/#246 related. No new blocking dependency.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions