Skip to content

fix(runtime): separate immutable delivery authority from operational state #235

Open

Description

@ebarti

Implementation progress — 27 October 2026

Candidate foundations; acceptance not yet qualified. #257 preserves control lineage and publication custody; #260 binds original execution authority; #261 adds registry observation epochs.

Current-head hosted checks and the issue's public acceptance still require verification. Registry epochs are an observation primitive; final native launch/reservation/GO accounting remains separate work in #243.

The referenced candidates are open and unmerged in stack #253. This progress map is non-closing; issue acceptance remains authoritative. Historical text below calling #238 open is superseded: #238 merged as c43cf0b.

Problem and evidence

Retained runs have been refused after runtime relocation, configuration/capacity changes, or later cleanup reconciliation even when the user's scope and retained candidate did not change. A Project update can also be overwritten by a competing projection writer. These are different facts with different owners; comparing or rewriting one composite state makes routine recovery look like a change in execution authority.

Existing repairs in #147, #224, #230 and #231 cover individual cases. The installed recovery work also separates shared installation identity from per-run grants. Preserve and consolidate those contracts rather than replacing them or weakening their checks.

Required behavior

  • Define the authoritative owner and version for immutable authorization (scope, accepted plan, permissions, endpoint and budget), current execution/effect custody, environment binding, and derived tracker/dashboard views.
  • Keep original authorization immutable. Operational changes must use explicit compatible transitions and must never silently broaden paths, grants, endpoint or product repair allowance.
  • Make tracker updates an idempotent projection of the current owner/revision. A stale writer cannot undo an operator hold or newer owner transition. Projection failure remains a projection problem and does not rerun implementation or erase accepted evidence.
  • Preserve historical verdicts when newer observations reconcile cleanup or publication. Record provenance and the newer observation separately.
  • Keep ordinary status reads bounded and available from persisted facts. Expensive proof reconstruction belongs to an explicit operation and must report when evidence is unavailable.
  • Supply a compatible migration for existing admitted histories and multiple retained continuations; do not require a blanket rewrite of historical specifications.

Acceptance

Exercise unchanged-source continuation across capacity change, runtime relocation, stale tracker writes and failed tracker transport through the public boundaries. Verify the original candidate, authorization, budget and historical results remain unchanged; changed authority and conflicting owners still refuse with a specific reason. Include concurrent-writer readback, not only a successful update response.

Document the ownership/version contract and migrate existing supported recovery paths incrementally.

Existing implementation to reuse

PR #238 is open at bfc918a545ce820f47c814e99a6fe5e92b1c3fc1, matching the inspected installed runtime. Reuse its separation of shared installation identity from original per-run scope and its preservation of original custody. This issue owns the consistent authority/projection contract across recovery paths.

Dependencies

No blocking prerequisite in this issue set. This establishes the state contract consumed by lifecycle recovery; independent operation and storage work can proceed in parallel.

Blocks:

Activity

  1. ebarti commented on Oct 26, 2026

    @ebarti
    OwnerAuthor

    Independent audit evidence, 2026-10-26: P2 — automatic unpublished retry loses an accepted timer-disable amendment. This belongs to #235's operational-state lineage contract, so no overlapping issue is opened.

    Verified against merged PR #238 tree at c43cf0bec931f4651fdeefdfc82a9e7d523c36b2, identical to the live clean bfc918a545ce820f47c814e99a6fe5e92b1c3fc1 tree. PR #229 is already merged at 494d6f8ec35955f88540692e911a8c2c71758f1e and its same-run disable/recovery behavior is useful existing functionality; this is a fresh automatic-successor failure.

    Observed reproduction through public control and real execution boundaries.

    1. Admit an owned Git/SQLite fixture with delivery_timeout_seconds=4; start the production workflow through ordinary service dispatch and hold its preparation activity.
    2. Call authenticated POST /api/runs/run-1/disable-limits using the current Temporal revision. It returns HTTP 200 with delivery_limits_disabled=true.
    3. Wait beyond the old four-second deadline. The original activity continues, proving the disable took effect.
    4. Release preparation into a controlled ConnectionError. The real workflow closes blocked, cleanup=confirmed, with the original checks.delivery_limits_control.disabled=true receipt intact.
    5. After authentic closure/finalization, release only the fixture's local tracker claim and run delivery_automatic_retry.retry_once(store); dispatch its one successor normally.
    6. The successor keeps the original admission timestamp and four-second policy, but its outbox has recovery_json=null. It immediately closes:
    {
      "original_disabled": true,
      "successor_outcome": "cancelled",
      "successor_error": "delivery elapsed time limit reached",
      "successor_has_disable_receipt": false,
      "preparation_launches": ["run-1"]
    }

    No successor preparation launched. The public command was completed and effective on the predecessor; automatic continuation silently undid it.

    Cause / owning layer. delivery_limits.py:85 records the revision-checked amendment in workflow checks. Initialization at line 173 restores it only from recovery state. Automatic successor construction copies frozen request fields without that operational receipt; admission retains the original clock, and dispatch starts with null recovery. The receipt exists but crosses no authenticated successor boundary.

    Expected invariant / smallest coherent fix. An automatically continued logical delivery must inherit its authenticated operator timer amendment. Bind the existing receipt into successor operational lineage and restore it before evaluating deadlines. Keep the frozen policy, original clock, attempt/repair accounting, custody checks and refusal of forged controls. Do not fix this by resetting the clock or requiring a second user command.

    Acceptance.

    • Repeat the sequence using public disable, real Temporal closure and ordinary retry/dispatch; successor preparation must perform useful work beyond the old deadline with the same authenticated disable receipt.
    • Restart between receipt persistence, successor admission and dispatch; overlap duplicate retry/response-loss requests and prove one owner/successor with unchanged budgets.
    • Negative controls: a run without the amendment still expires; forged/unrelated/stale-owner receipts cannot disable limits; existing same-run recovery remains valid.

    Evidence limits. Git operations, SQLite, Temporal, workflow update, clocks, closure readback, admission/dispatch and finalization were real. Provider preparation was a controlled owned activity; the fake-provider fixture enabled the existing resource-aware marker; fresh GitHub-base readback returned its exact local base; tracker claim release was fixture-local after real closure. No provider/GitHub writes, active delivery, production source, installed runtime or settings were changed.

    Open/closed issue searches for disable/timer/delivery-limit symptoms and all-state related PR searches identified #235 as owner. #245 is related clock/pause work, #246 owns integrated qualification, and #229 supplies the existing timer control. No new blocking prerequisite is needed for this bounded correction.

  2. ebarti commented on Oct 26, 2026

    @ebarti
    OwnerAuthor

    Independent audit evidence, 2026-10-26: P1 — a publication activity timeout can leave two live mutation bodies for one effect. Reusing #235 because it explicitly owns execution/effect custody.

    Confirmed with the unchanged production 180-second activity deadline and retry policy. Audited source c43cf0bec931f4651fdeefdfc82a9e7d523c36b2 (merged #238) has the same tracked tree as live clean bfc918a545ce820f47c814e99a6fe5e92b1c3fc1.

    Trigger and observed reproduction

    An owned isolated Temporal fixture invokes the production publication scheduler, activity, broker, real Git and SQLite effect journal. A private local gh stand-in supplies controlled responses; a local Git wrapper delays then executes real Git. No real GitHub requests occur.

    Sequential operations take:

    • PR-list readback: 55 seconds, below its 60-second bound.
    • git ls-remote: 100 seconds, below its 120-second bound.
    • PR creation stand-in: 80 seconds, below its 90-second bound.

    Their combined duration crosses the outer 180-second activity deadline. Temporal times out the async monitor and retries, but its original asyncio.to_thread body and subprocess continue.

    Observed process events:

    original create start: 1793025983.196233   PID 95573
    retry create start:    1793026009.566293   PID 96154
    original create end:   1793026063.203513
    retry create end:      1793026089.571321
    peak live mutation processes: 2
    overlap: 53.637 seconds
    publication effect rows: 1
    eventual effect state: complete
    

    At elapsed 249 seconds, the one shared effect row was already complete while the retry's second mutation command was still running. The workflow ultimately returned a publication receipt. A completed effect row therefore does not establish settled execution custody.

    This confirms duplicate mutation launch and two local execution owners, not two actual GitHub PRs. GitHub's same-head/base PR uniqueness is an existing useful safeguard.

    Owning code and cause

    The journal authenticates the effect's identity but does not establish who may still execute it. Monitor expiry is being treated as executor replacement even though the old executor is alive.

    Expected invariant / smallest coherent fix

    A replacement monitor may attach, wait or reconcile the original operation. It must not launch another mutation body while its predecessor is live or its outcome/custody remains unknown.

    Fix publication execution custody at the activity/effect boundary: durably identify and fence/join the original executor before a replacement may mutate; a stale executor must not start another effect or finalize a newer generation's row. Preserve exact candidate/head/branch checks, GitHub uniqueness/readback and authentic unknown outcomes. Retain whole-publication retries for confirmed pre-mutation transport failure. Raising timeouts alone does not establish exclusive ownership.

    Acceptance

    • Repeat the real default-deadline sequence above; each individual command stays within its own deadline.
    • Across timeout, worker replacement, cancellation, restart and lost completion, prove at most one live mutator per logical publication and no stale-generation completion.
    • A retry must attach/wait/reconcile while prior execution is live or unknown; late old execution cannot mutate after losing custody.
    • Confirmed pre-mutation transport failure still retries without another implementation turn or product repair debit.
    • Missing/changed candidates, requests, branches and owners still refuse; unknown remote outcomes stay unknown until exact readback.

    The fixture uses a minimal workflow entering the production publication boundary, a fake provider, and owned gh/Git latency adapters. It does not claim complete delivery, actual GitHub duplication or provider qualification. No production source/runtime/settings/active delivery was changed.

    Overlap and dependencies

    Open/closed issues were searched for publication/effect/timeout/owner/duplicate and all PR states for publication/publish-timeout/unknown-merge. #235 is the owning scope. #236 and #246 are related; no new blocking prerequisite is needed for this bounded fix.

    PR #149 is merged at 6caf78acb5606c5f29a1896b8d7e6db54377e6d8 and supplies useful bounded retry/readback behavior that must be preserved. PR #92 is closed unmerged, not a delivered fix. #238 is merged and still contains this composition gap.

  3. ebarti commented on Oct 26, 2026

    @ebarti
    OwnerAuthor

    Reassessment finding, 2026-10-26 — P2: an expired product clock cancels terminal-only tracker recovery before reconciliation starts. Reusing #235's frozen historical-outcome versus operational-state contract.

    Real public sequence on current admissions: create an owned predecessor through the repository's terminal checkpoint fixture, produce and seal its real Temporal history, then call authenticated POST /api/runs/run-1/reconcile-tracker. It returns 200, reconciliation_only=true. Ordinary dispatch starts its successor.

    With the original two-second delivery limit already expired, the successor schedules no terminal preflight/tracker activity and returns cancelled / delivery elapsed time limit reached. Its authentic retained terminal checkpoint still has blocked / Preserved failed gate. An otherwise identical no-delivery-timer control schedules preflight and tracker readback and preserves that blocked outcome.

    This was reproduced on current tracker_retry_version=1 admissions as well as supported historical admissions. The predecessor's outward outcome was still null in waiting_tracker; this finding concerns losing its frozen checkpoint outcome, not overwriting an already completed public result or rewriting historical bytes.

    Cause: delivery_limits.run:173 restores the old product clock and watcher before terminal-only recovery establishes settlement mode. It does not restore delivery_limits_closing; delivery_terminal_preflight is outside SETTLEMENT. The activity guard and independent watcher therefore trigger cancellation. The original terminal projection already enters settlement mode, but that mode does not survive the explicitly terminal-only successor.

    The documented contract freezes the transition and permits its successor only to confirm it.

    Smallest coherent correction: restore authenticated terminal settlement timer semantics before evaluating the successor's product clock. Preserve the original clock/history as evidence and a separate bounded reconciliation allowance. Adding only preflight to an allowlist is insufficient if the independent watcher can still cancel tracker readback. Do not reset product budgets or exempt ordinary product continuation from legitimate deadlines.

    Acceptance: repeat public reconcile, authentic closure, admission and dispatch with an expired product clock; preflight/readback must run and preserve the frozen outcome/error. Include restart/response loss, duplicate commands, still-pending remote confirmation and no-timer control. Keep checkpoint/candidate/owner/history/effect guards and bounded tracker recovery. The reproduced checkpoint was blocked; a delivered-checkpoint extension needs separate qualification.

    Seams: predecessor setup is the controlled terminal fixture; tracker transport and its eight-second deadline are controlled. Public HTTP, current admission shape, Git/SQLite, sealed Temporal history, dispatch, timers and projections are real. No provider/GitHub effects or production/runtime/settings/active-delivery changes.

    Open/closed issue and all-state PR searches identified #235's owning contract, with #240/#246 related. PR #229 and #146 are merged; #91 is closed unmerged. Audited main c43cf0b and installed bfc918a share a tree. No new blocker.

  4. ebarti commented on Oct 26, 2026

    @ebarti
    OwnerAuthor

    Reassessment finding, 2026-10-26 — P2: old same-iteration completion can label a newly failed gate completed. Reusing #235's derived-view contract; related #240/#246.

    Observed: a real Temporal workflow runs production _resume_ci and delivery_project after setup recovery. An actual owned local check exits 7. Workflow outcome is blocked, error local tests did not pass after setup recovery. Public GET /api/runs/run-1 returns checks.local.state=failed but phase_gates.local_checks.state=completed. The UI renders that completed state as a green check.

    The persisted sequence is ci_wait → checks_retry_started → blocked, all at product iteration 0. The older completion is authentic historical evidence, not the current gate result.

    Code/cause: delivery_store.py:4152 reduces completion events to type membership filtered only by iteration. Line 4181 checks that membership before current failed/unknown results. The setup-retry producer legitimately retains the iteration and runs a new local validation. Product iteration is therefore not a gate-attempt identity.

    Expected / smallest correction: current gate attempt/generation and observations must outrank historical completion for display. A newer start invalidates the prior completion as a statement about the current attempt; history remains immutable and visible separately. Preserve execution gates and budgets. The controller did block in this reproduction; this is false public gate presentation, not demonstrated execution bypass.

    Acceptance: earlier same-iteration success followed by failed, unknown, pending/running and finally successful retries must display each current state accurately. Preserve working previous-iteration isolation, honest unknown-without-old-completion behavior and original history. Include public detail/UI readback.

    Qualification seams: prior ci_wait was seeded via real store.project, and recovery-admission authentication was controlled. Git, SQLite, failing subprocess, production continuation/controller/projection, Temporal and public GET were real. This is not complete public continuation-admission or native/provider qualification. No production/runtime/settings/active-delivery or external effects changed.

    Open/closed issues and all-state PR searches found no narrower owning issue; PRs #58/#231 are merged and retain this reducer. Frozen PR #248 does not change delivery_store.py. No new blocker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions