Repository navigation
Description
Implementation progress — 27 October 2026
Candidate foundations; acceptance not yet qualified. #257 preserves control lineage and publication custody; #260 binds original execution authority; #261 adds registry observation epochs.
Current-head hosted checks and the issue's public acceptance still require verification. Registry epochs are an observation primitive; final native launch/reservation/GO accounting remains separate work in #243.
The referenced candidates are open and unmerged in stack #253. This progress map is non-closing; issue acceptance remains authoritative. Historical text below calling #238 open is superseded: #238 merged as c43cf0b.
Problem and evidence
Retained runs have been refused after runtime relocation, configuration/capacity changes, or later cleanup reconciliation even when the user's scope and retained candidate did not change. A Project update can also be overwritten by a competing projection writer. These are different facts with different owners; comparing or rewriting one composite state makes routine recovery look like a change in execution authority.
Existing repairs in #147, #224, #230 and #231 cover individual cases. The installed recovery work also separates shared installation identity from per-run grants. Preserve and consolidate those contracts rather than replacing them or weakening their checks.
Required behavior
- Define the authoritative owner and version for immutable authorization (scope, accepted plan, permissions, endpoint and budget), current execution/effect custody, environment binding, and derived tracker/dashboard views.
- Keep original authorization immutable. Operational changes must use explicit compatible transitions and must never silently broaden paths, grants, endpoint or product repair allowance.
- Make tracker updates an idempotent projection of the current owner/revision. A stale writer cannot undo an operator hold or newer owner transition. Projection failure remains a projection problem and does not rerun implementation or erase accepted evidence.
- Preserve historical verdicts when newer observations reconcile cleanup or publication. Record provenance and the newer observation separately.
- Keep ordinary status reads bounded and available from persisted facts. Expensive proof reconstruction belongs to an explicit operation and must report when evidence is unavailable.
- Supply a compatible migration for existing admitted histories and multiple retained continuations; do not require a blanket rewrite of historical specifications.
Acceptance
Exercise unchanged-source continuation across capacity change, runtime relocation, stale tracker writes and failed tracker transport through the public boundaries. Verify the original candidate, authorization, budget and historical results remain unchanged; changed authority and conflicting owners still refuse with a specific reason. Include concurrent-writer readback, not only a successful update response.
Document the ownership/version contract and migrate existing supported recovery paths incrementally.
Existing implementation to reuse
PR #238 is open at bfc918a545ce820f47c814e99a6fe5e92b1c3fc1, matching the inspected installed runtime. Reuse its separation of shared installation identity from original per-run scope and its preservation of original custody. This issue owns the consistent authority/projection contract across recovery paths.
Dependencies
No blocking prerequisite in this issue set. This establishes the state contract consumed by lifecycle recovery; independent operation and storage work can proceed in parallel.
Blocks:
- fix(recovery): expose typed lifecycle transitions and actionable blockers #240 — expose typed lifecycle transitions and actionable blockers.
Activity
ebarti commentedon Oct 26, 2026
Independent audit evidence, 2026-10-26: P2 — automatic unpublished retry loses an accepted timer-disable amendment. This belongs to #235's operational-state lineage contract, so no overlapping issue is opened.
Verified against merged PR #238 tree at
c43cf0bec931f4651fdeefdfc82a9e7d523c36b2, identical to the live cleanbfc918a545ce820f47c814e99a6fe5e92b1c3fc1tree. PR #229 is already merged at494d6f8ec35955f88540692e911a8c2c71758f1eand its same-run disable/recovery behavior is useful existing functionality; this is a fresh automatic-successor failure.Observed reproduction through public control and real execution boundaries.
delivery_timeout_seconds=4; start the production workflow through ordinary service dispatch and hold its preparation activity.POST /api/runs/run-1/disable-limitsusing the current Temporal revision. It returns HTTP 200 withdelivery_limits_disabled=true.ConnectionError. The real workflow closesblocked,cleanup=confirmed, with the originalchecks.delivery_limits_control.disabled=truereceipt intact.delivery_automatic_retry.retry_once(store); dispatch its one successor normally.recovery_json=null. It immediately closes:{ "original_disabled": true, "successor_outcome": "cancelled", "successor_error": "delivery elapsed time limit reached", "successor_has_disable_receipt": false, "preparation_launches": ["run-1"] }No successor preparation launched. The public command was completed and effective on the predecessor; automatic continuation silently undid it.
Cause / owning layer. delivery_limits.py:85 records the revision-checked amendment in workflow checks. Initialization at line 173 restores it only from recovery state. Automatic successor construction copies frozen request fields without that operational receipt; admission retains the original clock, and dispatch starts with null recovery. The receipt exists but crosses no authenticated successor boundary.
Expected invariant / smallest coherent fix. An automatically continued logical delivery must inherit its authenticated operator timer amendment. Bind the existing receipt into successor operational lineage and restore it before evaluating deadlines. Keep the frozen policy, original clock, attempt/repair accounting, custody checks and refusal of forged controls. Do not fix this by resetting the clock or requiring a second user command.
Acceptance.
Evidence limits. Git operations, SQLite, Temporal, workflow update, clocks, closure readback, admission/dispatch and finalization were real. Provider preparation was a controlled owned activity; the fake-provider fixture enabled the existing resource-aware marker; fresh GitHub-base readback returned its exact local base; tracker claim release was fixture-local after real closure. No provider/GitHub writes, active delivery, production source, installed runtime or settings were changed.
Open/closed issue searches for disable/timer/delivery-limit symptoms and all-state related PR searches identified #235 as owner. #245 is related clock/pause work, #246 owns integrated qualification, and #229 supplies the existing timer control. No new blocking prerequisite is needed for this bounded correction.
ebarti commentedon Oct 26, 2026
Independent audit evidence, 2026-10-26: P1 — a publication activity timeout can leave two live mutation bodies for one effect. Reusing #235 because it explicitly owns execution/effect custody.
Confirmed with the unchanged production 180-second activity deadline and retry policy. Audited source
c43cf0bec931f4651fdeefdfc82a9e7d523c36b2(merged #238) has the same tracked tree as live cleanbfc918a545ce820f47c814e99a6fe5e92b1c3fc1.Trigger and observed reproduction
An owned isolated Temporal fixture invokes the production publication scheduler, activity, broker, real Git and SQLite effect journal. A private local
ghstand-in supplies controlled responses; a local Git wrapper delays then executes real Git. No real GitHub requests occur.Sequential operations take:
git ls-remote: 100 seconds, below its 120-second bound.Their combined duration crosses the outer 180-second activity deadline. Temporal times out the async monitor and retries, but its original
asyncio.to_threadbody and subprocess continue.Observed process events:
At elapsed 249 seconds, the one shared effect row was already
completewhile the retry's second mutation command was still running. The workflow ultimately returned a publication receipt. A completed effect row therefore does not establish settled execution custody.This confirms duplicate mutation launch and two local execution owners, not two actual GitHub PRs. GitHub's same-head/base PR uniqueness is an existing useful safeguard.
Owning code and cause
asyncio.to_thread, without joining/settling the body when the activity monitor times out.The journal authenticates the effect's identity but does not establish who may still execute it. Monitor expiry is being treated as executor replacement even though the old executor is alive.
Expected invariant / smallest coherent fix
A replacement monitor may attach, wait or reconcile the original operation. It must not launch another mutation body while its predecessor is live or its outcome/custody remains unknown.
Fix publication execution custody at the activity/effect boundary: durably identify and fence/join the original executor before a replacement may mutate; a stale executor must not start another effect or finalize a newer generation's row. Preserve exact candidate/head/branch checks, GitHub uniqueness/readback and authentic unknown outcomes. Retain whole-publication retries for confirmed pre-mutation transport failure. Raising timeouts alone does not establish exclusive ownership.
Acceptance
The fixture uses a minimal workflow entering the production publication boundary, a fake provider, and owned
gh/Git latency adapters. It does not claim complete delivery, actual GitHub duplication or provider qualification. No production source/runtime/settings/active delivery was changed.Overlap and dependencies
Open/closed issues were searched for publication/effect/timeout/owner/duplicate and all PR states for publication/publish-timeout/unknown-merge. #235 is the owning scope. #236 and #246 are related; no new blocking prerequisite is needed for this bounded fix.
PR #149 is merged at
6caf78acb5606c5f29a1896b8d7e6db54377e6d8and supplies useful bounded retry/readback behavior that must be preserved. PR #92 is closed unmerged, not a delivered fix. #238 is merged and still contains this composition gap.ebarti commentedon Oct 26, 2026
Reassessment finding, 2026-10-26 — P2: an expired product clock cancels terminal-only tracker recovery before reconciliation starts. Reusing #235's frozen historical-outcome versus operational-state contract.
Real public sequence on current admissions: create an owned predecessor through the repository's terminal checkpoint fixture, produce and seal its real Temporal history, then call authenticated
POST /api/runs/run-1/reconcile-tracker. It returns 200,reconciliation_only=true. Ordinary dispatch starts its successor.With the original two-second delivery limit already expired, the successor schedules no terminal preflight/tracker activity and returns cancelled / delivery elapsed time limit reached. Its authentic retained terminal checkpoint still has blocked / Preserved failed gate. An otherwise identical no-delivery-timer control schedules preflight and tracker readback and preserves that blocked outcome.
This was reproduced on current
tracker_retry_version=1admissions as well as supported historical admissions. The predecessor's outward outcome was still null inwaiting_tracker; this finding concerns losing its frozen checkpoint outcome, not overwriting an already completed public result or rewriting historical bytes.Cause: delivery_limits.run:173 restores the old product clock and watcher before terminal-only recovery establishes settlement mode. It does not restore
delivery_limits_closing;delivery_terminal_preflightis outside SETTLEMENT. The activity guard and independent watcher therefore trigger cancellation. The original terminal projection already enters settlement mode, but that mode does not survive the explicitly terminal-only successor.The documented contract freezes the transition and permits its successor only to confirm it.
Smallest coherent correction: restore authenticated terminal settlement timer semantics before evaluating the successor's product clock. Preserve the original clock/history as evidence and a separate bounded reconciliation allowance. Adding only preflight to an allowlist is insufficient if the independent watcher can still cancel tracker readback. Do not reset product budgets or exempt ordinary product continuation from legitimate deadlines.
Acceptance: repeat public reconcile, authentic closure, admission and dispatch with an expired product clock; preflight/readback must run and preserve the frozen outcome/error. Include restart/response loss, duplicate commands, still-pending remote confirmation and no-timer control. Keep checkpoint/candidate/owner/history/effect guards and bounded tracker recovery. The reproduced checkpoint was blocked; a delivered-checkpoint extension needs separate qualification.
Seams: predecessor setup is the controlled terminal fixture; tracker transport and its eight-second deadline are controlled. Public HTTP, current admission shape, Git/SQLite, sealed Temporal history, dispatch, timers and projections are real. No provider/GitHub effects or production/runtime/settings/active-delivery changes.
Open/closed issue and all-state PR searches identified #235's owning contract, with #240/#246 related. PR #229 and #146 are merged; #91 is closed unmerged. Audited main c43cf0b and installed bfc918a share a tree. No new blocker.
ebarti commentedon Oct 26, 2026
Reassessment finding, 2026-10-26 — P2: old same-iteration completion can label a newly failed gate completed. Reusing #235's derived-view contract; related #240/#246.
Observed: a real Temporal workflow runs production
_resume_cianddelivery_projectafter setup recovery. An actual owned local check exits 7. Workflow outcome is blocked, errorlocal tests did not pass after setup recovery. PublicGET /api/runs/run-1returnschecks.local.state=failedbutphase_gates.local_checks.state=completed. The UI renders that completed state as a green check.The persisted sequence is
ci_wait → checks_retry_started → blocked, all at product iteration 0. The older completion is authentic historical evidence, not the current gate result.Code/cause: delivery_store.py:4152 reduces completion events to type membership filtered only by iteration. Line 4181 checks that membership before current failed/unknown results. The setup-retry producer legitimately retains the iteration and runs a new local validation. Product iteration is therefore not a gate-attempt identity.
Expected / smallest correction: current gate attempt/generation and observations must outrank historical completion for display. A newer start invalidates the prior completion as a statement about the current attempt; history remains immutable and visible separately. Preserve execution gates and budgets. The controller did block in this reproduction; this is false public gate presentation, not demonstrated execution bypass.
Acceptance: earlier same-iteration success followed by failed, unknown, pending/running and finally successful retries must display each current state accurately. Preserve working previous-iteration isolation, honest unknown-without-old-completion behavior and original history. Include public detail/UI readback.
Qualification seams: prior
ci_waitwas seeded via real store.project, and recovery-admission authentication was controlled. Git, SQLite, failing subprocess, production continuation/controller/projection, Temporal and public GET were real. This is not complete public continuation-admission or native/provider qualification. No production/runtime/settings/active-delivery or external effects changed.Open/closed issues and all-state PR searches found no narrower owning issue; PRs #58/#231 are merged and retain this reducer. Frozen PR #248 does not change delivery_store.py. No new blocker.
fix(runtime): preserve control lineage and publication custody (#257)
fix(api): preserve delivery operations across caller loss and restart (…
fix(delivery): bind authority to the original execution (#260)
fix(registry): detect stale authority observations (#261)