Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
24bac13
feat(runtime): expose durable queued pause controls
ebarti Oct 29, 2026
11a91d4
fix(recovery): report lifecycle custody blockers
ebarti Oct 29, 2026
7132fb7
fix(recovery): verify current checkpoint custody
ebarti Oct 29, 2026
3e1314e
fix(recovery): preserve assessment owner context
ebarti Oct 29, 2026
98386eb
fix(recovery): retain candidates after readiness failures
ebarti Oct 29, 2026
31d3f5a
fix(recovery): identify recorded command replays
ebarti Oct 29, 2026
c7bd401
test(recovery): exercise published candidate retry
ebarti Oct 29, 2026
ed0e162
fix(recovery): identify stopped candidate drift
ebarti Oct 29, 2026
35410b0
fix(recovery): identify stopped custody blockers
ebarti Oct 29, 2026
c31d6d8
fix(recovery): authenticate early unreviewed revisions
ebarti Oct 29, 2026
49c1375
fix(recovery): classify early revision prelaunch evidence
ebarti Oct 29, 2026
beb6994
fix(recovery): report publication custody and receipts
ebarti Oct 29, 2026
70ce906
fix(recovery): lock prelaunch resource evidence
ebarti Oct 29, 2026
882d162
fix(recovery): retain mixed revision review history
ebarti Oct 29, 2026
e72e5ab
fix(recovery): share historical repair admission checks
ebarti Oct 29, 2026
cc44243
fix(recovery): recheck historical work binding
ebarti Oct 29, 2026
adb8743
fix(recovery): back off retained readiness retries
ebarti Oct 29, 2026
efb7a5d
fix(recovery): preserve fake provider continuation
ebarti Oct 29, 2026
64f0ad3
fix(recovery): validate retained evidence on readback
ebarti Nov 1, 2026
8121ac2
fix(recovery): validate historical admission receipts
ebarti Nov 1, 2026
e5671ea
fix(recovery): bind retained state reads to opened files
ebarti Nov 1, 2026
bb5744c
Merge lifecycle admission into readiness retries
ebarti Nov 1, 2026
dcd4748
fix(recovery): integrate checkpoint and retry custody
ebarti Nov 1, 2026
479c56c
feat(runtime): guard staged publication handoffs
ebarti Nov 1, 2026
a6750a9
chore(integration): incorporate recovery foundations
ebarti Nov 1, 2026
c12c12a
fix(pause): reconcile known command refusals
ebarti Nov 1, 2026
e6cdc07
fix(pause): reject unaddressable operations
ebarti Nov 1, 2026
04cb084
chore(integration): incorporate merged recovery fixes
ebarti Nov 1, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 24 additions & 1 deletion runtime/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -276,7 +276,8 @@ operation ID carrying the same domain command and request returns the original
operation; a conflicting command ID is refused. Observations retain separate
identities so a fresh preflight can be followed by its distinct acknowledgement.

The service dispatches persisted pending work on restart. A previously running
The service dispatches persisted pending work on restart, except pending resume
operations described below. A previously running
operation is unknown once its exclusive owner is gone; it is not launched
again. `POST /api/operations/{id}/reconcile` checks supported original domain
receipts without creating another command. Temporal updates record the exact
Expand All @@ -292,6 +293,28 @@ CLI `operation-create --request envelope.json`, `operation --id ID`,
running service. The corresponding MCP tools and dashboard operation panel
expose the same state; none reissues an uncertain effect under a fresh ID.

### Queued pause controls

Pause currently applies only to explicitly admitted, untimed ordinary runs before
their workflow has started. `GET /api/runs/{id}/pause-control`, CLI
`pause-control --id RUN`, MCP `get_pause_control`, and the dashboard report the
same scope, persisted observed state and blockers. A `pause_requested` or
`pausing` state is not a completed pause. Running workflows, feature children,
native work and timed runs are not yet supported by this control.

The CLI `pause --id RUN --request FILE` and MCP `pause_run` accept an object
with caller-chosen stable `operation_id` and `command_id` values and the
`expected_revision` from pause control. `resume` and `resume_run` additionally
require its `epoch`; resume is an explicit
instruction. Both use the durable operation journal and require an already running
service. A resume operation still pending when the service restarts is held;
the user must explicitly resubmit the same operation to dispatch it. Preserve
the complete request after a timeout, inspect the operation and reconcile an
unknown result before attempting another command. A recovered
acknowledgement records the original intent; read pause control again for current
settlement or activation. The direct API and dashboard remain available with the
same command ID, revision and epoch semantics.

Routine CLI `run --id RUN` and MCP `get_run` read the versioned persisted
status projection. They do not reconstruct current execution authority. Use
CLI `run-detail --id RUN` or MCP `get_run_detail` for explicit full-detail
Expand Down
30 changes: 27 additions & 3 deletions runtime/src/devflow_temporal/delivery_activities.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
import sys
import threading
import time
from dataclasses import replace
from pathlib import Path
from typing import Any

Expand Down Expand Up @@ -65,12 +66,20 @@ def database_contention(exc: BaseException) -> bool:

class _DatabaseContentionInbound(ActivityInboundInterceptor):
async def execute_activity(self, input: ExecuteActivityInput) -> Any:
from .delivery_pause import PauseDeferred
from .delivery_pause_context import activate, split_activity_request

try:
# Only fresh ordinary histories opt in; legacy and merge workflows
# retain their existing protocols. SDK context selects the execution.
from . import delivery_execution_authority as authority
info = activity.info()
request = input.args[0] if input.args else None
args = list(input.args)
context = None
if len(args) == 1:
args[0], context = split_activity_request(args[0], info)
adjusted = replace(input, args=args) if context is not None else input
request = args[0] if args else None
spec = request.get("spec") if isinstance(request, dict) else None
if (info.workflow_type == "DevflowDeliveryWorkflow" and isinstance(spec, dict)
and spec.get(authority.VERSION) == 1):
Expand All @@ -81,10 +90,21 @@ async def execute_activity(self, input: ExecuteActivityInput) -> Any:
from .delivery_writer_custody import activity_custody

async with activity_custody(spec):
return await super().execute_activity(input)
with activate(context):
return await super().execute_activity(adjusted)
finally:
authority._ACTIVITY_AUTHORITY.reset(token)
return await super().execute_activity(input)
with activate(context):
return await super().execute_activity(adjusted)
except PauseDeferred as exc:
if (not exc.root_id or not exc.operation_key or not exc.step
or type(exc.epoch) is not int):
raise
raise ApplicationError(
str(exc), {"root_id": exc.root_id, "operation_key": exc.operation_key,
"step": exc.step, "epoch": exc.epoch},
type="PauseDeferred", non_retryable=True,
) from exc
except sqlite3.OperationalError as exc:
if not database_contention(exc):
raise
Expand Down Expand Up @@ -1099,6 +1119,7 @@ def _role_result(request, broker, workspace, review_diff, result):

@activity.defn(name="delivery_publish")
async def delivery_publish(request: dict[str, Any]) -> dict[str, Any]:
from .delivery_pause import PauseDeferred
from .delivery_writer_custody import to_thread

def execute() -> dict[str, Any]:
Expand All @@ -1109,6 +1130,9 @@ def execute() -> dict[str, Any]:

return publish(store, broker, request)
return broker.publish(request["iteration"], request["candidate"])
except PauseDeferred:
# The owner proved that this exact handoff never launched.
raise
except Exception as exc:
if (request["spec"].get("publication_readback_version") == 1
and broker.publication_may_have_effect is False):
Expand Down
86 changes: 77 additions & 9 deletions runtime/src/devflow_temporal/delivery_broker.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,25 +28,67 @@


def _run(argv: list[str], *, cwd: Path | None = None, timeout: int = 120) -> str:
from .delivery_pause_context import enabled as pause_enabled
from .delivery_publication_execution import current_mutation
from .delivery_publication_execution import inherited_descriptors as publication_descriptors
from .delivery_writer_custody import inherited_descriptors as writer_descriptors

nul_output = argv[0] == 'git' and '-z' in argv
result = subprocess.run(
argv, cwd=cwd, text=not nul_output, capture_output=True, timeout=timeout, check=False,
pass_fds=tuple(dict.fromkeys((*publication_descriptors(), *writer_descriptors()))),
)
stdout = result.stdout.decode('utf-8', 'surrogateescape') if nul_output else result.stdout
stderr = result.stderr.decode('utf-8', 'surrogateescape') if nul_output else result.stderr
if result.returncode:
pass_fds = tuple(dict.fromkeys((*publication_descriptors(), *writer_descriptors())))
mutation = current_mutation()
if mutation is None or not pause_enabled(mutation[0].broker.spec):
result = subprocess.run(
argv, cwd=cwd, text=not nul_output, capture_output=True, timeout=timeout, check=False,
pass_fds=pass_fds,
)
returncode, stdout, stderr = result.returncode, result.stdout, result.stderr
else:
from .delivery_pause_context import publication_handoff

owner, step = mutation
with publication_handoff(owner, step) as handoff:
try:
process = subprocess.Popen(argv, cwd=cwd, text=not nul_output,
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
pass_fds=pass_fds)
except (FileNotFoundError, PermissionError):
handoff.no_process_started()
raise
try:
stdout, stderr = process.communicate(timeout=timeout)
except subprocess.TimeoutExpired as exc:
process.kill()
stdout, stderr = process.communicate()
raise subprocess.TimeoutExpired(argv, timeout, output=stdout, stderr=stderr) from exc
except BaseException:
process.kill()
process.communicate()
raise
returncode = process.returncode
stdout = stdout.decode('utf-8', 'surrogateescape') if nul_output else stdout
stderr = stderr.decode('utf-8', 'surrogateescape') if nul_output else stderr
if returncode:
raise RuntimeError(
f"command failed ({result.returncode}): {argv[0]} {argv[1]}: "
f"command failed ({returncode}): {argv[0]} {argv[1]}: "
+ (stderr.strip() or stdout.strip())[:500]
)
# Preserve exact filenames, including CR/LF and leading/trailing whitespace.
return stdout if nul_output else stdout.strip()


def _staged_index_evidence(checkout: Path) -> dict[str, str]:
"""Observe supported index semantics without mutating or restaging."""
from .delivery_checkpoint_index import observe_index

def query(path, *args):
return _git(path, *args).encode("utf-8", "surrogateescape")

try:
return observe_index(checkout, query)
except ValueError as exc:
raise ValueError("staged publication index semantics are unproven") from exc


def _git(path: Path, *args: str) -> str:
return _run(
[
Expand Down Expand Up @@ -1708,6 +1750,12 @@ def publish(self, iteration: int, input_candidate: dict[str, Any]) -> dict[str,
return {"state": "pending", "reason": "original_" + exc.operation + "_readback"}

def _publish_owned(self, iteration, input_candidate, owner):
from .delivery_pause_context import enabled as pause_enabled

pause_stage = pause_enabled(self.spec)
if pause_stage:
# Unsupported index flags can cause Git to omit reviewed worktree bytes.
_staged_index_evidence(self.checkout)
# Only this invocation's confirmed pre-mutation failure can release uncertainty.
# Any retained original effect may belong to an earlier lost completion.
if self.spec.get("publication_readback_version") == 1:
Expand Down Expand Up @@ -1767,7 +1815,27 @@ def _publish_owned(self, iteration, input_candidate, owner):
if path in indexed or os.path.lexists(self.checkout / path))
# Already-staged removals are absent from both index and worktree;
# preserve them without passing an unmatched pathspec to git add.
if stageable:
stage = owner.mutation("stage") if pause_stage else {}
if stage.get("state") == "complete":
# Path-only diffs ignore untracked files and index flags. Bind every
# logical index entry recorded after staging, even for pure deletions.
if stage.get("evidence") != _staged_index_evidence(self.checkout):
raise ValueError("staged publication candidate changed during pause")
unstaged = set(_git(self.checkout, "diff", "--name-only", "-z").split("\0"))
staged = set(_git(
self.checkout, "diff", "--cached", "--name-only", "-z").split("\0"))
unstaged.discard("")
staged.discard("")
if unstaged or staged != set(changed):
raise ValueError("staged publication candidate changed during pause")
elif pause_stage:
# Even an already-staged deletion needs a recorded index before
# the commit fence, and an uncertain earlier stage cannot be replayed.
owner.mutate("stage", lambda: _git(
self.checkout, *(['--literal-pathspecs', 'add', '--all', '--', *stageable]
if stageable else ['ls-files', '--stage', '-z'])),
observe_completion=lambda: _staged_index_evidence(self.checkout))
elif stageable:
_git(self.checkout, '--literal-pathspecs', 'add', '--all', '--', *stageable)
if _git(self.checkout, "diff", "--cached", "--name-only"):
owner.mutate("commit", lambda: _git(
Expand Down
21 changes: 21 additions & 0 deletions runtime/src/devflow_temporal/delivery_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,27 @@ def reconcile_operation(self, operation_id: str) -> dict:
"POST", "/api/operations/" + quote(operation_id, safe="") + "/reconcile", {}
)

def pause_control(self, run_id: str) -> dict:
"""Read the persisted pause scope, state and blockers without starting a service."""
return self._request("GET", "/api/runs/" + quote(run_id, safe="") + "/pause-control")

def _pause_operation(self, run_id: str, action: str, payload: dict[str, Any]) -> dict:
keys = {"operation_id", "command_id", "expected_revision"}
if action == "resume":
keys.add("epoch")
if not isinstance(payload, dict) or set(payload) != keys:
raise ValueError("pause operation requires the exact operation and command fields")
return self.create_operation({
"operation_id": payload["operation_id"], "run_id": run_id, "kind": action,
"request": {key: value for key, value in payload.items() if key != "operation_id"},
})

def pause(self, run_id: str, payload: dict[str, Any]) -> dict:
return self._pause_operation(run_id, "pause", payload)

def resume(self, run_id: str, payload: dict[str, Any]) -> dict:
return self._pause_operation(run_id, "resume", payload)

def submit(self, payload: dict[str, Any]) -> dict:
return self._request("POST", "/api/runs", payload)

Expand Down
16 changes: 16 additions & 0 deletions runtime/src/devflow_temporal/delivery_control.py
Original file line number Diff line number Diff line change
Expand Up @@ -899,6 +899,9 @@ async def worker(
"operation",
"operation-reconcile",
"run-operations",
"pause-control",
"pause",
"resume",
"start",
"status",
"stop",
Expand Down Expand Up @@ -1002,6 +1005,19 @@ def _run(args: argparse.Namespace, parser: argparse.ArgumentParser) -> None:
}[args.command](args.id)
print(json.dumps(result, sort_keys=True, indent=2))
return
if args.command in {"pause-control", "pause", "resume"}:
if not args.id:
parser.error("--id is required for pause controls")
caller = read_only_client(Path(args.config).expanduser().resolve(strict=True))
if args.command == "pause-control":
result = caller.pause_control(args.id)
else:
if args.request is None:
parser.error("--request is required for pause and resume")
payload = json.loads(args.request.read_text(encoding="utf-8"))
result = getattr(caller, args.command)(args.id, payload)
print(json.dumps(result, sort_keys=True, indent=2))
return
if args.command == "capture-retained-closures":
if args.request is None:
parser.error("--request is required for explicit retained-history capture")
Expand Down
33 changes: 33 additions & 0 deletions runtime/src/devflow_temporal/delivery_mcp.py
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,39 @@ def read_evidence(run_id: str, evidence_id: str) -> dict:
"""Read one indexed, contained evidence artifact for a run."""
return read_only_client(config_path).evidence(run_id, evidence_id)

@server.tool(annotations=read)
async def get_pause_control(run_id: str) -> dict:
"""Read current pause scope, eligibility, observed state and blockers.

Only explicitly admitted before-start runs currently support pause.
This read does not start the service or authorize a transition.
"""
return await asyncio.to_thread(read_only_client(config_path).pause_control, run_id)

@server.tool(annotations=write)
async def pause_run(run_id: str, request_json: str) -> dict:
"""Request an eligible queued pause through a durable operation.

Supply stable operation_id and command_id values and the expected_revision
from pause control. A succeeded result acknowledges pause_requested, not
that work is already paused; a pending operation has not yet done so. Keep
the complete request stable and inspect the operation after uncertainty.
The service must already be running.
"""
return await asyncio.to_thread(
read_only_client(config_path).pause, run_id, json.loads(request_json))

@server.tool(annotations=write)
async def resume_run(run_id: str, request_json: str) -> dict:
"""Explicitly request resume of an eligible queued pause.

Supply stable operation_id and command_id values and the expected_revision
and epoch from pause control. A resuming acknowledgement is not completed activation. Keep
the request stable after uncertainty; the service must already run.
"""
return await asyncio.to_thread(
read_only_client(config_path).resume, run_id, json.loads(request_json))

@server.tool(annotations=write)
async def answer_decision(run_id: str, request_json: str) -> dict:
"""Answer a question, review a plan, or authorize the recorded feature merge."""
Expand Down
Loading
Loading