feat(a2a): orphaned-run revival seam — reconcile re-attaches instead of terminal-failing

[arcaputo3]

Problem

A serve restart/recycle drops in-memory runs. The tasks/get durable-completion safety net (reconcileOrphaned) then terminal-fails the task — "Task interrupted: no active run … Resend the message to retry" — even when the underlying work is still running (e.g. a Claude Managed Agents session, which survives serve recycles by design). Downstream (TJC-LP/tjc-agents TJC-1494): a Modal infra replacement of a CMA serve container false-failed 4 long email-driven runs in one day while their sessions kept working.

Change

New public A2AOrphanedRunReviver hook, consulted by reconcileOrphaned before failing a non-terminal task with no runtime bus:

• Some(run) → the server reserves the task's runtime slot (the existing registry mutex, so concurrent polls revive at most one run), re-loads the task to honor a concurrent terminal/cancel write, and forks run(publisher) with the exact same lifecycle as a normal execution (catchAll → terminal failed status; ensuring → finish + registry cleanup). The poll returns the task still working.
• None or a failed revive → the existing terminal orphan failure, verbatim.

Supporting changes:

• ResultManager.userMessage is now Option[A2AMessage] — revived runs have no in-flight PreparedRun; the history-repair message is reconstructed from persisted history (startRevivedRun).
• startExecution / startRevivedRun share the fork/failure/cleanup wrapper (forkManagedRun) — no behavior change on the normal path.
• A2ARole now derives CanEqual (consumers compiling with -language:strictEquality couldn't compare roles).
• Config threading: A2AServerCoreConfig.orphanedRunReviver (default None) + JVM A2AServerLive.Config field. JS config inherits the default — no JS-side surface change.

Tests

A2AServerCoreSpec (+5, runs on both platforms): revive re-attaches and completes; decline → terminal failure; reviver crash → terminal failure; concurrent polls fork exactly one run; a crashed revived run terminal-fails with the error detail. Full suites green: JVM 143/143, JS 190/190.

Consumer: TJC-LP/tjc-agents wires this for CMA agents (watch-only re-attach to the live session). Needs a release once merged so tjc-agents can pin it.

🤖 Generated with Claude Code

[claude]

Review: orphaned-run revival seam

Solid, well-scoped change with careful concurrency reasoning and genuinely good docstrings. The forkManagedRun extraction is a clean refactor that keeps the normal path behavior-identical while sharing the failure/cleanup lifecycle with the new path. The reserve-mutex approach for "at most one run forks" is correct — A2ARuntimeRegistry.reserve is atomic via Ref.Synchronized.modify, and it's the same mutex prepare contends on, so a concurrent resend can't double-fork. The reload-after-reserve to honor a concurrent terminal/cancel/delete write is the right instinct.

A few things worth considering:

Design / correctness

1. Revived runs bypass taskTimeout (documented, but the tradeoff deserves emphasis). The normal path wraps the body in executeRun, which applies Config.taskTimeout; a revived run invokes the reviver's run(manager) directly, so it has no server-side upper bound. This effectively trades the false-fail bug for a permanent-working failure mode: if a revived run hangs without ever publishing a stream-ending status, its bus stays live, so reconcileOrphaned will never fail it either (bus present → left alone). The docstring correctly says "bound the run yourself," but since the safety net can no longer rescue a hung revived run, this is the one place a buggy reviver silently defeats the whole self-healing property. Worth making sure the tjc-agents consumer wraps its re-attach in a deadline.

2. reviver.revive runs on the hot poll path, unreserved, once per concurrent poll. Documented ("keep it prompt, read-only, idempotent"), but two consequences are worth calling out for implementers:

   • Every tasks/get on an orphaned task pays the reviver's latency before the mutex — if revive does remote session I/O, that cost is on each poll until one wins the reserve. A frequently-polling client issues many concurrent revive calls.
   • The run returned by a losing poll is silently discarded and never invoked or cleaned up. For a watch-only re-attach closure this is harmless, but a reviver that acquires a resource while building the run value (vs. lazily inside the run) would leak it. A sentence in the trait doc making explicit that the returned run must be safe to drop unused would help.

3. Repeated revival if a run ends without a stream-ending status. The docstring notes the next poll reconciles again — good — but a reviver whose run consistently exits without a terminal/input-required status will be re-forked on every subsequent poll. Convergent only if the run is idempotent and eventually ends the stream. Just flagging this is a sharper requirement than "read-only."

Test coverage

The 5 new tests are well-chosen (success, decline, reviver-crash, one-run-under-race, revived-run-crash) and the race test correctly asserts the fork count via a gated promise. Two reviveOrphaned branches are still uncovered:

• task terminal-between-decision-and-reserve — the Some(fresh) if fresh.isStreamEnding => remove(key).as(fresh) branch (reviver said yes, but a concurrent terminal/cancel write landed before reserve). This is the exact race the reload guards against, so it'd be the most valuable to pin.
• task deleted-between-decision-and-reserve — the None => remove(key) *> taskNotFound branch.

Both are reachable by having the reviver's revive mutate the store (terminal-fail / delete the task) before returning Some(run).

Nits

• A2AServerLive.scala (the Server.Config indentation) and A2AServerDefaults.scala (the val alignment) contain reformatting churn unrelated to the feature — presumably scalafmt. Not blocking, just adds diff noise.
• startRevivedRun reconstructs userMessage via history.findLast(_.role == A2ARole.User), but since no TaskSnapshot is published on the revived path, ensureHistory never fires for revived runs — the value is effectively unused. Harmless, but a comment noting it's defensive-only (or dropping it) would reduce confusion. (This is what motivates the A2ARole derives CanEqual addition, a good cleanup regardless.)

Overall a careful, minimal seam that preserves the existing safety net's convergence guarantees. The concurrency invariants check out. My only substantive ask is to ensure the consumer bounds the revived run's lifetime given point (1), and ideally add the terminal-race test from the coverage section.

[arcaputo3]

Live consumer validation of the seam on TJC-LP/tjc-agents#115 (dev, orcagent): serve container recycled mid-run → first tasks/get on the fresh process logged Reviving orphaned non-terminal task … re-attaching a run, the task stayed working through the recycle, and completed ~10 minutes later with the expected answer. Pre-seam behavior was a terminal failed ("no active run"). Details in the tjc-agents PR comment.

🤖 Generated with Claude Code