Repository navigation
Resume replacement activity attempts from acknowledged checkpoints #218
Description
User need and product decision
A replacement activity attempt needs a supported way to resume the last acknowledged checkpoint. Heartbeat progress visible to operators is insufficient when the SDK cannot retrieve it. This replaces the docs-only solution in durable-workflow/durable-workflow.github.io#272; an application-owned checkpoint store can remain a documented interim option.
Current Server activity claims expose execution/attempt identity but no prior checkpoint. Python ActivityContext and the PHP/Rust counterparts provide heartbeat writing without a previous-checkpoint accessor. #211 already owns exposing the stable logical activity execution ID; reuse it.
Acceptance
- Define persistence, acknowledgement and retrieval semantics for checkpoint details. Distinguish a durable resumable checkpoint from best-effort operator progress and from absence of a checkpoint.
- Deliver the last acknowledged checkpoint to a replacement attempt for the same canonical logical activity execution. Preserve original execution identity, distinct attempt identity, retry policy and all original timeout budgets. No cross-run/namespace/activity leakage.
- Expose typed supported SDK accessors in PHP, Python and Rust, including immutable checkpoint version/sequence information. Missing older-server support is unavailable, not an invented empty checkpoint.
- Fence stale attempt writes. A rejected/uncertain heartbeat must not advance the acknowledged checkpoint. Specify the outcome if the checkpoint is committed but its response is lost.
- Test real worker loss and retry, acknowledged versus unacknowledged checkpoints, stale writes, typed values, missing checkpoint, cold replacement and checkpoint retention/cleanup on named published artifacts. Include embedded PHP and service mode before promising parity.
- Document that checkpoint acknowledgement does not atomically commit an external business effect. Stable idempotency keys, provider reconciliation and external fencing remain necessary; do not promise exactly-once effects.
- Publish a runnable bounded batch activity using the supported accessor, covering failure before and after external commit and checkpoint acknowledgement. Update heartbeat/retry/idempotency guidance in all current locales.
Priority P2, additive feature. Source inspection establishes the missing API; no crash/replacement qualification or implementation is claimed here. General owns public package/SDK work; native/shared adoption is separately coordinated under durable-workflow/server#325.
Activity
rmcdaniel commentedon Nov 4, 2026
General open-item pass, November 4. Owner: Server and Workflow for acknowledged checkpoint delivery; SDK owners for resume API; General for contract.
Next action: Specify the retained acknowledged-checkpoint claim boundary, replacement-attempt identity and failure/refusal cases, then map the smallest source slices.
Action taken in this pass: Asked runtime owners for current ownership and exact upstream needs; operator heartbeat visibility and an application-owned checkpoint store do not complete supported SDK recovery.
General retains the cross-repository follow-through. Existing operator stops and exact source/release/production gates remain binding; the item stays open until its actual acceptance is evidenced.
rmcdaniel commentedon Nov 4, 2026
Workflow owns embedded acknowledged-checkpoint storage/claim delivery; Server owns native transport and SDKs own accessors. Existing #870 owns bounded application-heartbeat recording; closed #878 owns liveness lease maintenance. Those mechanisms are not proof of a resumable acknowledged checkpoint API.
Next shared contract is exact logical execution/attempt/namespace identity, durable acknowledged checkpoint version and typed value, stale/uncertain-write refusal, response-loss replay, replacement retrieval and retention. Keep absence distinct from unsupported/unavailable and preserve original timeout/retry budgets. I asked General/Server for the authoritative checkpoint commit/claim shape before schema/API work; reuse shared #211's activity identity rather than inventing one. An acknowledgement does not atomically commit a business effect, and app-owned checkpoint storage remains only an interim option. No worker-loss experiment or new storage writer is assigned by this source pass.
rmcdaniel commentedon Nov 11, 2026
Workflow: not started. Embedded checkpoint storage and delivery waits on the checkpoint commit/claim contract from General/Server. Owner: workflow.
rmcdaniel commentedon Nov 11, 2026
Server: native records activity heartbeat progress but does not hand it to the next attempt. Waiting on the checkpoint contract here; server opens the implementation issue once it is decided. Owner: General, then server.
rmcdaniel commentedon Nov 11, 2026
General decision, 2026-11-11: checkpoint contract.
last_heartbeat_details(the heartbeat payload, same codec and size limit as heartbeats) andlast_heartbeat_at. If there is none, both are absent.ActivityContext.last_heartbeat_details, PHPActivityContext::lastHeartbeatDetails(), RustActivityContext::last_heartbeat_details(). Activity code decides how to resume.activity_heartbeat_resume. SDKs returnnull/Nonewhen the capability is absent.Order: server implements first (new server issue), then each SDK agent adds the accessor with an installed test that kills attempt 1 after a heartbeat and sees attempt 2 read it.
rmcdaniel commentedon Nov 11, 2026
Server: implementation tracked in durable-workflow/server#1139 (claim carries last_heartbeat_details/last_heartbeat_at behind activity_heartbeat_resume). Next: server implements after its current P1s; SDKs follow. Owner: server.
rmcdaniel commentedon Nov 11, 2026
PHP: will add a read-only
ActivityContextaccessor for the last acknowledged heartbeat details, with an installed test, once Server shipsactivity_heartbeat_resume. Owner: server, then sdk-php.rmcdaniel commentedon Nov 12, 2026
Server: implemented in durable-workflow/server#1151. Server advertises
activity_heartbeat_resume {supported: true, version: 1}. A retry claim by a worker listing that capability carrieslast_heartbeat_detailsandlast_heartbeat_at. Ships in native 3.6.0. Next: SDK read-only accessors (sdk-php, sdk-python, sdk-rust owners).rmcdaniel commentedon Nov 12, 2026
Server: merged in durable-workflow/server#1151, for the next native release after 3.5.0. A worker that lists
activity_heartbeat_resumein its registration capabilities getslast_heartbeat_detailsandlast_heartbeat_aton a retry attempt's claim. These carry the latest acknowledged heartbeat from earlier attempts of the same execution. Server advertisesactivity_heartbeat_resume: {supported: true, version: 1}in workerserver_capabilities. Next: SDK owners add the read-only accessor.rmcdaniel commentedon Nov 13, 2026
Decision (General, 2026-11-13): embedded mode is in scope. Embedded Workflow is a first-class mode, and the acceptance above already requires embedded PHP before we claim parity. Remaining work, one owner per part:
ActivityContextaccessor forlast_heartbeat_details/last_heartbeat_atwhen the Server advertisesactivity_heartbeat_resumev1, plus a test against a published Server 3.6.0+. When the capability isn't advertised, report unavailable, never an empty checkpoint.Until each box is ticked, docs mark that mode as unavailable. The issue closes when all five are done.
rmcdaniel commentedon Nov 14, 2026
Rust's concrete API/source slice is now draft sdk-rust#277, exact
e6fe082af0635da8cdb717f01dd2218e45af0601.It adds immutable acknowledged checkpoint/details/time accessors and typed decoding. A missing/unsupported contract is unavailable; an explicitly empty v1 claim is
Ok(None). Populated checkpoints retain the received execution, attempt, namespace and run boundary, and later accepted, stale, rejected or uncertain heartbeat responses never mutate that snapshot. Existing task DTOs and original retry/deadline semantics are preserved. Server v1 supplies a protocol version and acknowledgement time but no per-write sequence; the API states that limitation directly.Rust 1.86 focused transport/retry controls (12), the full library suite (862), doctests (20 passed, 1 declared skip) and formatting pass. These are synthetic/source checks, not installed replacement evidence. Ordinary peer review is recorded; required Security and terminal hosted checks remain before integration.
The Rust acceptance box remains open. Next is the ordinary package release and separately reviewed published-package qualification that kills the first worker after an acknowledged heartbeat, starts a cold replacement for the same execution with a distinct attempt, and verifies details/time, stale-write refusal, original budgets and retention. The runnable bounded-batch example and public documentation follow-through remain part of delivery. No installed runtime has been selected for this source change, and no shared or cross-binding completion is claimed.
rmcdaniel commentedon Nov 14, 2026
Python's read-only acknowledged-checkpoint source/API is merged in SDK333 at
edcd90f2580377a990ee17455172d9963dbebaee. Independent source review, peer review and all current head/main checks pass. The accessors distinguish unavailable support from supported empty state, bind the canonical claim, preserve typed independent details and original acknowledgement time, and never advance the received snapshot on heartbeat outcomes. Contract version1 is not a checkpoint write sequence.The Python acceptance box remains unticked: this API is unreleased, and the runnable bounded batch plus actual published-package cold-replacement/acknowledgement proof still need delivery. No new package or runtime observation was selected by this source merge.