Repository navigation
Complete Python and Rust worker-affinity parity #124
Description
Current support
Python local activities shipped in SDK 2.2.0, and typed worker sessions shipped in SDK 2.3.0. The published-package qualifications and exact artifacts are recorded in sdk-python#69, sdk-python#71 and the comments below. These capabilities remain present in SDK 2.5.0.
Python bounded sticky execution shipped in SDK 2.5.0 with Server 2.5.10 or newer. Enable it explicitly with positive sticky_cache_capacity, a retained history byte limit and TTL. The installed PyPI wheel passes seven connected scenarios covering paged warm reuse, entry/byte bounds, actual expiry, build refusal and graceful/SIGKILL replacement with cold replay. The tagged example reports two hits and one miss. The exact PHP 2.2.4 / Python 2.5.0 / Rust 3.3.3 tuple passes all 12 published lifecycle cells and 236 assertions in the original 30-second mixed cancellation cascade. The Python guide and runnable example and English/Ukrainian service-mode support matrix are deployed and verified through Site #176. Sample #160 selects this qualified tuple. Its prepared images are published to both registries and pass native AMD64/ARM64 qualification in run 37750483536.
Rust local activities shipped in SDK 3.1.0 with explicit Worker::local_activities(true) opt-in. The installed stable crate passed nine real Server local-activity cases, the complete published lifecycle matrix and the 236-assertion mixed-language cancellation cascade. The exact artifacts and commands are recorded in sdk-rust#65.
Rust typed worker sessions shipped in SDK 3.2.0, with explicit Worker::worker_sessions(true) opt-in. The fresh installed stable crate passes nine local-activity and six session scenarios against published Server 2.5.1 / Native 2.4.1, including physical worker SIGKILL, resource rebuilding, original TTL expiry and stale completion refusal. The complete published lifecycle matrix and all 236 mixed cancellation assertions pass. Exact artifacts and commands are recorded in sdk-rust#67.
The Rust guide and support matrix are deployed and verified through Site #160. The tagged runnable example and generated session API types are available. Session resources remain process-local and must be rebuilt after holder loss.
Rust SDK 3.4.0 adds bounded sticky execution through Worker::sticky_cache(StickyCacheOptions). Python and Rust caching are disabled by default and advertise support only when explicitly enabled. Entries use exact workflow/run/build identity, immutable encoded history, entry/byte bounds and TTL. Current-lease history validation preserves full cold replay after expiry, eviction, build mismatch or worker replacement. Live workflow objects and session resources are not retained in the history cache.
The Embedded documentation correction is complete separately in durable-workflow/durable-workflow.github.io#125. Documentation changes do not complete this implementation issue.
Implementation scope
Reuse the existing Server worker-affinity protocol and PHP SDK semantics with language-native execution.
- Python local activity execution, recorded replay and published-package qualification.
- Python typed worker-session lifecycle and published-package qualification.
- Python bounded sticky execution with cold-replay fallback.
- Rust local activity execution, recorded replay and published-package qualification.
- Rust typed worker-session lifecycle and published-package qualification.
- Rust bounded sticky execution with cold-replay fallback.
- Qualify remaining capabilities against the public scenarios: local retries/timeouts/cancellation and restart; session expiry, holder loss, reacquisition and shutdown; sticky eviction, build mismatch, replacement and cold replay.
- Keep unsupported SDK versions fail-closed and advertise each new capability only after implementation and published-package qualification.
- Publish runnable examples on the language SDK sites and update the service-mode support matrix for each completed increment.
Final Rust delivery
Rust SDK 3.4.0 is published through release 37758414632, source a176576283b16addf64b24f6b6cceac59b0d9b4f, from merged sdk-rust #77. Final supported-toolchain/package/docs/replay CI passes. The complete MySQL source qualification passes all 9 local-activity, 6 session and 8 sticky scenarios with caching enabled, plus the runnable example.
Published crate checksum: 9152e7104f1430cf2abf8545559b78ea5d548453e773e55355ebda9bff4db1eb.
- Installed crate 37758778999 passes 9 local-activity, 6 session and 8 sticky scenarios with caching enabled. The runnable example returns 42 and reports one hit, two misses and zero retained terminal entries. Scenarios include actual paged tail-fetch reuse, capacity/byte limits, real TTL expiry, build refusal, graceful replacement, physical SIGKILL/same-ID cold recovery and cached canonical cancellation with cleanup on a cold successor.
- Published lifecycle 37758782227 passes all 12 cells with no unproven cells. Exact tuple: PHP SDK 2.2.4, Python SDK 2.5.0, Rust SDK 3.4.0, Server 2.5.10, Workflow 2.5.3, CLI 2.2.0 and Waterline 2.3.1. Server index:
sha256:2043e097d6eb411632386b001385cb5d8145dac1e3b5b14df901ac333d81c5ec. - Mixed cancellation 37758785370 passes all 236 assertions under the original 30-second root budget, including heartbeat-independent activity stops, stale fencing, worker SIGKILL, cleanup replay and immutable duplicate identity/deadline.
- Site #177 deploys the English/Ukrainian affinity matrix, Rust guide and qualified install tuple. Deployment and merged-main checks pass. All four normal guide URLs, the live tuple, generated Rust API docs and tagged example are verified.
- Organization #158 updates the coverage inventory. Sample #161, source
675d90e673d8d68c8b47ae19c85ee2143c2ded8e, updates the consumer tuple and both Rust manifests/locks. All nine merged-main workflows pass. - Native prepared image qualification 37760070587 passes anonymous AMD64/ARM64 pull and startup, then promotes main. GHCR and Docker Hub immutable tag
sha-675d90e673d8d68c8b47ae19c85ee2143c2ded8e-run-37760070587-1and main return the same indexsha256:ea8bca5f0b9a3c76372ad52ca9e7875cb7ebea870ddf123da63683fd4df7e724.
Reproduce the installed worker checks with gh workflow run ci.yml -R durable-workflow/sdk-rust --ref main -f local_sdk_version=3.4.0 -f local_qualification=true -f session_qualification=true -f sticky_qualification=true. The workflow pins the qualified Server image and retains scenario logs, package provenance and crate checksums.
The Python/Rust affinity implementation and public delivery are complete. Prepared local cancellation supervision remains a separate profile governed by shared #136. Cloud qualification remains separate. The broader portable experiment audit continues in #122.
Activity
rmcdaniel commentedon Sep 24, 2026
Python local-activity implementation finding (2026-09-24): this is not a manifest flip.
sdk-pythonhas norecord_local_activitycommand or replay branch;replay()is synchronous, while normal Python activity handlers and heartbeat transport are async. PHP executes the registered handler within replay, encodes input before side effects, appends one terminal local record before later commands, then cold replay consumes persisted activity history. Server already accepts that wire command under the portable-affinity capability/protocol guard. Python also lacks the workflow-task lease-renewal client path needed while a local handler runs. A safe Python increment must bridge async handler execution without blocking polling, preserve the PHP wire/history semantics, and test retry/timeout/cancellation plus cold replay and uncertain acknowledgement before advertisinglocal_activities=true. Keep the capability fail-closed until those tests and published-package conformance pass. This is a technical handoff, not a completed capability.rmcdaniel commentedon Sep 24, 2026
Python local-activity parity is underway in sdk-python draft PR #68. The first commit prevents a PHP/Server local activity history event from being replayed as an ordinary remote activity; the second prerequisite is the lease-fenced workflow-task heartbeat client call. A new cold-history regression fixture runs through the official Replayer. Local Docker Python 3.12: 335 replay/client tests passed, focused corpus fixture passed, Ruff and mypy passed. The PR revision passed its regression-corpus, lint, Python 3.10/3.11/3.12, package, docs-build and public-boundary checks; integration is still running.
This is not Python local-activity support yet. The worker executor, attempt/heartbeat reports, retry/timeout/cancellation semantics, lease-loss handling, cold replay, and published-package conformance remain. The manifest stays
local_activities=false; do not merge the draft or treat it as capability parity until those contracts pass.rmcdaniel commentedon Sep 24, 2026
Incremental Python safety fix merged: sdk-python#68 at
178e966caaf1c4b7470e9432625eb833849da58e. Python now rejects Server local-activity history when workflow code attempts to replay it as an ordinary remote activity; the SDK also has the worker-scoped workflow-task lease heartbeat request. Python 3.10/3.11/3.12, integration, regression corpus, package, lint and public-boundary checks passed. This does not close #124: Python local-activity command/executor, failure and lease semantics, cold replay and published conformance remain, andlocal_activities=false.rmcdaniel commentedon Sep 24, 2026
Python local-activity execution is now in draft sdk-python#69. The worker path records attempts/heartbeats on
record_local_activity, fences workflow-task lease heartbeats, and cold-replays a committed local terminal event without rerunning the handler. Docker Python 3.12 replay/worker/corpus tests: 394 passed, 2 skipped; Ruff, mypy, public-boundary and regression-corpus checks passed locally. Still not supported for customers: manifest remainslocal_activities=false. The draft needs real Server conformance, lease/restart/failure and combined-command qualification before merge or an SDK release.rmcdaniel commentedon Sep 24, 2026
Python local-activity draft sdk-python#69: CI on 62a5ce7 passed 15 real-Server integration tests (one unrelated skip). Qualified success, transient retry, non-retryable failure, three timeout kinds, replacement-worker replay, accepted completion with its first acknowledgement lost, and cancellation of an in-flight run. On cancellation, Server invalidates the task lease; the next local heartbeat aborts and no local completion is recorded. An uncommitted completion after lease expiry remains unqualified.
local_activities=falsestays advertised. Still needed before merge/release: lease loss/reclaim, output/reference and combined-command behavior, plus published-package conformance.rmcdaniel commentedon Sep 24, 2026
Python local-activity parity update: sdk-python draft PR #69 now includes a real-Server worker-replacement test at 041ee0d. It executes the local activity, drops all workflow-task completion requests before Server receives them, verifies no ActivityCompleted event was committed, re-registers the same worker identity to reclaim the task, verifies the old attempt is rejected with workflow_task_attempt_mismatch, and completes on the replacement attempt. The activity executes twice (correct at-least-once behavior), while history contains one ActivityCompleted and one WorkflowCompleted. Local verification: 10/10 local-activity integration tests passed against isolated Server; 1,419 other unit tests passed, plus 2/2 Git-dependent scanner tests in a Git-equipped container. CI is running. This does not claim lease-timeout expiry, output/reference coverage, or mixed-command coverage; Python still advertises local_activities=false and the PR remains draft.
rmcdaniel commentedon Sep 24, 2026
Python parity progress: sdk-python draft PR #69 HEAD efdd329 adds configured local-activity argument/result offload and cold replay of a recorded external result, plus a regression ensuring a failed local-result upload does not commit a false workflow failure. Public CI run 36034914713 passed. This is wire/replay and failure-boundary proof, not yet a real-Server shared-store acceptance test. Python still advertises local_activities=false; lease-timeout recovery, mixed-command behavior, and end-to-end reference handling remain release gates.
rmcdaniel commentedon Sep 24, 2026
Python local-activity parity update: sdk-python#69 now has a real-Server external-payload reference test (95be461). It verifies runtime-mediated offload for local arguments and result, successful completion, and cold replay; all 11 local-activity integration tests pass. This is still an unpublished/draft capability (
local_activities=false). Remaining before enablement: lease-expiry recovery, mixed-command behavior, and cross-SDK conformance.rmcdaniel commentedon Sep 24, 2026
Python local-activity qualification advanced in sdk-python#69 commit 51bb069: real-Server mixed local/remote command replay passes; direct and runtime-mediated payload upload failures after local execution no longer turn into a terminal workflow failure or immediate task-failure loop. Public CI is running on this revision. Remaining: natural lease-expiry recovery and cross-SDK parity before advertising the capability.
rmcdaniel commentedon Sep 24, 2026
Python local activities are released in sdk-python 2.2.0 via PR #69. The normal Worker now advertises
local_activities=trueonly with an implementation ofrecord_local_activity; registered local handlers execute with bounded retry/timeout, task-lease fencing, durable terminal history, and cold replay. The PyPI-installed wheel passed 19 real-Server integration tests against published Server 2.4.0, with one optional CLI skip. The release PR records the exact wheel/image digests and live reference page. This completes the Python local-activity subtask only; Python worker sessions and sticky execution, and all three Rust capabilities, remain open under this issue. Cloud customer-runtime qualification for this SDK version has not been performed.rmcdaniel commentedon Sep 24, 2026
Python portion update (2026-09-24): local activities are already published, and typed worker sessions shipped in sdk-python 2.3.0 via PR #71. The 2.3.0 wheel installed from canonical PyPI passed 16 session/local-activity integration cases against the published Server 2.4.2 image; source CI passed Python 3.10/3.11/3.12 and Server integration. Sticky execution and all Rust capabilities remain open. Managed Cloud-cell qualification is separate; no Cloud session claim is made by this public release. The install/restart rehearsal surfaced an independent Workflow local-activity failure-payload defect, tracked in workflow#544. Do not close this parity issue until the remaining capabilities and published-artifact qualification are done.
rmcdaniel commentedon Oct 6, 2026
Rust local activities are implemented in merged sdk-rust#63. The exact crates.io 3.1.0-rc.1 consumer passes nine real Server cases on Rust 1.86, including committed/uncommitted recovery, actual SIGKILL and natural reclaim, stale publication refusal, timeouts, shutdown, cancellation without application heartbeats, external payloads and mixed commands: https://github.com/durable-workflow/sdk-rust/actions/runs/37451236677. Published mixed-language cancellation also retains all 236 assertions: https://github.com/durable-workflow/sdk-php/actions/runs/37450958076. Stable promotion is sdk-rust#65, with documentation in Site #159. Keep this issue open for sessions and sticky execution.
rmcdaniel commentedon Oct 6, 2026
Rust local activities are released in SDK 3.1.0. Enable them with
Worker::local_activities(true). The runnable example, API and support matrix are published and live verified.The exact crates.io archive is SHA-256
94de8ff705bd1a5192feed48391fd0eac75c36cf8e4beac6709a02ea1460c9a7, source5664fc4405d370b8f202f233e1809e4deffc1535. The fresh Rust 1.86 consumer and published-package checks passed:These use published Server 2.5.0 and Native 2.4.0. Async local handlers must yield to Tokio, and effects before durable acknowledgement must be idempotent. Prepared cooperative local supervision is a separate worker profile.
This completes the Rust local activity increment. The issue stays open for Rust sessions and sticky execution, and Python sticky execution. Next is typed Rust worker sessions with expiry, holder loss, cold reconstruction and shutdown qualification.
rmcdaniel commentedon Oct 8, 2026
Python 2.5.0 is published and its consumer delivery is verified.
The installed PyPI wheel passes all seven sticky scenarios against published
Server 2.5.10. The tagged runnable example reports two hits and one miss.
The exact PHP 2.2.4 / Python 2.5.0 / Rust 3.3.3 tuple passes all 12 published
lifecycle cells and the original 30-second cancellation cascade's 236 assertions.
The Python SDK guide and English/Ukrainian support matrix are live.
Site #176, organization #157 and Sample #160 are merged. Sample's complete
main scenarios pass. Its prepared image passes native AMD64/ARM64 anonymous
pull and startup qualification, and both registries' immutable and main
manifests match.
Prepared image tag:
sha-71f798e27a71fc6f1aa918e1e9be0a0dd30d99ab-run-37750483536-1Index:
sha256:0def41fa5e55db05908528e9f314cd42f1f0a486cea29c121a4ab1b622a3a631Published lifecycle
· Published cancellation
· Prepared image qualification
This issue remains open for Rust sticky execution. Its next increment is the
bounded history cache, current-claim page validation and cold fallback, followed
by published-crate recovery qualification and the Rust runnable example.