Implement service cooperative cancellation in Rust - #55
Merged
Merged
Conversation
Member
Author
Rust canonical history foundation and retained worker fixtureCurrent draft head is Local Rust 1.86 passes all 20 new request/history cases plus the existing cooperative saga case, formatting and the official Worker replay corpus. A new retained Request foundation Next action: connect canonical delivery to the actual authored replay boundary and shielded cleanup/saga flow, add claim/heartbeat observation and fenced delivery/refresh, then qualify actual remote callback lifetime, shutdown and cold owner/attempt reclaim against the exact Server candidate. This PR remains a draft. Protocol 1.19 and ordinary worker registration stay unchanged, and no Rust cooperative worker or published tuple capability claim is made yet. |
Member
Author
Rust canonical history foundation and retained worker fixtureCurrent draft head is Local Rust 1.86 passes all 20 new request/history cases plus the existing cooperative saga case, formatting and the official Worker replay corpus. A new retained Request foundation Next action: connect canonical delivery to the actual authored replay boundary and shielded cleanup/saga flow, add claim/heartbeat observation and fenced delivery/refresh, then qualify actual remote callback lifetime, shutdown and cold owner/attempt reclaim against the exact Server candidate. This PR remains a draft. Protocol 1.19 and ordinary worker registration stay unchanged, and no Rust cooperative worker or published tuple capability claim is made yet. |
Member
Author
Rust actual scalar replay and cleanupDraft #55 now has scalar replay implementation at The typed cancellation retains request and delivery metadata while the existing unit cancellation API remains available. Nested cleanup guards restore checks on drop and do not extend the Server deadline. Saga compensation is shielded and preserves the initiating cancellation through completed cleanup. A canonical request plus task flag without a committed delivery remains pending. Physical reopened conditions consume one authored occurrence, and adjacent logical conditions retain their separate cursors and scopes. Local Rust 1.86 passed 255 library tests and 24 integration/consumer/corpus tests. Final formatting and the actual official Worker corpus pass after workflow preflight was centralized in state construction. Twelve new scalar cases exercise five actual Worker call kinds with fresh cold contexts, recorded-call mismatch, request-only state, earlier completed forward results, completed and pending saga cleanup, physical condition reopens, adjacent conditions, late resolution, malformed/unconsumed boundaries and nested shields. A retained Next action: implement actual parallel/selection replay boundaries, then claim/heartbeat observation, fenced delivery/refresh and real remote callback lifetime, shutdown and cold owner/attempt replacement against the exact Server candidate. Keep this draft through the remaining per-language and exact published-tuple gates. Rust workers still use protocol 1.19 and advertise their existing capabilities. |
Member
Author
Rust actual scalar replay and cleanupDraft #55 now has scalar replay implementation at The typed cancellation retains request and delivery metadata while the existing unit cancellation API remains available. Nested cleanup guards restore checks on drop and do not extend the Server deadline. Saga compensation is shielded and preserves the initiating cancellation through completed cleanup. A canonical request plus task flag without a committed delivery remains pending. Physical reopened conditions consume one authored occurrence, and adjacent logical conditions retain their separate cursors and scopes. Local Rust 1.86 passed 255 library tests and 24 integration/consumer/corpus tests. Final formatting and the actual official Worker corpus pass after workflow preflight was centralized in state construction. Twelve new scalar cases exercise five actual Worker call kinds with fresh cold contexts, recorded-call mismatch, request-only state, earlier completed forward results, completed and pending saga cleanup, physical condition reopens, adjacent conditions, late resolution, malformed/unconsumed boundaries and nested shields. A retained Next action: implement actual parallel/selection replay boundaries, then claim/heartbeat observation, fenced delivery/refresh and real remote callback lifetime, shutdown and cold owner/attempt replacement against the exact Server candidate. Keep this draft through the remaining per-language and exact published-tuple gates. Rust workers still use protocol 1.19 and advertise their existing capabilities. |
Member
Author
|
Selection-handle replay is implemented at The actual Worker preserves its recorded winner, then interrupts the authored loser await with the original request/deadline and operation range. Matching the durable member metadata and resource identity rejects changed public handle fields. Late completion retains cancellation priority, request-only state stays pending, and changed/skipped/scalar-replaced boundaries or newly shielded delivery fail replay. Saga cleanup runs as the next durable command. Local Rust 1.86 formatting and all targets/features pass: 262 library cases and 24 integration/consumer/corpus cases. Seven new selection cases and a retained official-Worker fixture cover this addition. The fixture uses the existing selection workflow consumer without modifying its harness. Normal exact-head CI now qualifies the expanded corpus and baseline negative controls. Next implement actual parallel/selection-group delivery, then transport and callback lifetime/shutdown/cold-owner qualification against the exact Server candidate. This remains source-only draft work. Protocol 1.19 remains default and cooperative worker capability remains inactive. |
Member
Author
|
Authored parallel and selection-group cancellation replay is implemented at The group retains its whole authored span and validates every original durable leaf using the existing activity/child/timer/signal/condition future. Earlier partial results retain priority. Changed spans, nesting, later leaf details, scalar replacement and cleanup scope fail replay. The group returns the original typed cancellation directly, with no new work or wrapper failure replacing its identity. Late leaf failure and an uncommitted selection winner cannot replace delivery. Completed Saga cleanup consumes the durable slot after the entire group. Local Rust 1.86 formatting and all targets/features pass: 270 library cases and 24 integration/consumer/corpus cases. Eight new group cases and a retained cold-group fixture cover this addition; the fixture uses the unchanged existing official Worker consumer. Exact-head normal CI is running. The preceding selection head passes all normal CI, including corpus growth 18→21 and all three negative controls against unchanged production source Next integrate claim/heartbeat observations, fenced delivery and bounded canonical refresh, then qualify actual remote callback lifetime, shutdown and cold owner replacement. Reopened group/selection conditions remain a real-Server scenario gate. Protocol 1.19 stays default and cooperative worker capability stays inactive throughout this draft work. |
Member
Author
|
All normal CI gates pass at current head The retained official-Worker replay corpus grows 18→22 and all four new fixtures fail on unchanged published-production source Next implement the fenced worker transport and bounded canonical refresh against Server candidate |
Member
Author
|
Fenced cancellation transport is implemented at Delivery sends the actual selected task, lease owner and attempt with worker credentials, and verifies the acknowledgment's durable run, original request and complete authored range. Canonical refresh uses the original opaque Server token, one five-second budget across all pages, bounded page sizes and a 128-page cap. Changed identity, malformed pages, missing or non-progressing tokens and token cycles are rejected. The caller's original snapshot and observation remain unchanged. A delivery acknowledgment alone does not authorize workflow-code cancellation. Local Rust 1.86 formatting and all targets/features pass: 278 library cases and 24 integration/consumer/corpus cases. Eight new HTTP transport cases cover the exact fence and credentials, malformed or changed acknowledgments, fresh opaque-token history, lost-acknowledgment canonical proof, bad pages/identity, cycles and page limits, invalid inputs and role credentials, ownership refusal and the shared budget across two slow pages. These are transport-level checks, not a real Server claim qualification. Next integrate actual claim/heartbeat observations and the worker's delivery, refresh and replay coordinator, then complete callback lifetime/publication fencing, shutdown, owner/attempt replacement and reopened group/selection condition qualification against Server candidate |
Member
Author
|
Private authored-call delivery intent is implemented at Given the actual claim, original observation and a proven canonical request, the registered Worker's replay suspends at the eligible authored scalar, parallel/select or selection-handle call. It exports private intent without returning an application-visible cancellation error. Preserve preceding commands, validate original metadata before proposing delivery, and reject commands authored beyond a suspended boundary. Late completion or an uncommitted selection winner cannot resume workflow code before delivery. A prior committed winner still replays normally before the loser await. Shields keep cleanup pending without extending the original deadline. Local Rust 1.86 formatting and all targets/features pass: 291 library cases and 24 integration/consumer/corpus cases. Thirteen new cases cover new and recorded calls for all five scalar kinds, late resolution, earlier commands and replayed side effects, committed delivery with the original observation, ignored pending calls, prohibited later commands, prior winner and pending/late loser, all seven changed public handle fields, new/scheduled/partially completed nested groups and uncommitted selection winners, later-leaf mismatch, physical condition reopening, shielded cleanup and preflight refusal before workflow code. The corpus retains all four controls against unchanged production source, with no consumer harness change. This private replay path is not yet wired to registration or the poll/heartbeat coordinator. Next connect the actual claim/heartbeat observation, fenced delivery, canonical proof and cold replay, then complete remote callback lifetime/publication fencing, shutdown, owner/attempt replacement and real-Server reopened group/selection condition gates. Ordinary workers retain protocol 1.19 and do not advertise cooperative capability. This PR remains a draft through per-language and exact published-tuple qualification. |
Member
Author
Rust heartbeat source qualificationSDK Rust draft #55 head Explicit protocol 1.20 heartbeat sends the actual task, owner and attempt with worker credentials and one five-second budget. It accepts a positive renewal only for that exact claim, a leased task, valid expiry, active run and null reason. Repeated observations preserve the original cancellation request identity, requested timestamp, cleanup deadline and opaque refresh token. The five new HTTP cases cover no/new observations, equivalent timestamps with original text/token retained, malformed or changed acknowledgments and request identity, rejected input before transport, ownership refusal and request budget expiry. While inspecting the connected Server gate, a controlled two-process interleaving reproduced an existing heartbeat race in Server #291 head These are source and HTTP transport results. Ordinary Rust workers still use protocol 1.19 and do not advertise cooperative cancellation. Real worker orchestration, callback lifetime, shutdown/replacement, reopened-condition and exact published-tuple gates remain required. |
Member
Author
|
Cooperative claim acquisition and heartbeat observation carrier is implemented at The explicit protocol 1.20 poll retains the actual task owner, attempt and original cancellation observation in a read-only companion type. Existing public WorkflowTask fields and ordinary protocol 1.19 Worker behavior are unchanged. History pages use that exact claim and encoded task path. Unreadable poll replies retry with the same acquisition ID. Poll and paginated history share one long-poll budget plus five seconds, bounded to 128 pages. Claim heartbeats retain the first request identity, timestamps, cleanup deadline and opaque refresh token. A request first seen in heartbeat is captured. Invalid or refused renewal cannot mutate the original claim or observation. A task flag or observation alone exposes no cancellation error to application code. Local Rust 1.86 container qualification on 2026-10-01 13:40–13:42 UTC passes formatting and Server #293 is delivered in Server 2.4.36 with exact published before/after regression and all 12 published lifecycle cells passing. Server cancellation draft #291 head Next action: connect this actual claim carrier to private delivery intent, canonical refresh/proof and replay before application cancellation is exposed. Preserve pending command prefixes, then qualify actual callback ownership and physical lifetime, shutdown/replacement and reopened condition/selection/group history against the real Server candidate. These HTTP transport tests are source qualification. No capability activation, published cooperative tuple claim or stable release is made. |
Member
Author
|
Implemented and qualified the Rust Worker delivery coordinator at The actual managed poll path preserves one acquisition ID across retry and retains the Server's exact immutable owner and attempt. With an original pending observation, it refreshes canonical history before replay. Earlier pending commands commit on the predecessor claim. Its successor replays their durable results before delivery. Without a prefix, delivery is followed by canonical refresh and an exact intent match before workflow code can observe cancellation, including after a lost acknowledgment. Seven new HTTP cases exercise that actual Worker path. They cover ordering, canonical proof, refused or changed history, ownership loss, prefix completion and successor replay, shielded waiting, typed unhandled cancellation, Saga cleanup replay by a fresh Worker, and poll acquisition identity. The terminal command uses the existing typed non-retryable Qualification on 2026-10-01:
PR #55 remains a draft. Ordinary Rust Workers retain protocol 1.19 and no cooperative capability. The next work is actual activity callback lifetime and publication fencing, shutdown and owner/attempt replacement, followed by qualified registration, real Server scenarios, and the exact published cross-language tuple. |
Member
Author
|
Implemented and qualified activity callback ownership and lifetime fencing at The gated actual Rust Worker activity path checks the original owner and activity attempt before invoking a callback, around authored heartbeats, and before success or failure publication. Status uses explicit protocol 1.20, worker credentials, the encoded task path and one five-second budget. It renews neither leases nor progress. Continuation requires a leased task, running attempt and activity, valid execution deadlines, and any required active worker session. Callback contexts share irreversible abandonment. Ownership loss, delivered cancellation, failed observation, shutdown or a dropped poll future prevents subsequent result, failure and heartbeat publication. A genuine application failure still publishes through the existing failure path while the claim remains valid. Pending asynchronous callback futures are dropped on ownership loss and shutdown before deregistration. Ten new HTTP cases exercise the actual Worker path, including exact transport/role/namespace, malformed and changed acknowledgments, missing or conflicting claim fields, expired lease and application deadlines, session replacement, backend interruption, heartbeat refusal, valid completion/failure, suppressed late outcomes, changed public context fields, cloned contexts after abandonment, callback Drop, actual Worker shutdown/deregistration, poll-future cancellation and the five-second observation budget. Qualification on 2026-10-01:
Next: integrate compatible protocol 1.20 registration, then qualify against actual Server #291 with cold claim replacement and reopened condition/group/selection scenarios. Complete the accepted per-language and exact published tuple gates before capability activation. PR #55 remains a draft and ordinary Workers retain protocol 1.19 without cooperative capability. Rust's existing local-activity and worker-affinity refusals remain explicit. |
Member
Author
Actual Server qualification passed for explicit Rust WorkersRust The connected run builds the official Dockerfile from exact Server The side-effect case commits its prefix, drops the original Worker, and registers a different Worker that replays the recorded result before delivery at sequence 2. Saga cleanup also runs under a different Worker, records its actual activity heartbeat and completion, and produces one canonical cancelled result under the original request. Every scenario checks request/delivery identity and rejects ordinary completion/failure history. The duplicate preserves the original cleanup deadline. This exposed and qualified the Server successor fix. Before that fix, the prefix was durable but the waiting run had no successor to deliver cancellation. Raw results, runtime image identity, and Native package provenance are retained for seven days. CI teardown removed its task containers, network, and image. The local reproduction stack, test images, and dependency caches have also been removed. Next: qualify actual managed-loop callback fencing and real process/attempt replacement, including the remaining group/selection wait cases, then the complete published tuple. The two fresh-Worker cases above do not establish process-kill recovery. The PR remains a draft. Ordinary published workers remain on protocol 1.19 without cooperative capability. |
Member
Author
Complete Rust scope-admission Source qualificationhttps://github.com/durable-workflow/sdk-rust/actions/runs/37122437842 passes completely at Exact Server Connected Source passes 17 cases, zero failures/ignored, one separately filtered fixture, in 368.50 seconds. It covers managed remote stop without application heartbeats, physical drop/join, cloned context and stale publication fences, scalar/parallel/selection delivery, metadata prefix before canonical delivery, child-first recovery, cold cleanup replay and deadline/termination interruption. Bounded remote Abandon retains its original finite total lifetime. Separate killed Activity-owner reclamation cases have their own longer recovery window, rather than a +30-second claim. The Rust cold workflow-cleanup fixture replaces a dropped Worker under the ten-second workflow lease. It is distinct from the actual actively leased PHP workflow-cleanup SIGKILL in the mixed PHP/Python/Rust source proof. The separately retained ordinary 60-second Server lease proof remains attributed to its earlier tuple. This source run alone is not the requested published mixed-language cascade. The new guard's six local tests and controlled official-consumer proof are retained at #55 (comment). They demonstrate prior application execution on unsupported scoped history and candidate refusal without a completion/failure HTTP request. Complete connected scenario logs and source/package/image provenance are retained here. The full run supersedes the earlier pending status. Scope body execution, scoped delivery and selective callback supervision remain next steps before protocol freeze. Rust 3.0.0 still follows the approved major release plan and exact published-artifact gates. Package version/default protocol remain unchanged, scope execution stays unadvertised, PR #55 remains draft and shared #136 remains open. No Cloud target or deployment changed. Archive part 1/2, 40,224 bytes total. SHA-256 |
Member
Author
Rust scope-admission connected Source evidenceContinuation of the retained raw source run 37122437842 archive. Archive part 2/2, 40,224 bytes total. SHA-256 |
Member
Author
Hosted scoped-origin source evidence, part 00 of 00-03Python CI 37168773035 and Rust CI 37168657510 completed successfully. PR heads are Python Archive: |
Member
Author
Hosted scoped-origin source evidence, part 01 of 00-03Python CI 37168773035 and Rust CI 37168657510 completed successfully. PR heads are Python Archive: |
Member
Author
Hosted scoped-origin source evidence, part 02 of 00-03Python CI 37168773035 and Rust CI 37168657510 completed successfully. PR heads are Python Archive: |
Member
Author
Hosted scoped-origin source evidence, part 03 of 00-03Python CI 37168773035 and Rust CI 37168657510 completed successfully. PR heads are Python Archive: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the Rust consumer for shared cancellation #136.
Supported release boundary
Whole-run cooperative requests, immutable context/deadline, Activity/Child
policies, callback-future supervision independent of application heartbeats,
physical stop acknowledgement, stale publication fencing and shielded cleanup
recovery. Ordinary workers remain protocol 1.19. Cooperation opts into 1.20.
General cancellable scopes remain a disabled source preview. Portable local
activities, worker sessions and sticky execution remain unsupported here.
Qualification and migration
Connected qualification
passes all 27 live cases, including actual remote callback stop and cleanup
SIGKILL/replay under the original deadline. The current metadata/release-tooling
head
eb4e4f4980521c50e28f9c0db789912b1a088e8apasses every ordinary gate,Rust 1.86/stable, package, release-entrypoint and docs/corpus checks.
The implementation also passes the required source mixed cascade.
Rust 3's public fields and enum variants require struct-literal/exhaustive-match
migration. The major-release decision is approved.
The migration guide and current-version release notes are included.
Publication
Publish 3.0.0-rc.1 for Server 2.5.0-rc.1 through the protected crates.io workflow,
including archive/provenance and fresh Rust 1.86 consumer verification.
Published mixed qualification and stable acceptance remain required before #136 closes.