Managed worker exits when a workflow-task report races a cancellation (409 run_closed)

Problem

A managed Rust worker exits when its workflow-task completion races a cancellation. Native Server 3.0.4 answers HTTP 409 with reason: run_closed and the worker's run() returns the error. Every later task on that worker then goes unprocessed.

Found by installed-native acceptance of published 4.11.1 against native 3.0.4 (linux/amd64, lab3). Every lifecycle case completed correctly, but same_worker_running_after_cases was false. The worker's last line was:

Error: Http { status: 409, body: {"task_id":"01M6HD5ST1R1VXKMXBZ9JTBS3Z","workflow_task_attempt":1,"error":"Workflow run is already closed.","reason":"run_closed", ..., "stop_reason":"run_cancelled","cancel_requested":true,"can_continue":false,"run_status":"cancelled","run_closed_reason":"cancelled","task_status":"cancelled","lease_owner":"qualification-installed"} }

It is timing-dependent. The same cell passed for 4.11.0 on arm64.

Why it isn't covered

Keep Rust workers polling after matching workflow-failure timeout receipts #146 settles a matching terminal run_timed_out receipt.
Keep managed workers polling after a non-recorded task_not_leased receipt #159 settles a matching non-recorded task_not_leased receipt.
run_closed after cancellation, termination or completion by another path is neither.
The native receipt also lacks recorded and run_id, the same shape gap as server#994.

Expected

A workflow-task completion or failure report that receives a terminal-run 409 for the same task and attempt settles, and the worker keeps polling. The run is already closed, so redelivery can't help and nothing is lost.
Match exact fields only: same task_id and workflow_task_attempt, a terminal reason (run_closed), and run_id when present. Other 409s still surface, and direct client calls keep the original error.
Coordinate the receipt shape with server#994: native should send recorded:false and run_id on every not-recorded refusal, including run_closed. Decide whether the SDK requires them or accepts the current native shape for run_closed.
Add a regression test for the cancel-during-report race, plus installed-native acceptance, which already catches it.

Security settlement contract for this defect:

Do not accept native 3.0.4's current incomplete run_closed body. A managed worker may settle only HTTP 409 with reason: run_closed, recorded === false, and exact matching task_id, workflow-task attempt, and run_id. Require the terminal run/stop state expected for the refusal. Missing fields do not count.
Coordinate with server#994 first so native emits that full not-recorded receipt for cancel/terminate/other terminal-run refusals. Do not infer recorded:false from the status or reason.
The settlement is managed-worker-only: it returns to polling without a retry loop. Direct completion/failure calls retain the original HTTP error. Every mismatched field, recorded receipt, nonterminal state, or other 409 remains fatal.
Tests must cover exact success plus each absent/mismatched field and terminal-state mismatch, prove the worker keeps polling after the exact receipt, and prove no outcome/history/retry/lease state is advanced by the client. The installed-native cancellation/report race and same-worker assertion are the release counterfactual.

4.11.1 surfaced this honestly and remains unqualified on the lifecycle cell. The fix belongs in a new patch release after the receipt contract exists; do not rewrite or reinterpret the immutable 4.11.1 result.

Security scope decision after reconciling published PHP 2.11.8 source, the shared late-completion corpus and sdk-rust#157's live receipt:

PHP has two paths. Its ownership guard emits task_not_leased / run_closed with exact task and attempt but without recorded or run_id. A rarer engine-outcome race after both guards pass emits the full not-recorded receipt seen in Managed worker exits on a non-recorded workflow-task 409 (task_not_leased) #157. Therefore Keep managed workers polling after a non-recorded task_not_leased receipt #159 remains a valid fix for the race it actually observed; it does not settle the smaller guard response.
For this issue, keep the settlement fail-closed: require recorded == false and exact task, attempt and run identity. Do not interpret a missing field as false or infer run identity from reason, stop_reason or task status. server#994 will provide that native receipt for Server 3.0.5, after which this issue needs its counterfactual and the installed-native lifecycle re-run.
Do not fold a narrower PHP-only body into this change. PHP Server 2.11.8 is frozen and its smaller guard response remains a compatibility gap, but accepting it safely would require an explicit, separately reviewed SDK product contract (including how the client binds the response to the published PHP implementation). General should track/prioritize that separately if support is desired.

This keeps the failed 4.11.1 lifecycle gate open until exact native qualification passes; the patch-specific cancellation-clock replay does not close that gate.

Product decision (General), with Server's PHP 2.11.8 path mapping (server#994 6613551175 and follow-up):

PHP 2.11.8 lease-guard path (guardWorkflowTaskOwnership, before the transaction and again under the task lock) returns {task_id, workflow_task_attempt, error, reason, task_status} with no recorded and no run_id. Deterministic late completions and cancel races take this path.
PHP 2.11.8 engine-outcome path (the engine reports completed:false after the guards passed) returns recorded:false and run_id (the Managed worker exits on a non-recorded workflow-task 409 (task_not_leased) #157 body). Keep managed workers polling after a non-recorded task_not_leased receipt #159 already settles it.

Decision: the SDK will not infer recorded:false or run_id from reason or status (Security contract #170 6613137060). So managed Rust workers on PHP 2.11.8 can still exit on the lease-guard path. There will be no further PHP Server releases (2.11.8 is final). The fix comes through native 3.x: server#994 makes native send the exact not-recorded receipt for complete and fail, duplicate and checkpoint reports across all terminal statuses. A new Rust patch will then settle exact run_closed receipts, and it must pass same-worker installed-native acceptance. Upgrade guidance for Rust users on PHP Server: move to native Server 3.x. The SDK docs will say so when the patch ships.

Reopened deliberately. The source fix merged via #176 at Security-cleared exact head 6ee0c1e (merge commit 33dfa74; parents e773b61 + 6ee0c1e; merge tree c7561548 equals head tree). Security's clearance (#176) covers source merge only.

This stays open until same-installed-worker lifecycle acceptance passes against published native Server 3.0.5. Until then:

no version bump;
no tag;
no crate publication;
no stable-manifest change;
no qualified claim.

Rust 4.11.0 stays the last fully qualified release.

Security gate-order ruling for the #170 / 4.11.2 release path: use the pre-publication artifact route. Do not tag or publish 4.11.2 before lifecycle acceptance.

The acceptance harness currently proves a registry-installed crate only after publication. Publishing first and yanking on failure would still leave an immutable, unqualified version in the public registry. That is not the intended gate.

Add a separately reviewed pre-publication mode with these properties:

Build cargo package from the exact clean release head after the 4.11.2 version/product-train/CHANGELOG changes are final. Bind the resulting .crate SHA-256, internal version, exact file set and VCS commit metadata to that head.
Run the same-installed-worker lifecycle acceptance against published native Server 3.0.5 using only the unpacked bytes of that exact local package. Keep the current server/index/manifest and same-worker assertions; no path dependency on other checkout bytes.
Bind publication to that accepted artifact. The release workflow must reproduce the package and refuse unless its checksum equals the accepted checksum before cargo publish (or publish the exact accepted artifact if the supported tooling permits it).
After publication, read back crates.io and require its checksum and source/tag identity to match the accepted artifact before the release is considered complete.
No tag, public release, stable-manifest change or qualified claim until the pre-publication acceptance passes. A failure remains a source defect, not a public-yank workflow.

The harness and workflow changes need their own exact-head review and counterfactuals, including a one-byte package drift, wrong VCS commit/version, extra/missing file, and a publish-time repackage mismatch.

The pre-publication installed-package acceptance required by this issue passed against exact published native Server 3.0.5 on arm64, including cancellation and later work on the same still-running worker. Both lifecycle/deadline histories and independent cleanup passed. The exact 4.11.2 package is bound to frozen source feb49025e28516f00377e3fe335e9276895f313b and checksum fbd4aeae15dc3ab0b677bf04fcdcc22b13e6cf18c93296e648172424b38d6d81.

Draft approval PR #197 holds the unchanged receipt and exact summary for Security review; #177 has the run handoff. This issue remains open through independent approval and publication/readback. Nothing is tagged or published, and the stable manifest remains at the previously qualified release.

Closing after the required publication and independent installed acceptance, rather than the earlier source merge alone.

The fix shipped in Rust SDK 4.11.2, with source feb49025e28516f00377e3fe335e9276895f313b and approved crate checksum fbd4aeae15dc3ab0b677bf04fcdcc22b13e6cf18c93296e648172424b38d6d81. Security accepted the registry-installed lifecycle/deadline cells against published native Server 3.0.5, including independent durable histories, the same installed worker surviving the cases, unchanged binary, and complete cleanup. General's final integration/readback accepted the release as shipped. Today's registry readback remains non-yanked with that exact checksum.

The source fix and refusal/mismatch controls are in #176. Settlement still requires the full exact non-recorded receipt. Native 3.0.4 and PHP Server 2.11.8's smaller lease-guard responses remain refused, as the explicit product/security decision above requires. Rust workers needing this recovery should use the fixed SDK with native Server 3.0.5 or later. Downstream pin adoption is recorded separately on #177; this closure does not claim that every consumer has upgraded.

Keep managed workers polling after an exact run_closed receipt

Add pre-publication installed-native acceptance and checksum-bound pu…

Navigation Menu

Repository navigation

Managed worker exits when a workflow-task report races a cancellation (409 run_closed) #170

Problem

Why it isn't covered

Expected

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Managed worker exits when a workflow-task report races a cancellation (409 run_closed) #170

Description

Problem

Why it isn't covered

Expected

Activity

rmcdaniel commented on Nov 3, 2026

rmcdaniel commented on Nov 3, 2026

rmcdaniel commented on Nov 3, 2026

rmcdaniel commented on Nov 3, 2026

rmcdaniel commented on Nov 3, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions