Recover expired workflow leases while a remote activity remains leased

[rmcdaniel]

Customer outcome

Recover an expired workflow task while an older remote activity is still leased, so accepted workflow work can reach a replacement worker without waiting for the activity's separate five-minute lease.

Confirmed defect

Found while qualifying the real Python remote worker in shared cancellation work. Python head 88f0e31bf1736271deaabcc67d74df9ce98df491, Server 073a516bbd4063f57d4ad65ad732ec423131c8ff, published Workflow 2.3.0, MySQL 8.0 and Redis 7 reproduce it in candidate CI and a fresh isolated local stack.

1. Run the actual Python worker and block a remote activity without user heartbeats.
2. Kill its process, leaving its already-issued workflow long poll on the Server.
3. Request cooperative cancellation. That outstanding poll claims the new workflow task for the dead owner.
4. The configured ten-second workflow lease expires. A different replacement worker polls the same queue.
5. Server invokes the package's attemptRepair(), but the summary selects the older, still-leased activity. Repair repeatedly records repair_not_needed with activity_task_leased. The expired workflow task stays leased and the replacement receives no claim.

The local snapshot shows the activity lease expiring at 06:59:28.256350Z, while the workflow lease expired at 06:54:38.541218Z. At 06:54:44.542725Z that workflow task was still leased at attempt 1 and repair count 0. Four successive repair events reported repair_not_needed. The kill qualification fails before receiving the replacement workflow claim. This is a native recovery-selection defect, not evidence for increasing the fixture timeout.

Work and verification

• Make an expired workflow lease visible to the native repair decision even when another task remains legitimately in flight.
• Preserve the existing activity owner, attempt, lease and history while reclaiming only the expired workflow task.
• Cover the fresh-lease case, replacement claim fencing and the combination of an active remote activity with pending workflow work.
• Run the Workflow quality cycle and supported database gates. Rerun the real Python SIGKILL scenario against the exact corrected package and Server source.
• Publish the qualified patch through the normal artifact contract and update dependent qualification. Candidate cooperative protocol activation remains a separate shared release gate.

Delivered and verified artifacts

The native correction merged in #600 and shipped as Workflow 2.3.1, exact source fb3f3e59a4342fdebf8ced6160798906c3ee4387. Supported database/quality gates and published package/embedded-upgrade verification pass. Server 2.4.35 incorporates that package at published index sha256:49560f7f861271931125348a9d01638cd722e13ee976b5b732ef122548f15dab. Its release retains the unchanged ordinary-protocol before/after recovery probe and all twelve passing published PHP/Python/Rust lifecycle cells with hashes and raw artifacts. The corrected actual Python SIGKILL candidate scenario also passes against the pinned corrected native source.

Sample App's main app and portable tuple were updated in #129, with bounded ARM64 publication repair #130. #132 completes the secondary Laravel microservice and bundled CLI correction. Its protected publication 36840788892 passes both native builds and anonymous published startup qualification. Actual contents on AMD64 and ARM64 confirm Workflow 2.3.1 in both Laravel apps and CLI 2.1.3. GHCR/Docker Hub main and immutable indexes all resolve anonymously to sha256:671498a741b86386cdebf603479e0c87f5f2777ebb7a86c6bfc76f33bd4a5977 at merged source 11ee4c7143a37cbfc22981af547120e293339933.

The final published consumer gate is complete. All merged task branches were deleted and their GitHub absence verified. Cooperative protocol activation remains the separate shared published-artifact gate in durable-workflow/.github#136. Cloud qualification stays in its private owning record. No production or customer namespace is used in this qualification.

[rmcdaniel]

The isolated reproduction identifies the failure as Workflow #599. A still-issued workflow poll claims the new task for the killed process. Native repair then hides that expired workflow lease behind the older activity lease and repeatedly returns repair_not_needed.

A native regression using the actual workflow and activity bridges fails on the published baseline, while its fresh-lease case passes. Prioritizing expired leases in the native run summary makes both cases pass, with 33 assertions checking that only the workflow claim changes and the activity task, execution, attempt and history stay unchanged.

The unchanged Python SIGKILL case passes in 18.28 seconds when the isolated Server loads that corrected source. Its 30-second claim bound is unchanged. The replacement workflow task has attempt 2 and repair count 1, commits canonical delivery/cleanup, and the dead activity owner's late completion and failure are rejected. This local source result is the counterfactual for the defect. The full cooperative source suite, Workflow quality cycle, patch publication and exact dependent candidate gate are still running or pending. Protocol 1.20 remains opt-in.

Next action: qualify and publish the Workflow repair patch, update the Server candidate to that exact package, then rerun Python's entire connected candidate suite and retire the local qualification resources.

[rmcdaniel]

The full Python cooperative source suite now passes 18/18 cases in 109.18 seconds, including the unchanged SIGKILL recovery drill. The isolated Server uses candidate 073a516bbd4063f57d4ad65ad732ec423131c8ff with the corrected native RunSummaryProjector loaded readonly. Python remains at 88f0e31bf1736271deaabcc67d74df9ce98df491. The corrected file SHA-256 is 7344a3b2d905aba37868abfb5f863e9ba4f24999c00a71bc6371d2ef37cca639.

The package regression separately passes both expired and fresh workflow-lease cases with 33 assertions. ECS and PHPStan pass. The remaining native unit, coverage, feature and supported-database gates are in progress. Patch publication and the exact dependent Server/Python candidate run remain the next release steps. This result has no default protocol activation or production deployment.

The completed local HTTP/queue-worker/MySQL drill containers are being removed now. Its Redis service remains only for the active native quality checks and will be removed at that qualification's handoff, together with task images, worktrees and scratch evidence after the results are retained here.

[rmcdaniel]

Workflow 2.3.1 is published and verified at immutable source fb3f3e59a4342fdebf8ced6160798906c3ee4387.

• Merged PR #600 retains the exact qualified source tree. Its remote head branch is deleted.
• Full source qualification passes supported database, Laravel compatibility, quality, corpus and coverage gates. Both new lease-state cases pass on MySQL and PostgreSQL. Combined coverage is 88.25% against 87.85%.
• Packagist's source and dist references match the immutable release commit. Published verification passes package identity, platform/protocol resolver contracts and all 16 supported Laravel/PHP upgrade combinations.
• The release permanently retains the scoped native and Python candidate counterfactual JUnit files. Downloaded XML assets match the uploaded hashes. Native corrected JUnit SHA256 is bba0ed02d494fdd998cb9e51e85fa370599d1f69d83f6afe5f28df4dd5c894a8 and Python SIGKILL JUnit is 815d03a0ef22aba19726b2f2329d73c590260f0622b0f3df4d52937ee8f3dd73.

The self-hosted follow-through is Server PR #292, exact head e27db3096ca59a25b5a5fce88757d5ca31fb2daf. It changes only the Workflow dependency graph entry and prepares Server 2.4.35 / chart 0.1.131. Source contracts and chart rendering pass; feature, chart install, HTTP topology, database rolling-upgrade and bounded polling gates are active.

Keep this issue open. Next qualify/publish that image, verify the installed engine and focused recovery path plus published PHP/Python/Rust lifecycle tuple, then update Sample App. Waterline's exact Workflow pin is development-only, and unchanged SDKs need no alignment release. Update the cooperative Server/Python candidate tuple after this corrected engine adoption. No Cloud deployment or protocol-default change is part of this release.

[rmcdaniel]

The focused check also reproduces #599 through published Server 2.4.34 and PHP SDK 2.1.6, using ordinary protocol 1.19 and a signal rather than the cooperative candidate. It leaves the accepted signal's workflow claim unacknowledged while an older remote activity remains leased. There is no database lease edit, source mount, forced repair or extended fixture timeout.

Exact image: durableworkflow/server@sha256:1b660a7228be15a777b46dfb720ba85b083cf8ee70d6d1cd29853118bb1b2715, source d9b1a5610b8a53d6a2ed7fe6c97c38e6f7283348, embedded Workflow 2.3.0 at bce4365d76b3e5ff3a9f765f8a7831e6997e5da7. MySQL is sha256:679e7e924f38a3cbb62a3d7df32924b83f7321a602d3f9f967c01b3df18495d6 and Redis is sha256:0637954999d01b7c9ce9167db2da50656e2590d3b884f1c600c5f63bb6e6773c. Workflow lease is eight seconds, activity lease is 300 seconds, dispatch is poll, and memory/swap limits are equal. The SDK is freshly installed from its exact published package.

The replacement receives no claim within 20 seconds after workflow expiry. Four native repair events report repair_not_needed, matching the source root cause. Total drill time is 31.16 seconds. The completed baseline HTTP/database/Redis stack and network are being removed immediately. The same probe, configuration and bound will run against the corrected published image.

Server PR #292's full feature gate passes 2,252 tests / 47,202 assertions with unchanged worker OpenAPI version 23. Repository contracts, MySQL HTTP topology, both database rolling-upgrade checks and chart rendering pass. Final chart install and polling smoke are the remaining source gates. After they pass, merge and publish Server 2.4.35, verify its exact installed native source, rerun this focused check and the published PHP/Python/Rust lifecycle tuple, and finish Sample App adoption before closing #599.

[rmcdaniel]

Published native recovery delivery verified

Server 2.4.35 is published from d0692b171a0cabe6eefdc8bfc0f82d249e16128d, with Docker Hub and GHCR index sha256:49560f7f861271931125348a9d01638cd722e13ee976b5b732ef122548f15dab. Protected publication passes AMD64/ARM64 image identity, first readiness, source-free Compose, protocol convergence and Helm 0.1.131 anonymous pull/install. The actual installed image contains Workflow 2.3.1 at upstream fb3f3e59a4342fdebf8ced6160798906c3ee4387 and the corrected projector bytes. All three provenance copies match. Default protocol remains 1.19.

The identical ordinary-protocol published PHP probe fails on untouched Server 2.4.34 and passes on 2.4.35. After natural workflow lease expiry, a replacement reclaims the same workflow task at the next attempt while the older active activity remains unchanged. Stale workflow completion is rejected, the original activity owner completes successfully, and history has exactly one activity completion and one workflow completion. No database lease edits or installed-code patches are used.

Published lifecycle run 36832820422 passes all 12 cells against Workflow 2.3.1, Server 2.4.35, PHP SDK 2.1.6, Python SDK 2.3.7, Rust SDK 2.1.2, CLI 2.1.3 and Waterline 2.0.9. There are no blocked or unproven cells.

The release permanently retains publication/chart evidence, the installed-image assertion probe, before/after raw recovery JSON, the identical PHP reproducer, full lifecycle evidence, a report with the frozen tuple and commands, and SHA256SUMS. Every downloaded release asset matches its upload hash. Completed local test stacks, anonymous volumes, images, SDK dependencies and merged native/stable Server branches are removed.

Sample App adoption remains in PR #129 through its exact-head gates and any devcontainer publication. The new cooperative protocol is still a separate draft qualification under shared #136.

[rmcdaniel]

Development-image follow-through needs the bounded publisher correction

The adopted source/main consumers pass. Image publication run 36833874541 completed AMD64 and completed ARM64's build/startup smoke, but ARM64's 20-minute cap cancelled the job while it was uploading image layers. Assembly, architecture qualification and main promotion remained skipped, so the downstream published development image is not yet verified. The prior successful identical recipe took approximately five minutes per architecture.

Sample App PR #130 keeps AMD64's 20-minute cap and gives ARM64 a bounded 30-minute cap, with the same runners, image contents, immutable identities, native qualification and promotion gates. Next action is finish its normal checks, merge the reviewed configuration change and complete/verify the exact publisher. The native issue stays open through that result. No local image stack or paid host is being held for this wait.

[rmcdaniel]

ARM64 publication repair sample-app#130 merged as 605a0b5524affeedf0c6782ebb3a07c2f16d8df2 after all source gates passed. The merged tree matches reviewed head 96d61e7c68d101862d56e12b36d0b246cb7ec514. Both merged head branches are deleted, and the remote branch's absence is verified.

Protected main publication 36836885653 is rebuilding both architectures. Next action is verify native AMD64/ARM64 qualification, immutable index identity, anonymous registry availability and promoted package pins. Workflow 2.3.1 and Server 2.4.35 already have their exact published-artifact recovery and twelve PHP/Python/Rust lifecycle receipts. This final affected Sample App image outcome remains open until publication is verified.

[rmcdaniel]

Delivered and verified

The recovery fix is complete across the affected published public consumers.

• Workflow #600 ships as Workflow 2.3.1, source fb3f3e59a4342fdebf8ced6160798906c3ee4387. Supported native database/Laravel/quality checks and all 16 published package-upgrade combinations pass. The release retains the regression and corrected real Python SIGKILL source evidence.
• Server #292 ships as Server 2.4.35, index sha256:49560f7f861271931125348a9d01638cd722e13ee976b5b732ef122548f15dab, with Helm 0.1.131. The release permanently retains the unchanged ordinary-protocol before/after recovery probe, installed-image identity, all twelve passing exact published PHP/Python/Rust lifecycle cells and raw artifacts/checksums. The corrected image reclaims the expired workflow task while preserving the active activity owner/attempt/history, fences the old workflow completion and commits exactly one completion for each task. The baseline fails the same bounded probe.
• Sample App #132 is merged and protected publication 36840788892 passes both native builds and anonymous published startup qualification. Actual contents on both architectures confirm Workflow 2.3.1 in both Laravel apps and CLI 2.1.3. GHCR/Docker Hub main and immutable sha-11ee4c7143a37cbfc22981af547120e293339933-run-36840788892-1 all resolve anonymously to sha256:671498a741b86386cdebf603479e0c87f5f2777ebb7a86c6bfc76f33bd4a5977. The detailed consumer receipt is retained on Sample App Queue connection config #131/Emit exception #132.

All merged task branches were deleted and their GitHub absence verified. Public qualification uses isolated stacks and exact published artifacts. Cooperative protocol activation remains open in durable-workflow/.github#136, and Cloud qualification remains in its private owning record. Closing this native recovery defect after the final published consumer verification.