Fence workflow heartbeats against concurrent lease replacement

[rmcdaniel]

Customer problem

A workflow task heartbeat must renew only the caller's current owner and attempt. The current Server controller checks ownership before the separate Native renewal transaction. If Native reclaims an expired lease between those operations, the old heartbeat can renew the replacement lease and return a positive acknowledgment under the old owner and attempt.

A controlled two-process HTTP interleaving reproduced this against Server draft #291 at c32434cc784a1bf870dcf19568bef944a320f6a7 with published Native Workflow 2.3.1. The original guard passed, Native actually reclaimed the task to a replacement owner and attempt, and the original heartbeat returned HTTP 200, renewed: true, attempt 1 and the original owner. No lease row was manually changed. Current main has the same validation/renewal separation. This is a source reproduction, not a published-image or deployment claim.

Acceptance

• Lock the namespace-scoped task, validate current ownership and registration, and renew within the same transaction. Preserve existing response schemas, refusal reasons and lock-pressure handling.
• Add a deterministic two-process regression using actual Native claims and renewal. Verify that a positive acknowledgment matches the persisted owner and attempt, and that a concurrent poll cannot replace a claim renewed under the lock.
• Retain wrong-owner, wrong-attempt, expired lease, missing task, stale registration and backend-pressure coverage. Pass the owning repository's normal CI.
• Deliver the existing protocol fix independently of cooperative cancellation. Keep default protocol 1.19 and ordinary capabilities unchanged.
• Publish a patch image through reviewed Server changes and verify the exact artifact and affected published PHP/Python/Rust worker behavior. Record versions, digests and results before closing.

Next action: extract the locked heartbeat and ordinary-protocol concurrency regression onto current main, qualify it, and carry the same fix into #291 before Rust coordinator integration.

[rmcdaniel]

Heartbeat ownership interleaving reproduced and fixed at source

The ordinary protocol 1.19 regression fails against unchanged main d0692b171a0cabe6eefdc8bfc0f82d249e16128d with locked published Native Workflow 2.3.1. Its two HTTP kernels perform actual Native status, claim and renewal. The fixture only provides an IPC barrier and clocks crossing the one-second lease expiry. It does not edit lease rows or fabricate Native results.

Observed baseline:

{"heartbeat":{"workflow_task_attempt":1,"lease_owner":"original","renewed":true},"replacement_claim":{"workflow_task_attempt":2,"lease_owner":"replacement"}}

The same Native task ID was present in both responses. The positive heartbeat therefore identified a claim that no longer owned the task.

The fix in #294 head 226025ffe3e0c1af98a77c01932a50f41799d8d8 holds the namespace-scoped task lock across ownership validation and Native renewal. Local Pint and 79 focused protocol, ownership/error, success and poll-pressure cases pass with 4,286 assertions. The controlled race now retains the original durable owner/attempt and the replacement poll returns no task. Command: php vendor/bin/phpunit --filter 'WorkflowTaskHeartbeatRaceTest|WorkerProtocolOwnershipErrorContractTest|WorkerProtocolSuccessContractTest|SqliteWorkerPollLockPressureTest|WorkerPollBackpressureTest'.

Cooperative cancellation draft #291 carries the same locked renewal, plus the pending observation read inside that transaction, at 6be6cd39 (full exact revision available in its CI). Its 24 focused cancellation protocol and concurrency cases pass locally with 545 assertions, preserving the original request/deadline and one canonical request without inventing delivery.

Normal CI is running for both heads. This source result does not claim a published image or Cloud deployment. Next finish exact-head CI/review, publish Server 2.4.36/Helm 0.1.132 independently of cooperative cancellation and verify the affected published worker cells. Rust coordinator qualification then uses the requalified #291 head. Default protocol 1.19 and ordinary capabilities remain unchanged.

[rmcdaniel]

Published Server 2.4.35 reproduces the ownership defect

The same committed regression ran inside the unchanged published Server image durableworkflow/server@sha256:49560f7f861271931125348a9d01638cd722e13ee976b5b732ef122548f15dab with its own PHP 8.3.35, Laravel and Native runtime code. PHPUnit 11.5.55 and Mockery 1.6.12 were mounted as separate test tools, together with the repository's tests. No application or vendor file was replaced. Reflection receipts verify that the controller and Native bridge load from the artifact, and both artifact provenance records name Native 2.3.1 at fb3f3e59a4342fdebf8ced6160798906c3ee4387.

The two actual HTTP kernels reproduce:

{"heartbeat_status":200,"heartbeat":{"workflow_task_attempt":1,"lease_owner":"original","renewed":true,"reason":null},"replacement_claim":{"workflow_task_attempt":2,"lease_owner":"replacement"}}

Both responses identify the same actual Native task. The unchanged published image therefore fails the ownership assertion, with 16 assertions reached. SQLite clocks and an IPC barrier control the interleaving; the harness does not edit any lease, owner or attempt row.

The original full CI result exposed an overly strict test assertion, not a valid positive acknowledgment. PHP 8.3 uses deferred SQLite transactions even when transaction_mode is configured, allowing replacement to win. The fixed transaction retries and returns a correct owner-mismatch refusal. The regression now requires either a positive acknowledgment matching the durable original claim or that exact refusal together with proof of the replacement claim. The same change preserves the cancellation request and cleanup deadline in #291. This still fails the old published image.

Next complete revised-head CI and review, publish Server 2.4.36, repeat this exact-image regression and verify the published PHP/Python/Rust lifecycle tuple before closing #293.

[rmcdaniel]

Delivered in Server 2.4.36 through merged #294 at 898c51375ddcb2c25588eb63e41fed26a3fd61b2.

• The unchanged published Server 2.4.35 image fails the controlled ownership race. The unchanged published 2.4.36 image passes 1 case and 21 assertions, using its own controller and published Native 2.3.1 code.
• Protected image/chart publication passes both supported image architectures, bare-image readiness, source-free Compose bootstrap, protocol catalog convergence and anonymous clean Helm installation.
• Published PHP/Python/Rust lifecycle qualification passes all 12 required cells, with no unproven cells or findings. Exact tuple: Server 2.4.36, Native 2.3.1, PHP SDK 2.1.6, Python SDK 2.3.7, Rust SDK 2.1.2, CLI 2.1.3 and Waterline 2.0.9.
• Server OCI index: sha256:74cdf7feae85052151de905d877a2e1ba5fcbfabf9ef50f3a8aab7c285685204. Helm 0.1.132 package: sha256:67b8c865c836039ec2639bd068a1b001b7062e9bde5361d45adbd6469cb808e2.
• Full reviewed-source feature CI passes 2,253 cases and 47,227 assertions. The normal ownership/refusal, pressure, protocol/corpus, polling, replay/query, MySQL/Postgres upgrade and chart gates passed before merge.

The qualification report and raw before/after evidence with reproduction tooling are release assets. Both were downloaded again and their SHA-256 hashes matched the uploaded originals. The evidence archive hash is 3d4e27fe5debdf595cf5bab2495ca4d97cdb830156c7a72355001fcb768663b9.

Default protocol remains 1.19. No SDK release or database migration is required. Helm's immutable default is updated. Inspection found no affected public consumer pinned to the preceding Server 2.4.35 image that requires a separate update. Cloud deployment qualification remains separately owned.

The merged remote branch was deleted and verified absent. The fix is also carried into cancellation draft #291 at 24b54c9f8b3f84cad93ec58d5d7cd2f95160458d, where further cancellation coordinator and source qualification remain visible. This issue's published artifact acceptance is complete.