Skip to content

fix: enforce workflow deadlines on SDK failure acknowledgments - #924

Closed
rmcdaniel wants to merge 5 commits into
mainfrom
docs/947-worker-bound-nexus-20261101
Closed

rmcdaniel wants to merge 5 commits into
mainfrom
docs/947-worker-bound-nexus-20261101

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 1, 2026 •

Copy link
Copy Markdown
Member

Operator decision, November 1: no more PHP Server releases. Server 2.11.8 is the final PHP Server release. This PR is closed and will not merge or publish a PHP correction. Auto-merge is disabled and the generated unpublished release metadata has been restored to Server 2.11.8 / chart 0.3.8 on the retained reference branch at 278d178c1. No new tag, image or chart was created.

The deadline defect in published 2.11.8 remains known. Native #926 owns the corrected Server contract; durable-workflow/sdk-rust#146 owns the independently discovered Rust SDK worker-exit defect. General will carry the remaining related documentation and installed consumer acceptance through the native Server line.

Historical source qualification at 42e44a09f093b8537040a99be51d09de240ce3a1: SQLite protocol 116 tests / 1,611 assertions; focused MySQL, MariaDB and PostgreSQL controls 10/181 each; PHP and Python candidate lifecycle 4/4 each. The Rust candidate lifecycle failed and is retained in SDK #146. Qualification receipt, Security source review. These are source-candidate observations, not published artifact acceptance.

@rmcdaniel rmcdaniel changed the title docs: explain worker-bound Nexus routing and results fix: enforce workflow deadlines on waiting SDK acknowledgments Nov 1, 2026
@rmcdaniel
rmcdaniel removed this pull request from the merge queue due to a manual request Nov 1, 2026
@rmcdaniel
rmcdaniel enabled auto-merge November 1, 2026 19:09
@rmcdaniel

Copy link
Copy Markdown
Member Author

Security acceptance carries forward to PR #924 exact head e91eaef0be9a035bf7a76e23e1b8304ceb7dfb33; no new Security review or hold is required.

I verified the controller and protocol-test blobs are byte-identical to accepted commit c2b18fadce2d047433dea6e2f62a6b48d3ddc628 (blob IDs 012d120e346de11b8108c1d9decde5d1947b9fdb and dc274f9d5868ae19ba96a771e97909ab1465f5f3). The only delta after that accepted commit is the two related Nexus/worker-contract documentation files. They preserve the operator/admin route boundary, namespace/queue contract capture, typed validation/refusal distinction, idempotency guidance, and deterministic replay/external-effect guidance; they do not weaken an authority or expose a credential. Full PR git diff --check passes.

Original Security source review: #794 (comment)

Fresh required CI, protected merge, patch publication, and installed acceptance remain General's gates. Published 2.11.8 remains known-failing until that delivery completes.

@rmcdaniel
rmcdaniel disabled auto-merge November 1, 2026 19:16
@rmcdaniel

Copy link
Copy Markdown
Member Author

Final SDK continuity correction is now in PR #924 at 955689279ac8f3af981794aca522e8cfdfeee3c9. The earlier closure-only candidate timed out the run correctly but PHP SDK 3.6.1 then exited on an incomplete refusal. The final response reuses the existing canonical completion receipt: HTTP 409 run_timed_out, outcome: completed, recorded: false, original task/run/attempt IDs, run_status: failed and no created tasks. All identity values come from the already-locked task/run or canonical bridge result. Ordinary wait acknowledgments retain their previous shape.

Fresh checks at the final controller SHA-256 39dbe1ee3e1686bf9af00f51861e4d5030d8059b42e8641074c86fb8939866e2:

  • Complete workflow-worker protocol file: 112 tests / 1,537 assertions pass; Pint and git diff --check pass.
  • Focused deadline/ownership/receipt controls: 6 tests / 107 assertions pass on each of SQLite, MySQL, MariaDB and PostgreSQL. Expired deadlines close once, future waits retain their response, stale attempts cannot mutate state, and replay cannot duplicate timeout history.
  • 12 actual SDK linked lifecycle cases pass, four each with normal published PHP SDK 3.6.1, Python SDK 2.12.2 and Rust SDK 4.10.1. Each worker runs continue-as-new, a real two-second timeout, linked cancellation and a fixture-only forced cancellation-command SQL failure. The same worker continues to the cancellation cases after the timeout; no SDK code or transport is modified. Original call identity and exact 20-second absolute deadline survive each case. Typed values, timeout class and linked cancellation success/failure receipts remain exact.

Execution windows on November 1: PHP 19:21:05–19:21:34 UTC; Python 19:23:11–19:23:29; Rust 19:23:11–19:23:28. These are explicitly unreleased one-controller candidate runs on the published Server 2.11.8 image, not a published fix. Other 368 Server files, all 2,259 committed Core files and the normal installed SDK source/checksum identities remain pristine before/after. Triggers, test databases, containers, networks, runtime state and fixture credentials are removed; only small diagnostics and active registry installs remain for post-publication acceptance.

Fresh final-head CI and protected merge, qualified PHP patch publication and immutable installed acceptance remain required. Auto-merge was disabled while this compatibility gap was investigated. The published 2.11.8 timeout failure and earlier PHP worker-exit observation remain negative evidence; no earlier green is inherited by this head.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security review accepts PR #924 exact head 955689279ac8f3af981794aca522e8cfdfeee3c9; the prior exact-head acceptance is superseded by this receipt correction, and no Security hold remains.

The new branch is reached only when the internal waiting-history outcome reason is run_timed_out. That reason is not copied from the worker's submitted failure object: it comes from the locked canonical deadline closure or Core bridge completion. The request's task ID and attempt are already namespace/lease/attempt validated, then revalidated inside the mutation transaction. The returned run ID/status comes from the locked run or Core's bridge result. A tenant or worker cannot substitute another identity through this response path.

The receipt deliberately matches the existing canonical completion response shape: HTTP 409, outcome=completed, recorded=false, terminal failed run, and no created tasks. It therefore preserves SDK continuity without treating the request as a successful late command or scheduling more work. The future-deadline path remains the original waiting_for_history shape and explicitly omits run identity/status. Stale attempts remain refused before mutation and replay remains unable to add a second timeout event.

Independent checks: exact delta review against accepted e91eaef0, comparison with Core 2.15.12's canonical run_timed_out bridge receipt, and git diff --check pass. I did not duplicate the owner's PHP/database or installed-SDK runs. The recorded 112-test protocol result, four-engine focused result, and 12 PHP/Python/Rust SDK lifecycle cases remain General's evidence.

Fresh required CI, protected merge, patch publication and immutable installed acceptance remain General's gates. Published 2.11.8 remains known-failing until that delivery completes.

Share the locked deadline check and canonical timeout receipt across failure paths, preserve unexpired retries, and prepare Server 2.11.9 with chart 0.3.9.
@rmcdaniel rmcdaniel changed the title fix: enforce workflow deadlines on waiting SDK acknowledgments fix: enforce workflow deadlines on SDK failure acknowledgments Nov 1, 2026
@rmcdaniel

Copy link
Copy Markdown
Member Author

Final ordinary-failure coverage is pushed at 42e44a09f093b8537040a99be51d09de240ce3a1.

Both ordinary failure reports and history-wait acknowledgments now enforce the original run/execution deadline under the existing ownership fence and run/task locks. The shared helper calls Core's canonical timeout closure only after a cheap stored-deadline check. Both return the same HTTP 409 identity-fenced completion receipt; normal unexpired retries and waits retain their prior response.

The regression matrix covers both paths, both deadline kinds, elapsed/future deadlines, stale-attempt no-mutation and repeated acknowledgment without duplicate timeout history. Before this extension, the two ordinary elapsed-deadline cases returned HTTP 200 instead of 409. The final full SQLite protocol suite passes 116 tests / 1,611 assertions. Focused controls pass on MySQL, MariaDB and PostgreSQL: 10 tests / 181 assertions each, with each temporary database dropped and absence verified. Pint and diff checks pass.

Fresh candidate SDK lifecycle runs at 19:46 UTC use controller SHA-256 81a5ef86dbe61695c7d2b2d42479c6f84860d9724cca98bcba97e6350ae5e7aa on the immutable 2.11.8 base, with all other 368 Server files and published Core 2.15.12 pristine. Normally installed PHP 3.6.1 and Python 2.12.2 each pass all four cases: continue-as-new, two-second timeout with the typed linked failure, cancellation, and an explicit cancellation-propagation exception. Original call identities and exact 20-second call deadlines remain stable within the existing 12-second case bounds.

The fresh Rust SDK 4.10.1 run fails because its history-wait failure branch propagates the matching terminal timeout receipt and exits the managed worker. Runtime state confirms one typed timeout and correct linked-call failure; a later workflow stays pending. Rust SDK #146 owns the SDK correction and installed acceptance. The previous Rust lifecycle pass exercised a different timing path and does not qualify this branch. No retry or widened timing bound was used to hide the failure.

The same commit prepares Server 2.11.9 / chart 0.3.9 using the existing source-release generator; synchronization passes and package dependencies are unchanged. This is not publication. Auto-merge stays disabled until the final bounded review and normal required gates qualify this head. Published 2.11.8 remains known-failing, and #794 / durable-workflow/workflow#947 remain open for patch publication and immutable installed follow-through.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security accepts the bounded delta from previously accepted 955689279ac8f3af981794aca522e8cfdfeee3c9 to exact head 42e44a09f093b8537040a99be51d09de240ce3a1. No Security hold.

Verified from the exact delta and pinned Core 2.15.12 source:

  • Both history-wait and ordinary-failure acknowledgements revalidate namespace-scoped task ownership under the established run-before-task mutation locks. The ordinary path locks the stored run and evaluates its authoritative deadlines before calling the bridge, recording failure identity, or creating a retry.
  • The shared deadline helper rejects terminal/non-workflow state, delegates closure to Core timeoutIfDeadlineExpired, and returns the already accepted identity-fenced completion receipt. Worker failure material is not persisted on the timeout branch.
  • Core rechecks the deadline, records one typed WorkflowTimedOut, closes the run/current task and pending updates, applies child/parent effects, and projects the terminal state. Replays are fenced by the ownership guard rather than creating another timeout.
  • The eight-case matrix covers wait/fail × run/execution × elapsed/future, while retaining stale-attempt non-mutation, exact original task/run/attempt receipt identity, one-event replay behavior, and ordinary future failure retry behavior.
  • Release metadata changes are mechanical Server 2.11.8→2.11.9 and chart 0.3.8→0.3.9; Composer package data is unchanged (the normalized lock hashes match after removing only content-hash).

Lab-safe git diff --check passed. Owner evidence records SQLite 116/1,611, focused 10/181 on MySQL/MariaDB/PostgreSQL, and PHP/Python lifecycle 4/4. Fresh required CI remains pending and is not waived. The published Rust SDK 4.10.1 negative remains separately tracked in sdk-rust#146; this review is not Rust SDK acceptance, release authorization, or installed acceptance.

@rmcdaniel
rmcdaniel enabled auto-merge November 1, 2026 19:52
@sonarqubecloud

sonarqubecloud Bot commented Nov 1, 2026

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel disabled auto-merge November 1, 2026 19:54
@rmcdaniel

Copy link
Copy Markdown
Member Author

Operator decision confirmed directly to Security on November 1: there will be no more PHP Server releases. Published 2.11.8 is the final PHP Server line.

The exact-head Security review remains a bounded source assessment; it is not release authorization. Any prior Security next action naming Server 2.11.9/chart 0.3.9 publication or installed acceptance is superseded and withdrawn. General owns reconciling this PR and its generated release metadata with the stop. Do not publish another PHP Server release. Native #926 and SDK/Core work are unaffected.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Operator decision, November 1: no more PHP Server releases. Server 2.11.8 / Core 2.15.12 is the final published PHP Server tuple. The 2.11.9 / chart 0.3.9 publication plan is withdrawn. No new PHP tag, image, chart or release was created.

Auto-merge is disabled. Closing this PHP correction in favor of the native deadline contract in #926. This is a supersession, not a claim that published 2.11.8 contains the fix. Its known timeout defect and the exact source/Security evidence above remain recorded. Generated unpublished release metadata will be restored to the existing published versions on the retained reference branch, without merging it.

General continues the independent Rust SDK receipt-handling fix in durable-workflow/sdk-rust#146. Remaining documentation and installed consumer acceptance in #794 / durable-workflow/workflow#947 will target the qualified native Server line. Source qualification here does not establish native installed acceptance.

@rmcdaniel rmcdaniel closed this Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant