Repository navigation
Schedule a signal received during another signal’s leased replay #956
Description
Completed and accepted: final installed-image receipt and Security acceptance. Published Server 2.11.7/Core 2.15.7 passes all eight named-signal/kernel cases (152 assertions) and both real HTTP Python 2.12.1 cold-process timing cases, without product overlays. Normal image/chart publication passed. Kernel and network scopes are explicitly separated in the receipt.
Earlier diagnosis, counterfactuals and source/publication evidence follows.
General owns this PHP Core defect following the concurrent-wait review in durable-workflow/server#836. It is separate from #953's retained-checkpoint cursor fix and #857's terminal-boundary contract.
An accepted signal that resolves an already-open signal wait can arrive while another signal's workflow task is leased. When that earlier replay finishes waiting, Core does not enqueue a delivery turn for the newly received signal. The result depends on whether the workflow definition is available locally.
Reproduced using the normal published Composer Core 2.15.5, source/dist 164599419ec4cef275927ad23da1db81233dc514, PHP 8.3.36 and the canonical Laravel/SQLite test profile in Server source. The probe uses Core's real WorkflowStub start/signal admission, task claim, history snapshot, WorkflowFiberRunner and completion bridge. It does not seed an unrelated leased task or fabricate signal history.
The authored workflow waits for an ordinary mixed parallel group:
all([
static fn () => signal('advance'),
static fn () => signal('finish'),
static fn () => timer(60),
]);Reproduction:
- Start the workflow and complete its first task with the runner's three emitted commands. Both signal waits and the timer are committed.
- Accept
finish, claim its generated signal-resume workflow task and read that task's history. The runner is still waiting foradvanceand the timer. - Accept
advancewhile that task is leased. It is accepted, received and bound to the originaladvancewait. - Complete the claimed
finishtask with the runner's empty command list. The completion succeeds and the run stays waiting, butcreated_task_idsis empty and there is no ready workflow task foradvance.
The four-case regression probe has 51 assertions, one missing-delivery failure, three passing controls, zero errors/skips:
- Definition available, signal during lease: missing immediate delivery task.
- Definition available, signal after completion: delivery task created.
- Definition unavailable but durable declared-signal contract retained, signal during lease: delivery task created.
- Same durable-contract case, signal after completion: delivery task created.
Source explanation: SignalWaits::forRun folds SignalReceived into status: resolved, source_status: received. DefaultWorkflowTaskBridge::markRunWaiting calls PendingMessageTask::createForRun(... includeReceivedProjectedSignalWait: false), which excludes that wait. The alternative declared-signal eligibility path applies only when the workflow definition is unavailable, explaining the control result. The ordinary Server acknowledgement has the same exclusion and is independently reproduced below. The embedded executor retains its run lock throughout execution, so its admission boundary differs; no embedded defect is claimed.
This establishes missing immediate scheduling. It does not establish permanent loss: a later timer/other event may cause another replay. Maintenance-repair behavior and actual installed SDK/Server image acceptance have not been qualified by this probe.
Acceptance:
- Schedule the newly admitted eligible signal when the current replay relinquishes the run, including an already-open named wait and locally available definitions.
- Preserve the original signal/wait/command identities, command ordering and claim authority. Do not repeatedly requeue an already-attempted or unconsumed signal, wake unrelated buffered signals, or alter closed/cancelled wait semantics.
- Cover both signal-resume completion and the ordinary waiting acknowledgement, plus positive/negative timing and definition-availability controls. Qualify any corresponding embedded executor path.
- Publish the qualified Core fix, integrate it normally into Server, and record exact published consumer acceptance. Coordinate the changed reference with the Rust owner without modifying frozen native fixtures as part of PHP triage.
Core 2.15.6 is published and verified: release, source/dist a0eed69785139c6e093be7a4b5534ca1f4008fb9. Its tree exactly matches qualified head 20ce409c. The full main build passes 23 jobs with two conditional skips; all 17 published-package/Laravel upgrade jobs pass. All five required local commands pass: ECS, PHPStan, unit 5,160/72,773, coverage 5,160/72,961 at 77.24%, PostgreSQL features 2,549/25,076. Zero errors/failures; unit/coverage have 43 skips and 15 notices, features have 18 skips. Source identity 1/42 and policy 68 tests pass. Publication receipt.
Server PR #843 merged through the queue at 9fc29d437b8198c24d426659543a9f59506f35f1 on 2026-10-28 19:18:56 UTC, exact reviewed head 65b710ed4b765211a735abbd1e3fe7bdab84ecd6. The PHP fix and normal Composer Core 2.15.6 integration are now on main with exact source/dist identity. Combined HTTP/checkpoint/published-library qualification passes 293 tests / 11,881 assertions, zero errors/failures and eight database-specific skips under SQLite. PostgreSQL regressions plus all four repair lock-order mutations pass eight / 192, zero skips. The original-controller overlay still fails the target case with the new Core: four / 71, one failure and three controls, zero errors/skips. The published-library authored signal-resume probe passes four / 52. Pint passes. Exact-head hosted checks and Security review passed. Required merge-group PHP, shared fixtures and Sonar App checks passed before merge. Main PHP feature/readiness/TLS qualification passed. The aggregate fails only on the newly corrected Symfony Mime dependency audit, addressed by #845/#846; this is not a published Server-image or SDK acceptance result.
Embedded assessment: RunWorkflowTask::executeClaim locks task and run and calls WorkflowExecutor::run within one SqliteTransaction::forTask transaction. The split external lease/history/admission window is not equivalent to that embedded path. No executor change or embedded reproduction is claimed. Detailed scope receipt.
Installed acceptance is completed by the final Server 2.11.7/Core 2.15.7 receipt. This issue is closed; no unchanged SDK release is required merely for version alignment.
Activity
rmcdaniel commentedon Oct 28, 2026
The ordinary Server acknowledgement is also confirmed and fixed in source. A real HTTP start/register/poll/complete sequence opens two independent signal waits and a timer. The production timer handler creates an ordinary workflow task; a signal is then admitted during its lease or after its waiting acknowledgement. These are protocol-level independent waits, not an SDK-authored mixed parallel-group claim.
With normal published Core 2.15.5 and the original controller, the four timing/definition cases produce 71 assertions: one missing-delivery failure, three passing controls, zero errors/skips. Removing the received-wait exclusion from the ordinary acknowledgement passes all four / 72 assertions. The related worker success, worker protocol and repair suites pass 174 tests / 5,328 assertions with eight database-specific race cases skipped under SQLite. Claim/attempt/namespace validation is unchanged.
The corresponding embedded path has a different admission boundary:
RunWorkflowTask::executeClaimlocks both task and run, invokesWorkflowExecutor::run, and finishes inside the sameSqliteTransaction::forTasktransaction. The split external claim/history/admission window reproduced above is therefore not equivalent to that embedded execution path. No embedded executor change or reproduced embedded defect is claimed.Core PR #957 is at
20ce409cd5279868a9d95b57939ebb355c454f3e. Runtime, PHP tests and release metadata are identical to0b404682; the subsequent change only corrects the post-completion scheduling classification. All 68 policy tests pass, actual base-to-head validation passes, and the exact-head hosted checks are terminal green. Security and independent source/hosted reviews report no finding: #957 (comment) and #957 (comment).Full ECS and PHPStan pass. Full unit qualification is still running. An earlier run inherited the cached test image's file cache default and was interrupted; it is not qualification evidence. The current run explicitly selects Redis cache. Coverage and the full feature suite follow after unit qualification. Core 2.15.6 is planned metadata only; no publication or installed consumer result is claimed.
rmcdaniel commentedon Oct 28, 2026
Qualification correction: the full PostgreSQL/unit invocation with Redis cache stopped after 3,092 tests / 49,093 assertions with one failure and one error. The failure is the existing redrive test's strict integer assertion against a Redis-returned string; that test file is byte-identical to published 2.15.5. The error is a 15-second cold-process timeout in the schedule tracing test. Neither result is a qualification pass, and the timeout's underlying cause is not established by that run.
I had applied the feature profile's Redis cache selection to the unit suite. The unit/coverage workflow uses the testbench array cache default and file-backed SQLite. Rechecking the two failing controls with array cache passes two tests / 101 assertions without source changes. Full
composer unitandcomposer coverageare now running in separate exact-source checkouts with precreated SQLite files, explicit array cache, and separate Redis services for tests that use Redis directly. Coverage uses PCOV 1.0.12. An initial coverage startup without the SQLite file also failed immediately; its partial result is excluded.The earlier focused PostgreSQL signal regression and ordering controls remain nine tests / 150 assertions passing. Full feature qualification still follows. Core #957 remains draft, and 2.15.6 is not published.
rmcdaniel commentedon Oct 28, 2026
The file-backed SQLite/array-cache unit rerun also stopped on a cold-process timeout: 746 tests / 4,016 assertions, one admission-tracing error, zero assertion failures. The parallel coverage attempt showed the same family of timeout errors and was deliberately stopped; it is not pass evidence.
The local invocation omitted CI's fast
TMPDIRsetting, so the tracing fixture created its independent SQLite databases in the container's ordinary temporary filesystem. WithTMPDIRon the same fast temporary storage used by CI, all nine admission/schedule controls pass 689 assertions in 29.3 seconds, with the existing 15-second process timeouts unchanged. Their existing stage journal reaches application, migration and schema readiness. This establishes the corrected control result, not a final full-suite result.Full unit and PCOV coverage are running again with that complete profile, in independent checkouts and Redis services. Full PostgreSQL 17.11 feature qualification has also started in a third isolated checkout with its own database and Redis service. All use exact head
20ce409cd5279868a9d95b57939ebb355c454f3e. No product source, timeout or assertion was changed during this diagnosis. All full-suite outcomes remain pending; #957 is draft and 2.15.6 is unpublished.rmcdaniel commentedon Oct 28, 2026
Core 2.15.6 is published and verified: https://github.com/durable-workflow/workflow/releases/tag/2.15.6 . Its immutable annotated tag
addcade36689b03f7a321c282641e72e04897e1fresolves to merged sourcea0eed69785139c6e093be7a4b5534ca1f4008fb9. Packagist source/dist references are that same commit and the package's source-release metadata is 2.15.6.Full main qualification completed with 23 successful jobs and two existing conditional skips. Published-package verification passed all 17 package/Laravel upgrade jobs. All five required local commands also passed as recorded above.
Server #843 is being integrated normally against current Server main, preserving its reviewed PHP files. Composer updates exactly one package from 2.15.5 to 2.15.6; no other runtime/development dependency changed. The installed package reports 2.15.6, reference
a0eed697, and matching source metadata, without repository overrides or vendor patches. Combined HTTP/retained-checkpoint qualification is running before the updated head is pushed.This publishes the Core bridge fix. The ordinary Server acknowledgement fix, normal source integration and exact published Server/SDK acceptance remain separate requirements. #956 stays open. This change adds no wire/schema requirement and does not require unchanged SDKs to receive aligned version numbers.
rmcdaniel commentedon Oct 29, 2026
Installed consumer follow-through found a separate replay defect in published Python SDK 2.12.0: SDK Python #223.
The real SDK Worker handles
finish, computes a reopened condition while still waiting foradvance, then acceptsadvanceimmediately before sending that completion under its live claim. The Server schedules and leases the next delivery turn; the SDK then rejects an unconsumedConditionWaitOpened(sequence 2)while attempting to complete. Sending the second signal immediately after the same completion passes and completes the run once.The unchanged published Server 2.11.5 / Core 2.15.1 and uploaded 2.11.6 candidate / Core 2.15.7 have the same result with the same normal SDK 2.12.0 installation: one during-lease failure and one passing after-completion control, zero test errors/skips. Full immutable image/source identity and canonical event order are recorded in #223. This establishes a pre-existing replay/receiver-order gap rather than a new 2.11.6 regression.
This experiment exercises the SDK's actual remote condition-reopening path. It is separate from this issue's locally available named-wait source regression and does not replace that source proof. An initial observer expected an empty waiting acknowledgement; inspection showed the supported SDK correctly emits another condition descriptor, so the final experiment checks that actual completion path. No SDK/Server implementation or history rows were changed for the installed experiments.
General owns #223 and will qualify/publish the SDK correction, preserve the determinism fence and wait/receiver identities, then rerun the exact installed consumer. Keep #956 open through that acceptance; source integration and the Server's ability to schedule the delivery turn do not prove successful SDK replay.
rmcdaniel commentedon Oct 29, 2026
The separate Python replay blocker is delivered: SDK Python 2.12.1 is published and durable-workflow/sdk-python#223 is closed. Final publication and installed receipt records the exact source/tag/wheel, successful trusted publisher, all 39 installed runtime files, and eight real before/after timing cases across Server 2.11.5 and 2.11.6, including replacement in a separate fresh process.
Those cases exercise the supported remote-definition SDK condition-reopening path. They preserve the original signal identities and complete each original run once. They do not establish this issue's distinct locally available named-wait or ordinary waiting-acknowledgement installed acceptance.
Keep #956 open. Next: qualify those exact named-wait/ordinary-acknowledgement paths against the corrected, fully qualified published Server reference. Server 2.11.6 has a separate release-blocking bridge-idempotency defect, durable-workflow/server#861; its narrow fix is in durable-workflow/server#862 and the replacement release is 2.11.7. No additional unchanged SDK release or Native/frozen-fixture edit is needed for this handoff.
rmcdaniel commentedon Oct 29, 2026
Published-image acceptance is complete for #956. The fix first shipped in Core 2.15.6 and remains in published Core 2.15.7 (
3aa98b8473b130dbc141457f76ccc32e4aa58062) inside stable Server 2.11.7. Exact amd64 child:sha256:be7c2288ff935396262d743590530b204fac11b76582e56a525cc81a8da840d3, sourcef55f90e54a213872f1c164a1b22c082d859c18aa. The normal image/chart publisher is terminal successful.The final installed-image kernel profile passes 8 tests / 152 assertions, zero failures/errors/skips, in 4.735 seconds. Four cases exercise ordinary waiting acknowledgement; four exercise authored signal-resume completion. Each crosses signal-during-lease versus after-completion and locally available definition versus durable-contract-only controls. The authored test/workflow come from exact published Core; the only harness adaptation adds Laravel
RefreshDatabasefor Server's test application.The image's actual application/controller, Core completion bridge, history projection and fiber runner execute. No
/appor/app/vendoroverlay is used. Reflection and loader guards prove product classes resolve from the image; the external loader declares no App/Workflow/Illuminate namespaces and cannot resolve the checked product classes. Original signal/wait/command identities are preserved; delivery applies once and leaves no extra ready workflow turn. This is kernel-only evidence, including canonical timer driving with the test clock, separate from network HTTP and local arm64 execution.On the same final image, normal published Python SDK 2.12.1 passes both real HTTP cold-process timing cases, zero failures/errors/skips, in 7.37 seconds: signal before the earlier acknowledgement and the after-acknowledgement control. Each replacement is a separate fresh process with a different PID and its own registration. Both signals are received/applied once and the workflow completes once. No history fabrication, maintenance rescue or changed runtime limits. This remote-definition/condition-reopening consumer remains distinct from the locally available named-wait kernel cases. Python 2.12.1 separately delivers SDK #223.
The original Core/controller counterfactuals, timing/definition controls, full quality cycle and published-package upgrade matrix remain in earlier receipts. Embedded execution holds the run lock across execution, so no equivalent executor defect/change is claimed. Other supported SDK scopes are recorded in #953; no unchanged SDK release is required merely to match Core.
Selected #956 acceptance is complete. Completed runtime fixtures, one-offs and exact task networks are removed and absence verified. Server #861 has separate Bridge verification and final hold resolution.
rmcdaniel commentedon Oct 29, 2026
Security installed acceptance — PASS
The final published-image receipt satisfies this issue's selected acceptance boundary on stable Server
2.11.7/ Core2.15.7.The exact immutable amd64 child passes the no-overlay/no-shadow kernel profile: 8 tests / 152 assertions, zero failures/errors/skips. On the same child, published Python SDK
2.12.1passes both real-HTTP cold-process timing cases with fresh replacement processes and exactly-once signal/completion behavior.This accepts the scoped fix and closes #956. It does not make a local arm64 execution claim or widen acceptance to unrelated SDK/Native/Cloud deployment work.