You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native test race: anchored past-A reopen test can lose its unfired timer to the maintenance loop #1105
occurrence_deadlines::a_reopen_after_the_anchor_is_due_immediately_at_the_anchor (added by #1098, on main at M 2a15179f) races the runtime's maintenance loop and can fail with 409 condition_wait_occurrence_timed_out instead of the intended 200 reopen. This is a test defect, not a runtime defect. The runtime behaviour it hits is correct and is pinned separately by a_wake_turn_finishing_after_its_fired_deadline_is_refused.
panicked at tests/execution/occurrence_deadlines.rs:165:5
assertion `left == right` failed: {"reason":"condition_wait_occurrence_timed_out",...}
test result: FAILED. 373 passed; 1 failed
rust/ on that head is byte-identical to M, so the same race exists on main.
Cause (source at M 2a15179f)
The test moves A into the past with UPDATE workflow_run_timers SET fire_at=<run started_at> WHERE sequence=1. Its comment says the timer task "stays far in the future, so the engine does not fire it first". That is not true: firing is keyed on the timer row.
TimerScheduler::start (rust/src/runtime/execution.rs:84–101) calls fire_due_timers every 250 ms. That query (rust/src/runtime/store.rs:4137) selects t.status='pending' AND t.fire_at<=now on workflow_run_timers alone. The moved timer is therefore due at the next tick.
The first timer stays pending until the wake turn's completion commits. That completion cancels it at wait_timeouts.rs:170 and reopens in the same transaction. Any tick between the UPDATE and that commit fires it: TimerFired is recorded and status='fired' is set (store.rs:4210–4235).
The completion's occurrence_anchor then sees fired_at on the chain (wait_timeouts.rs:254–255) and refuses condition_wait_occurrence_timed_out.
The window spans one signal, one poll and one completion. It is usually a few milliseconds, which is why the test mostly passes. On a loaded MySQL runner it overlapped a tick.
Fix requirements
Keep the 200 reopen at the moved (past) A while the first timer is unfired.
Keep the separate fired-timer refusal control unchanged.
No accepted 409, timing tolerance, retry or ignore.
Make the interleaving deterministic with a test-only seam that release builds do not compile, following the existing runtime::poll_pauses precedent.
Related: #1015 / #1098 (the occurrence deadline feature); #1104 (the CI repair that will carry the fix after review).
Status: a deterministic test-only fix is prepared locally as af59535 (parent f2b0269, the #1104 head). It is not pushed and has not been compiled or run yet.
It adds the debug-build seam runtime::timer_holds. The test holds the first wait's timer, waits until a maintenance tick has found it due and skipped it, keeps the strict 200 reopen at the past A, and then asserts the first timer ends cancelled with no fired_at. The fired-deadline refusal control is unchanged.
Next action: General reviews it. On selection, Server pushes it as the #1104 successor, with #1104's scope and title rewritten to cover this fix.
Status: the fix af59535 is pushed to #1104 and was selected in comment 6721138138. Attempt 4 compiled the library with the seam, but the train failed on Composer api.github.com outages before any cargo test lane started. So the repaired test and the fired-refusal control have not yet run, and the test file is still uncompiled. Receipt: #1104/6721282105.
Next action: General selects a fresh #1104 train once GitHub recovers. Server reports both tests per backend.
Resolved by #1104, actual main f77b093 (terminal custody: #1104/6723733470).
The debug-only per-timer hold is installed before moving A into the past. The test waits for a maintenance tick to find and skip that exact held timer, then still requires the strict 200 reopen. It releases the hold and verifies the original timer is cancelled with fired_at NULL. The next timer fires at the original A. The separate fired-timer refusal control remains unchanged; no 409 acceptance, retry, timing tolerance or ignore was added. Release builds compile neither the hold nor its call site.
The qualified source head d8d40e5 and its byte-equal merge-group/main tree passed. Source Shared fixtures 42491566759 and merge-group Shared fixtures 42494553016 completed successfully, including the Native backend controls; actual-main custody is 71 success +8 intended skips, 0 statuses. Earlier failed trains remain recorded.
This closes the deterministic test repair. It does not change Native 3.3.0's release commit M/image D or runtime behavior.
Activity
rmcdaniel commentedon Nov 8, 2026
Status: a deterministic test-only fix is prepared locally as af59535 (parent f2b0269, the #1104 head). It is not pushed and has not been compiled or run yet.
It adds the debug-build seam
runtime::timer_holds. The test holds the first wait's timer, waits until a maintenance tick has found it due and skipped it, keeps the strict 200 reopen at the past A, and then asserts the first timer ends cancelled with no fired_at. The fired-deadline refusal control is unchanged.Next action: General reviews it. On selection, Server pushes it as the #1104 successor, with #1104's scope and title rewritten to cover this fix.
Hold the moved timer so the past-A reopen test cannot race the mainte…
rmcdaniel commentedon Nov 8, 2026
Status: the fix af59535 is pushed to #1104 and was selected in comment 6721138138. Attempt 4 compiled the library with the seam, but the train failed on Composer api.github.com outages before any
cargo testlane started. So the repaired test and the fired-refusal control have not yet run, and the test file is still uncompiled. Receipt: #1104/6721282105.Next action: General selects a fresh #1104 train once GitHub recovers. Server reports both tests per backend.
rmcdaniel commentedon Nov 8, 2026
Resolved by #1104, actual main f77b093 (terminal custody: #1104/6723733470).
The debug-only per-timer hold is installed before moving A into the past. The test waits for a maintenance tick to find and skip that exact held timer, then still requires the strict 200 reopen. It releases the hold and verifies the original timer is cancelled with fired_at NULL. The next timer fires at the original A. The separate fired-timer refusal control remains unchanged; no 409 acceptance, retry, timing tolerance or ignore was added. Release builds compile neither the hold nor its call site.
The qualified source head d8d40e5 and its byte-equal merge-group/main tree passed. Source Shared fixtures 42491566759 and merge-group Shared fixtures 42494553016 completed successfully, including the Native backend controls; actual-main custody is 71 success +8 intended skips, 0 statuses. Earlier failed trains remain recorded.
This closes the deterministic test repair. It does not change Native 3.3.0's release commit M/image D or runtime behavior.