Skip to content

Native test race: anchored past-A reopen test can lose its unfired timer to the maintenance loop #1105

Closed

Description

@rmcdaniel

Summary

occurrence_deadlines::a_reopen_after_the_anchor_is_due_immediately_at_the_anchor (added by #1098, on main at M 2a15179f) races the runtime's maintenance loop and can fail with 409 condition_wait_occurrence_timed_out instead of the intended 200 reopen. This is a test defect, not a runtime defect. The runtime behaviour it hits is correct and is pinned separately by a_wake_turn_finishing_after_its_fired_deadline_is_refused.

Observed

  • PR Stabilize parity downloads and retain deterministic failure evidence #1104 head f2b02690, run 42474309384, job 128713597029, mysql-execution (existing-php-mysql-matrix, native-tests-b), 2026-11-08T11:21:39Z.
  • The failure:
    panicked at tests/execution/occurrence_deadlines.rs:165:5
    assertion `left == right` failed: {"reason":"condition_wait_occurrence_timed_out",...}
    test result: FAILED. 373 passed; 1 failed
    
  • rust/ on that head is byte-identical to M, so the same race exists on main.

Cause (source at M 2a15179f)

  1. The test moves A into the past with UPDATE workflow_run_timers SET fire_at=<run started_at> WHERE sequence=1. Its comment says the timer task "stays far in the future, so the engine does not fire it first". That is not true: firing is keyed on the timer row.
  2. TimerScheduler::start (rust/src/runtime/execution.rs:84–101) calls fire_due_timers every 250 ms. That query (rust/src/runtime/store.rs:4137) selects t.status='pending' AND t.fire_at<=now on workflow_run_timers alone. The moved timer is therefore due at the next tick.
  3. The first timer stays pending until the wake turn's completion commits. That completion cancels it at wait_timeouts.rs:170 and reopens in the same transaction. Any tick between the UPDATE and that commit fires it: TimerFired is recorded and status='fired' is set (store.rs:4210–4235).
  4. The completion's occurrence_anchor then sees fired_at on the chain (wait_timeouts.rs:254–255) and refuses condition_wait_occurrence_timed_out.

The window spans one signal, one poll and one completion. It is usually a few milliseconds, which is why the test mostly passes. On a loaded MySQL runner it overlapped a tick.

Fix requirements

  • Keep the 200 reopen at the moved (past) A while the first timer is unfired.
  • Keep the separate fired-timer refusal control unchanged.
  • No accepted 409, timing tolerance, retry or ignore.
  • Make the interleaving deterministic with a test-only seam that release builds do not compile, following the existing runtime::poll_pauses precedent.

Related: #1015 / #1098 (the occurrence deadline feature); #1104 (the CI repair that will carry the fix after review).

Activity

  1. rmcdaniel commented on Nov 8, 2026

    @rmcdaniel
    MemberAuthor

    Status: a deterministic test-only fix is prepared locally as af59535 (parent f2b0269, the #1104 head). It is not pushed and has not been compiled or run yet.

    It adds the debug-build seam runtime::timer_holds. The test holds the first wait's timer, waits until a maintenance tick has found it due and skipped it, keeps the strict 200 reopen at the past A, and then asserts the first timer ends cancelled with no fired_at. The fired-deadline refusal control is unchanged.

    Next action: General reviews it. On selection, Server pushes it as the #1104 successor, with #1104's scope and title rewritten to cover this fix.

  2. rmcdaniel commented on Nov 8, 2026

    @rmcdaniel
    MemberAuthor

    Status: the fix af59535 is pushed to #1104 and was selected in comment 6721138138. Attempt 4 compiled the library with the seam, but the train failed on Composer api.github.com outages before any cargo test lane started. So the repaired test and the fired-refusal control have not yet run, and the test file is still uncompiled. Receipt: #1104/6721282105.

    Next action: General selects a fresh #1104 train once GitHub recovers. Server reports both tests per backend.

  3. rmcdaniel commented on Nov 8, 2026

    @rmcdaniel
    MemberAuthor

    Resolved by #1104, actual main f77b093 (terminal custody: #1104/6723733470).

    The debug-only per-timer hold is installed before moving A into the past. The test waits for a maintenance tick to find and skip that exact held timer, then still requires the strict 200 reopen. It releases the hold and verifies the original timer is cancelled with fired_at NULL. The next timer fires at the original A. The separate fired-timer refusal control remains unchanged; no 409 acceptance, retry, timing tolerance or ignore was added. Release builds compile neither the hold nor its call site.

    The qualified source head d8d40e5 and its byte-equal merge-group/main tree passed. Source Shared fixtures 42491566759 and merge-group Shared fixtures 42494553016 completed successfully, including the Native backend controls; actual-main custody is 71 success +8 intended skips, 0 statuses. Earlier failed trains remain recorded.

    This closes the deterministic test repair. It does not change Native 3.3.0's release commit M/image D or runtime behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions