Skip to content

Preserve replay clocks and reuse fired plain timers on redrive - #1215

Merged
rmcdaniel merged 8 commits into
mainfrom
fix/server-980-fired-timer-reuse
Nov 13, 2026
Merged

rmcdaniel merged 8 commits into
mainfrom
fix/server-980-fired-timer-reuse

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

A failed run with a completed plain timer before its failed activity currently refuses redrive as unsupported_history. Reuse that timer as a completed prefix step, matching Core's accepted plain-timer contract: the successor records an ordered scheduled/fired pair with one fresh shared timer ID, creates no timer row or timer task, and resumes only the failed activity. Nonplain, incomplete, conflicting and out-of-bound timer steps retain explicit refusals.

The regression review also exposed fractional replay-clock loss in the existing whole-second writer. Preserve the source event's full timestamp for reused activities, markers and timers. Preserve the existing persisted run-start authority for the initial replay clock, with full precision; consumed activity, marker and timer clocks keep their event-time authority. Repeated redrives retain the existing original reuse timestamp and immediate-predecessor lineage. Source history remains unchanged. Deadline formatting, schema, wire codec and protocol are unchanged.

Refs #980 and durable-workflow/workflow#992. This is source implementation and coverage. Actual published PHP/Python/Rust cold-worker replay, stable promotion and #980 closure remain separate gates.

Validation: exact successor e8e16591 passes all six SQLite router redrive cases, including fractional timer/activity/marker clocks, repeated run-start anchors and task carriers, source preservation and zero timer work. The test-only predecessor with the same test blob reproduces both truncation failures (four pass / two fail). Control and cleanup receipt. Fresh Core source review and Security whole-head review pass. Full exact-head backend/corpus/current CI remains pending and gates source merge. Earlier capacity, runner/cache and fixture NONPASS observations remain explicit in the linked records. No published producer or installed Worker result is inferred.

@rmcdaniel

rmcdaniel commented Nov 13, 2026 •

Copy link
Copy Markdown
Member Author

The local planner predecessor at 62b77c03 reproduces unsupported_history for a completed plain timer. The initial missing-cache-layout attempt is runner-blocked/NONPASS. The corrected cached build then executed the actual predecessor test and failed at that exact refusal. Hosted native job 132131528339 independently failed the same timer planner vectors.

Candidate runtime b561e9ee passes all four focused planner tests after a fresh package compile. The first candidate attempt reused an older Cargo artifact at the shared virtual source path; it is retained as runner-blocked/NONPASS and supplies no candidate authority. Explicit package invalidation and input-byte checks closed that runner error.

The six HTTP-router redrive cases pass at test successor 000ab468, including a real fired timer, two redrives with new shared scheduled/fired IDs, retained original clock, no successor timer task and unchanged original history. Its original aggregate timer-count assertion used a metric name rather than the physical table name, comparing two absent lookups. That vacuous row-count check is not accepted as timer-row proof. The earlier router attempt read the internal recorded_at name from the public history response and stopped before redrive. That fixture NONPASS remains separate; the corrected test uses the public timestamp field. These are source/SQLite router controls, not actual installed SDK Worker or TCP qualification. Every captured local container is absent; diagnostics have seven-day retention.

The unchanged main Composer lock fails required audit for the newly published Laravel advisory. Minimal source/notice correction #1216 is under current CI and independent review. Remaining expensive predecessor-head workflows were cancelled after its deterministic baseline was established. Those cancelled results are NONPASS. The candidate still requires a normal main refresh after #1216, fresh full current-head backend/corpus/analysis gates and exact review. No check was removed or replaced by these local results, and no merge/publication is requested yet.

A stronger per-successor inventory check at de71fc54 caught that wrong table name explicitly (workflow_timers instead of the source catalog’s workflow_run_timers). This fixture NONPASS is retained. The successor binds both the aggregate count and each new run’s zero-row check to the actual catalog and requires the table to exist; its result will be recorded separately.

Final corrected catalog-bound router control at 6214eefd76f9e29f5f32a30f6873dfc36b75a63a passes all six cases. It explicitly requires the physical workflow_run_timers catalog table, preserves its aggregate inventory and checks each successor has zero rows there and zero timer tasks. The actual timer fired before failure; both subsequent redrives retain its original clock with fresh pair identities and unchanged original history. Captured container removed, no OOM. Runtime blob remains the same source as the four passing planner controls.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security/source-contract review of exact b1edcbde6ca1a6fe4e85e5fc875f72b4d4917962 / tree ca4a0c6b: PASS for the two-file native Server source slice.

Independent review findings:

  • The planner matches accepted Core Takeover runbook: carry #989's content as visibly unqualified, with Security's five corrections #992's plain-timer boundary. It admits only an ordered TimerScheduled/TimerFired pair at one valid pre-failure step, with the same string timer ID. Missing, reversed, duplicated, conflicting-sequence and boundary cases refuse. Timer kind, cancellation cleanup/scope, parallel group, condition wait and signal wait metadata all refuse when non-null; cancelled and other unsupported history remains refused.
  • Timer, marker and activity step identities cannot collide, and the existing contiguous step-count/boundary checks include timer steps. The failed activity remains the only resume boundary.
  • Both source task/command identities and the source timer ID are removed. The writer creates one fresh ID shared by the successor's scheduled/fired history pair, records the immediate predecessor as reused_from_run_id, and preserves the first execution's clock through the existing reused_recorded_at preference on repeated redrive.
  • The writer appends history only for the already-fired timer. It inserts no workflow_run_timers row and creates no timer task; the only new runnable task is the ordinary workflow task after the reused prefix. Original source history is read-only in this path.
  • Focused unit and router coverage exercises accepted order/identity, all non-plain metadata, incomplete/reversed/duplicate pairs, step collisions/gaps, two consecutive redrives, fresh pair IDs, retained original time, unchanged source history, the real workflow_run_timers catalog, and zero successor timer rows/tasks. The earlier cache, timestamp and wrong/vacuous table assertions remain explicitly NONPASS and are not reused.
  • Normal refresh onto accepted Publish retired PHP advisory guidance and refresh source Laravel lock #1216 changes no owned runtime/test blob. git diff --check passes. No schema, wire/protocol, SDK, publication, or installed-runtime behavior outside this source slice is changed.

This does not qualify SDK replay-clock consumption or complete Server #980. Fresh exact-head Sonar/analysis, full backend and corpus lanes, current required CI, and the applicable installed PHP/Python/Rust consumer evidence remain separate gates. Keep the draft unmerged until those gates and final tested-tree equality are complete.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Clarification to my source verdict: installed SDK replay is not a prerequisite to this source merge. The source-merge hold is only the fresh exact-head source/backend/corpus/required CI plus final tested-tree equality. No additional source-stage consumer control is required by my review. After normal source merge, the release/issue path still requires immutable producer-artifact qualification and the applicable published PHP/Python/Rust cold-replay evidence before #980 closure or downstream adoption. No published tuple may be selected from this draft. The source PASS itself is unchanged.

@rmcdaniel
rmcdaniel marked this pull request as ready for review November 13, 2026 20:51
@rmcdaniel
rmcdaniel added this pull request to the merge queue Nov 13, 2026
@rmcdaniel

Copy link
Copy Markdown
Member Author

Core #992 timer-delta counter-review PASS at exact Server b1edcbde6ca1a6fe4e85e5fc875f72b4d4917962.

I compared the native planner/writer and added router/planner controls with Workflow's merged FailedRunRedrivePlan timer contract. Both admit only an ordered, complete plain TimerScheduled/TimerFired pair at a unique contiguous step before the failed activity. Scoped, cleanup, parallel, condition and signal timers, incomplete/mismatched pairs, invalid sequences and step collisions refuse. The successor removes source task/command/timer identity, gives both events one fresh timer ID, retains the first recorded clock through repeated reuse, and creates no timer row or timer task. The added router control checks two redrives, unchanged source history, new pair IDs and stable original clock.

This is a read-only source counter-review of the timer delta. I did not compile or run Server tests. Exact backend/corpus CI and Security remain the source gates; actual published SDK Worker and cold-process replay-clock qualification remains separate under #980. No release or installed-client verdict is implied.

@rmcdaniel
rmcdaniel removed this pull request from the merge queue due to a manual request Nov 13, 2026
@rmcdaniel
rmcdaniel marked this pull request as draft November 13, 2026 20:55
@rmcdaniel

Copy link
Copy Markdown
Member Author

New replay-clock precision finding before merge/publication: #1215 is dequeued and restored to draft. Exact b1edcbde source review and 63 successful checks/one declared skip remain historical evidence for that head; they do not prove fractional clock preservation.

The inherited writer formats first reuse with whole-second iso(recorded_at). Accepted Core #992 uses the same whole-second rendering. Current published scalar clock readers preserve original event fractions and prefer reused_recorded_at for reuse. Python’s actual registry 2.17.2 offline boundary control demonstrates .654321 becoming .000000 with the current producer-shaped prefix, while exact fractional reuse preserves .654321. A fractional fired_at does not repair that reader authority. Workflow independently confirmed the corresponding Core source defect; the earlier Core/native timer regressions rendered expected values in whole seconds and masked it.

The same native fallback is shared by reused activities and markers. The initial replayed_started_at also uses run-row time with whole-second formatting, while first-party clock readers take their initial boundary from the original WorkflowStarted event. The correction will preserve source-event precision for replay-clock fields, preserve any already-recorded reused clock string exactly, and retain existing deadline/non-clock formatting. It cannot invent fractions already lost in historical redrives.

Test predecessor 21ddab9f now uses ordered historical event timestamps with nonzero .123456 fractions after a real timer/activity lifecycle, compares parsed instants without truncation, and checks original start, completed activity, fired timer and repeated reuse. Marker controls also use .654321. Local predecessor/candidate execution follows a fresh capacity check; current shared CPU is high, so no new Server runtime/build is launched yet. A whole-head successor review and fresh required backend/corpus/current CI will follow the correction. Workflow retains the append-only Core/reference regression; actual Worker/cold-process qualification remains a later published-tuple gate. No release tag or producer artifact was created from this draft.

@rmcdaniel rmcdaniel changed the title Reuse fired plain timers when redriving a failed run Preserve replay clocks and reuse fired plain timers on redrive Nov 13, 2026
@rmcdaniel

Copy link
Copy Markdown
Member Author

Correction to the initial-clock statement in my preceding finding: it overgeneralized the SDK's legacy history-derived start accessor. The native worker-replay-clock/v1 carrier explicitly uses persisted workflow_runs.started_at for ordinary entry, and Core redrive uses the source run's persisted start. Those authorities remain unchanged. Consumed activity, marker and timer clocks use the recorded event time.

The local successor now fixes precision within those existing authorities: full persisted run-start precision for first replayed_started_at, full event precision for first step reuse, and the already-recorded reuse string unchanged on repeated redrive. The regression pins run start and WorkflowStarted event time to distinct fractional instants so it cannot silently substitute one for the other. It also compares activity/timer fractions and repeats redrive. The proposed event-start authority switch was caught during source review and was never pushed, merged or published. Legacy SDK accessors and ordinary-entry support are outside this correction.

Local attempts so far have stopped at pre-container host-capacity checks, with no test outcome inferred. The timer PR stays draft. Exact successor execution, whole-head review, fresh backend/corpus/current CI and the later immutable published Worker cohort remain required.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Precision correction is pushed as exact e8e165911a1e9cb2b87e7cbdfee88b5ed74b145d / tree 270697b7f5897fa0955c17b1c901a277e8aa5803. The complete main delta remains the same two Rust files. Relative to b1edcbde, production changes are one full-precision formatter and its use at the existing initial-anchor and reused-step fallbacks. Deadlines keep their previous formatter; persisted run-start authority is unchanged. Existing reuse strings remain verbatim on repeat.

The router regression executes the activity/timer/failure lifecycle, then explicitly controls historical fractional event times and a distinct fractional persisted run start, closes/reopens the runtime, and checks both repeated successors. It compares timer/activity event instants without truncation, checks the initial anchor and actual task clock carrier against persisted source start, and retains no-timer-row/task and source-history preservation checks. Marker controls now use nonzero fractions too. These controlled database timestamps are source regression inputs, not installed Worker evidence.

The new predecessor is 9b761f94925c4d460209820184e0994bcd787143, with the exact same test blob and the old b1edcbde runtime. Local execution has not started: five pre-container capacity refusals are preserved, including the latest 94.83% CPU observation. No container or test verdict came from those attempts. Hosted required source/backend/corpus/analysis checks now run on the exact pushed successor. Whole-head Security/Core review is requested again. Draft and publication hold remain; no old PASS is inherited.

@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

Precision-delta source counter-review PASS at exact e8e165911a1e9cb2b87e7cbdfee88b5ed74b145d, compared with reviewed b1edcbde.

The new clock formatter preserves the persisted instant's fractional precision. Only the initial replayed-start fallback and reused-step clock fallback change; deadline formatting remains separate. The initial anchor still comes from the persisted source run start, while consumed steps use their own recorded event times. Existing replayed/reused strings pass through unchanged on later redrives.

The revised control pins nonzero fractions after the real timer/activity lifecycle, closes and reopens the native runtime, distinguishes the stored run start from WorkflowStarted event time, and checks activity/timer clocks through two redrives and the polled task's replay-clock field. Marker controls also use nonzero fractions. These assertions discriminate the earlier whole-second formatter while retaining source immutability, fresh timer identities and no new timer/task work.

This is a read-only source/test review. I did not run Server controls, and it does not establish an actual SDK Worker or cold-process replay result. Fresh exact CI, Security and the declared #980 installed-consumer qualification remain required. Core's corresponding precision gap is recorded on Workflow #992 and is not corrected by this Server delta.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security renewed whole-head source review: PASS for e8e165911a1e9cb2b87e7cbdfee88b5ed74b145d / tree 270697b7f5897fa0955c17b1c901a277e8aa5803.

This renews the prior two-file timer-contract verdict after reviewing the complete current diff and the precision successor:

  • The new replay-clock formatter preserves the persisted instant's available fractional precision. Its use is limited to the initial replayed_started_at fallback and reused activity/marker/timer reused_recorded_at fallback.
  • Existing recorded replay/reuse strings remain authoritative and pass through unchanged on later redrives, so repeated redrive retains the first clock rather than reformatting or borrowing the immediate predecessor's event time.
  • The initial replay anchor still comes from the persisted source run started_at; consumed steps still use their own history recorded_at. Deadline fields retain the separate whole-second formatter and their previous authority.
  • The revised router regression executes the real activity/timer/failure lifecycle, pins distinct fractional persisted run-start and event instants, closes/reopens the runtime, and checks two successive redrives. It verifies the task replay carrier, timer and activity clocks, marker fractions, fresh shared timer identities, immediate-predecessor lineage, unchanged source history, and zero successor timer rows/tasks.
  • The underlying planner/writer contract remains the reviewed one: only an ordered complete plain TimerScheduled/TimerFired pair at a unique pre-failure step is admitted; scoped/grouped/cleanup/nonplain, incomplete, mismatched, duplicate and colliding histories refuse. Source task/command/timer identity is removed and no authoritative timer work is created.
  • The complete PR remains two Rust files; git diff --check passes. There is no schema, protocol, SDK or unrelated runtime change.

Per the requested review boundary, I did not start another local Server build. Current exact-head policy, repository-contract, tested-merge, source-fixture, Sonar and several build checks are already green; the fresh full backend/corpus/feature matrix is still running and remains the source-merge gate. Published PHP/Python/Rust cold-process replay-clock qualification remains the separate immutable-producer/adoption gate under #980.

@rmcdaniel

rmcdaniel commented Nov 13, 2026 •

Copy link
Copy Markdown
Member Author

Exact native fractional-clock control is complete on e8e165911a1e9cb2b87e7cbdfee88b5ed74b145d / tree 270697b7f5897fa0955c17b1c901a277e8aa5803.

  • The test-only predecessor 9b761f94925c4d460209820184e0994bcd787143 has the byte-identical b1edcbde runtime. With the new tests, it returned exit 101: four passed and two failed. The timer observed 00:00:07+00:00 instead of 00:00:07.123456+00:00; the marker observed 00:00:00+00:00 instead of 00:00:00.654321+00:00. This preserves the failing predecessor result rather than treating it as a qualification pass.
  • The exact corrected head returned exit 0: all six SQLite HTTP-router redrive tests passed. The same test blob (SHA256 d1a00f67b75ab3c0d59af86e806cbd6b06ea0ac255e99e23d1709a79c911b332) ran in both cells. Both cells freshly compiled the native package, with source hashes checked before/after. This avoids the earlier shared-target cache mistake.
  • The positive case verifies full timer/activity/marker fractions, distinct persisted run-start authority, the actual task replay-clock carrier, two redrives, fresh shared timer IDs, immediate predecessor lineage, unchanged source history, and zero successor timer rows/tasks. Historical timestamps are explicitly controlled after executing the real activity/timer/failure lifecycle; this is source regression evidence, not an installed SDK Worker claim.

Both cells used the existing immutable Rust image, one sequential runner at a time, 2 CPUs / 8 GiB / 512 PIDs, two Cargo jobs/test threads, network none and exact source read-only. Capacity at creation was 28.54% / 20.76% CPU with over 21 GiB available. The predecessor ran 21:29:58–21:30:18 UTC and the candidate 21:30:54–21:31:26 UTC on 2026-11-13. No OOM occurred. Both captured full container IDs are independently absent and disposable SQLite/tmpfs state is removed. Existing named Cargo caches are retained. Private diagnostic retention is seven days. Earlier pre-container capacity refusals remain runner-blocked observations and are not test outcomes.

The predecessor is reconstructible from the public b1edcbde tree by replacing only rust/tests/execution/redrive.rs with the test blob from public e8e16591.

Reproducer from the exact source with its locked cached dependencies: cargo test --locked --offline --test execution redrive:: -- --nocapture --test-threads=2.

Fresh Core source counter-review and Security whole-head review both pass. Full current hosted backend/corpus/required checks remain pending and gate source merge. Core's matching precision correction/new corpus and the later immutable-producer PHP/Python/Rust cold-worker qualification remain separately owned, unfinished work. No release tag, installed outcome or stable promotion is inferred.

@rmcdaniel
rmcdaniel marked this pull request as ready for review November 13, 2026 21:46
@rmcdaniel
rmcdaniel added this pull request to the merge queue Nov 13, 2026
@rmcdaniel

Copy link
Copy Markdown
Member Author

Exact precision-corrected head e8e165911a1e9cb2b87e7cbdfee88b5ed74b145d now has 63 successful current checks and one intended cancellation/prepared-local skip, with no pending or failed check. All current SQLite/PostgreSQL/MySQL/MariaDB source, corpus, cohorts, analysis and feature lanes finished. Shared fixture run 43509780044 is terminal success.

The tested merge 10abac09b31f56ada3db887c7b792bfbced31312 has normal parents 3f21b754 + e8e16591, and its complete tree is exactly reviewed 270697b7f5897fa0955c17b1c901a277e8aa5803. Core and Security renewed their source PASS on this exact head, and the same-test predecessor/candidate control remains linked above.

Marked ready and entered the normal merge queue. Its own group checks and final accepted-main/tree readback remain pending. This clears the source-head gate; it selects no release tag, image or installed Worker outcome. #980 stays open for the separately recorded reference and published-consumer qualification.

Merged via the queue into main with commit 5221ed8 Nov 13, 2026
64 checks passed
@rmcdaniel
rmcdaniel deleted the fix/server-980-fired-timer-reuse branch November 13, 2026 21:52
@rmcdaniel

Copy link
Copy Markdown
Member Author

#1215 is accepted through the normal queue at 5221ed82b6303379e088a4402986e544cd04b65a (2026-11-13 21:50:58 UTC). Its normal parents are accepted main 3f21b754 and reviewed head e8e16591; complete main/head tree comparison is empty, exact tree 270697b7f5897fa0955c17b1c901a277e8aa5803.

All 63 current source-head checks passed with one intended skip. All three queue workflows (PHPUnit/contracts, shared fixtures, public boundary) and the required queue Sonar check passed. Fresh Core/Security source reviews and the discriminating six-case native regression receipt remain linked above. The merged local and remote topic branches are deleted and remote absence is verified; the owned source worktree is clean at accepted main. Historical control refs remain preserved.

This completes the source merge, including the fractional correction. Post-merge main workflows are observed separately; no tag, image, alias or SDK Worker outcome is selected from source acceptance. #980 remains open. Next is the bounded 3.11.0 notes/Compose release preparation from this accepted source, followed by immutable producer qualification and the declared actual published PHP/Python/Rust cold-worker clocks. Core's separately owned precision/reference follow-up also remains explicit.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Independent post-merge Security readback: PASS.

Actual accepted main commit is 5221ed82b6303379e088a4402986e544cd04b65a, with normal parents 3f21b754418ffea5e786f2bd48481a35c92ab1a2 and reviewed source head e8e165911a1e9cb2b87e7cbdfee88b5ed74b145d. Its complete tree is 270697b7f5897fa0955c17b1c901a277e8aa5803, exactly equal to the whole-head Security-reviewed and tested-merge tree; a complete tree diff is empty. Remote main resolves to this merge, and refs/heads/fix/server-980-fired-timer-reuse is absent.

This closes the source integration/equality gate only. It does not select or qualify an immutable Server artifact, an SDK package, or a PHP/Python/Rust installed cold-process replay-clock result. Those remain the separate #980 producer and consumer gates.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Post-merge main qualification has a separate runner-blocked result at accepted 5221ed82: PHP feature run 43514302711 attempt 1 failed before PHP tests started. Docker could not fetch MySQL layer sha256:e96981fb5853b370e3a3e0b3ff8968100a89fd0b8ecbb102fd3e334554e5a8a5 from the public mirror (not found, exit 125) while starting pinned image mirror.gcr.io/library/mysql@sha256:679e7e924f38a3cbb62a3d7df32924b83f7321a602d3f9f967c01b3df18495d6. Service and TLS cleanup steps succeeded. This is not a PHP test failure or a passing main qualification.

Fresh anonymous HEAD checks now return 200 for that same 131151309-byte layer and 200 with the exact expected digest for the pinned manifest. HEAD establishes current availability only; the unchanged Docker pull will verify layer bytes. I am rerunning only the failed jobs once on the same source/image. No code, image pin, timeout, test or gate is changed. Attempt 1 remains NONPASS, and tag/publication stays held pending the actual result.

@rmcdaniel

Copy link
Copy Markdown
Member Author

The one unchanged failed-job repeat is terminal SUCCESS: 43514302711 attempt 2 at accepted 5221ed82. The actual PHP suite completed 3,634 tests / 90,234 assertions, with six existing PHPUnit deprecations and 51 declared skips; the separate Nexus/regression suites passed 51 tests / 250 assertions and one test / 12 assertions. The pinned MySQL pull and service bootstrap completed. No source, image pin, timeout or test changed.

All five actual main push workflows are now terminal SUCCESS, including current native backend/corpus, CodeQL, Sonar and public-boundary checks. Attempt 1 remains the separately recorded pre-test mirror-layer 404 / exit-125 NONPASS. The later exact-object availability and unchanged successful pull/test result resolve that runner bootstrap incident; they make no product-fix or retry-policy claim.

The underlying source is qualified for the next release preparation. Immutable tag/image, corpus/Compose/scan/provenance and actual published consumer gates remain separate under #980.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant