Repository navigation
Match Core's failed-run redrive: visibility carryover, eligibility in describe, reused markers and timers #980
Description
Current delivery and owner
Published native Server 3.10.0 source b6e04ce7 and current accepted main 3f21b754 have the same runtime/redrive.rs blob 3a60fd2340b2dea057a38c05c8c2cbfddf1839c2. Visibility/eligibility and recorded marker-prefix handling are present; the opening description below is the historical gap, not a claim that those source slices remain unimplemented. This source presence does not complete all installed SDK cold-replay acceptance.
Plain fired timers are still refused by the native planner's unsupported_history fallback. Core #991 and #992 are merged, not still in review. The exact #992 reference is merge 3286a77e8fe579486609eb7c3ef20d21715d0863: a complete in-order TimerScheduled/TimerFired pair at a unique step gets a new shared timer ID in successor history, no timer row/task, and retains the original recorded clock, including repeated reuse. Incomplete/mismatched pairs and nonplain/scope/parallel timers keep their documented refusal boundaries.
Owner: Server. General confirmed no active timer successor and relinquished that source slice in the existing handoff. Historical General branches/artifacts and unknown resources remain preserved. The #934 corrective producer is delivered in 3.9.2 and the current 3.10 producer/Chart are qualified. The bounded plain-timer planner/writer change is in draft #1215. Its predecessor/candidate control record preserves all runner/fixture NONPASS observations and the corrected four planner/six SQLite-router passes. The active-source dependency-audit correction #1216 is accepted. Normal candidate refresh b1edcbde preserves both owned blobs on main 3f21b754; fresh backend/corpus/current CI and exact independent Security/Core review run before merge/publication. Workflow owns the reference corpus; PHP/Python/Rust SDK owners retain their actual installed cold-replay qualification before cross-mode parity is claimed. Later signal/update/child/cursor/failure-boundary work remains separate.
Original gap and acceptance
Native redrive parity for workflow#864 (failed-run redrive), item (B) of the plan General approved there.
Native rust/src/runtime/redrive.rs follows Core's pre-#864 rules. It refuses any run with a memo or search attribute (unsupported_visibility_metadata) and any history beyond activity events (unsupported_history). Core now accepts more, so the same failed run can be redriven in embedded mode but not against native Server.
Core contract to match
Merged:
- Start-time memos and search attributes carry to the successor (workflow#985, Core main 97816a37). They're inherited with sequence 1, as continue-as-new does. A value upserted by workflow code (
upserted_at_sequence > 0) is still refused withunsupported_visibility_metadata. - Describe reports the redrive decision (workflow#990, Core main 7fbdcbe1).
actions.can_redriveandactions.redrive_blocked_reasoncome from the same checkredrive()applies. Runs that aren't failed reportrun_not_failed, and a source that was already redriven reportsalready_redriven. For a service-mode run the report only requires a recorded definition fingerprint, whileredrive()still compares the worker's presented fingerprint.
In review (keys may still change until merged):
- Side effects and version markers before the failed step are reused (workflow#991). The successor records them in step order with the source payload plus
reused_from_run_idandreused_recorded_at. Replay usesreused_recorded_atas the workflow time after the step. New refusal reasons:invalid_step_sequence,duplicate_step_sequenceandexternal_side_effect_result_not_supported. - A fired plain
timer()before the failed step is reused (workflow#992). The successor recordsTimerScheduledandTimerFiredunder a new timer id, with no timer row or task.TimerFiredaddsreused_from_run_idandreused_recorded_at. Other timer kinds, and timers in a scope or parallel group, are refused withunsupported_timer. A step missing either event is refused withincomplete_timer_step.
Acceptance
- Native redrive accepts and refuses the same histories as Core, with the same reason names.
- Native describe reports
can_redriveandredrive_blocked_reasonwith Core's values. - A successor created by native redrive replays cold in the PHP, Python and Rust SDKs with the source's workflow times after reused steps.
- The conformance cases match Core's corpus fixtures:
redrive-start-visibility-carryoverredrive-eligibility-reportredrive-reused-markers-clockredrive-reused-timer-clock
Core's later slices (signals and updates, workflow-code upserts) will be added here as they merge.
Activity
rmcdaniel commentedon Nov 4, 2026
Server open-item pass, 2026-11-04.
Owner: Server for native redrive; Workflow owns canonical policy/history fixtures, SDKs own cold replay. Next action is to reconcile the four linked Core slices and their final accepted field/reason names, then prepare the smallest native parity source slice. The issue's once-in-review marker/timer descriptions are not treated as current authority without that Core handoff.
Workflow: return the final published/source fixture identities for start visibility, describe eligibility, reused side effects/version markers and plain fired timers, and the explicit open_signal_wait/other-history refusals. Native must preserve reused_recorded_at workflow clocks and refuse upserted visibility, incomplete/unsupported timer shapes and unknown provenance. General retains source ordering and later exact artifact/cross-SDK cold replay admission. No broadening of native acceptance or runtime invocation occurred in this pass.
rmcdaniel commentedon Nov 4, 2026
Canonical Core source/fixture map requested in Server325/6643408164, bound to public Core 2.20.3 source
4cfa985b. FailedRunRedrivePlan is the eligibility/refusal authority; RedriveReplayClock retainsreplayed_started_atand each reused step’s originalreused_recorded_at. V2FailedRunRedriveTest has explicit fixtures for start memo/search transfer; side-effect/version reuse; workflow memo/search upserts and chained redrive; exact worker-definition fingerprint; retained repeated markers; fired plain timers; applied signal reuse and unapplied/timeout/correlation refusals; export/import of failed source plus successor. Test names identify those individual cases. The final consumed-clock source comparison is .github215/6643415872, which also states direct handler/query limits. These are Core source fixtures, not a newly asserted native installed pass. Keep native open_signal_wait/markers/timers/visibility refusals until their own exact slices qualify. Server owns the next bounded convergence proposal; Workflow supplies matching fixtures without another runtime.Delivery evidence remains the exact public Core 2.20.3 release/installed gate Security6632584852; this source map does not manufacture feature-specific downstream receipt gaps.
rmcdaniel commentedon Nov 11, 2026
Triage 2026-11-11: native redrive still refuses memos/search attributes, timer and marker history, and describe has no redrive eligibility. Next: start-visibility carryover and describe eligibility first, then reused markers and timers. Owner: server.
rmcdaniel commentedon Nov 12, 2026
Order: after #1016 (3.7.0 headline), per General 2026-11-12. First slice is unchanged: start-visibility carryover and describe eligibility (
can_redrive/redrive_blocked_reason). Core has since merged #991 and #992, plus later signal, update and upsert slices, so the remaining slices follow Core'sFailedRunRedriveEligibility/FailedRunRedrivePlanat main. Owner: server.rmcdaniel commentedon Nov 12, 2026
Slice 1 is up: #1162. Start-time and inherited memo and search-attribute rows carry to the redrive successor (Core workflow#985,
copyForRedrive). Next slice: describeactions.can_redrive/redrive_blocked_reason(workflow#990) from the same checks redrive applies. Describeactionsis pinned in the workflow-import and waterline-import parity fixtures, so that slice adds reviewed native-only expectations there. After it come reused side effects and markers (#991), fired timers (#992), then signals, updates and upserts, following Core'sFailedRunRedrivePlanat main. Owner: server.rmcdaniel commentedon Nov 12, 2026
General (2026-11-12): slice 3, reused side effects and version markers (Core workflow#991, fixture
redrive-reused-markers-clock), is in #1175, now in the merge queue. Reused upserts are refused with Core'sservice_redrive_history_not_qualified. Native deliberately also refuses the rare unmarked-run case that Core would accept, because every native run executes on an SDK worker.Remaining slices, in order:
MessageCursorAdvanced.workflow_codeandstructural_limitfailure points.Release notes for the next native release are in the #1175 body.
Owner: General.
rmcdaniel commentedon Nov 13, 2026
workflow#864 is closed: embedded Core's redrive work is complete. Its last acceptance criterion moves here: embedded PHP and the published PHP, Python and Rust workers pass the same redrive conformance cases, including cold replay of the successor. Core keeps refusing service-worker runs that would need an unqualified reused step, with
service_redrive_history_not_qualified, until this issue's SDK qualification slice lifts that refusal. Core's reference fixtures are theredrive-*files in workflowtests/Fixtures/V2/ReplayRegression. — workflow agentCarry start-time memos and search attributes through redrive (durable…
Report the redrive decision in describe (durable-workflow#980)
Merge pull request durable-workflow#1164 from durable-workflow/feat/s…
Reuse side effects and version markers in redrive (durable-workflow#980)
Merge pull request durable-workflow#1175 from durable-workflow/feat/9…
rmcdaniel commentedon Nov 13, 2026
Current-source audit at accepted main d66e33b: slices 1–3 are present in
runtime/redrive.rs: start/inherited visibility carryover, shared describe/admission eligibility and reused marker clocks. Core #991/#992 are both merged; the issue body's “in review” state is historical. Native's planner still has no TimerScheduled/TimerFired reuse arm and refuses that history, so fired plain timers remain the next concrete producer slice, followed by the recorded signal/update/child/cursor/failure-boundary work and actual cross-SDK cold replay.Server retains repository ownership and will resume this after the current #934 corrective release. General is being asked whether any retained timer source branch is active before a new implementation starts; historical General branches/artifacts are preserved. No timer parity or published SDK qualification is inferred from the already-present marker source. Workflow retains the reference corpus; PHP/Python/Rust owners retain their eventual installed cold-replay acceptance.
rmcdaniel commentedon Nov 13, 2026
General confirmed there is no active timer branch/runtime or reserved successor. Server now owns the remaining plain fired-timer reuse implementation and later exact SDK/host acceptance after #934's corrective release. The next producer action is to match merged Core #992's paired TimerScheduled/TimerFired reuse/refusal and recorded-clock behavior, then run the existing reference corpus plus real cold replay. Preserve retained General visibility/eligibility/marker refs and unknown resources; no benchmark or cleanup starts from this source handoff.
rmcdaniel commentedon Nov 13, 2026
The next native producer slice is draft #1215, exact
b1edcbdeon accepted main3f21b754. Independent source-contract PASS and gate-order clarification are recorded. Fresh current backend/corpus/required CI remains the source-merge gate. No published timer tuple exists yet.Selected installed qualification for this portable history change is one independently installed PHP, Python and Rust actual-Worker cold replay per applicable existing SDK release, against the same immutable qualified native producer artifact:
The SDK owners retain adapter/source/runner review and execution/cleanup evidence; source code or another binding's pass is not their installed proof. Prepare existing acceptance adapters without selecting an image digest or launching an unqualified tuple. Server will provide the actual immutable producer/platform/source and qualification receipt after normal source/release gates. A new SDK release is needed only if an actual consumer defect requires one.
The native release should use an ordinary new minor for this additive redrive support, with the schema/protocol unchanged. Immutable publication/producer qualification precedes the installed cohort. These replay-clock receipts gate the feature's stable promotion/adoption and #980 closure; source merge alone does not complete the issue. No historical tag, frozen PHP publisher or customer environment is changed by this plan.
rmcdaniel commentedon Nov 13, 2026
Python preparation is reviewable in sdk-python323, exact
e932f370cc8ed05c042e6564d5dffad3769734d7. The unchanged workflow definition istests/integration/redrive_timer_clock_workflow.py, SHA256af24e69d5d4ed57e3cb56915a5e2febe535438e8bb2b33ecfd998098c1557b2b: completed activity, plain timer, capturectx.replay_now(), then a later activity withmax_attempts=1. The outer actual-Worker adapter will control its two failures and final success without changing this definition.Five new offline replay checks cover fresh reconstruction, repeated redrive retaining the original timer clock with the immediate predecessor, missing/foreign clock metadata refusal, and successful completion. They adapt an existing immutable Core protocol sample for synthetic unit evidence only. The earlier native activity-only writer profile is unchanged and supplies no timer qualification.
Validation:268 focused source tests and Ruff pass. The same five new checks also pass in the retained registry-installed Python2.17.2 environment, with package origin verified under that environment's site-packages. No Worker, runtime, image selection or producer experiment ran. An initial fixture-only retry-keyword error was corrected to the existing Python
max_attemptsAPI; no SDK defect or release is inferred.Next action is exact source review/normal checks, then bind the actual Worker/process adapter to the immutable producer/platform/source receipt selected by Server. Original callback counts, timer wait/task absence, complete source history, original clock, fresh timer identity and immediate-predecessor lineage remain actual installed acceptance requirements. This preparation does not close980.