Skip to content

Recover service workers from transient Server outages - #69

Merged
rmcdaniel merged 1 commit into
mainfrom
fix/transient-worker-poll-recovery
Oct 7, 2026
Merged

rmcdaniel merged 1 commit into
mainfrom
fix/transient-worker-poll-recovery

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

Change

Adds Worker::recover_transient_outages(true) for long-running service workers. Retryable task-poll and worker-heartbeat outages retain the original poll request identity and retry with capped exponential backoff. Shutdown interrupts retry waits. An in-flight poll still completes and settles its leased response.

The option is disabled by default, preserving the existing bounded retry contract. run_once stays bounded, and max_retries = 0 disables ordinary retries. Registration, deregistration, authentication, malformed protocol/codec responses, handler execution and task settlement are outside the recovery option.

A fatal workflow or query poll error also signals the shared stop flag so a heartbeat recovering from a different outage cannot hide that error. Activity lanes already signal this flag.

Version and changelog prepare the routine 3.2.1 patch release. This extends the existing worker rather than adding an external restart loop.

Validation

  • All 68 focused worker tests, formatting and the warning-free cargo check --all-targets pass in a Rust 1.86 container.
  • Strict Clippy reports lints in unchanged code, including existing session validation and test helpers. It is not the repository's normal CI gate.
  • Regression cases cover all three poll families and heartbeat recovery beyond the finite budget, unchanged request identity, throttled retries, prompt poll/heartbeat shutdown, fatal authentication during heartbeat recovery, bounded one-shot calls and explicitly disabled retries.
  • Existing bounded-run and authentication checks remain.
  • Ordinary CI passes Rust 1.86/stable, package, documentation, regression corpus, Avro protocol and target-branch qualification. Public-boundary checks pass.
  • SDK 3.2.1 publication verifies the exact crates.io package against source. Registry checksum: 9d0482e9e070aefa6844a16132ca35eea8dcebc1524859f879961e57fc0985ae.
  • Sample App #152 now opts in to recovery. Its published-package timer run and merged-main repeat pass all twelve PHP/Python/Rust cases, including the physical Server restart that exposed the failure. The Rust process recovers without manual restart and completes the original pending timer history.

Refs #68 and durable-workflow/.github#122. Published-package qualification and Sample App follow-through are complete.

@rmcdaniel
rmcdaniel marked this pull request as ready for review October 7, 2026 22:43
@rmcdaniel
rmcdaniel merged commit f94b6bc into main Oct 7, 2026
11 checks passed
@rmcdaniel
rmcdaniel deleted the fix/transient-worker-poll-recovery branch October 7, 2026 22:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants