Long-running worker exits on transient Server poll outage and strands waiting workflows

[rmcdaniel]

Observed failure

Published Rust SDK 3.2.0's long-running workflow worker exits during a bounded Server restart. The same PHP 2.2.3 and Python 2.4.2 workers recover and finish their pending timers. Rust's timer remains waiting after Server is healthy again because the worker process has exited.

Reproduced in the published-artifact timer experiment on Sample App #152, job 113047763895. Server 2.5.9, MySQL 8 and Redis 7. Normal completion and worker SIGKILL/cold replay pass in all three languages. PHP and Python Server-restart cells pass. Rust exits with WorkerShutdown whose primary error is a transport connection refusal from POST /api/worker/workflow-tasks/poll; deregistration also fails while the endpoint is offline. The Rust workflow remains waiting until the observer's 120-second timeout. Cancellation cells have not run in this failed job.

Acceptance

• Long-running worker poll loops recover from bounded transient connection, timeout and retryable service failures with bounded backoff.
• Shutdown remains responsive during backoff. Permanent authentication, capability, protocol and malformed-response failures remain explicit.
• Do not hide handler failures, stale completion or lost authority as transient poll failures.
• Add focused regression tests for recovery and shutdown, run normal Rust checks, publish a qualified patch, then repeat the exact published timer experiment.

This is a product finding from durable-workflow/.github#122. Do not make the experiment pass by restarting the exited worker manually or by weakening its Server-restart assertion.

[rmcdaniel]

Shipped SDK 3.2.1 through #69. Long-running service workers can enable Worker::recover_transient_outages(true) to recover retryable poll and worker-heartbeat outages with capped backoff and responsive shutdown. The default remains bounded, and one-shot calls, authentication/protocol failures, handlers and task settlement retain their existing contracts.

Package publication verified the exact crates.io artifact. Sample App #152 enables the option and passes all twelve PHP/Python/Rust timer cases, both before merge and on merged main. During a physical Server restart across the timer's original deadline, the Rust process recovers without manual restart and completes the original timer history. Worker SIGKILL/cold replay and cooperative cancellation also pass in each language.

Exact tuple: Server 2.5.9 at sha256:11c11e11498b63e644d906e43f49810a1bc0b5298f750734fe0edb279c622bca, Workflow 2.5.3, Waterline 2.3.1, CLI 2.1.3, PHP SDK 2.2.3, Python SDK 2.4.2 and Rust SDK 3.2.1. Published Sample App startup qualification passes amd64 and arm64. Both public registries serve the merged revision at image index sha256:32948877cf048baffa5369ef85105bf47d7c7d4611945e505151bc8b45cb1060.

The recovery defect and affected public consumer are delivered. The wider experiment audit remains open in durable-workflow/.github#122.