Repository navigation
Add storage-pressure admission before backing storage is exhausted #131
Description
Problem
A runtime can reach its backing filesystem or native writer quota while logical namespace quotas still permit operations. In a native MySQL reproduction, committed data remained recoverable, but a full quota caused Table is full failures in maintenance and command writes; a worker task completion returned HTTP 500. Logical object/row limits are not a physical storage reserve.
Implementation scope
Add an opt-in, deployment-neutral storage-pressure input to Server. An operator-owned local health snapshot reports the resource identity, observation time and whether the resource is normal, draining or fenced. Server must fail closed on missing, stale or malformed configured observations, without needing writes to the already constrained database.
- Normal: preserve current behavior.
- Draining: reject new producers and new task claims before mutation, while allowing bounded acknowledgements for existing leases and operator recovery/inspection.
- Fenced: reject ordinary mutations before they execute; preserve read-only inspection and explicit recovery access.
- Keep authentication, role, namespace and protocol validation intact. Responses must not expose local paths or internal resource identifiers.
- Reuse worker polling/backoff and task/attempt identities. Never describe an ambiguous completion as uncommitted, blindly replay application activity effects, or claim success after a storage rejection.
- Pause new internal queue work while the resource is constrained. Document scheduler integration through the same observation rather than inventing another scheduler.
The storage operator owns authoritative measurement, reserve sizing, growth and recovery. This input is backpressure, not a replacement for native quotas, a byte reservation algorithm, or a guarantee against arbitrary write amplification between observations.
Acceptance
- Implement the optional admission input, request classifications and observable reasons.
- Test defaults, fresh normal/draining/fenced states, stale/malformed/missing snapshots and wrong resource identity.
- Prove rejection precedes persistent mutation; completed history remains inspectable; existing request/task identities are preserved.
- Verify PHP, Python and Rust poll behavior against pressure and recovery. Source-level contracts and published-package recovery passed. Rust external-payload readback was also fixed and published in sdk-rust#37 / 2.0.3.
- Qualify concurrent producers and bounded outstanding completions against a real capacity-limited database, without deleting committed data. Native MySQL proof: 40 accepted producers, 16 draining refusals, eight completed leases, native 1114, same-volume restart/growth and unchanged data. The native test also passed on the published Server image; deployment-wide quota/reserve qualification remains operator-owned.
- Publish the supported configuration/protocol and package changes after normal public checks. Server 2.3.0, PHP SDK 2.0.7, Python SDK 2.0.3 and Rust SDK 2.0.3 passed the published-tuple recovery experiment.
References: RabbitMQ disk alarms demonstrates why admission must precede disk exhaustion and why polling alone cannot guarantee an absolute bound; InnoDB error handling distinguishes statement rollback on full tablespaces from whole-transaction rollback.
Implementation merged in #132; published in Server 2.3.0. The reusable published-SDK experiment merged in #136 at 54556af636570f28b42144fd9f6bc340c938e824. Native full-capacity/growth and PHP/Python/Rust acknowledgement recovery, external-result consumption and fresh-process post-restart verification passed.
Admission remains opt-in. This completes the deployment-neutral Server contract, not any deployment's authoritative collector, all-writer reserve sizing, host quota, backup/restore or growth qualification. Operators must qualify those before enablement.
Activity
rmcdaniel commentedon Sep 7, 2026
PHP worker inspection found two concrete storage-admission gaps: pressure-rejected polls rotate their request ID, and pressure-rejected task acknowledgements terminate the worker. I am fixing these in sdk-php using the existing retry/backoff path, with tests for unchanged poll/task/attempt identities and handler execution counts. Python and Rust qualification remains required before #132 is released or admission is enabled.
rmcdaniel commentedon Sep 7, 2026
PHP worker support is merged in durable-workflow/sdk-php#61. It preserves poll identity through entry or mid-poll storage refusal, retries only the already-computed completion/failure acknowledgement, and keeps shutdown/cancellation/expired-lease behavior explicit. Local-activity heartbeat refusal cannot be recorded as an application failure.
Validation: 48 focused cases; full PHP 8.4 suite 575 tests / 3,365 assertions (one existing live-runtime memo test skipped); PHP 8.1-8.4, Laravel/Symfony compatibility, static analysis, and cold-replay checks all green in public CI. The old worker fails the new poll/acknowledgement tests, and the new cold-replay consumer fails with 503 against the old source while passing on the candidate.
Remaining before release or rollout: Python and Rust pressure recovery; real native storage-limit/reserve qualification including all writers; Cloud collector/scheduler/repair handling. This merge does not release the PHP package, close #131, or authorize enabling #132.
rmcdaniel commentedon Sep 7, 2026
Python inspection confirms that bounded HTTP retries can exhaust during pressure, after which poll loops create a new poll ID and activity completion drops the computed result. Runtime-owned payload offload also means the retry must happen below serialization/upload, around the already-prepared HTTP request. I am extending the existing Python transport retry policy only within a running worker context, including the separate query thread, while preserving direct Client behavior and graceful shutdown.
rmcdaniel commentedon Sep 7, 2026
Python recovery is merged in durable-workflow/sdk-python#57 (0def8d81b7860d4c9ca85e62d81657f30b32e37d). The existing transport policy now retains prepared worker requests through validated storage admission pauses, including activity heartbeats and the separate query thread. Direct Client/control-plane behavior stays bounded.
Validation: 1,370 non-integration tests; 80 focused storage/cold-replay cases; strict mypy/Ruff; Python 3.10-3.12, package and portal checks. Public Server integration: 6 passed, 1 existing CLI-dependent query test skipped. The new cold-completion consumer fails with the base transport policy (four replays) and passes on the fix (one replay, byte-identical acknowledgements).
Rust is next: inspection confirms bounded poll retries and direct completion/failure acknowledgement paths still need storage recovery. Shutdown must remain interruptible during registration, worker/task heartbeat and acknowledgement pauses. After Rust: published SDK artifacts and real capacity-limited database/reserve qualification across all writers. Server admission remains disabled; neither this merge nor the PHP merge closes #131.
rmcdaniel commentedon Sep 7, 2026
Rust implementation is underway. Its current bounded poll/heartbeat retry path preserves a poll ID only until the retry budget exhausts; completion/failure acknowledgements are one-shot. I am adding validated storage-refusal retry below serialization on a worker-owned Client clone, leaving direct clients/control-plane requests unchanged. The same stop flag will interrupt pressure during registration, task acknowledgements and heartbeats; storage interruption must not become an activity failure.
rmcdaniel commentedon Sep 7, 2026
Rust recovery is merged: durable-workflow/sdk-rust#34, commit 7edcf4351b0e24ec5f356996b1ded869e4534fc5.
All three first-party SDK source changes are now merged: PHP #61, Python #57, Rust #34. Rust passed 210 all-target/all-feature tests and all required public checks on Rust 1.86 and stable, packaging, docs, Avro interoperability, and the regression corpus. The cold-completion fixture fails against the previous implementation and passes with recovery: seven storage refusals, eight identical completion requests, one replayed handler invocation. Authentication, cancellation, malformed refusals and shutdown remain enforced.
This completes source-level SDK contract coverage, not live capacity qualification. None of these recovery changes has been published as a package yet, and Server storage admission remains opt-in and unenabled. Remaining work: finish the Server PR and real capacity-limited concurrent-writer/outstanding-completion experiment, publish the compatible patches, and qualify the published tuple. Do not infer a safe reserve or full-disk guarantee from HTTP fault tests.