Skip to content

Back off persistent worker poll failures - #203

Merged
rmcdaniel merged 8 commits into
mainfrom
fix/189-poll-error-backoff
Oct 23, 2026
Merged

rmcdaniel merged 8 commits into
mainfrom
fix/189-poll-error-backoff

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

Workers currently retry persistent authentication, HTTP and transport poll failures every second, multiplying requests and warnings during outages. Workflow, activity and query polling now back off independently with fresh jitter, a ceiling that doubles from 1 to 30 seconds, and reset after any successful response. HTTP 401/403 diagnostics name credential/permission failures; unchanged warnings repeat at most once per minute per poller while every failure still records an error metric.

Typed capacity waits and storage-admission propagation retain their existing paths. Failure waits release poll reservations and remain interruptible by shutdown. Worker documentation describes the limits and recovery.

Addresses #189. Initial unchanged-source evidence: 21 behavior controls fail; three storage-admission controls pass. Current focused qualification: 59 tests pass, including real Client/HTTP transport mapping, task and empty recovery, jitter, cap, warning limits, reservation release and shutdown. Ruff passes; mypy passes all 36 source files on functional candidate dceb721. Full local Python 3.12 functional qualification passes 3,183 tests / two skips / 105 integration deselections in 158.64 seconds; final focused worker/storage/backoff qualification passes 399 tests / two skips. The later cold-history compatibility control passes independently; these full/focused counts belong to their earlier exact revisions, not a whole-final-head claim. Fresh hosted normal CI remains required. A test-only Sonar warning about multiple possible invocations inside an exception assertion was corrected without changing the gate.

Publication and installed-package acceptance remain required before closing #189. The existing registry-side Trusted Publisher dependency in #197 still applies; this PR neither creates a release tag nor changes the prepared security-only version metadata.

The original hosted corpus refusal is retained. A new, actually exercised cold-history fixture proves recovery preserves the original task attempt/run and recorded activity result without rescheduling the activity. The unchanged corpus validator now accepts growth from 65 to 66 replay fixtures, and all 54 corpus guard tests pass. No guard or timeout was weakened.

@rmcdaniel
rmcdaniel marked this pull request as ready for review October 23, 2026 07:25
@rmcdaniel

Copy link
Copy Markdown
Member Author

The original final-head CI run 39918720501 is retained as failed: all three Python versions reject the new corpus fixture because I omitted its workflow from the standalone corpus runner's explicit registration list. The focused worker test exercised it, but did not cover that separate consumer.

Test-only correction c3cf6ef37eb010b6b5fe928da22463e3b8c77f4d registers the same reproducer without changing the fixture, source behavior, guard, skip policy or timeout. The combined corpus/backoff control now passes 121 tests locally; fresh normal CI is 39919997631. This does not turn the original failed run into a pass.

@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel merged commit 6752909 into main Oct 23, 2026
18 checks passed
@rmcdaniel
rmcdaniel deleted the fix/189-poll-error-backoff branch October 23, 2026 08:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant