feat(queue): debounce exact-review dispatch and shed low-priority enqueues under backlog#621
Merged
Merged
Conversation
…ueues under backlog
Contributor
|
Blocking — do not shed Above the proposed 300-pending limit, this workflow accepts Please exempt |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Two churn sources spotted during the Jul 13-15 backlog incident: (1) contributors editing a PR body and pushing a commit seconds apart produce two dispatches for the same item — the second run cancels the first mid-flight (maintainers dubbed it the re-review double-trigger); (2) pending depth is unbounded (peaked >1,400 on Jul 15) — recovery/backfill enqueues keep piling in while the queue is already saturated.
What
EXACT_REVIEW_DISPATCH_DEBOUNCE_MS(default 45s) before becoming dispatchable; merged revisions extend the wait, capped atEXACT_REVIEW_DISPATCH_DEBOUNCE_MAX_MS(default 3 min) from first enqueue so chatty items still dispatch. Explicit commands and publication items dispatch immediately — including through merges: immediacy is computed from the merged decision, so a later plain webhook can never re-debounce a pending command (regression-tested).EXACT_REVIEW_PENDING_SOFT_LIMIT(default 300), new low-priority enqueues (failed_review_shard_recovery,artifact_retention_recovery,source_drift_requeue) return HTTP 202{ok, shed:true, reason:"backpressure"}. User/webhook events, commands, publications, revision merges, and unknown future sources are never shed. Callers in sweep.yml treat shed as log-and-skip./statsand lane health; counter persisted via a guardedALTER TABLEon the DO meta table (deploy-compatible with existing state).Proof
node --test test/dashboard-worker.test.ts test/exact-review-health.test.ts test/sweep-workflow.test.ts— 192 pass (debounce hold/extend/cap, command+publication bypass, command-merge immediacy regression, shed classification incl. never-shed cases, counter in stats, workflow shed handling).pnpm run check:limitsgreen.