Skip to content

feat(queue): debounce exact-review dispatch and shed low-priority enqueues under backlog#621

Merged
steipete merged 1 commit into
mainfrom
feat/exact-review-enqueue-backpressure
Jul 16, 2026
Merged

feat(queue): debounce exact-review dispatch and shed low-priority enqueues under backlog#621
steipete merged 1 commit into
mainfrom
feat/exact-review-enqueue-backpressure

Conversation

@steipete

Copy link
Copy Markdown
Contributor

Why

Two churn sources spotted during the Jul 13-15 backlog incident: (1) contributors editing a PR body and pushing a commit seconds apart produce two dispatches for the same item — the second run cancels the first mid-flight (maintainers dubbed it the re-review double-trigger); (2) pending depth is unbounded (peaked >1,400 on Jul 15) — recovery/backfill enqueues keep piling in while the queue is already saturated.

What

  • Dispatch debounce: ordinary fresh enqueues wait EXACT_REVIEW_DISPATCH_DEBOUNCE_MS (default 45s) before becoming dispatchable; merged revisions extend the wait, capped at EXACT_REVIEW_DISPATCH_DEBOUNCE_MAX_MS (default 3 min) from first enqueue so chatty items still dispatch. Explicit commands and publication items dispatch immediately — including through merges: immediacy is computed from the merged decision, so a later plain webhook can never re-debounce a pending command (regression-tested).
  • Backpressure shed: when pending depth ≥ EXACT_REVIEW_PENDING_SOFT_LIMIT (default 300), new low-priority enqueues (failed_review_shard_recovery, artifact_retention_recovery, source_drift_requeue) return HTTP 202 {ok, shed:true, reason:"backpressure"}. User/webhook events, commands, publications, revision merges, and unknown future sources are never shed. Callers in sweep.yml treat shed as log-and-skip.
  • Shed counter + pending depth surfaced in /stats and lane health; counter persisted via a guarded ALTER TABLE on the DO meta table (deploy-compatible with existing state).

Proof

  • node --test test/dashboard-worker.test.ts test/exact-review-health.test.ts test/sweep-workflow.test.ts — 192 pass (debounce hold/extend/cap, command+publication bypass, command-merge immediacy regression, shed classification incl. never-shed cases, counter in stats, workflow shed handling).
  • pnpm run check:limits green.
  • Autoreview (Codex Sol, xhigh): two accepted findings fixed (merged-decision immediacy in both merge paths); one final finding rejected after code-read — it claimed shed retains the expired publication, but the item is deleted before the shed branch (now stated in a comment).

@steipete
steipete requested a review from a team as a code owner July 16, 2026 05:31
@brokemac79

Copy link
Copy Markdown
Contributor

@steipete

Blocking — do not shed source_drift_requeue. #616 relies on that exact source action to recover the legacy tuple-less publisher artifacts introduced by #592.

Above the proposed 300-pending limit, this workflow accepts { shed: true } as success, so the publisher completes without admitting the fresh review that must produce the target comment and labels. With the live queue currently well above that limit, this would silently drop part of the incident recovery.

Please exempt source_drift_requeue. Before shedding failed_review_shard_recovery or artifact_retention_recovery too, prove each has an independent durable re-admission path.

@steipete
steipete merged commit 9fa4341 into main Jul 16, 2026
7 checks passed
@steipete
steipete deleted the feat/exact-review-enqueue-backpressure branch July 16, 2026 05:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants