Skip to content

fix(queue): unblock parallel exact result publication#614

Merged
brokemac79 merged 1 commit into
mainfrom
codex/exact-publisher-cap-24
Jul 15, 2026
Merged

fix(queue): unblock parallel exact result publication#614
brokemac79 merged 1 commit into
mainfrom
codex/exact-publisher-cap-24

Conversation

@brokemac79

@brokemac79 brokemac79 commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Summary

Replace the single global exact-review publisher with bounded publication admission (default 24) while preserving target-serialized comment routing.

Incident evidence

  • #592 (30502fdf4e8a, merged 2026-07-15 10:38:45Z) introduced the split exact-review publisher and its global clawsweeper-exact-review-publisher concurrency group.
  • #611 restored per-target publishing for the normal review path only; it did not remove the exact publisher's global serialization.
  • #612 preserved/requeued durable exact-review lease tuples, but did not increase final-publication capacity.
  • The test issue #613 reached the successful producer run (29454686822), proving the read-only review stage can run. Its final publisher path then exposed the separate durable-publication choke point.
  • Live queue snapshot at 2026-07-15T23:25:29.922Z: 2,638 pending, 0 dispatching, 16 leased; openclaw/openclaw accounts for 2,342 pending rows. The 64-item handoff fleet has at least 48 unoccupied slots. This PR fixes the publisher bottleneck, not that separate review-admission/execution under-utilization.

Implementation

  • Add EXACT_REVIEW_PUBLICATION_MAX_CONCURRENT, defaulting to 24 and clamped to the 64-item exact-review fleet cap.
  • Admit and wake up to that many durable publication items; remove the workflow-wide publisher concurrency group that serialized them to one.
  • Preserve the existing repair-comment-router.yml target-repository concurrency boundary. Exact publishers now hand every current, nonterminal verdict to that router rather than concurrently writing its shared ledger inline.
  • Add queue, workflow, and routing-predicate tests, plus operator documentation.

No permission scope is broadened: the existing write-capable publisher token still owns GitHub mutation, and this change only uses the job's pre-existing actions: write permission to queue the existing router workflow.

Validation

Passed locally:

pnpm run build:all
node --test test/dashboard-worker.test.ts test/repair/event-apply-proof.test.ts test/clawsweeper.test.ts
node --test --test-name-pattern="exact event review hands immutable artifacts to the queue-bounded publisher lane|exact event workflow binds all work to canonical claim" test/sweep-workflow.test.ts
pnpm run lint:dashboard
pnpm run lint:repair
pnpm run lint:scripts
pnpm exec oxfmt --check <11 changed files>
git diff --check

The focused test run passed 200 behavioural tests; the exact-workflow selection and all changed-file format/lint checks passed.

Codex review closeout:

  • An initial branch review found a real P1: parallel publishers could race the shared comment-router ledger. This patch removes the inline ledger writer and routes through the existing per-target serialized router.
  • Final codex review --uncommitted: clean.
  • Final codex review --base origin/main: clean — “No actionable defects were identified in the diff. The bounded publication admission and deferred routing changes are consistently wired through the queue, workflow, and tests.”

Read-only local ClawSweeper review:

pnpm run review -- --local-range --target-repo openclaw/clawsweeper --base origin/main

It reviewed e0e44656f856..eff4017042fdef, found no discrete correctness or security finding, and made no GitHub mutation. Artifact: .git/clawsweeper/reviews/local-range-1784157727786-15652/0.md.

actionlint is not installed on this host. The full workflow-helper suite has five baseline Windows/WSL failures because this Docker Desktop WSL distribution lacks /bin/bash; those tests exercise Linux-hosted Bash helpers outside this change. The targeted exact-workflow assertions above passed.

Risks and rollout

This is a bounded throughput change, not a claim that all current backlog causes are fixed. After deployment, retained exact publication items are automatically eligible for the 24-way lane; no manual replay is needed for that publisher backlog.

Monitor for 10–15 minutes:

  1. publisher leases approaching 24 while durable comments/labels appear;
  2. repair-comment-router.yml running serially per target without ledger conflicts;
  3. total pending and especially openclaw/openclaw pending declining;
  4. the separate idle-fleet symptom (16 active / at least 48 free in the cited snapshot). If it persists, treat it as the next root-cause investigation rather than increasing this cap again.

Links

#592 · #611 · #612 · #613 · producer run 29454686822 · publisher run 29455750741

@brokemac79
brokemac79 requested a review from a team as a code owner July 15, 2026 23:27
@brokemac79
brokemac79 merged commit 648e468 into main Jul 15, 2026
10 checks passed
@brokemac79
brokemac79 deleted the codex/exact-publisher-cap-24 branch July 15, 2026 23:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant