fix(queue): unblock exact review publication

[brokemac79]

Summary

• give exact-review artifact publication its own serial GitHub Actions concurrency lane instead of placing it behind every bulk state publisher
• recover both new and already-persisted unclaimed publication dispatches after 15 minutes, with stale lease tuples rejected safely
• use the existing five-minute dashboard Worker cron as a guaranteed post-deploy queue heartbeat, and correct the false future/zero-age dispatch telemetry

This fixes the confirmed path where a review completed successfully but its final ClawSweeper comment and labels could not be published. It deliberately does not bulk-rewrite every pending review row; the separate stale-pending recovery/capacity follow-up remains necessary if the producer backlog does not drain normally.

Incident evidence

At 2026-07-15T18:00:34Z, the live queue API reported:

• pending=1961, dispatching=1, leased=15
• oldest pending item 2026-07-15T11:01:02Z
• 48 of 64 review slots shown as available
• a false oldest_dispatching_at=2026-07-22T13:10:22Z with age zero
• next_wake_at=2026-07-15T20:05:52Z

The false future timestamp came from deriving the dispatch start by subtracting the normal six-minute lease from the seven-day publication lease introduced with #592. It made a multi-hour stuck handoff look healthy, but it was telemetry fallout rather than the cause.

The publication failure itself is directly visible in Actions:

• run 29418496158, Review event item openclaw/openclaw#107676@publish:29409874087:1, was created at 13:16:24Z and was still pending at 18:00Z
• its only relevant job, 87362614333, had no runner and had never completed
• an earlier exact publisher, job 87336341784, was created at 11:07:50Z but did not start until 13:14:06Z
• a representative bulk publisher, job 87350685065, was created at 12:22:22Z and did not start until 16:10:32Z

The live GitHub concurrency endpoint initially showed 19 members in clawsweeper-state-publisher, with the exact publisher last behind 18 generic publishers. At 18:00Z it still showed three members in order: one running generic publisher, one pending generic publisher, then exact job 87362614333. That proves the final publisher was queued behind bulk state work, not waiting for Codex or a runner after a claim. The current endpoint is here.

Representative producer runs succeeded:

• openclaw/openclaw#107785 producer run 29411365434, created 11:21:51Z, completed successfully
• openclaw/openclaw#107787 producer run 29412181317, created 11:35:41Z, completed successfully

Those runs reached real Codex completion, artifact upload, and durable publication enqueue. This rules out a global Codex/GPT, Cloudflare intake, token, or GitHub permission outage as the cause of the missing final comments/labels. It does not claim that every individual review run was healthy.

The issue timestamps are also not a reliable merge boundary: #107785 received its recorded labels at 10:41Z, and #107787 has a durable review comment at 10:46Z, before the later producer runs above. The Actions concurrency evidence provides the actual causal boundary.

Causal timeline

• 00:11:35Z: Vincent's #586 revert merged. It predates the publisher regression and is not on the causal path.
• 10:38:45Z: Peter's #592 merged as 30502fdf4e8acdcbf022302278af0e78a307eb65. It placed exact event-review-publish and generic publish jobs in clawsweeper-state-publisher, both serial with queue: max, and extended unclaimed publication leases to seven days to tolerate that queue.
• 10:55:39Z: #593 merged. It forwards exact-review bundle CLI commands; it does not separate or prioritize publishers.
• 11:07:50Z: exact publisher job 87336341784 entered the shared group and waited over two hours.
• 11:21Z / 11:35Z: representative #107785/#107787 producer runs started and later succeeded.
• 12:22:22Z: representative generic publisher job 87350685065 entered the same group and waited nearly four hours.
• 13:16:24Z: run 29418496158 entered for an already-produced exact artifact and remained pending with no runner.
• 13:17:47Z: #596 merged. Its Landlock fallback and exact-claim conflict handling are independently valid, but it does not change the shared publisher group.
• 15:44:33Z: #601 merged as 68f83c25fc564ebfb76ac75ed3d64a7cb1c07bb8. It stops dispatcher pauses from repeatedly deferring the whole pending backlog. It came after the publisher starvation and is not its cause.
• 16:32:18Z: live queue sample: pending=1717, dispatching=1, leased=19, with the same future dispatch timestamp.
• 18:00:34Z: live queue sample: pending=1961, dispatching=1, leased=15; exact run 29418496158 was still last in the shared publisher group.

Why #601 is retained

#601 was evaluated specifically because it was not the publication root cause. Reverting it would restore a separate confirmed defect: a paused dispatcher rewriting retained pending rows' nextAttemptAt on every pass. The focused regression exact-review queue defers retained backlog until a paused dispatcher retry passes on this branch. This PR therefore keeps #601 and documents the distinction rather than combining a known regression into the incident repair.

Other nearby work is also not causal:

• #589 and #591 are open and unmerged
• #581 explicitly reports inaccurate re-review acknowledgement wording, not queue latency; it is independently valid but separate from this outage

Implementation

1. Exact event-review-publish jobs use clawsweeper-exact-review-publisher; bulk publish jobs remain in clawsweeper-state-publisher. Both lanes stay serial and non-cancelling.
2. Unclaimed publication handoffs use an effective 15-minute lease (or a longer configured ordinary dispatch lease). Claimed executions retain the existing execution lease and are not shortened.
3. Existing seven-day rows are capped using their persisted dispatchedAt. Reclamation, claim validation, queue wake scheduling, and stats all use the same effective expiry.
4. If an old queued job arrives first, its expired lease is persisted back to pending, an immediate alarm is scheduled, and the stale claim receives 409 lease_not_active.
5. If no request arrives, the normal five-minute dashboard Worker cron calls the queue stats heartbeat, reclaims the old row, and re-arms the alarm. The new dispatch uses the dedicated lane.
6. Handoff health no longer accepts a derived start time in the future, so a long legacy lease falls back to the real dispatchedAt and reports its actual age.

Automatic recovery and backlog scope

After merge, the normal dashboard push workflow deploys the Worker because this PR changes dashboard/**. Within one five-minute cron interval, an old unclaimed seven-day publication row is re-admitted and redispatched. New exact publisher jobs use the dedicated lane immediately. A late job from the old shared lane cannot publish twice because its lease tuple has been replaced.

This should restore final comments and labels for completed reviews without a manual Actions dispatch or gate change. It will not immediately reprocess all pending=1961 producer rows. Those remain governed by normal exact-review capacity plus #601; bounded stale-pending recovery and the observed capacity under-utilization should be diagnosed separately before any bulk mutation.

Validation

Windows focused proof on the final branch:

corepack pnpm build:all
node --test test/dashboard-worker.test.ts test/exact-review-health.test.ts
# 117 passed, 0 failed
node --test --test-name-pattern "exact event review hands immutable artifacts to one dedicated publisher" test/sweep-workflow.test.ts
# 1 passed, 0 failed
corepack pnpm lint
corepack pnpm exec oxfmt --check <six touched files>
git diff --check origin/main...HEAD

The queue regression recreates both live rollout paths: a persisted publication whose stored lease is seven days in the future but whose dispatchedAt is 16 minutes old is recovered by the scheduled heartbeat; a second expired legacy dispatch is recovered when its stale job claims first. Both are redispatched with new tuples, and stale claims return 409.

Hosted-Linux proof used Crabbox local-container, image mcr.microsoft.com/playwright:v1.60.0-noble, lease cbx_02442d834d3d:

• build, lint, and six-file format check passed
• queue/dashboard health tests: 117 passed
• dedicated publisher workflow assertion: 1 passed
• continuous reconcile-records state-race tests: 3 passed
• pinned actionlint v1.7.12 passed for sweep.yml (archive SHA-256 8aca8db96f1b94770f1b0d72b6dddcb1ebb8123cb3712530b08cc387b349a3d8)

Five unrelated full sweep-workflow Bash-fixture tests cannot run natively on this Windows host because its default docker-desktop WSL distro has no /bin/bash; the touched workflow assertion and state-race proof passed in the Linux container. No platform policy or production shell surface was changed.

Codex review closeout

• codex review --uncommitted initially found the valid rollout gap that old seven-day leases were not migrated; fixed and proved
• the next pass found that a stale claim returned 409 without persisting/scheduling recovery; fixed and proved
• the next pass found that deployment alone did not instantiate the Durable Object or re-arm its old alarm; the five-minute scheduled heartbeat was added and proved
• final codex review --uncommitted: clean, no actionable findings
• final codex review --base origin/main: clean, no actionable findings; confirmed the dedicated lane and recovery do not affect claimed executions

Risks and rollout

• Exact and generic state publishers may now overlap, at most one in each lane. The existing tuple-aware reconcile-records writer handles disjoint state races; all three continuous race tests passed on Linux.
• A GitHub delay longer than 15 minutes can produce another dispatch attempt, but only the current lease tuple can claim and publish.
• This touches .github/workflows/sweep.yml; OAuth tokens without workflow scope may not be able to update the branch. The maintainer branch push succeeded, and CI/actionlint still need to remain green.
• No gates, direct workflow dispatches, Actions reruns, or manual state mutations are part of rollout.
• No changelog change; release-note context is contained here.

Links

• Live queue API: https://clawsweeper.openclaw.ai/api/exact-review-queue
• Root regression: feat: serialize exact review publication #592
• Stuck exact publisher: https://github.com/openclaw/clawsweeper/actions/runs/29418496158
• GitHub concurrency behavior: https://docs.github.com/en/actions/how-tos/write-workflows/choose-when-workflows-run/control-workflow-concurrency
• Related, retained backlog fix: fix(queue): preserve pending backlog across dispatcher pauses #601
• Adjacent repairs reviewed: revert: restore pre-campaign ClawSweeper state #586, fix: forward exact review bundle commands #593, fix: recover repair and exact-review workers #596

[brokemac79]

@clawsweeper review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

• State: Complete
• Detail: The item is closed; no stale verdict was published.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/29482549560
• Updated: 2026-07-16T08:12:59.225Z