fix(publish): bound serialized writer contention

[brokemac79]

Summary

• Feed structured state_contention publication completions into the existing adaptive capacity controller.
• Clamp serialized state publication to a four-worker incident cohort, independently of a higher configured adaptive minimum, and never let another failure raise that cap.
• Give that four-worker cohort a bounded 366-second acquisition budget: three two-minute predecessor lease TTLs plus one complete jittered poll.
• Show state publish contention in the operator dashboard instead of mislabeling the condition as GitHub 5xx pressure.

Problem and causal proof

PR #722 added one global remote-ref lease around state mutation, but publication admission retained its adaptive 24–48 worker range. The two controls were not coordinated: healthy publishers serialized behind one owner, while later members of the 48-worker cohort exhausted the old 180-second acquisition budget and returned retryable_failure/state_contention.

This is not a GitHub-token failure. Post-#724 run 29781604190 eventually published openclaw/openclaw#111766: it waited about 186 seconds through earlier owners, acquired the state lease, held it for about 44 seconds, released it cleanly, and completed as publication_applied. In the same post-merge period, other publishers failed at 180 seconds while the observed owner continued renewing normally. Those failures were then retried or dead-lettered, so apparent queue resolution could be dominated by disposal rather than delivery.

The production baseline immediately after #724 was 607 pending, 47/48 active, -40/hour 15-minute net drain, and zero publications in the prior 15 minutes. The lane later showed successful publications again, but state-contention retries and dead letters continued to dominate enough resolutions that delivery was not recovered.

Related issue #725 independently identified the same single global lease bottleneck and proposes broader per-item sharding. This PR is the narrow incident repair and remains compatible with that future architecture. Credit to @yetval for that independent report, and to the authors of #722/#724 for the serialization and state-safety work this patch coordinates.

Implementation

1. A structured retryable_failure with reason state_contention becomes publication-capacity failure feedback.
2. State contention sets the admission ceiling to four even when EXACT_REVIEW_PUBLICATION_MIN_CONCURRENT is configured higher.
3. A later GitHub transient/rate-limit failure cannot raise an existing lower ceiling; only the established clean-success recovery path may reopen capacity.
4. The default state lease acquisition timeout becomes 366 seconds. Four admitted workers means the last waiter can observe three full 120-second predecessor TTLs plus the acquisition loop's maximum five-second backoff and one-second jitter.
5. Dashboard pressure text names state contention directly.

No workflow dispatch, queue replay/reset, dead-letter replay, gate change, state layout change, retry-budget increase, or broad refactor is included.

Validation

Focused and broad checks

pnpm run build:repair
pnpm run build:dashboard
node --test test/dashboard-worker.test.ts
  156 passed, 0 failed
node --test --test-name-pattern "state publish lease" test/repair/git-publish.test.ts
  5 passed, 0 failed
pnpm run lint:repair
pnpm run lint:dashboard
pnpm run lint:scripts
pnpm exec oxfmt --check dashboard/exact-review-queue.ts dashboard/worker.ts src/repair/git-publish.ts test/dashboard-worker.test.ts
git diff --check origin/main...HEAD

The controller regression uses EXACT_REVIEW_PUBLICATION_MIN_CONCURRENT=12, proves state contention still sets capacity/ceiling to 4, then completes another publisher with github_transient and proves the ceiling remains 4. The renderer regression proves the dashboard says state publish contention, not GitHub 5xx.

Docker-backed Crabbox contention proof

Provider: local-container (Docker, mcr.microsoft.com/playwright:v1.60.0-noble)

• Lease cbx_447c46b1810a: ran the production commit and patched checkout in parallel against separate bare Git origins with four simultaneous real remote-ref lease writers, each holding the critical section for 70 seconds.
• Production 180-second baseline: 3/4 writers completed; the fourth failed with StatePublishContentionError at 180,051 ms.
• Patched checkout: 4/4 writers acquired and released; the final writer completed at 281,187 ms. CRABBOX_PROOF=PASS.
• The new capacity assertion failed on the production commit (adaptive, expected throttled) and passed on the patch.
• Lease cbx_9469821b180a: the complete test/repair/git-publish.test.ts production-surface file passed in Linux (exit 0).

The final 366-second value is six seconds safer than the successful Crabbox checkout: committed review identified that an exact 360-second deadline could miss the boundary because the loop starts before jitter and polls strictly before its deadline.

Review closeout

• codex review --uncommitted reviewed the initial dirty patch.
• Accepted findings added full-TTL polling slack, made the four-worker cap independent of configured minimum, preserved a lower cap across later failure kinds, and labeled the dashboard reason accurately. Focused proof and the full dashboard suite were rerun after the fixes.
• The branch was rebased onto actual origin/main commit 6444e678dccb3cce1505827674b05a1f2f7ce397 after fix(dashboard): bound Bay terminal records to tide #726 landed, and overlapping dashboard tests were rerun.
• Final command: codex review --base origin/main
• Final result: no actionable finding; contention feedback, capacity cap, recovery behavior, lease timeout, and dashboard rendering are internally consistent.

Risks and rollout

• Existing active publishers are not cancelled. On deployment, the first structured state-contention completion contracts new admission; already-running work drains naturally.
• Four workers still serialize state writes, but keep non-critical publication work overlapped. The measured post-fix(publish): coordinate every state branch writer #724 critical section was about 44 seconds, which is below the observed arrival interval needed to begin draining the backlog.
• The existing clean-success recovery controller may probe upward after its cooldown and success threshold. Any renewed state contention immediately returns the ceiling to four, and no failure path can raise it.
• Recovery must be judged by successful published + superseded delivery matching arrivals with declining pending depth; dead-letter disposal is not recovery.
• Broader sharding from Observational note: single state-publish lease is stalling review completion; consider per-item shard of the lease and state layout #725 can replace this coordination later without changing record semantics.

Baseline noise

The native-Windows aggregate Git-publish file contains pre-existing mixed-platform fixture failures, including /usr/bin/env assumptions and shared Git/CWD race cases. Per the platform-scope policy, this PR does not port that Linux-hosted surface. The same complete file passed in Docker-backed Crabbox Linux, and the native focused lease subset passed 5/5.

[brokemac79]

@clawsweeper review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

• State: Complete
• Detail: The item is closed; no stale verdict was published.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/29808900626
• Updated: 2026-07-21T07:03:31.444Z