fix(publish): recover exact state races without full reset

[brokemac79]

Summary

• rebuild an exact publication tuple directly on the latest remote state tree
• renew the exact-state publication lease after a lost push race
• update only paths changed by that tuple instead of resetting the production-sized state worktree

Problem and causal proof

#722 serialized exact state mutations, but it did not restore publication throughput. At the 2026-07-20T16:29:45Z post-merge baseline the publication lane still had 354 pending, 352 ready, 40/40 leases occupied, an oldest item age of 13,786 seconds, and net drain of -52/hour over 15 minutes and -56/hour over 60 minutes.

A 2026-07-20T16:40:52Z workflow inventory separated the 40 occupied publication slots into 28 pre-#722 publishers, 12 post-#722 executions, and one dispatch reservation. Thirty-seven of the 39 running publishers were already inside Publish event result and apply safe close. Ten post-#722 publishers then exhausted the new 180-second lease acquisition bound.

The lease was not deadlocked and GitHub access was not failing. The active owner repeatedly lost state compare-and-swap races to pre-lease publishers, then ran git reset --hard across a 434,644-file state worktree. Individual resets took roughly 25-30 seconds; one observed owner recorded 22 state races and 21 cannot lock ref failures. That work allowed its two-minute lease to expire while it was still recovering, leaving new writers queued behind an owner that could no longer complete under its lease. Affected logs contained no 401/403 or token-access signature.

The incident predates #712, and high publication concurrency predates the #711/#712 window. However, the renewed capacity stall immediately after #712 is high-confidence attributable to #712's synchronous missing-object/history recovery: every losing publisher could perform serial 60-second refetch, deepen, and unshallow operations before retrying. #714 showed the object-fetch premise was invalid, #716 moved no-base reconciliation onto the remote head, and #720 removed hydration from one exact path. The remaining full-worktree reset in exact race recovery is the first fault addressed here.

Implementation

• For bounded exact-tuple reconciliation, resolve the latest remote commit/tree and rewrite only the selected tuple entries with mktree --missing, then create a child commit with commit-tree. Arbitrary remote sibling paths are preserved.
• Reconcile the local checkout with read-tree, update-ref, and materialization/removal of only paths changed between the prior and replacement trees. The exact recovery path no longer performs a full git reset --hard.
• Give every lease commit a unique generation and renew ownership with a compare-and-swap push when an exact state push loses a race. Ambiguous atomic-push results remain recoverable only when the same owner still controls the lease.
• Keep the existing two-minute lease TTL, three-minute acquisition bound, state-contention retry classification, and broad reconciliation behavior unchanged.

This does not repair #711's separate broad no-common-base replacement risk. That path is not required to release the currently blocked exact publication leases and should be handled independently.

Validation

Crabbox Docker behavioral proof

Crabbox local-container, node:24-bookworm, lease cbx_d76c6b7f57f2, exit 0:

• copied the new forced-race regression onto unmodified origin/main; it failed as expected
• patched branch passed the forced exact race, eight-writer serialization, and shallow-losing-writer focused cases
• all 52 test/repair/git-publish.test.ts cases passed
• the forced race proves the default ttl_ms: 120000, a lease renewal before recovery, no post-race git reset, the bounded tuple-tree rebuild, and preservation/materialization of both the local tuple and an unleased concurrent remote sibling

Proof command:

C:\Users\marti\.local\bin\crabbox.exe run --provider local-container --local-container-image node:24-bookworm --no-hydrate --timing-json --script C:\clawsweeper-work\orchestrator\csw-049-crabbox-proof.sh

Static and build proof

• pnpm run build:all
• pnpm run lint:repair
• pnpm exec oxlint test/repair/git-publish.test.ts --deny-warnings --report-unused-disable-directives -D correctness
• pnpm exec oxfmt --check src/repair/git-publish.ts test/repair/git-publish.test.ts
• git diff --check

All passed. A separate test/sweep-workflow.test.ts probe passed 66/67 cases; its only failure was an untouched fixture requiring jq, which is absent from the stock unprivileged Node/Playwright Crabbox images. The authoritative widened proof is the complete 52-case publisher suite in Crabbox.

Retrospective #712 review

• codex review --commit 25135198f921b062b6d276931ca794d0222129c6 identified the synchronous recovery/history-fetch amplification, the invalid assumption that GitHub would serve unavailable object IDs, and a Windows-only historical test portability defect.
• A read-only local ClawSweeper range review of exact merge 25135198f921b062b6d276931ca794d0222129c6 against parent 611b3f165d05254c9ec20ee69953167c517e6c7b completed with high confidence but reported no actionable finding and rated the patch correct. Production evidence and the retrospective Codex review therefore expose a review gap; the local review added no further code requirement.
• This patch addresses the relevant runtime findings. The hard-coded /usr/bin/git test portability defect is unrelated to the production incident and is intentionally excluded.

Codex review closeout

• First codex review --uncommitted found three actionable issues: an overlong lease TTL weakened crash recovery, that TTL still could not cover every broad recovery fetch, and the state-tree diff used Git's default 16 MiB output buffer.
• Accepted fixes restored the original 120-second TTL, scoped renewal and tree-only reconciliation strictly to the bounded exact path while preserving the broad path, and raised the state-tree diff buffer to 256 MiB.
• Focused and full Crabbox proofs were repeated after those fixes.
• Final codex review --uncommitted: clean; no actionable findings.
• Final codex review --base origin/main: clean; "The bounded reconciliation rebuild and lease-renewal changes preserve the selected record tuples and maintain the lease across a lost push race. No actionable regressions were identified in the diff."

Risk and rollout

• Pre-merge/legacy publishers do not respect the lease and may continue colliding until they time out or finish. This repair makes a new exact publisher's recovery bounded and prevents a full state checkout reset from consuming its lease.
• A crashed owner still becomes recoverable after the unchanged two-minute TTL. Renewal uses compare-and-swap and cannot extend or delete a different owner's lease.
• Remote siblings are preserved by construction; the new regression forces an unleased concurrent sibling write and verifies both tuples in the resulting remote tree and local checkout.
• No capacity, queue budget, retry count, gate, token scope, workflow dispatch, state reset, replay, requeue, or GitHub API behavior changes.

Credit

This repair preserves and narrows the remote-head reconciliation work from Peter Steinberger's #711, #712, #714, #716, and #721, together with the bounded exact recovery from #720 and lease fencing from #722.

[brokemac79]

@clawsweeper review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

• State: Complete
• Detail: The item is closed; no stale verdict was published.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/29792514003
• Updated: 2026-07-21T01:14:10.373Z