Skip to content

fix(core): retry omitted-revision mutations under contention - #164

Merged
BrainerVirus merged 6 commits into
mainfrom
bugfix/engine-revision-retry
Oct 4, 2026
Merged

BrainerVirus merged 6 commits into
mainfrom
bugfix/engine-revision-retry

Conversation

@BrainerVirus

@BrainerVirus BrainerVirus commented Oct 3, 2026 •

Copy link
Copy Markdown
Owner

Slice S2b (docs/workit-next/plan.md, design §0 item 14).

Problem

Mutations that omit expectedRevision/expectedWorkspaceRevision read the revision before taking the store lock, so under contention they returned revision_conflict even though the caller never asked for CAS.

Change

  • Whole-operation retry. task, policy, evidence, finding, worker, writer and state import re-run the whole operation when the store rejects the commit as a revision clash: fresh read, policy, requirement and candidate checks, then commit. Nothing is resolved silently under the lock.
  • Only revisions the engine filled are retried. A clash on a revision the caller omitted is retried; one the caller supplied keeps strict CAS. So a caller who passes the current expectedRevision and omits the workspace revision survives an unrelated task.start. A clash without actual-revision details (for example an import whose source export changed) is never retried.
  • Store labels workspace conflicts correctly. mutateWorkspace/mutateTaskAndWorkspace now report expectedWorkspaceRevision/actualWorkspaceRevision. The recovery paths are unchanged, to avoid overlapping fix(core): bound recovery copies and add workit gc #154.
  • Decisions retry only the commit. decision, observeDecision and observeStandingDecision verify and retire the native receipt once. On a revision clash they re-read the task, re-check that it is not closed, re-verify the standing approval, and retry just mutateTask.
    • A standing decision records the provenance from that re-verification, so the receipt's configDigest matches the rule that is live at commit.
    • Inside the lock, the update refuses a closed task again, and the check of already-used receipts prevents a double consume.
    • This covers OpenCode and Pi, which record every decision through observeDecision.
  • Bounded. At most 8 attempts, with a random 1 to 4·n ms pause between them. Retries stop once the lock budget (defaultLockTimeout, 250 ms in-process) has elapsed, measured from the end of the first attempt, so a slow first lock wait still leaves at least one retry. The result is retryable busy.
  • state.recover never retries: it is an operator CAS over snapshot bytes.
  • Bootstrap text (methods.ts): an omitted revision absorbs a concurrent write (re-read, re-check, reapply), and persistent contention returns busy. revision_conflict means a revision you passed is stale.

Notes

  • The observeStandingDecision retry rarely fires in practice, because auto-approval passes explicit revisions.
  • Lead-writer takeover can now happen on a retry. If another lead session acquires the writer between attempts, the retried writer.acquire sees a lead-held writer and transfers it, as a sequential call would. Worker-held writers still return writer_conflict.
  • I did not add a skip of the candidate re-capture on retry: detecting "nothing in scope changed" needs the same hashing (S5's mtime cache is the place for it).

Tests (test/workit-core/revision-retry.test.ts, 20)

  • Whole-operation retry
    • A competing write with no revisions passed: the call succeeds on attempt 2.
    • An explicit stale revision returns revision_conflict after 1 attempt.
    • A requirement added between attempts returns requirements_unsatisfied.
  • Bounds
    • With interference on every attempt and a large budget, the call returns busy after exactly 8 attempts.
    • With slow interference, the deadline ends the loop before the attempt cap.
    • A first attempt slower than the budget still gets a retry and succeeds.
  • Partial revisions
    • Explicit task revision plus an unrelated start: retried, succeeds.
    • Explicit workspace revision plus a task write: retried, succeeds.
    • A stale explicit workspace revision returns revision_conflict with workspace detail keys, after 1 attempt.
  • Not retried
    • An import whose source export changed returns revision_conflict with 1 importTask call.
    • state.recover with a concurrent workspace write is not retried.
  • Decisions
    • Under a concurrent write, a receipted decision is recorded once, the verifier runs once, and the receipt is kept.
    • Task closed between attempts (receipted, stated and plain decisions): invalid_transition, and nothing lands on the closed task.
    • When the retry's re-read misses the closure, the in-lock check still refuses the decision.
    • Standing rule removed between attempts: permission_denied, nothing recorded.
    • Standing rule changed but still live: the decision records the re-verified receipt (the new configDigest).
  • Multi-process: 4 processes × 30 calls produce only ok/busy; the recorded findings exactly match the reported successes and the task count matches.

Mutation checks:

  • Retry disabled: 4 fail.
  • Retry explicit revisions too: 2 fail.
  • Cap raised to 50: 1 fails.
  • Details filter dropped: 4 fail.
  • Retry only when both revisions are omitted: 3 fail.
  • state.recover wrapped: 1 fails.
  • Decision commit not retried: 1 fails.
  • Deadline removed: 1 fails.
  • Store workspace label reverted: 2 fail.
  • Recheck closure check removed: 3 fail.
  • In-lock closure check removed: 1 fails.
  • Standing re-verify removed: 2 fail.
  • First verification's provenance kept: 1 fails.
  • Deadline started before the first attempt: 1 fails.

Measured (finding.record, 50 calls/process, Linux)

revision_conflict busy (lock) busy (retry cap) ok
main, 4 procs 10–14 21–24 – 162–169
this PR, 4 procs 0 5–10 0–2 190–193
main, 8 procs 33–55 58–165 – 202–287
this PR, 8 procs 0 48–68 9–13 319–343

Findings always equal ok.

Verified on Node 24: lint, format:check, typecheck, knip, bun run test (1258 pass), bun run test:packaging (247 pass).

🤖 Generated with Claude Code

BrainerVirus and others added 6 commits October 3, 2026 18:47
A call that omits expectedRevision and expectedWorkspaceRevision is not
asking for compare-and-swap, yet the engine resolved revisions before
taking the store lock, so losing a race surfaced revision_conflict.

The engine now re-runs such a call (fresh read, policy, requirement and
candidate checks, then commit) when the store rejects it with a CAS
conflict, up to 8 attempts with jittered backoff, then returns retryable
busy. Explicit revisions keep strict CAS. Recovery and native-receipt
decisions never retry. The bootstrap guidance now says revision_conflict
means an explicit revision is stale.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A revision the caller omitted is now retried even when the other one was
passed explicitly; the store reports workspace conflicts with workspace
detail keys so the engine can tell which revision lost the race.

Decisions retry only their commit, re-reading the revision and repeating
the record-dependent checks, so a host receipt retired before the commit
is not burned by contention; the in-lock receipt check still prevents a
double consume. Retries also stop once the lock budget has elapsed, which
bounds blocking to about twice that budget.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g on retry

A decision now refuses a closed task inside the mutateTask update as well
as in the retry re-check, and a standing decision records the provenance
of the re-verification it passed on retry rather than the first one. The
retry deadline starts after the first attempt, so a slow first lock wait
still leaves at least one retry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BrainerVirus
BrainerVirus merged commit c483927 into main Oct 4, 2026
5 checks passed
@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 2.2.1 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant