Skip to content

fix(core): bound recovery copies and add workit gc - #154

Merged
BrainerVirus merged 23 commits into
mainfrom
bugfix/bounded-recovery
Oct 3, 2026
Merged

BrainerVirus merged 23 commits into
mainfrom
bugfix/bounded-recovery

Conversation

@BrainerVirus

@BrainerVirus BrainerVirus commented Oct 3, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #150 (bugfix/store-lock-reclaim). Review only the top two commits.

CI note: this PR targets a non-main branch, so the CI workflows do not run on it. They will run once it is retargeted to main after #150 merges. Local verification is below.

What

  • Recovery copies are capped. Each task or workspace record keeps its newest 3 copies (RECOVERY_COPIES_PER_RECORD) in .workit/recovery/. Pruning runs after each copy, always keeps the copy just written, and is best-effort: a pruning failure never fails the mutation. When an existing copy is saved again with the same bytes, its mtime is refreshed so it counts as newest.

  • New workit gc [--dry-run] [--json] (TaskStore.collectGarbage). It runs in the current directory and:

    • prunes every record's copies down to the cap,
    • removes *.tmp files older than 1 h that crashed writers left in recovery/, tasks/ and .workit/,
    • dedupes stored candidates where it is safe: duplicate copies of the same content-addressed candidate ID collapse to their latest position, so lookups by ID and candidates.at(-1) are unchanged. This only happens in paused tasks. Active tasks (skippedActive) are left alone so a live session's revision is not bumped under it. Closed tasks (skippedClosed) are never rewritten, because they are immutable history (docs/workit-v1/contracts.md).

    It never deletes tasks/*.json or workspace.json. --dry-run is strictly read-only: no lock, no mkdir, no .gitignore rewrite. The root resolves like task commands (WORKFLOW_WORKSPACE_ROOT, then cwd).

  • state.recover is no longer advertised. I checked the claim in code first. The operation requires OperationContext.nativeRecovery, and no production host sets it. The only sources are workit-cli/src/task.ts deps.nativeRecovery, which index.tsx never passes, and two core tests. The OpenCode, Pi, MCP (Cursor and Codex) and CLI hosts all go through paths without it, so in production the operation can only return permission_denied. Changes:

    • operationJsonSchema and boundedOperationJsonSchema now use the new advertisedOperationSchemas, where state is export|import. MCP and Pi read their schemas from these.
    • OpenCode's tool shape uses advertisedOperationSchemas.
    • The CLI drops state recover.
    • parseOperation and the engine path are unchanged, so embedders that do supply nativeRecovery, and the existing continuity tests, keep working.
  • README, AGENTS.md and CHANGELOG are updated.

Why

The audit (§3.4, §6 #3 and #10, §7.2 #2) measured 818 MB and 5,752 files in recovery/ for 74 tasks, growing about quadratically with mutations. It also found that state.recover was advertised on every host but could not succeed anywhere.

Given/When/Then covered (test/workit-core/recovery-gc.test.ts)

  • Given 20 writes to one task, Then at most 3 recovery copies remain and the latest prior bytes are kept
  • Given 1,000 writes to one task (soak, opt-in with WORKIT_SOAK=1; measured 7.9 s and passing)
  • Given a recovery dir with N stale copies per record, When gc runs, Then each record keeps only the newest copies up to the cap (40 task copies and 25 workspace copies leave 3 + 3)
  • Given gc --dry-run, Then it reports what it would remove and writes nothing at all (unchanged .workit listing and mtimes; no .gitignore or lock created)
  • Given gc runs, Then every current read returns the same records (byte-identical tasks/*.json and workspace.json, equal listTasks)
  • Given paused, active, and closed tasks with duplicate stored candidates, When gc runs, Then only the paused task collapses to its latest positions (the closed record's bytes are unchanged)
  • Given a stale leftover temp file, When gc runs, Then it is removed while fresh temp files stay
  • Given stale recovery copies, When workit gc --json runs, Then it prunes to the cap and reports the count
  • Given no host supplies native recovery authority, Then state.recover is not advertised but stays parseable for the engine

Existing tests updated, none deleted:

  • task-commands.test.ts: the CLI action count goes from 24 to 23.
  • mcp/server.test.ts: the advertised-action parity excludes state.recover.

Verification (measured, Node 24.20.0, Bun 1.3.14, in the worktree)

  • Red before the change: on the S1 code, 1,000 mutateTask writes left 1000 recovery files. With this change they leave 3. The time for 1,000 writes is fsync-bound and stayed in the same range across runs (S1: 7.8 s / 22.8 s, S2: 8.9 s / 38.9 s on a noisy disk; no consistent regression). The new test file could not even load against S1 because collectGarbage and RECOVERY_COPIES_PER_RECORD did not exist yet.
  • Green after: 8/8 recovery-gc tests, MCP server 17/17, CLI task-commands green.
  • End to end with the built CLI (node packages/workit-cli/dist/index.js): a checkout seeded with 6,000 stale copies (3,000 task + 3,000 workspace) went through workit gc in 0.6 s, with 5,994 copies removed (11.8 MB) and 6 kept. task inspect output was byte-identical before and after. A later task progress succeeded and the directory stayed at 6 files.
  • bun run lint ✅, format:check ✅, typecheck ✅, knip ✅, build ✅.
  • Full bun test after the review fixes: 1598 pass / 1 skip (soak) / 1 fail. The one failure is AR-14, which already fails on origin/main because its repoRoot resolves to the parent of the checkout (details in fix(core): reclaim stale metadata locks and report contention as busy #150).

Risks

  • Pruning orders copies by mtime with nanosecond precision. A filesystem with coarse mtimes could tie-break by name, but the copy just written is always kept.
  • The first write after upgrade still lists the whole recovery/ directory, but only lstats the copies of that one record. Running workit gc once clears old backlogs.
  • Candidate dedupe bumps the revision of the paused tasks it rewrites. A stale explicit expectedRevision held for such a task gets revision_conflict.

🤖 Generated with Claude Code

BrainerVirus and others added 3 commits October 3, 2026 14:49
A .workit/metadata.lock left by a dead process bricked the store: every
mutation returned recovery_required and nothing could clear it. Contention
between live writers (timeoutMs 0, no retries) was also mapped to
recovery_required, which the bootstrap treats as a stop condition.

- Reclaim a lock whose owner is provably gone: dead pid, a pid reused by a
  different process (start time differs), a foreign-host lock past a 10 min
  TTL, or an unreadable lock past 30 s. Abandoned reclaim guards past 30 s
  are cleared as well.
- Retry a live holder for up to 2 s, then return the new retryable `busy`
  code. Losing a reclaim race retries instead of failing.
- Genuine data-integrity failures (corrupt or foreign records) and the
  explicit state.recover protocol keep returning recovery_required.
- `workit doctor` gains a workspace_lock check; `workit doctor --fix-lock`
  clears a stale lock and abandoned guard, never a live one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review findings on the lock reclaim change:

- Lock identity now includes the Linux pid-namespace inode and boot id,
  folded into `host` (`host#pidns:bootid`) so older strict schemas still
  parse it. A container sharing the hostname, another boot, or a legacy
  lock is treated as foreign (TTL rules only) instead of being checked
  against this host's process table.
- `doctor --fix-lock` clears under the same `.reclaim` guard writers take,
  skips when a reclaim is in progress, and re-verifies the bytes just
  before removal, so it can no longer delete a lock a writer just took.
- The wait budget is 250 ms by default (in-process hosts) and 2 s in the
  CLI process; a lost reclaim race retries within the remaining budget.
- /proc stat start time is parsed after the last ")"; macOS/FreeBSD use
  `ps -o lstart=`. `doctor --fix-lock --force` (with --yes or a TTY
  confirmation) is the explicit escape hatch; `--fix-lock` honours
  WORKFLOW_WORKSPACE_ROOT like task commands.
- Bootstrap: a revision_conflict on a call without expectedRevision is
  retryable contention.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BrainerVirus
BrainerVirus force-pushed the bugfix/bounded-recovery branch from 737ffdd to e8f869c Compare October 3, 2026 18:43
…iable ones

- A lock from the same host and pid namespace but another boot id is
  stale at once: its owner died with the previous boot.
- doctor's workspace_lock warns when an unverifiable lock (other host,
  pid namespace, or an older Workit's plain-hostname lock) has blocked
  writes for over 30 s, with the exact `workit doctor --fix-lock --force
  --yes` command.
- Regression test: a plain-hostname lock naming live pid 1 with a
  mismatched start time (a container's lock) stays busy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BrainerVirus
BrainerVirus force-pushed the bugfix/bounded-recovery branch from e8f869c to 8be531b Compare October 3, 2026 19:05
…ks carry namespace identity

On macOS and Windows there is no pid-namespace identity, so a
plain-hostname lock is the host's own format and pid checks apply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BrainerVirus
BrainerVirus force-pushed the bugfix/bounded-recovery branch from 8be531b to 1b3b439 Compare October 3, 2026 19:27
BrainerVirus and others added 3 commits October 3, 2026 16:59
Resolve task-store.ts: keep main's task index; the metadata lock schema
now lives in store-lock.ts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every snapshot replacement copied the previous bytes into
.workit/recovery/ with no cap (818 MB / 5,752 files measured in one
checkout), and state.recover was advertised everywhere although no
shipped host supplies the native recovery authority it requires.

- Keep the newest 3 recovery copies per task/workspace record; pruning
  runs after each copy and never fails the mutation.
- `workit gc [--dry-run] [--json]` prunes copies beyond the cap, removes
  temp files older than 1 h, and collapses duplicate stored candidates
  (same content-addressed ID, latest position kept) in paused/closed
  tasks. Live task and workspace records are never deleted.
- Stop advertising state.recover in host tool schemas (MCP, Pi, OpenCode)
  and the CLI; parseOperation and the engine path stay for embedders
  that supply nativeRecovery.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Candidate dedupe now rewrites paused tasks only; closed tasks are
  immutable history and are reported as skippedClosed.
- `workit gc --dry-run` no longer takes the lock or initializes storage
  (no mkdir, no .gitignore rewrite).
- `workit gc` resolves the workspace root like task commands
  (WORKFLOW_WORKSPACE_ROOT, then cwd).
- The bounded-copies test uses 20 writes; the 1,000-write version is an
  opt-in soak behind WORKIT_SOAK=1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BrainerVirus
BrainerVirus force-pushed the bugfix/bounded-recovery branch from 1b3b439 to 432470f Compare October 3, 2026 20:02
BrainerVirus and others added 6 commits October 3, 2026 17:30
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d replace

Under multi-process contention Windows briefly refuses to rename over, or
open, a file another process is reading or renaming (EPERM/EACCES/EBUSY).
That surfaced as storage_error (and could surface as recovery_required on
read). Retry for about a second before reporting the error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BrainerVirus
BrainerVirus changed the base branch from bugfix/store-lock-reclaim to main October 3, 2026 21:17
BrainerVirus and others added 8 commits October 3, 2026 18:18
main's tree at cae7b00 is identical to the #150 head (37eb9dc) this
branch already contains, so the resolution keeps this branch's files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 3-process stress test required more than 100 of 120 writes to land;
a windows-latest run on main got 84 ok and 36 busy under the 250 ms
in-process budget and failed. The invariant is zero recovery_required
and only retryable codes (busy, revision_conflict); keep a weak
progress floor of one process's worth (40 ok).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On windows-latest the 3-process contention test still saw storage_error.
Route the remaining write-path file operations through the bounded
Windows retry, and treat lock-path sharing violations as contention:

- lock acquisition retries EPERM/EACCES/EBUSY within its budget, and
  lockFailure maps them to busy instead of storage_error
- lock release retries (a failed unlink left the lock held in-process)
- storage initialization writes .gitignore only when it is missing or
  different instead of rewriting it on every mutation
- snapshot and recovery-copy reads and the recovery mtime refresh retry

The contention test's workers now report the code and detail of any
unexpected result. The OpenCode SDK pin doctor test runs hermetically
(node and bun only on PATH) so it no longer reaches npm or gh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Take main's index.tsx; `workit gc` moves into the router as
verbs/gc.ts with the shared envelope, registered next to doctor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With a read-only or ACL-denied .workit, Windows EPERM/EACCES on the lock
create looked like contention: the acquire retried the whole budget and
returned busy "held by another Workit call" with a --fix-lock hint while
no lock file existed. A sharing-class denial now counts as contention
only when a lock file is present; otherwise, after two quick re-checks
for a holder that was mid-delete, it fails fast as storage_error with a
permissions/read-only-attribute hint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…bing

On windows-latest the contention test showed the previous heuristic was
wrong: a create denied while the released lock is still pending delete
is already invisible to lstat, so "no visible holder" also covers real
contention. When a sharing-class denial finds no holder, probe whether
the lock directory accepts a new file; only an unwritable directory is
reported as storage_error with the permissions hint. gc also sweeps
stale `.probe` leftovers, and the 20-write bounded-copies test gets an
explicit timeout for slow Windows fsync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BrainerVirus
BrainerVirus merged commit a8334f5 into main Oct 3, 2026
5 checks passed
@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 2.2.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant