fix(core): bound recovery copies and add workit gc - #154
Merged
Merged
Conversation
A .workit/metadata.lock left by a dead process bricked the store: every mutation returned recovery_required and nothing could clear it. Contention between live writers (timeoutMs 0, no retries) was also mapped to recovery_required, which the bootstrap treats as a stop condition. - Reclaim a lock whose owner is provably gone: dead pid, a pid reused by a different process (start time differs), a foreign-host lock past a 10 min TTL, or an unreadable lock past 30 s. Abandoned reclaim guards past 30 s are cleared as well. - Retry a live holder for up to 2 s, then return the new retryable `busy` code. Losing a reclaim race retries instead of failing. - Genuine data-integrity failures (corrupt or foreign records) and the explicit state.recover protocol keep returning recovery_required. - `workit doctor` gains a workspace_lock check; `workit doctor --fix-lock` clears a stale lock and abandoned guard, never a live one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review findings on the lock reclaim change: - Lock identity now includes the Linux pid-namespace inode and boot id, folded into `host` (`host#pidns:bootid`) so older strict schemas still parse it. A container sharing the hostname, another boot, or a legacy lock is treated as foreign (TTL rules only) instead of being checked against this host's process table. - `doctor --fix-lock` clears under the same `.reclaim` guard writers take, skips when a reclaim is in progress, and re-verifies the bytes just before removal, so it can no longer delete a lock a writer just took. - The wait budget is 250 ms by default (in-process hosts) and 2 s in the CLI process; a lost reclaim race retries within the remaining budget. - /proc stat start time is parsed after the last ")"; macOS/FreeBSD use `ps -o lstart=`. `doctor --fix-lock --force` (with --yes or a TTY confirmation) is the explicit escape hatch; `--fix-lock` honours WORKFLOW_WORKSPACE_ROOT like task commands. - Bootstrap: a revision_conflict on a call without expectedRevision is retryable contention. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BrainerVirus
force-pushed
the
bugfix/bounded-recovery
branch
from
October 3, 2026 18:43
737ffdd to
e8f869c
Compare
…iable ones - A lock from the same host and pid namespace but another boot id is stale at once: its owner died with the previous boot. - doctor's workspace_lock warns when an unverifiable lock (other host, pid namespace, or an older Workit's plain-hostname lock) has blocked writes for over 30 s, with the exact `workit doctor --fix-lock --force --yes` command. - Regression test: a plain-hostname lock naming live pid 1 with a mismatched start time (a container's lock) stays busy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BrainerVirus
force-pushed
the
bugfix/bounded-recovery
branch
from
October 3, 2026 19:05
e8f869c to
8be531b
Compare
…ks carry namespace identity On macOS and Windows there is no pid-namespace identity, so a plain-hostname lock is the host's own format and pid checks apply. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BrainerVirus
force-pushed
the
bugfix/bounded-recovery
branch
from
October 3, 2026 19:27
8be531b to
1b3b439
Compare
Resolve task-store.ts: keep main's task index; the metadata lock schema now lives in store-lock.ts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every snapshot replacement copied the previous bytes into .workit/recovery/ with no cap (818 MB / 5,752 files measured in one checkout), and state.recover was advertised everywhere although no shipped host supplies the native recovery authority it requires. - Keep the newest 3 recovery copies per task/workspace record; pruning runs after each copy and never fails the mutation. - `workit gc [--dry-run] [--json]` prunes copies beyond the cap, removes temp files older than 1 h, and collapses duplicate stored candidates (same content-addressed ID, latest position kept) in paused/closed tasks. Live task and workspace records are never deleted. - Stop advertising state.recover in host tool schemas (MCP, Pi, OpenCode) and the CLI; parseOperation and the engine path stay for embedders that supply nativeRecovery. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Candidate dedupe now rewrites paused tasks only; closed tasks are immutable history and are reported as skippedClosed. - `workit gc --dry-run` no longer takes the lock or initializes storage (no mkdir, no .gitignore rewrite). - `workit gc` resolves the workspace root like task commands (WORKFLOW_WORKSPACE_ROOT, then cwd). - The bounded-copies test uses 20 writes; the 1,000-write version is an opt-in soak behind WORKIT_SOAK=1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BrainerVirus
force-pushed
the
bugfix/bounded-recovery
branch
from
October 3, 2026 20:02
1b3b439 to
432470f
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d replace Under multi-process contention Windows briefly refuses to rename over, or open, a file another process is reading or renaming (EPERM/EACCES/EBUSY). That surfaced as storage_error (and could surface as recovery_required on read). Retry for about a second before reporting the error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 3-process stress test required more than 100 of 120 writes to land; a windows-latest run on main got 84 ok and 36 busy under the 250 ms in-process budget and failed. The invariant is zero recovery_required and only retryable codes (busy, revision_conflict); keep a weak progress floor of one process's worth (40 ok). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On windows-latest the 3-process contention test still saw storage_error. Route the remaining write-path file operations through the bounded Windows retry, and treat lock-path sharing violations as contention: - lock acquisition retries EPERM/EACCES/EBUSY within its budget, and lockFailure maps them to busy instead of storage_error - lock release retries (a failed unlink left the lock held in-process) - storage initialization writes .gitignore only when it is missing or different instead of rewriting it on every mutation - snapshot and recovery-copy reads and the recovery mtime refresh retry The contention test's workers now report the code and detail of any unexpected result. The OpenCode SDK pin doctor test runs hermetically (node and bun only on PATH) so it no longer reaches npm or gh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Take main's index.tsx; `workit gc` moves into the router as verbs/gc.ts with the shared envelope, registered next to doctor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With a read-only or ACL-denied .workit, Windows EPERM/EACCES on the lock create looked like contention: the acquire retried the whole budget and returned busy "held by another Workit call" with a --fix-lock hint while no lock file existed. A sharing-class denial now counts as contention only when a lock file is present; otherwise, after two quick re-checks for a holder that was mid-delete, it fails fast as storage_error with a permissions/read-only-attribute hint. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…bing On windows-latest the contention test showed the previous heuristic was wrong: a create denied while the released lock is still pending delete is already invisible to lstat, so "no visible holder" also covers real contention. When a sharing-class denial finds no holder, probe whether the lock directory accepts a new file; only an unwritable directory is reported as storage_error with the permissions hint. gc also sweeps stale `.probe` leftovers, and the 20-write bounded-copies test gets an explicit timeout for slow Windows fsync. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
🎉 This PR is included in version 2.2.0 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #150 (
bugfix/store-lock-reclaim). Review only the top two commits.CI note: this PR targets a non-
mainbranch, so the CI workflows do not run on it. They will run once it is retargeted tomainafter #150 merges. Local verification is below.What
Recovery copies are capped. Each task or workspace record keeps its newest 3 copies (
RECOVERY_COPIES_PER_RECORD) in.workit/recovery/. Pruning runs after each copy, always keeps the copy just written, and is best-effort: a pruning failure never fails the mutation. When an existing copy is saved again with the same bytes, its mtime is refreshed so it counts as newest.New
workit gc [--dry-run] [--json](TaskStore.collectGarbage). It runs in the current directory and:*.tmpfiles older than 1 h that crashed writers left inrecovery/,tasks/and.workit/,candidates.at(-1)are unchanged. This only happens in paused tasks. Active tasks (skippedActive) are left alone so a live session's revision is not bumped under it. Closed tasks (skippedClosed) are never rewritten, because they are immutable history (docs/workit-v1/contracts.md).It never deletes
tasks/*.jsonorworkspace.json.--dry-runis strictly read-only: no lock, no mkdir, no.gitignorerewrite. The root resolves like task commands (WORKFLOW_WORKSPACE_ROOT, then cwd).state.recoveris no longer advertised. I checked the claim in code first. The operation requiresOperationContext.nativeRecovery, and no production host sets it. The only sources areworkit-cli/src/task.tsdeps.nativeRecovery, whichindex.tsxnever passes, and two core tests. The OpenCode, Pi, MCP (Cursor and Codex) and CLI hosts all go through paths without it, so in production the operation can only returnpermission_denied. Changes:operationJsonSchemaandboundedOperationJsonSchemanow use the newadvertisedOperationSchemas, wherestateisexport|import. MCP and Pi read their schemas from these.advertisedOperationSchemas.state recover.parseOperationand the engine path are unchanged, so embedders that do supplynativeRecovery, and the existing continuity tests, keep working.README, AGENTS.md and CHANGELOG are updated.
Why
The audit (§3.4, §6 #3 and #10, §7.2 #2) measured 818 MB and 5,752 files in
recovery/for 74 tasks, growing about quadratically with mutations. It also found thatstate.recoverwas advertised on every host but could not succeed anywhere.Given/When/Then covered (
test/workit-core/recovery-gc.test.ts)WORKIT_SOAK=1; measured 7.9 s and passing).workitlisting and mtimes; no.gitignoreor lock created)tasks/*.jsonandworkspace.json, equallistTasks)workit gc --jsonruns, Then it prunes to the cap and reports the countExisting tests updated, none deleted:
task-commands.test.ts: the CLI action count goes from 24 to 23.mcp/server.test.ts: the advertised-action parity excludesstate.recover.Verification (measured, Node 24.20.0, Bun 1.3.14, in the worktree)
mutateTaskwrites left 1000 recovery files. With this change they leave 3. The time for 1,000 writes is fsync-bound and stayed in the same range across runs (S1: 7.8 s / 22.8 s, S2: 8.9 s / 38.9 s on a noisy disk; no consistent regression). The new test file could not even load against S1 becausecollectGarbageandRECOVERY_COPIES_PER_RECORDdid not exist yet.node packages/workit-cli/dist/index.js): a checkout seeded with 6,000 stale copies (3,000 task + 3,000 workspace) went throughworkit gcin 0.6 s, with 5,994 copies removed (11.8 MB) and 6 kept.task inspectoutput was byte-identical before and after. A latertask progresssucceeded and the directory stayed at 6 files.bun run lint✅,format:check✅,typecheck✅,knip✅,build✅.bun testafter the review fixes: 1598 pass / 1 skip (soak) / 1 fail. The one failure isAR-14, which already fails onorigin/mainbecause itsrepoRootresolves to the parent of the checkout (details in fix(core): reclaim stale metadata locks and report contention as busy #150).Risks
recovery/directory, but onlylstats the copies of that one record. Runningworkit gconce clears old backlogs.expectedRevisionheld for such a task getsrevision_conflict.🤖 Generated with Claude Code