fix(core): reclaim stale metadata locks and report contention as busy

AGENTS.md

@@ -115,7 +115,7 @@ process around it.
   `gh`/`glab` CLI identity, not separate GitHub/GitLab token files.
   Doctor also fails when OpenCode's frozen `@latest` package cache lags the
   published `workit-opencode` (delete the cache dir and restart OpenCode).
-- Task state lives under `.workit/` in the session directory, even when that directory is not a Git repository; never edit it directly. Git/hosting actions accept `cwd` for an action-time target checkout without prior attachment; the coordinator task keeps the writer, while a conflicting writer in the target checkout blocks mutation. A managed action holds the target metadata lock through settlement so another writer cannot acquire during the effect. Use the eight shared operation families (`workit_task`, `workit_policy`, `workit_evidence`, `workit_finding`, `workit_decision`, `workit_worker`, `workit_writer`, `workit_state`) with closed `action` enums. CLI surface is `workit <family> <action>` (hyphenated actions). There are no `workit flow` aliases.
+- Task state lives under `.workit/` in the session directory, even when that directory is not a Git repository; never edit it directly. A stale `metadata.lock` (dead/reused pid in the same host, pid namespace and boot; anything else only past its TTL) is reclaimed by the next write or cleared with `workit doctor --fix-lock` (`--force --yes` for an unverifiable lock); contention with a live holder returns retryable `busy`, and `recovery_required` is reserved for genuine state damage. Git/hosting actions accept `cwd` for an action-time target checkout without prior attachment; the coordinator task keeps the writer, while a conflicting writer in the target checkout blocks mutation. A managed action holds the target metadata lock through settlement so another writer cannot acquire during the effect. Use the eight shared operation families (`workit_task`, `workit_policy`, `workit_evidence`, `workit_finding`, `workit_decision`, `workit_worker`, `workit_writer`, `workit_state`) with closed `action` enums. CLI surface is `workit <family> <action>` (hyphenated actions). There are no `workit flow` aliases.
 - `task.list` defaults to a compact, 20-item active/paused projection; bounded closed/all history is opt-in with `status` and `limit`, and `task.inspect` defaults to `summary`. Closed views evaluate their captured closure candidate and carry no current writer lease.
 - Workit tracking is optional: direct investigation, questions, non-Git work, and routine reversible edits need no task, policy assessment, or writer calls. Start one compact task only when handoff, dependencies, concurrent actors, or meaningful decisions make continuity useful; assess or reassess when the relevant rules/evidence require it. `policy.preview` stays read-only.
 - Routine authorized branch/commit work chooses native host Git/shell tools from the outset when managed coordination or reconciliation is unnecessary. Resolve target conventions; never switch paths after a denial or uncertain managed effect to evade safeguards. A local-commit endpoint creates no PR-readiness or task-closure ceremony. The distributed bootstrap carries this routing guidance.

CHANGELOG.md

@@ -22,6 +22,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- A `.workit/metadata.lock` left by a dead process (or a reused pid, or a
+  foreign-host/namespace lock past its TTL) no longer bricks the store: the next
+  write reclaims it. Locks carry the pid namespace and boot id so a container
+  sharing the hostname is never judged by this host's process table, and a
+  lock from before a reboot is reclaimed at once.
+  Contention with a live writer retries briefly (250 ms in-process, 2 s in the
+  CLI) and returns the retryable `busy` code instead of `recovery_required`.
+  `workit doctor` reports a stale lock (`workspace_lock`); `workit doctor
+  --fix-lock` clears it under the reclaim guard, and `--force --yes` clears an
+  unverifiable lock explicitly.
 - Compact task context retains the newest decisions and surfaces bounded,
   redacted choice summaries instead of selecting an arbitrary UUID-ordered set.
 - Distributed cross-repository guidance binds unfinished work to its checkout,

README.md

@@ -203,6 +203,8 @@ workit upgrade --apply --confirm  # apply a reviewed preview
 workit upgrade --cli --apply --confirm  # also update an existing global CLI
 workit launch pi --auto-upgrade --  # update before starting Pi
 workit doctor            # offline installation health report (--json for machines)
+workit doctor --fix-lock # clear a stale .workit metadata lock (WORKFLOW_WORKSPACE_ROOT or cwd)
+workit doctor --fix-lock --force [--yes]  # clear a lock whose owner cannot be verified
 workit <family> <action> [--payload <json|@file|->] [--task <id>] [--confirm] [--json]
 workit action <operation> --payload <JSON> [--preview] [--confirm] [--json]
 workit handoff --task <id> [--json]
@@ -403,7 +405,19 @@ The canonical target, relevant Git/remote state, and effective `gh`/`glab`
 account are checked again before a remote effect. The coordinator owns the
 Workit writer; an independently held writer in the target checkout remains a
 real conflict, and managed actions hold that checkout's Workit metadata lock
-through effect settlement so a writer cannot acquire mid-action. New branch
+through effect settlement so a writer cannot acquire mid-action. A metadata
+lock whose owner is gone (dead or reused pid) is reclaimed by the next write.
+A lock records its host plus, on Linux, its pid namespace and boot id; a lock
+from another host, container namespace, boot, or an older Workit version cannot
+be checked against this process table and is reclaimed only after a 10-minute
+TTL (a lock from the same host and pid namespace but an earlier boot is
+reclaimed at once). `workit doctor` warns when such an unverifiable lock has
+blocked writes for over 30 s and prints `workit doctor --fix-lock --force --yes`. A write that meets a live holder retries briefly (250 ms inside host
+plugins and the MCP server, 2 s in the CLI) and then returns the retryable
+`busy` code, never `recovery_required`. `workit doctor` warns about a stale
+lock and `workit doctor --fix-lock` clears it under the same reclaim guard
+writers use; `--force` (with `--yes` or an interactive confirmation) is the
+explicit escape hatch for a lock whose owner cannot be verified. New branch
 setup shows both the existing local base SHA and remote base SHA in its
 approval, rechecks them, and creates only from an approved commit. Workit does
 not reject Git-valid branch names or user commit

packages/workit-cli/src/index.tsx

@@ -8,6 +8,12 @@ import { createLogger } from "@brainervirus/workit-core/src/core/logger";
 import { EVENT, errorDetail } from "@brainervirus/workit-core/src/core/boundary";
 import { setDiagnosticLogger } from "@brainervirus/workit-core/src/core/config";
 import { runDoctor } from "@brainervirus/workit-core/src/core/doctor";
+import {
+  clearStaleMetadataLock,
+  inspectMetadataLock,
+  setDefaultLockTimeout,
+} from "@brainervirus/workit-core/src/core/store-lock";
+import { createInterface } from "node:readline/promises";
 import {
   applySetupPreview,
   buildSetupPreview,
@@ -24,7 +30,7 @@ import {
 } from "@brainervirus/workit-core/src/core/uninstall";
 import { applyWizardBranchPolicy } from "./logic";
 import { runCutoverCommand } from "./cutover-cli";
-import { runActionCommand, runTaskCommand, TASK_FAMILIES } from "./task";
+import { runActionCommand, runTaskCommand, TASK_FAMILIES, workspaceRootFor } from "./task";
 import { externalActionHelp } from "@brainervirus/workit-core/src/core";
 import { runLaunchCommand, runUpgradeCommand } from "./upgrade";
 
@@ -58,6 +64,8 @@ Usage:
   workit upgrade   Preview upgrades (--apply --confirm; --hosts=a,b; --cli for the CLI)
   workit launch <host> [--auto-upgrade] [-- args]  Upgrade before host startup
   workit doctor    Verify the offline installation health (add --json for a machine-readable report)
+                   --fix-lock clears a stale .workit metadata lock in the workspace root
+                   --fix-lock --force [--yes] clears it even when its owner cannot be verified
   workit uninstall Remove workit host registrations interactively (~/.config/workit is kept)
   workit cutover   Preview or apply an explicit v1 cutover (apply requires --confirm)
 ${COMMAND_DESCRIPTIONS.map(([cmd, desc]) => `  ${cmd.padEnd(helpColumn)}${desc}`).join("\n")}
@@ -310,11 +318,54 @@ export async function runUninstall() {
 
 // `workit doctor` (DG-07): offline engine, human or --json report, exit code
 // reflects the health. Never writes the report to stderr (the logger owns that).
-function runDoctorCommand(args: string[]) {
-  const report = runDoctor({ host: "cli", cwd: process.cwd() });
+// Explicit escape hatch for a lock whose owner cannot be verified (no process
+// start time, a foreign pid namespace): show the holder, then require --yes or
+// an interactive confirmation.
+async function confirmForcedLockClear(root: string, args: string[]): Promise<boolean> {
+  const lock = inspectMetadataLock(root);
+  if (!lock.present) return true;
+  const owner = lock.owner
+    ? `pid ${lock.owner.pid} on ${lock.owner.host} (start ${lock.owner.processStart ?? "unknown"})`
+    : "unreadable lock";
+  console.log(`fix-lock --force: ${lock.path} is held by ${owner}: ${lock.reason}`);
+  if (args.includes("--yes")) return true;
+  if (process.stdin.isTTY !== true) {
+    console.log("fix-lock --force: refusing without --yes outside an interactive terminal");
+    return false;
+  }
+  const rl = createInterface({ input: process.stdin, output: process.stdout });
+  try {
+    const answer = await rl.question("Remove this lock even if its holder may be alive? [y/N] ");
+    return /^y(es)?$/i.test(answer.trim());
+  } finally {
+    rl.close();
+  }
+}
+
+async function runDoctorCommand(args: string[]) {
+  // --fix-lock runs first so the report reflects the cleaned state. Without
+  // --force it removes only a lock whose owner is provably gone.
+  const root = workspaceRootFor();
+  const force = args.includes("--force");
+  let fixLock: ReturnType<typeof clearStaleMetadataLock> | null = null;
+  if (args.includes("--fix-lock")) {
+    if (force && !(await confirmForcedLockClear(root, args))) process.exit(1);
+    fixLock = clearStaleMetadataLock(root, { force });
+  }
+  const report = runDoctor({ host: "cli", cwd: process.cwd(), workspaceRoot: root });
   if (args.includes("--json")) {
-    console.log(JSON.stringify(report, null, 2));
+    console.log(JSON.stringify(fixLock ? { ...report, fixLock } : report, null, 2));
   } else {
+    if (fixLock) {
+      const what = fixLock.cleared
+        ? `cleared ${force ? "" : "stale "}lock ${fixLock.path} (${fixLock.reason})`
+        : fixLock.state === "absent"
+          ? "no metadata lock to clear"
+          : `kept lock ${fixLock.path}: ${fixLock.skipped ?? fixLock.reason}`;
+      console.log(
+        `fix-lock: ${what}${fixLock.guardCleared ? "; removed abandoned reclaim guard" : ""}`,
+      );
+    }
     console.log(
       `workit doctor — ${report.ok ? "healthy" : "problems found"} (${report.offline ? "offline" : "online"})`,
     );
@@ -334,6 +385,9 @@ if (import.meta.main) {
   const args = process.argv.slice(2);
   const [subcommand] = args;
   setDiagnosticLogger(logger);
+  // The CLI owns its process, so a contended write may wait longer than an
+  // in-process host could afford before reporting busy.
+  setDefaultLockTimeout(2_000);
   logger.info(EVENT.initialization, { host: "cli", command: subcommand });
   // The CLI owns its process: uncaught failures are logged and surfaced with a
   // nonzero exit instead of a silent crash (DG-04).
@@ -351,7 +405,7 @@ if (import.meta.main) {
   } else if (subcommand === "init") {
     await runInit();
   } else if (subcommand === "doctor") {
-    runDoctorCommand(args);
+    await runDoctorCommand(args);
   } else if ((TASK_FAMILIES as readonly string[]).includes(subcommand)) {
     process.exit(await runTaskCommand(args));
   } else if (subcommand === "action") {

packages/workit-cli/src/task.ts

@@ -46,6 +46,10 @@ import type { Provenance } from "@brainervirus/workit-core/src/core/task-contrac
 import { canonicalJson, type Result } from "@brainervirus/workit-core/src/core/task-contract";
 
 export const TASK_FAMILIES = OPERATION_FAMILIES;
+
+/** The Workit store root for a CLI command: explicit root, WORKFLOW_WORKSPACE_ROOT, then cwd. */
+export const workspaceRootFor = (deps: { root?: string; cwd?: string } = {}): string =>
+  deps.root ?? process.env.WORKFLOW_WORKSPACE_ROOT ?? deps.cwd ?? process.cwd();
 export const TASK_ACTIONS = {
   task: ["start", "list", "inspect", "revise", "progress", "pause", "resume", "close"],
   policy: ["assess", "preview", "explain"],
@@ -390,7 +394,7 @@ export async function runTaskCommand(argv: string[], deps: TaskCliDeps = {}): Pr
       );
     return parsed.usage ? 2 : 1;
   }
-  const root = deps.root ?? process.env.WORKFLOW_WORKSPACE_ROOT ?? deps.cwd ?? process.cwd();
+  const root = workspaceRootFor(deps);
   let observedConfirmation = parsed.parsed.observedConfirmation;
   if (needsConsent(parsed.parsed) && !parsed.parsed.confirmed) {
     const consent = await observeConsent(deps);
@@ -650,7 +654,7 @@ export async function runActionCommand(argv: string[], deps: TaskCliDeps = {}):
     else printHuman(parsed, deps);
     return 2;
   }
-  const root = deps.root ?? process.env.WORKFLOW_WORKSPACE_ROOT ?? deps.cwd ?? process.cwd();
+  const root = workspaceRootFor(deps);
   let resolved = resolveExternalActionRequest(root, parsed.data);
   if (!resolved.ok) {
     if (json) jsonResult(outOf(deps), resolved);

packages/workit-core/src/core/doctor.ts

@@ -19,6 +19,7 @@ import {
 import os from "node:os";
 import path from "node:path";
 import { SUPPORT_MATRIX } from "./support-matrix";
+import { inspectMetadataLock } from "./store-lock";
 import { bundleHashOfFile, isEphemeralCachePath } from "./runtime-identity";
 import { EVENT } from "./boundary";
 import { getDiagnosticLogger, isConfigObject } from "./config";
@@ -61,6 +62,7 @@ export type DoctorCheckId =
   | "duplicate_registration"
   | "malformed_config"
   | "workspace_mismatch"
+  | "workspace_lock"
   | "credential_metadata"
   | "github_identity"
   | "gitlab_identity"
@@ -110,6 +112,8 @@ export type DoctorOptions = {
   /** Checkout containing packages/ (monorepo or share clone). */
   dev?: string;
   cwd?: string;
+  /** Workit store root for the lock check (default: WORKFLOW_WORKSPACE_ROOT, then cwd). */
+  workspaceRoot?: string;
   opencodeConfig?: string;
   /** OpenCode npm `@latest` package cache root (test seam). */
   opencodePackageCacheDir?: string;
@@ -129,6 +133,7 @@ type Resolved = {
   configDir: string;
   stateDir: string;
   cwd: string;
+  workspaceRoot: string;
   dev: string | null;
   opencodeConfig: string;
   opencodePackageCacheDir: string;
@@ -179,6 +184,7 @@ const resolve = (options: DoctorOptions): Resolved => {
     configDir,
     stateDir,
     cwd,
+    workspaceRoot: options.workspaceRoot ?? env.WORKFLOW_WORKSPACE_ROOT ?? cwd,
     dev,
     opencodeConfig:
       options.opencodeConfig ?? path.join(home, ".config", "opencode", "opencode.json"),
@@ -1693,6 +1699,44 @@ const checkManagedContentConflict = (res: Resolved): DoctorCheck => {
   };
 };
 
+// The checkout's `.workit/metadata.lock`. Writes reclaim a stale lock by
+// themselves, so a stale lock is a warning with an explicit cleanup command.
+const BLOCKING_LOCK_WARN_MS = 30_000;
+const checkWorkspaceLock = (res: Resolved): DoctorCheck => {
+  const lock = inspectMetadataLock(res.workspaceRoot);
+  const fix = "workit doctor --fix-lock";
+  if (lock.guard === "abandoned")
+    return {
+      id: "workspace_lock",
+      status: "warn",
+      detail: `abandoned lock reclaim guard at ${lock.path}.reclaim`,
+      fix,
+    };
+  if (lock.state === "absent")
+    return { id: "workspace_lock", status: "pass", detail: "no metadata lock held" };
+  if (lock.state === "stale")
+    return {
+      id: "workspace_lock",
+      status: "warn",
+      detail: `stale metadata lock at ${lock.path}: ${lock.reason}`,
+      fix,
+    };
+  // An unverifiable owner (other host, pid namespace, or an older Workit's
+  // lock) that has blocked writes this long needs an explicit decision.
+  if (lock.state === "unknown" && (lock.ageMs ?? 0) > BLOCKING_LOCK_WARN_MS)
+    return {
+      id: "workspace_lock",
+      status: "warn",
+      detail: `metadata lock at ${lock.path} has blocked writes for ${Math.round((lock.ageMs ?? 0) / 1000)}s and its owner cannot be verified: ${lock.reason}`,
+      fix: "workit doctor --fix-lock --force --yes",
+    };
+  return {
+    id: "workspace_lock",
+    status: "pass",
+    detail: `metadata lock ${lock.reason} (writes retry, then report busy)`,
+  };
+};
+
 const RUN_CHECKS: Array<(res: Resolved) => DoctorCheck> = [
   checkRuntime,
   checkVersions,
@@ -1705,6 +1749,7 @@ const RUN_CHECKS: Array<(res: Resolved) => DoctorCheck> = [
   checkDuplicateRegistration,
   checkMalformedConfig,
   checkWorkspaceMismatch,
+  checkWorkspaceLock,
   checkCredentialMetadata,
   checkGithubIdentity,
   checkGitLabIdentity,

packages/workit-core/src/core/methods.ts

@@ -110,6 +110,11 @@ Start a record once for an explicit tracked objective; assess or reassess only
 when policy selection or changed evidence/constraints requires it. Omitted
 expectedRevision and expectedWorkspaceRevision use current values; explicit
 values are still concurrency-checked, so never copy revisions between calls.
+A busy result means another live Workit call holds the checkout lock: retry the
+same call; it is not a recovery condition. A lock left by a dead process is
+reclaimed on the next write, and \`workit doctor --fix-lock\` clears it on demand.
+A revision_conflict on a call that omitted expectedRevision is contention too:
+re-read the record and retry the call.
 A solo edit does not need writer acquisition; use it when concurrent checkout
 writers need coordination. Record only observed facts and checks. Evidence can
 become stale when its bound candidate changes; reconcile findings against the