feat(doctor): report and reclaim abandoned response-state temps

devlog/_plan/260819_response_state_temp_reclaim/021_audit_round3.md

@@ -0,0 +1,60 @@
+# Audit round 3 — phase 2 implementation
+
+Reviewer: independent `explorer`, read-only, against `24a901d5c`. Verdict:
+**GO-WITH-FIXES (blockers=5)**. Main-agent judgment: **near-pass** — all five folded,
+none rebutted.
+
+## Confirmed
+
+- **The dry run shares one predicate.** The `dryRun` branch sits AFTER every gate
+  (basename, pid/seq sanity, inspect failure, isFile + grace, boot floor, self-pid,
+  liveness), so `eligible` is by construction the exact set that would reach `unlink`.
+  The drift risk the plan named is closed.
+- **The default path deletes nothing** and needs no running server: the only syscalls are
+  `readdir`/`lstat`/`realpath`, and `getConfigDir()` is pure string resolution.
+- **The layer stands alone** at its own tip.
+
+## Blocker 1 (accepted) — report and reclaim disagreed in MAGNITUDE
+
+The predicate agreed; the budget did not. The report was bounded by `maxEntries` (4096)
+while the reclaim used the default `maxCleanups` (512). On the reported ~816-file backlog
+doctor would say "816 reclaimable", then free 512 and print that, leaving 304 with no hint
+that another run was needed.
+
+Fixed twice over: the doctor reclaim now passes a matching budget, AND a partial pass
+prints how many remain with an instruction to run again. The second half matters because
+any budget can still be exceeded.
+
+## Blocker 4 (accepted, the most serious) — the safety property had no test
+
+`formatResponseTempLines` tests feed literal objects to a pure formatter, so none of them
+can observe deletion. Nothing covered the call site: **inverting the report/reclaim
+ternary would have left the whole suite green.** The flagship property — "doctor does not
+delete by default" — was claimed by three accept criteria and demonstrated by none.
+
+Fixed with an end-to-end `describe` that seeds a stale temp in an isolated
+`OPENCODEX_HOME`, runs `runDoctor([])`, asserts the file SURVIVES, then runs the flag and
+asserts it is gone. That test fails if the default is ever inverted.
+
+## Blocker 5 (accepted) — the CLI told a lie to its own target reader
+
+Both the CLI string and the docs promised locked files "are retried automatically". True
+only while a proxy runs and ticks — but this command exists for the operator whose proxy
+will NOT start. Reworded to "retried on the next reclaim — automatically while the proxy
+runs, otherwise re-run this command", in the CLI and the docs, with a regression test
+asserting the phrase "retried automatically" never appears.
+
+## Blockers 2 and 3 (accepted) — discoverability
+
+The flag had no help text, and a typo (`--reclaim-response-temp`) silently degraded into a
+report, so an operator would read "nothing to reclaim" as an answer to a question they
+never asked. Added to `ocx help`, and any unrecognized `--reclaim*` argument now warns.
+
+## Non-blocking, recorded
+
+- `bytesRemoved` under-counts against `eligibleBytes` when another process wins an ENOENT
+  race. Defensible — we did not free those bytes — and left as-is.
+- The "none abandoned" line now names that it covers response-state temps specifically,
+  since the sibling producers (B9 in `002`) remain unreclaimed by design.
+- i18n: only the English page was added, matching the existing convention for
+  `windows-memory.md`. Locale readers fall back to English; no contradiction is introduced.

devlog/_plan/260819_response_state_temp_reclaim/022_phase2_verification.md

@@ -0,0 +1,36 @@
+# Phase 2 verification
+
+Full suite on `macmini-cf` at `a2cec13db` (worktree `/tmp/ocx-reclaim`).
+
+## Full suite — 13397 pass, 1 fail
+
+`bun run test` → `Ran 13410 tests across 850 files [474.20s]`.
+
+The single failure is `update-npm-cache-preflight > runs the real worker protocol against
+npm's configured cache path`, already proven pre-existing in `012` by running that file at
+the unmodified base `59964ad77` (10 pass / 1 fail, identical). It depends on a working
+`npm config` on the host.
+
+The 7 GUI `react` module-load errors seen in the phase-1 run are absent here — that run
+had an incomplete `gui/node_modules`, confirming they were environmental as recorded.
+
+## Focused — 171 pass, 0 fail
+
+`bun test tests/doctor.test.ts tests/responses-state.test.ts tests/state-store-sweeper.test.ts`
+→ 171 pass, 506 assertions, on both the workstation and `macmini-cf`.
+
+`bun run typecheck` clean; `bun run privacy:scan` passed.
+
+## What the new end-to-end tests actually pin
+
+Audit round 3's sharpest finding was that inverting the report/reclaim ternary in
+`runDoctor` would have left the entire suite green. The added
+`doctor reclaim wiring (end to end)` block seeds a real stale temp in an isolated
+`OPENCODEX_HOME` and asserts:
+
+- `runDoctor([])` leaves the file ON DISK and prints "reclaimable";
+- `runDoctor(["--reclaim-response-temps"])` removes it and prints "Reclaimed 1";
+- `runDoctor(["--reclaim-response-temp"])` (typo) warns and removes nothing.
+
+The first of those fails if the default is ever inverted, which is the property three
+accept criteria claimed and none previously demonstrated.

docs-site/astro.config.mjs

@@ -155,6 +155,7 @@ export default defineConfig({
           collapsed: true,
           items: [
             { label: "Windows Memory Growth", translations: { fr: "Augmentation de la mémoire sous Windows", ko: "Windows 메모리 증가", "zh-CN": "Windows 内存增长", "zh-TW": "Windows 記憶體增長", ru: "Рост памяти в Windows", ja: "Windows メモリ増加", tr: "Windows Bellek Artışı" }, slug: "troubleshooting/windows-memory" },
+            { label: "Disk Usage from Temp Files", translations: { fr: "Espace disque et fichiers temporaires", ko: "임시 파일 디스크 사용량", "zh-CN": "临时文件磁盘占用", "zh-TW": "暫存檔磁碟用量", ru: "Использование диска временными файлами", ja: "一時ファイルのディスク使用量", tr: "Geçici Dosya Disk Kullanımı" }, slug: "troubleshooting/disk-usage-temp-files" },
           ],
         },
         { label: "Contributing", translations: { fr: "Contribuer", ko: "기여하기", "zh-CN": "贡献", "zh-TW": "貢獻", ru: "Как внести вклад", ja: "コントリビュート", tr: "Katkıda Bulunma" }, slug: "contributing" },

docs-site/src/content/docs/troubleshooting/disk-usage-temp-files.md

@@ -0,0 +1,82 @@
+---
+title: Disk Usage from Temp Files
+description: What responses-state.json.ocx.*.tmp files are, why they could accumulate, and how to reclaim them.
+---
+
+Some users found many gigabytes of files named like
+`responses-state.json.ocx.<pid>.<seq>.tmp` in their opencodex home
+(`~/.opencodex` by default), growing after every reboot.
+
+## What these files are
+
+opencodex keeps a continuation cache so `previous_response_id` chains survive a
+proxy restart. It writes that snapshot atomically: content goes to a temp file
+first, then replaces the real file in one step. That is what stops a crash
+mid-write from leaving a half-written snapshot.
+
+The temp is normally removed the instant the swap completes. If the process dies
+between the two steps, the temp survives.
+
+Each file can be up to 24 MB because the snapshot is rewritten whole, not
+appended to. A few hundred abandoned files therefore add up quickly.
+
+**They are cache, not durable state.** Deleting them costs nothing except that
+in-flight conversation chains may re-send context once. No configuration,
+credentials, or history live in these files.
+
+## Why they could accumulate
+
+A cleanup already existed, but it ran at one moment only: when a proxy loaded
+the continuation cache for the first time, which happens *before* that process
+writes anything. Two consequences followed.
+
+A proxy that crashed and restarted swept too early to see the temp its
+predecessor had just left — there is a 15-minute grace period so a file being
+written right now is never touched — and it never looked again for the rest of
+its life. Each restart then added one more file.
+
+Worse, the cleanup skipped any file whose owning process ID was still alive.
+After a reboot the operating system routinely reissues the same process IDs, so
+an old file could be permanently mistaken for one belonging to a running
+process. That is why the growth tracked reboots.
+
+## What opencodex does now
+
+The cleanup repeats on the proxy's normal background timer instead of running
+once at startup, so a running proxy reclaims abandoned files on its own. It also
+ignores the process-ID check for files older than the current boot, since no
+running process can own those.
+
+The safety rules are unchanged: a file younger than 15 minutes is never removed,
+and the proxy never removes a file it is writing itself.
+
+## Reclaiming files that already accumulated
+
+If the proxy runs, this happens automatically within a minute or two.
+
+If the proxy will **not** start — the case where the pile grows fastest — check
+and reclaim from the command line:
+
+```bash
+ocx doctor
+```
+
+The "Response-state temp files" section reports how many files are reclaimable
+and how much space they hold. It only reports; it changes nothing.
+
+To actually remove them:
+
+```bash
+ocx doctor --reclaim-response-temps
+```
+
+Both commands work without a running proxy. Files currently locked by another
+process are reported rather than forced. They are retried on the next reclaim —
+automatically while the proxy is running, otherwise the next time you run this
+command.
+
+If a very large backlog exceeds one pass, the command says how many files remain
+so you can run it again.
+
+This covers response-state snapshot temps specifically. Other components write
+their own temp files with a similar name, and those are not touched here.

src/cli/doctor.ts

@@ -25,6 +25,11 @@ import { collectOrcaCodexHomeDiagnostic, resolveCodexHomeDir as resolveCodexHome
 import { scanCodexAgentRolesWithTomlModelFallback } from "../codex/subagent-model-fallback";
 import { findCodexOnPath, isWindowsInteropDir } from "../codex/shim";
 import { countPendingOpencodexHistory } from "../codex/history-provider";
+import {
+  inspectAbandonedResponseStateTemps,
+  reclaimAbandonedResponseStateTemps,
+  type ResponseStateTempRecoveryResult,
+} from "../responses/state";
 import {
   CodexUserIdentityRefusal,
   probeCodexCoordinatorNamespace,
@@ -678,6 +683,57 @@ export async function fetchServiceMemory(
 
 const mb = (bytes: number): string => `${Math.round(bytes / (1024 * 1024))}MB`;
 
+export const RECLAIM_RESPONSE_TEMPS_FLAG = "--reclaim-response-temps";
+/** Matches the dry run's entry bound so report and reclaim agree on a large backlog. */
+const RESPONSE_TEMP_RECLAIM_MAX_CLEANUPS = 4_096;
+/** Names the subsystem: other components mint temps with the same shape and are not covered. */
+const CLEAN_RESPONSE_TEMP_LINE = "  ok  No abandoned response-state temp files.";
+
+/**
+ * Render the abandoned-temp section (testable without console capture).
+ *
+ * Report is the DEFAULT and reclaim is opt-in: `doctor` is a diagnostic an operator runs
+ * to understand a machine, so deleting files as a side effect of asking a question is the
+ * wrong default even for cache files.
+ *
+ * Counts come from `eligible`/`eligibleBytes`, never `matched`: `matched` is incremented
+ * before the file-type, age, boot-floor, and liveness gates, so reporting it would tell an
+ * operator that live-pid temps and young temps are "abandoned".
+ */
+export function formatResponseTempLines(
+  result: ResponseStateTempRecoveryResult,
+  reclaimed: boolean,
+): string[] {
+  if (reclaimed) {
+    if (result.removed === 0 && result.failed === 0) return [CLEAN_RESPONSE_TEMP_LINE];
+    const lines = [`  ok  Reclaimed ${result.removed} abandoned response-state temp file(s), ${mb(result.bytesRemoved)} freed.`];
+    if (result.failed > 0) {
+      // Never "retried automatically": this command exists for the operator whose proxy will
+      // NOT start, and in that state nothing retries anything.
+      lines.push(`  !!  ${result.failed} file(s) could not be removed (in use or locked). Retried on the next reclaim — automatically while the proxy runs, otherwise re-run this command.`);
+    }
+    // `truncated`, not `eligible > removed + failed`: outside a dry run every eligible entry
+    // is unlinked or failed on the same iteration it is counted, so those two are always
+    // equal and the comparison never fired. An operator with a backlog past the budget was
+    // told the reclaim had finished.
+    if (result.truncated) {
+      lines.push("  !!  Cleanup budget reached; files remain. Run the command again to continue.");
+    }
+    return lines;
+  }
+  if (result.eligible === 0) return [CLEAN_RESPONSE_TEMP_LINE];
+  const lines = [
+    `  !!  ${result.eligible} abandoned response-state temp file(s), ${mb(result.eligibleBytes)} reclaimable.`,
+    "      These are interrupted snapshot writes (continuation cache only) and are safe to remove.",
+    "      Reclaim them with: ocx doctor --reclaim-response-temps",
+  ];
+  // The dry run skips the cleanup budget but is still bounded by the entry cap, so a large
+  // enough backlog makes this a floor rather than a total. Say so instead of letting an
+  // operator size the problem from a truncated count.
+  if (result.truncated) lines.push("      Scan stopped at its entry budget; the real total is higher.");
+  return lines;
+}
+
 /** Render the doctor "Memory / runtime" section lines (testable without console capture). */
 export function formatServiceMemoryLines(report: ServiceMemoryReport): string[] {
   const lines: string[] = [];
@@ -805,6 +861,26 @@ export async function runDoctor(args: string[] = []): Promise<void> {
     console.log(`  ${row.exists ? "ok " : "-- "} ${row.label}: ${row.path}${flags ? `  (${flags})` : ""}`);
   }
 
+  // Runs without the proxy on purpose: the worst accumulation happens when the proxy will
+  // not start, which is exactly when the in-process periodic reclaim never ticks.
+  const reclaimTemps = args.includes(RECLAIM_RESPONSE_TEMPS_FLAG);
+  console.log("\nResponse-state temp files");
+  // A typo must not silently degrade into "nothing to reclaim" — the operator would read the
+  // report as an answer to a question they never actually asked.
+  for (const arg of args) {
+    if (arg !== RECLAIM_RESPONSE_TEMPS_FLAG && /^--reclaim/.test(arg)) {
+      console.log(`  !!  Unrecognized flag ${arg}; did you mean ${RECLAIM_RESPONSE_TEMPS_FLAG}? Reporting only.`);
+    }
+  }
+  for (const line of formatResponseTempLines(
+    // The reclaim budget matches the report budget: a report bounded by entries and a removal
+    // bounded by a smaller cleanup cap would tell an operator 816 and then silently free 512.
+    reclaimTemps
+      ? reclaimAbandonedResponseStateTemps({ maxCleanups: RESPONSE_TEMP_RECLAIM_MAX_CLEANUPS })
+      : inspectAbandonedResponseStateTemps(),
+    reclaimTemps,
+  )) console.log(line);
+
   const orcaHome = collectOrcaCodexHomeDiagnostic();
   console.log("\nCodex app home targeting");
   console.log(`  ${orcaHome.mismatch ? "!! " : "ok "} Effective Codex home: ${orcaHome.effectiveCodexHome}`);

src/cli/help.ts

@@ -36,6 +36,8 @@ Usage:
                               Refresh Codex's model cache from the active catalog
   ocx status                  Check proxy server status
   ocx doctor                  Diagnose environment/network issues (WSL, proxy, ChatGPT reachability)
+  ocx doctor --reclaim-response-temps
+                              Reclaim abandoned response-state temp files (works without a running proxy)
   ocx debug <scope>           provider/usage/injection/claude on|off|status|reset
   ocx login <provider>        OAuth or API-key provider login
   ocx logout <provider>       Remove a stored OAuth login