Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
93351c9
docs(plans): C6 small unbounded growth, items verified on main (#1278)
Juliusolsson05 Sep 27, 2026
dfa0c73
fix(storage): empty proxy parent dirs are removed (rmdir, not rm), an…
Juliusolsson05 Sep 27, 2026
2f7d40c
fix(paste-debug): cap open journal writers at 64 like dictation's (#1…
Juliusolsson05 Sep 27, 2026
d70e072
fix(window): bound the closed-window tombstones at 256 (#1278)
Juliusolsson05 Sep 27, 2026
d5101fa
fix(conversations): sweep Codex heads of gone rollouts; LRU-bound the…
Juliusolsson05 Sep 27, 2026
83c3a4e
fix(storage): an unreadable legacy ledger protects every legacy bundl…
Juliusolsson05 Sep 27, 2026
00a5ba9
fix(storage): one malformed legacy ledger row no longer stops debug p…
Juliusolsson05 Sep 27, 2026
6150238
docs(plans): row 13 moved into #1417; overlap with #1411 narrowed to …
Juliusolsson05 Sep 27, 2026
380434a
fix(storage): trust a ledger parse only if the file is unchanged acro…
Juliusolsson05 Sep 27, 2026
44d557a
fix(paste-debug): flush joins an in-flight append; a re-created write…
Juliusolsson05 Sep 27, 2026
3635167
fix(window): bound tombstones by age (10 min), not count, so a mass c…
Juliusolsson05 Sep 27, 2026
5734da8
fix(conversations): sweep Claude summaries per listed directory; LRU-…
Juliusolsson05 Sep 27, 2026
6209149
docs(plans): #1417 round-1 review disposition
Juliusolsson05 Sep 27, 2026
c79cc0c
fix(storage): an absent legacy ledger is unknown and protects every l…
Juliusolsson05 Sep 27, 2026
349e72b
fix(conversations): sweep summaries of Claude project directories tha…
Juliusolsson05 Sep 27, 2026
67cdaba
fix(window): age tombstones on a monotonic clock (#1417 review round 2)
Juliusolsson05 Sep 27, 2026
d003f0e
fix(paste-debug): a failed append keeps its batch (bounded), never re…
Juliusolsson05 Sep 27, 2026
02c7c91
docs(plans): #1417 round-2 disposition
Juliusolsson05 Sep 27, 2026
0de1778
Merge remote-tracking branch 'origin/main' into fix/c6-small-unbounded
Juliusolsson05 Sep 27, 2026
26ee43a
test(storage): a ledger replacement only the inode distinguishes is r…
Juliusolsson05 Sep 27, 2026
00a6ccb
test(paste-debug): a failed batch is retried ahead of lines appended …
Juliusolsson05 Sep 27, 2026
1c0702f
Merge remote-tracking branch 'origin/main' into fix/c6-small-unbounded
Juliusolsson05 Sep 27, 2026
8b3928d
docs(debug-retention): the second-stat comment describes the round-2 …
Juliusolsson05 Sep 27, 2026
d374537
Merge origin/main (batch Q) into fix/c6-small-unbounded
Juliusolsson05 Sep 27, 2026
6cd622f
Merge origin/main (batch R) into fix/c6-small-unbounded
Juliusolsson05 Sep 27, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions docs/plans/2026-09-27-c6-small-unbounded.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# C6 small unbounded growth (#1278)

Source: the Stage 3 C6 hunt (`temp/quality-loop/hunt-c6.md`, rows 7–14 and part of 5). Each item was verified on origin/main `5e22c7b0` by a read-only Explore pass, then re-read by hand at the fix site. Every fix has a test that fails without it (fail-first, or the fix removed as a mutation).

**Owner rule (2026-09-27): "do not delete stuff often."** Fixes bound *memory* and repair a cleanup that was already intended. None of them adds a new deletion policy for user-visible data.

| # | Item | Verdict on main | Change | Test |
|---|---|---|---|---|
| 1 | `pasteDebugJournal.ts` `journals` Map | Real. A paste id is a fresh UUID and `dispose()` has no caller, so there was one writer per paste forever. | Cap at 64 with oldest-first eviction, and make `flushAll` await evicted flushes. The same bound and shape as `dictationJournal` (#1276). | New `pasteDebugJournal.test.ts`. Mutant: dropping the evicted-flush drain goes red. |
| 2 | Empty proxy parent dirs | **A bug, not a missing feature.** `removeEmptyParents` called `rm(dir, { recursive: false })`, which always throws EISDIR (confirmed on Node 24.14.1), so no parent was ever removed. The author's machine has 2,978 empty dirs, walked on every prune. | `rmdir`, which removes only an empty dir and does it atomically: a concurrent new run makes it fail with ENOTEMPTY. `root + sep` keeps sibling roots out of scope. | Real-filesystem test. Red on main; the old `rm` as a mutant goes red. |
| 3a | Legacy `saved-debug-bundles.jsonl` re-parsed on every prune | Real (18.4 MB, every 5 min). Nothing appends to it any more. | Cache the parsed set by file identity: inode + ctime + mtime + size, so a rename-replace or a same-size edit with the mtime set back still re-parses. **Steering q109:** a failed read is `'unknown'`, never cached, and protects every legacy bundle for that prune; only ENOENT means "no ledger". This also closes main's one-prune window, where a transient read failure classified hand-saved bundles as deletable. | Parse-count test; a real temp ledger that is unreadable and then readable; a failure with unchanged identity that must not be cached; a same-size edit with the mtime set back. All three q109 tests are red at `d5101fa5`. Mutants red: caching `'unknown'`, an mtime+size key, `'unknown'` → legacy. |
| 3a′ | #1251 row 13 (moved here from #1411): one malformed legacy-ledger row stopped all pruning | Real: a JSON-valid non-entry (`null`, a non-string `bundlePath`) threw, which rejected `collectArtifacts`. | Rows are shape-checked (`parseManualLegacyBundlePaths`); a non-string reason counts as manual (kept). It ships here because row 13 and steering q109 change the same loader. | A parser test plus a test through the real loader and cache (the #1411 review's surviving 'loader returns empty' mutant is now red). |
| 3b | `autosaved-debug-bundles.jsonl` append-only | **Mostly moot** (corrected by review of #1417, c): normal autosaves return before `appendDebugBundleSaved` (`debugBundle.ts:226-232`), and this machine has no autosave ledger at all. | **Residual.** Nothing to trim in practice, and trimming a ledger would be a deletion policy. | none |
| 4 | WorkflowBridge `runsBySession` / `latestLifecycleByRunId` | Grows with the durable workflow-mcp store (`start()` reloads every stored run). | **Residual:** bounded by the store once #1275's retention (90 days, resumable never deleted) lands. Pruning the maps alone bounds nothing. | none |
| 5 | `sessionManager` `lastActivityAt` | Intentional; WHY at `sessionManager.ts:1095-1099` (telemetry asks about exited panes). One number per session id. | None. | none |
| 6 | Conversation caches | Grow with the transcript store plus deleted transcripts, not over time. | **Codex `heads`:** entries for files the (scope-independent) walk no longer finds are swept; this is exact. **Search `promptCache`:** LRU 1024, the same bound as the prompt folder's, via a new shared `LruMap` that now also backs `promptFolder.ts` (its private LRU functions are gone). **Claude `summaries`, codex `rolloutPaths`:** residual; a scoped discovery cannot tell a deleted transcript from an unwalked one, and a count cap below the store size would re-read the store on every full discovery. | Codex heads sweep test (mutant red); `lruMap.test.ts`; `promptFolder` suite 13/13 unchanged. |
| 7 | `cli-update-logs/` | Real but tiny. This machine has 13 files and 52 KB since 2026-09-03. `cliUpdateOrchestrator.ts:57-61` keeps them on purpose ("the diagnostic value of a two-year-old failed-update log is nonzero"). | **None.** Adding age deletion here would contradict both the WHY and the owner's rule. | none |
| 8 | `windowRegistry` `retiredWebContentsIds` | Real. The comment's "cleared with the registry" only ever happened in the test reset. | Cap at 256 by insertion order. Only a save dequeued moments after `closed` needs a tombstone, and webContents ids are never reused. | Red on main (id 1 still resolved after 300 closes). |
| 9 | `worktree-activity-index.json` rewritten whole | Derived cache, rebuilt from current candidates, so it grows with the corpus and not over time. | **Out of scope:** #767 item 3 (save on an all-cache-hit refresh) is the tracked remainder. | none |

## Overlap with #1411 (manager q109)

Row 13 of #1251 moved here, so `debugRetention.ts` and its test now change only in this PR. #1411 still shares `codex.ts` (different hunks: #1411 guards `fromHead`, this PR sweeps `heads` in `walkRollouts`) and `codex.system.test.ts`. Both insert a test before the same anchor, so the second PR to merge resolves a trivial conflict there.

## Review round 1 (a, b, c codex at `6150238a`): all FIX-BEFORE-MERGE

| Finding | Verdict | Change |
|---|---|---|
| **a, major:** the ledger renamed away between the identity `stat` and the read. The loader's ENOENT "no ledger" was cached under the identity of the file that WAS there, so a manual bundle became deletable (this machine's ledger holds 14 manual saves). | valid | A second `stat` after the load. The parse is trusted only if the identity is unchanged; otherwise `'unknown'` (protective, not cached). Fail-first for "still away" and "renamed back". |
| **a survivor:** `startsWith(root)` without `sep` | valid | A sibling `proxy-old/` fixture; the mutant is red. |
| **a survivor:** `ino` dropped from the identity | declined | A portable test cannot change only the inode: a rename-replace also moves ctime. `ino` stays as defence in depth. |
| **b + c, major:** an evicted paste writer with an append IN FLIGHT escaped the shutdown drain (`draining` made the second drain return at once), and a re-created writer could append first. | valid | `flush()` joins the in-flight append, then writes the rest. A re-created writer's first append waits for the evicted writer's flush (the reader takes a session's start from the first line). A gated-append test; both mutants (no join, no ordering) red. |
| **b, major:** the 256 count cap on tombstones could drop a queued final save during a mass close | valid | The bound is now age: a tombstone older than 10 minutes is swept when another window closes. A mass close keeps them all. Red under the count cap. |
| **c, major:** scoped Claude discovery kept summaries of files gone from directories it had just listed | valid | Each discovery sweeps summaries under the directories it enumerated that are not in the listing; unwalked directories are untouched. Own-corpus test. |
| **c, minor:** Codex `rolloutPaths` unbounded | valid | `LruMap(4096)`, twice the recorded 2,023 indexed threads. A miss only costs the existing SQLite lookup. |
| **c:** plan 3b stale | valid | Corrected above. |
| **b survivors:** promptFolder's test hook using `get` instead of `peek`; a search-cache cap of 1 | declined | Neither is a behaviour of the app. The hook is test-only inspection, and the cap is a performance contract no test times; both are shared `LruMap` semantics, pinned in `lruMap.test.ts`. |

## Review round 2 (a, b, c codex at `62091493`): all FIX-BEFORE-MERGE (final round)

| Finding | Verdict | Change |
|---|---|---|
| **a, major:** a ledger absent for the whole prune (moved aside before it) read as "no ledger", so manual bundles became deletable | valid | **An absent ledger is `'unknown'` too:** it protects every legacy bundle and is never cached. Stated cost: with no ledger at all, pre-split legacy bundles are never aged out. Fail-first. |
| **a, major:** a whole Claude project directory removed kept its summaries forever | valid | The sweep also drops summaries whose project directory is missing from the scope-independent root listing, or vanished (ENOENT) while being listed. A failed root listing, or EACCES on a directory, stays unknown and keeps them. Fail-first. |
| **b, major; a and c, minor:** tombstone age used the wall clock, so a backward step plus correction dropped a fresh tombstone, and a future-dated one blocked the sweep | valid | `performance.now()` (monotonic). Tests fake only `performance`, plus a wall-clock step test; the `Date.now` mutant is red. |
| **c, major; a and b, minor:** a failed paste append dropped its batch, the timer's failure was an unhandled rejection, and a joining flush resolved as success | valid | A failed batch goes back in front of the queue (order holds), and the timer path catches and logs. A flush retries after the write it joined, so both callers reject on a dead disk. The retry queue is bounded at 1000 lines, and dropped lines are reported as one `ERROR journal:dropped-lines` line once writes work. The failed-batch and false-success tests are red at the previous head; a new test pins the bound. |
| **a / b / c survivors:** the `rolloutPaths` bound (Map, or the cap size) | declined | Pinning it needs over 4096 indexed rows or reaching into the private map; the mechanism is the shared `LruMap`, pinned in `lruMap.test.ts`. A miss only costs the existing SQLite lookup. |
| **b survivors (from round 1):** the promptFolder hook, a search-cache cap of 1 | declined | Unchanged reasons: a test-only hook and an untimed performance contract. |
28 changes: 5 additions & 23 deletions src/main/conversations/prompts/promptFolder.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import { open, stat } from 'fs/promises'

import { performanceService } from '@main/performance/PerformanceService.js'
import { asRecord, parseJsonRecord } from '@shared/lib/asRecord.js'
import { LruMap } from '@shared/lib/lruMap.js'

// Incremental user-prompt reader over an append-only provider transcript.
//
Expand Down Expand Up @@ -106,26 +107,7 @@ const HEAD_CWD_WINDOW_BYTES = 64 * 1024
* across providers in practice, but we prefix to be safe. Bounded:
* every transcript ever listed or searched used to stay here forever
* with its full prompt list. */
const promptCache = new Map<string, CacheEntry>()

function cacheGet(key: string): CacheEntry | undefined {
const entry = promptCache.get(key)
if (entry === undefined) return undefined
// Re-insert so Map iteration order doubles as LRU order.
promptCache.delete(key)
promptCache.set(key, entry)
return entry
}

function cacheSet(key: string, entry: CacheEntry): void {
if (promptCache.has(key)) promptCache.delete(key)
promptCache.set(key, entry)
while (promptCache.size > PROMPT_CACHE_MAX_ENTRIES) {
const oldest = promptCache.keys().next().value as string | undefined
if (oldest === undefined) break
promptCache.delete(oldest)
}
}
const promptCache = new LruMap<string, CacheEntry>(PROMPT_CACHE_MAX_ENTRIES)

/** Test-only hooks: the cache is module state by design (one per process). */
export function __resetPromptFolderCacheForTests(): void {
Expand All @@ -135,7 +117,7 @@ export function __promptFolderCacheEntryForTests(
kind: AgentProviderKind,
sessionId: string,
): { parsedFrom: number; parsedTo: number; prompts: number; cwd: string; lastBytesRead: number } | null {
const entry = promptCache.get(cacheKey(kind, sessionId))
const entry = promptCache.peek(cacheKey(kind, sessionId))
if (!entry) return null
return {
parsedFrom: entry.parsedFrom,
Expand Down Expand Up @@ -219,7 +201,7 @@ async function extractPromptsUnlocked(
need: need === 'all' ? -1 : need,
})
const key = cacheKey(kind, sessionId)
let entry = cacheGet(key)
let entry = promptCache.get(key)
let size: number
let mtime: number
try {
Expand Down Expand Up @@ -300,7 +282,7 @@ async function extractPromptsUnlocked(
entry.mtime = mtime
entry.size = size
entry.lastBytesRead = bytesRead
cacheSet(key, entry)
promptCache.set(key, entry)
span.end({
result: 'parsed',
bytes: bytesRead,
Expand Down
9 changes: 8 additions & 1 deletion src/main/conversations/service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ import { defaultOpencodeDataDir, OpencodeConversationSource } from './sources/op
import { GrokConversationSource } from './sources/grok.js'
import { PiConversationSource } from './sources/pi.js'
import type { ConversationSource, SourceConversation } from './sources/types.js'
import { LruMap } from '@shared/lib/lruMap.js'

// The single consumer of the catalog and the single owner of caches.
//
Expand Down Expand Up @@ -51,6 +52,7 @@ const DISCOVERY_FRESH_MS = 3_000
// half the bytes keeps the first search near the budget, and everything the
// tail window misses is still searchable by label, first prompt and title.
const SEARCH_PROMPT_ROWS = 150
const SEARCH_PROMPT_CACHE_MAX_ENTRIES = 1024
const SEARCH_PROMPTS_PER_ROW = 40
const SEARCH_BYTES_PER_ROW = 128 * 1024

Expand All @@ -62,7 +64,12 @@ export class ConversationService {
private discovery: Discovery | null = null
private inflight: { key: string; promise: Promise<Discovery> } | null = null
private discoveries = 0
private readonly promptCache = new Map<string, { at: number; texts: string[] }>()
// Search prompts per row, LRU-bounded (#1278): every row ever searched stayed
// here for the life of the process. Same bound and eviction as the prompt
// folder's cache (prompts/promptFolder.ts), and for the same reason it is
// far above SEARCH_PROMPT_ROWS: every row one search folds must still be
// cached when the next keystroke arrives, or search thrashes and re-reads.
private readonly promptCache = new LruMap<string, { at: number; texts: string[] }>(SEARCH_PROMPT_CACHE_MAX_ENTRIES)

constructor(private readonly deps: {
sources: ConversationSource[]
Expand Down
42 changes: 40 additions & 2 deletions src/main/conversations/sources/claude.system.test.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { mkdir, readdir, writeFile } from 'node:fs/promises'
import { join } from 'node:path'
import { mkdir, readdir, rm, writeFile } from 'node:fs/promises'
import { dirname, join } from 'node:path'
import { afterAll, beforeAll, describe, expect, it } from 'vitest'

import { resolveFamily } from '../family.js'
Expand Down Expand Up @@ -107,4 +107,42 @@ describe('Claude conversation source', () => {
expect(prompts[i - 1]!.timestamp ?? 0).toBeGreaterThanOrEqual(prompts[i]!.timestamp ?? 0)
}
})

// Review of #1417 (c): `summaries` is keyed by file and was never pruned, so
// a transcript deleted from a directory that discovery keeps listing kept
// its parsed head and user texts for the life of the process. Its own corpus:
// this test deletes a file, and the shared one is read-only by convention.
it('forgets the summary of a transcript gone from a directory it listed (#1278)', async () => {
const own = await setup()
const summaries = (own.source as unknown as { summaries: Map<string, unknown> }).summaries
const family = await resolveFamily('/fixture/repo', 'repository', { listWorktrees: own.listWorktrees })
const first = await own.source.discover({ scope: 'repository', family })
const gone = first.find(row => row.file && summaries.has(row.file))!.file!
const unrelated = [...summaries.keys()].length

await rm(gone)
const second = await own.source.discover({ scope: 'repository', family })
expect(second.map(row => row.file)).not.toContain(gone)
expect(summaries.has(gone)).toBe(false)
expect(summaries.size).toBe(unrelated - 1)
})

// Review of #1417, round 2 (a): a whole project directory removed (a pruned
// worktree's transcripts) never reached the per-directory sweep.
it('forgets the summaries of a project directory that is gone (#1278)', async () => {
const own = await setup()
const summaries = (own.source as unknown as { summaries: Map<string, unknown> }).summaries
const family = await resolveFamily('/fixture/repo', 'repository', { listWorktrees: own.listWorktrees })
await own.source.discover({ scope: 'repository', family })
const counts = new Map<string, number>()
for (const file of summaries.keys()) counts.set(dirname(file), (counts.get(dirname(file)) ?? 0) + 1)
const [goneDir, goneCount] = [...counts.entries()].sort((a, b) => b[1] - a[1])[0]!
expect(counts.size).toBeGreaterThan(1)

await rm(goneDir, { recursive: true })
const before = summaries.size
await own.source.discover({ scope: 'repository', family })
expect([...summaries.keys()].some(file => dirname(file) === goneDir)).toBe(false)
expect(summaries.size).toBe(before - goneCount)
})
})
Loading
Loading