perf: skip copying and indexing sidechain conversations to reduce disk usage

[minyek]

perf: skip copying and indexing sidechain conversations to reduce disk usage

Description

The primary goal of this PR is to reduce episodic-memory's on-disk footprint. The plugin copies and indexes sidechain conversations (subagent dispatches, Warmup/ai-title stubs, prompt-suggestion sidechains) that search already excludes and never returns. In my install that was ~40% of the SQLite DB and ~25% of the conversation archive — disk spent on data no query can reach. This PR stops copying and indexing them so that storage isn't consumed going forward. It also avoids the embedding work for those exchanges, but the motivation is storage, not speed — sync latency is roughly unchanged. If a sidechain type later proves worth searching, it can be added with summarisation and query support.

• sync.ts copies every discovered .jsonl from ~/.claude/projects/ into ~/.config/superpowers/conversation-archive/, including nested subagent dispatches at <project>/<session>/subagents/agent-*.jsonl and top-level harness stubs at <project>/agent-*.jsonl (Warmup, prompt-suggestion sidechains, ai-title).
• indexer.ts then embeds every parsed exchange into a 384-float vector and inserts rows into exchanges and vec_exchanges, regardless of isSidechain.
• search.ts unconditionally excludes them via AND e.is_sidechain = 0 in both vector and text query paths.

This is a storage change, not a behavior change: search results are identical before and after, because the AND e.is_sidechain = 0 filter (originally requested in #42) already hides these exchanges from every query and has no off switch. That left an asymmetry — the data is filtered out of every result, yet still embedded, written to the DB, and copied to disk. This PR removes that wasted storage by stopping at copy time and index time, matching the filter that search already applies.

Why These Files Exist

All isSidechain: true, all legitimate Claude Code artifacts:

• Explicit subagent dispatches — Agent/Task tool transcripts under <project>/<parent-session>/subagents/. The agent's report-back already lives in the parent transcript and is searchable; the sidechain holds only the agent's internal trace (files read, intermediate reasoning).
• Warmup and harness records — top-level <project>/agent-aXXX.jsonl. Prompt-cache warmup stubs ({"content":"Warmup"}), prompt-suggestion sidechains, ai-title stubs. Most parse to 0 exchanges.
• /btw asides — Claude Code's /btw ("by the way") command asks a quick question off the main thread; its records are isSidechain: true and never join the main conversation. We considered surfacing these but intentionally don't: like every sidechain here they're already excluded from search by the is_sidechain = 0 filter, so this PR doesn't change their searchability, and they're a small volume of deliberately ephemeral content — /btw exists precisely to keep asides out of history — so summarising and indexing them for search isn't worth the cost.

None have semantic value for cross-session search as currently surfaced.

Change

Two layers:

1. Skip sidechain exchanges at index time. A single exported indexExchanges() in src/indexer.ts owns the embed + insert loop and short-circuits on isSidechain, skipping both the embedding generation and the DB insert — symmetric with the existing AND e.is_sidechain = 0 filter in src/search.ts. All five indexing paths route through it (full reindex, single session, unprocessed backlog, background sync, and verify/repair), so the filter can't be applied to some paths and missed on others. The two that matter most are syncConversations — the background path that runs on every session start — and repairIndex; both would otherwise index inline sidechain exchanges from mixed-content files. The helper returns the inserted (non-sidechain) count, which the user-facing "Exchanges: N" logs report.

2. Skip sidechain files at copy time. New isSidechainFile() in src/sync.ts reads the file in chunks and inspects whole (newline-terminated) records until one carries an isSidechain flag, returning that value. It reads complete records rather than a fixed byte window, so a subagent file whose opening record is large (a big initial prompt with attachments) is classified correctly instead of truncated; the scan is bounded (1 MB) so a pathologically large file is never read in full. Conservative on every error path (unreadable file, an unparseable record, or no record carrying the field within the cap) so ambiguous and mixed-content files still get copied and handled by the per-exchange filter above. The check is gated behind a cheap mtime comparison, so unchanged files aren't re-read on every sync. Sidechain files are counted as skipped, never copied, and don't enter filesToIndex or filesToSummarize.

Schema groundwork for the is_sidechain column stays in place, so a future opt-in feature could lift the search filter and re-index from archive.

Tests

test/indexer-sidechain.test.ts (new, 2 tests) — the manual reindex path (indexUnprocessed):

• Mixed transcript (one sidechain + one non-sidechain exchange) → COUNT(*) FROM exchanges is 1, COUNT(*) WHERE is_sidechain = 1 is 0.
• All-sidechain transcript → COUNT(*) FROM exchanges is 0.

test/sync.test.ts (5 new tests):

• Top-level agent-a01.jsonl Warmup stub → not copied.
• Nested <session>/subagents/agent-a01.jsonl subagent dispatch → not copied.
• Regular non-sidechain conversation → copied as before.
• Mixed-content file (non-sidechain first record + inline sidechain exchanges) synced with indexing enabled → file is copied, but only the non-sidechain exchange lands in the DB. This pins the background syncConversations path, which the indexer test above doesn't exercise; it indexed both rows before this fix.
• Sidechain file whose first record is larger than the read chunk → still skipped at copy time (guards the complete-record read described above).

The sync test fixture now sets EPISODIC_MEMORY_CONFIG_DIR to isolate from any pre-existing exclude.txt that would have pre-filtered subagents/ dirs at discovery and masked the test signal. Existing test/incremental-indexing.test.ts and test/exclude-nested.test.ts still pass.

Validation

Beyond the unit suite, the change was exercised against real transcripts in an isolated scratch environment (real files read-only; scratch DB and archive, summaries off):

• A 41-file sample (32 subagent sidechains + 9 main conversations) synced to 9 copied, 32 skipped, 0 sidechain rows indexed; the main conversations were indexed and returned from search as expected, and no real conversation was mis-skipped.
• This is where the large-first-record case surfaced. One subagent file had an 8 401-byte opening record. The earlier fixed-window peek truncated it mid-record, mis-classified it as non-sidechain, and copied it — search stayed correct (the per-exchange filter indexed 0 of its rows), but the file-level archive saving was lost. Reading complete records fixes it: the file is now skipped at copy time. That motivated layer 2's complete-record read and is locked in by a regression test.

Environment

• episodic-memory v1.4.2

For existing installs

This PR only prevents new sidechain data. Existing archived files and DB rows from prior versions stay until cleaned up. To reclaim space, run from the plugin directory:

# 1. Delete sidechain rows from the DB (handles vec_exchanges and tool_calls).
node -e "
  const { initDatabase, deleteExchange } = require('./dist/db.js');
  const db = initDatabase();
  const ids = db.prepare('SELECT id FROM exchanges WHERE is_sidechain = 1').all();
  const tx = db.transaction(rows => { for (const {id} of rows) deleteExchange(db, id); });
  tx(ids);
  db.exec('VACUUM');
  console.log('Deleted ' + ids.length + ' sidechain rows');
"

# 2. Remove archived nested-dispatch directories.
find ~/.config/superpowers/conversation-archive -type d -name subagents -exec rm -rf {} +

Step 2 covers <session>/subagents/ dirs; top-level <project>/agent-aXXX.jsonl Warmup stubs from prior versions are smaller and can be left in place or removed manually. After this PR lands, no new sidechain files are written, so the cleanup is one-shot.

Safe to skip — search results are unaffected either way (sidechain rows weren't returned before, aren't now). Worth running if disk pressure matters; ~40% DB and ~25% archive reduction in my install.