fix(kb): honour --dry-run when indexing, and page the chunk scan

[sroussey]

Two correctness bugs in the knowledge base, one on the write side and one on the read side.

1. --dry-run wrote to the knowledge base for real

What was wrong. secKnowledgeBase.ts:160-193 already recognised the hazard: the KB's three storages are constructed directly against getDb() and never pass through createStorage, so no ReadOnlyTabularStorage wrapper stands between them and a committed row. It guards the DDL (refusing under a dry run when a table is missing, skipping setupDatabase()) and the kb_index row write. Nothing guarded the ingest: IndexFilingSectionsTask.execute had no isDryRun() check at all, and KbAddDocumentTask → KnowledgeBase.upsert lands straight on the raw SqliteTabularStorage / SqliteVectorStorage.

The concrete failure. On a database that has been indexed once (so the three tables exist), sec --dry-run index --company AAPL has runCommand print "Dry run — no data will be written", then embeds every unindexed Apple filing and commits kb_document and kb_chunk rows. sec --dry-run ask "…" is worse: the implicit pre-index (src/cli/groups/ask.ts:112-116) does the same for up to DEFAULT_ASK_INDEX_LIMIT = 25 filings before the question is read. The added test confirms it against the unfixed code — expect(upsert).not.toHaveBeenCalled() fails with Number of calls: 1.

What changed. IndexFilingSectionsTask.execute now branches on isDryRun() after the selection (which is a read, so the run can still say what it would do) and before the first upsert: it prints Would index N filing(s) into the knowledge base. and returns { success: true, indexed: 0, sections: 0, skipped, truncated }. That is the shape BootstrapDownloadTask and IngestAdvSnapshotTask already use. getSecKnowledgeBase() is still called first, so its existing dry-run refusal for an index that does not exist yet is untouched.

2. sec ask retrieval was an unbounded full scan of kb_chunk

What was wrong. The chunk store was @workglow/sqlite's SqliteVectorStorage, whose similaritySearch is const allEntities = (await this.getAll()) || [] followed by a JS cosineSimilarity per row — and SqliteTabularStorage.getAll() with no options is a bare SELECT * FROM `kb_chunk` with no LIMIT. Every question materialised the entire index.

The concrete failure. This is not hypothetical scale; the product directs users at it. IndexFilingSectionsTask's own doc says a corpus "takes hours to days", and sec ask prints "Run sec index for the full build" when it truncates. Follow that advice over a few thousand converted filings — hundreds of thousands of chunks, each row carrying its text plus a vector stored as a JSON string — and the next sec ask loads all of it into the heap and hydrates every row before scoring one query. The user pays days of CPU embedding to produce an index that cannot be queried.

What changed. PagedChunkVectorStorage (src/kb/PagedChunkVectorStorage.ts) overrides similaritySearch to read kb_chunk in 512-row pages, scoring as it goes and keeping only the best topK. Pages are ordered by the primary key, because LIMIT/OFFSET with no ORDER BY is free to hand back a row twice and skip another. secKnowledgeBase.ts constructs that instead of the base class.

What this does not fix, stated where it matters. There is no approximate-nearest-neighbour index here, so every question still scores every chunk: this bounds memory, not time. That is written on the class, and sec ask's truncation advice now says so too — a larger index is a slower question, and --company / --form scope the build. Adding an ANN index is a @workglow/sqlite change, not one this repo can make.

Tests

Three new cases, all run against the unfixed code first:

• src/kb/secKnowledgeBaseSearch.test.ts — seeds 1200 chunks (at a 4-wide vector, via SEC_EMBEDDING_DIMENSIONS, so no model is loaded), spies on the SQLite connection's prepare, and asserts every read of kb_chunk carries a LIMIT while the ranking still returns the three best rows — which are seeded last in key order, so only a scan that reaches the end finds them. Failed before (expected 1 to be greater than 1: one unbounded read), passes after.
• IndexFilingSectionsTask under --dry-run > embeds and persists nothing — failed before (upsert called once), passes after.
• … > still indexes when this is not a dry run — the control; passed before and after, so the guard cannot be a blanket disable.

Verified

• bunx vitest run src/kb src/task/kb — 41 passed
• bunx vitest run (full suite) — 151 files passed, 3 skipped; 1486 tests passed, 21 skipped
• bun run typecheck — clean
• bun run lint — clean
• bunx oxfmt --check — all matched files correctly formatted
• bun run build — both binaries bundle

Nothing was skipped. All checks were run in a fresh worktree after bun install.

Not verified

The paging is exercised over 1200 rows at a 4-wide vector, not against a real multi-hundred-thousand-chunk index at 768 dimensions — no such corpus exists here, and building one is the multi-day job this fix is about. The property under test is that the reads are bounded and the ranking is unchanged, which is what the size-independent part of the claim rests on.

🤖 Generated with Claude Code

https://claude.ai/code/session_01T8DU24G9GWuRUJncc5TJFZ

---

Generated by Claude Code