Select the filings to index in the database, not by sifting headers

[sroussey]

Closes #351.

limit bounded the number of filings indexed, and an already-indexed one was skipped without counting toward it:

if (limit !== undefined && indexed >= limit) { truncated = true; break; }
const docId = `${header.accession_number}:${header.doc_file}`;
if (input.force !== true && (await kb.getDocument(docId)) !== undefined) {
  skipped += 1;
  continue;                       // ← costs a KB round trip, buys no progress
}

So --limit 5 against a corpus that is already indexed read every converted document and probed the knowledge base once per row before concluding it had nothing to do. ask builds the index implicitly, so that ran on every question.

Take the issue's preferred option, not its fallback

The issue offers an examined budget as a fallback. selectFilingsToConvert.ts already answers the same question about conversion as an anti-join, three directories away, so this copies that shape rather than inventing a second one.

selectDocumentsToIndex:

SELECT d.*
  FROM `filing_document` d
  LEFT JOIN `kb_document` k
    ON k.`doc_id` = d.`accession_number` || ':' || d.`doc_file`
 WHERE …scope… AND d.`section_count` > 0 AND k.`doc_id` IS NULL
 ORDER BY d.`filing_date` DESC, d.`accession_number` DESC
 LIMIT ?

since moved into the query in the same change — filtered in the loop, it cost a full read of everything older than the cutoff just to discard it.

Four things that are not just the anti-join

• truncated is observed, not inferred. The selector is asked for limit + 1. Filling the limit exactly no longer reads as "there is more behind this".
• No index yet is an answer, not an error. The three knowledge-base tables are built lazily by the first command that opens the index, and this runs before that — so a database with converted filings and no index has no table to join against. The join is dropped and every document needs work.
• Documents with no sections are excluded. They embed nothing, so they never enter kb_document and would be re-selected forever; under a selection-time limit they would also spend it on work that cannot happen. section_count is written in the same transaction as the section rows, so it is the same answer reading them gives.
• skipped stays exact, through one COUNT over the same join. It is what sec index prints and what decides whether it suggests asking a question or converting filings — reporting 0 on a fully indexed corpus would send the operator to sec update documents, which has nothing to do.

The progress denominator was wrong, and this fixes it as a side effect

It used documentRepo.count(scope) — every candidate, including the already-indexed ones — so a run indexing three filings out of three hundred candidates drew at one percent. The honest denominator is the work selected, which is now known before the loop starts. That COUNT is the one skipped replaces, so the query count is unchanged.

The repository fallback consults no knowledge base

It is reached only where there cannot be one. A non-durable document repository is invisible to getDb(), so opening the index there would read a real database this caller never wrote to and report the wrong documents as already indexed; on Postgres getSecKnowledgeBase refuses by name, so the index does not exist at all. Naming the candidates and letting the caller fail where the knowledge base itself refuses is the honest behaviour for both.

Verified

• 9 new cases in selectDocumentsToIndex.sqlite.test.ts. The discriminating ones were watched failing first — removing the section_count > 0 guard turns that case red; the limit case fails against the old counting.
• The two task tests that pinned the streaming mechanism are replaced by ones that pin the outcome: no repository read method is called at all on the SQLite path, getDocument is never called per row, and skipped still reports 3 on a fully indexed corpus.
• format-check, lint, tsc --noEmit, and src/task src/kb src/cli src/config (648 passed, 21 skipped) all clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LWp6Z6wvAPDaDCjAFcTSj6

---

Generated by Claude Code