Skip to content

sec ask --company/--form/--since/--accession do not narrow retrieval — they are appended to the prompt; and AskTask has no test #353

Description

@sroussey

The four scope flags on sec ask are described to the user as filters:

$ ./dist/sec.js ask --help
  --company <cik|ticker|name>  Narrow to one issuer
  --form <form>                Narrow to one form type, e.g. 10-K
  --since <date>               Only filings on or after this date (YYYY-MM-DD)
  --accession <accession>      Narrow to one filing

They narrow nothing. AskTask.execute passes the whole knowledge base to AiChatWithKbTask with no filter and turns the scope into a sentence (src/task/kb/AskTask.ts:105-115, 149-165):

const scope = describeScope(input);
 prompt: scope === undefined ? input.question : `${input.question}\n\n(${scope})`,

function describeScope(input: TaskPorts<AskTaskInput>): string | undefined {
  
  return parts.length === 0 ? undefined : `Restrict your answer to: ${parts.join(", ")}`;
}

The JSDoc above it is candid about this —

Retrieval here is over one knowledge base with no per-query filter, so the scope steers the model rather than the search. Said plainly instead of implied: a narrowing that only lives in a flag is a narrowing the answer can silently ignore.

— which is the right instinct written in the wrong place. The candour is in a comment nobody running the CLI reads; the help text still says "Narrow to one issuer", and the citations printed under the answer can be from any issuer in the index. --accession is the sharpest case: a user asking about one filing gets excerpts from whatever the embedder ranked highest across the whole corpus, presented as [1] 10-K · 2024-11-01 · 0000320193-24-000123.

This is the same class as #349 (grounding enforced in the prompt rather than in code), on a smaller blast radius.

Ask

Either honour the flags or stop calling them narrowings.

  1. Preferred — filter at retrieval. IndexFilingSectionsTask already writes cik, accession, docFile, form and filingDate into each document's metadata (src/task/kb/IndexFilingSectionsTask.ts:109-117) precisely so a citation can name its filing; the same fields are what a metadata filter would key on. If AiChatWithKbTask cannot take one, retrieve through kb.similaritySearch directly and pass the excerpts as context.
  2. Failing that — reword the help ("Steer the answer toward…", not "Narrow to"), and print a one-line note under an answer whose references fall outside the requested scope. A wrong-issuer citation the user can see is recoverable; one presented as scoped is not.

Second half: AskTask is untested

$ ls src/task/kb/
AskTask.ts  IndexFilingSectionsTask.test.ts  IndexFilingSectionsTask.ts

164 lines, the command the re-founding is named for, and no test file. Everything in it that is checkable without a model is: describeScope's output for each combination of flags, the one-shot human connector's decline response (:23-30), the reference mapping at :136-142, and the empty-references path from #349. A stub for AiChatWithKbTask covers all of them.

Verify

sec index --limit 5
sec --json ask "..." --accession <an-accession-not-in-those-5> | \
  node -e 'const d=JSON.parse(require("fs").readFileSync(0));console.log(d.references.map(r=>r.title))'
# today: titles from other filings

Found during the 2026-09-07 review. Snapshot: workglow-dev/prdanalysis/grades/2026-09-07/sec-detailed.md.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions