Skip to content

The document sweep has no terminal state: a permanently unconvertible filing is re-fetched on every run, forever #337

Description

@sroussey

ConvertFilingDocumentsTask gets the sweep discipline right — cancellation re-thrown, one bad filing skipped rather than fatal, context.disown in finally (src/task/document/ConvertFilingDocumentsTask.ts:139-166). But a skipped filing writes no filing_document row, and selectFilingsToConvert's anti-join keys on exactly that row:

LEFT JOIN `filing_document` d
  ON d.`cik` = f.`cik` AND d.`accession_number` = f.`accession_number`
 AND d.`converter_version` = ? AND d.`is_primary` = 1
… if (options.force !== true) clauses.push("d.`accession_number` IS NULL")

So a filing that can never convert — a members list with no prose after the binary-envelope refusal (7defb7b), a document the parser produces no structure for — is re-selected on the next sweep and every sweep after it, at one HTTP fetch plus a full parse each time. There is no dead letter, no negative marker and no CONVERTER_NO_STRUCTURE row for this tier (that reason code exists in DEAD_LETTER_REASON_CODES but the document sweep does not write one).

This is the mirror of the class c9bb423 just fixed — there a missing primary row made the filing re-select forever after a successful conversion — and it is bounded by the converter version, so the cost is per-version rather than unbounded in time. But it is the same anti-join with no terminal state, and the corpus is CONVERTIBLE_FORMS over hundreds of thousands of filings.

Ask. Give the sweep a way to record "tried at this converter version, produced nothing" that the anti-join respects, so an operator's next sync documents moves on. Two shapes fit what the repo already does:

  • a filing_document row with is_primary set and zero sections plus a reason column, so the anti-join is unchanged; or
  • an extraction_dead_letter row under a document-tier extractor id with CONVERTER_NO_STRUCTURE, version-gated like every other, so retry-dead-letters is the recovery path after a converter fix.

Either way, --force should still re-try it.

Found during the 2026-08-31 review. Snapshot: workglow-dev/prdanalysis/grades/2026-08-31/sec-detailed.md §4.7.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions