Replace six per-source parsers with LLM extraction - #214
Merged
Conversation
Add the two frontier tickets for #205 (LLM-based extraction): - extraction_cache table keyed on (sha256, extractor_version) so re-runs skip documents whose content and prompt haven't changed - ExtractionClient: OpenRouter API client with Nemotron 3 Ultra (free) default, GPT-5.6-Luna fallback, retry with backoff, placeholder credential guard, flex tier for OpenAI models - tb extract --dry-run <sha256> CLI command for one-off testing - 26 new tests (5 cache + 21 extraction), 826 total passing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RiYypnPbDwFtp9zAp7tz6W
capture_files now compares the new capture's file count against the existing bundle's top-level entry count before calling build_bundle. A re-capture with strictly fewer files is refused — the existing bundle stays, partials are kept for a future retry, and a log message names both counts. First captures and equal-or-larger re-captures are unaffected; _MIN_CAPTURE_RATIO is unchanged. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RiYypnPbDwFtp9zAp7tz6W
…#209) Adds extract_corpus() to iterate corpus documents through the LLM client, gated by machine-classification labels and cached in extraction_cache with result_json. Validation compares cached results against ground-truth labels. CLI wires --corpus, --limit, --labels, --validate flags on tb extract. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RiYypnPbDwFtp9zAp7tz6W
…counts (#210) check_declared_counts compares each contract's declared_submissions against the actual bid count, flagging shortfalls while keeping overshoots. Flags are stored in extraction_cache.result_json under _flags and surfaced in the validate CLI output. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RiYypnPbDwFtp9zAp7tz6W
split_document breaks docs exceeding max_chars on double-newline boundaries; each chunk is extracted independently and the contracts merged. dedup_contracts collapses contracts sharing a reference, keeping the one with more bids. extract_corpus tracks split/chunks stats. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RiYypnPbDwFtp9zAp7tz6W
All six corpora extracted through GPT-5.6-Luna. Per-corpus recall against incumbent parsers: EP 98.6%, Zoo 96.3%, award_summary 88.9%, committee 100%, TRCA 74.3% (partial join coverage). Composite has no per-document join — corpus-level only. Adds --model flag for forcing a specific OpenRouter model, --report flag for generating a validation report comparing LLM vs incumbent parser output, and corpus_validation_report() in extraction.py. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RiYypnPbDwFtp9zAp7tz6W
backfill_from_extraction() maps cached LLM extractions to the store's agency_bid/agency_award, bid, and composite_award tables using the rebuild pattern (delete old source rows, insert new). Wired as `tb extract --corpus <name> --backfill`. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RiYypnPbDwFtp9zAp7tz6W
…ion (#213 phase B) Six corpus files (TRCA, EP, Zoo, award_summary, committee, composite) had their parse functions deleted and store functions rewired to call extract_and_backfill(), which reads from the extraction_cache populated in #212. Infrastructure code (download, scrape, cache, discovery) is kept intact. Deleted modules: pdf_tables.py, agency_report.py (dead once parsers gone). Deleted test files: test_composite_awards.py, test_composite_reports.py, test_pdf_tables.py, test_agency_report.py. Remaining test files keep only infrastructure tests. -3,028 lines removed, 734 tests pass. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RiYypnPbDwFtp9zAp7tz6W
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
extraction_cachetable) and orchestrator that sends board-report text to GPT-5.6-Luna via OpenRouter, validates against declared bid counts, and stores structured JSON keyed on(sha256, extractor_version)agency_award,agency_bid,agency_solicitation,composite_award,bid) from the cached extractions — Phase Apdf_tables.py,agency_report.py) — Phase Bstore_*function to callextract_and_backfill(), which skips extraction when all docs are cached and no API key is setWhat this changes
One extraction path instead of six. A City PDF reformat that would have silently broken one parser now degrades gracefully through the model's general reading ability. Net -3,470 lines of parser code removed across the branch.
Full corpus: 1,722 documents extracted, validated against ground truth (declared bid counts in the documents themselves), total cost $6.08.
Download/scrape/cache/discovery infrastructure is untouched — only the parse-and-store seam changed.
Commits
de500d054c0b90968224398c531c3c152f164a552fe936004Also includes
2af4f89(fix: refuse to overwrite an Ariba bundle with fewer files, #199).Test plan
Closes #213, closes #209, closes #210, closes #211, closes #212.
🤖 Generated with Claude Code
https://claude.ai/code/session_01RiYypnPbDwFtp9zAp7tz6W