docs: measurement corpus — human ground truth, machine labels, and two protocols - #206
Merged
Conversation
…recall Every prior evaluation in this repo used a parser's own output as ground truth, so that parser scored 100% by construction and anything disagreeing with it scored as wrong. These labels break that circularity. 15 excerpts, labelled from scratch (no candidate names shown — pre-populating them would have baked in the union of the parser's and the model's assumptions, making any bidder they both missed invisible), blind to which machine proposed what, with per-contract attribution. Incumbent trca_board parser against Alex's 10 documents / 89 companies: recall 57% precision 88% (all 7 unmatched rows adjudicated non-errors) single-contract documents 96% multi-contract documents 44% The failure is structural, not transcriptional: it reads a bid list correctly whenever there is one, and loses over half the record whenever a document holds several contracts, because it anchors on "received from the following", takes that list and stops. D09 holds four contracts and yields 25%. This retires #203's conclusion that TRCA should keep its regex because cells could not be attributed to a solicitation — the regex has the same attribution failure, invisible only because the regex WAS the ground truth. Labels carry a corrections block: three source-decided fixes applied with Alex's permission, and an explicit list of what was deliberately left alone, including the still-open definitional question of whether a pre-qualification submitter with no tabulated price counts as a bid. Gabe's set is outstanding, so there is no inter-rater agreement figure yet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W53WHx8mm2UHuLFAQWeF62
The three companies excluded in D01 and the three in D10 looked like one definitional question. They are opposite cases, and the reports say so: - D01 is TWO-STAGE. Six firms made pre-qualification submissions; tender documents "were issued to the following three (3) Proponent(s)". Buildscapes, Pine Valley and Shoreline never received tenders and never bid. Excluding them is correct and the parser counting them is an error. - D10 had eight tenders SUBMITTED, three "disqualified because of the incomplete submission package". Those three did bid and belong in the record with no valid price — the archive's existing Non-Compliant convention (#94). Rule: a company is a bidder if it SUBMITTED A BID for that contract. Pre-qualified then eliminated, no; submitted then disqualified, yes. Also reverts a name change I made in error: D01's results table (the actual bid record) reads "Limited"; only the pre-qualification list reads "Ltd", so the labeller's original was verbatim from the right place. Parser vs corrected labels: recall 59% (54/92), precision 93%. Its 3 confirmed precision errors are all one class — pre-qualification list read as a bid list — and its recall deficit is entirely structural: 96% on single-contract documents against 44% on multi-contract ones. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W53WHx8mm2UHuLFAQWeF62
…was reverted) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W53WHx8mm2UHuLFAQWeF62
…RAGE OVERLAPS Gabe's 10 documents (69 rows) landed, 5 overlapping with Alex's. Raw company-set agreement across the overlap is 69%, which is misleading. Per contract: contracts both labellers read 42/44 companies agreed = 95% companies in contracts only one labeller reached 13 The 2 residual disagreements are name variants where both forms appear verbatim. Every substantive disagreement is contract COVERAGE, not reading. On D06 Gabe found 1 of 3 contracts; on D09, 1 of 4 — both large meeting packages, and exactly where the incumbent parser also stops at the first bid list. Three consequences worth recording: - The labels are trustworthy where they overlap: reading a bid list is not the ambiguous part of this task. - The hard part is finding all the contracts, for humans as much as for software. That reframes the failure as a property of the documents rather than of regex. - Ground truth on multi-contract packages must be assumed INCOMPLETE, Alex's included. Two independent readers both under-covered. Recall measured against these labels is an upper bound on the labels, not on the documents — which means the models' 100% means they matched Alex's coverage, not that they exceeded it. Independent corroboration of one earlier adjudication: Alex's D10 omitted three disqualified tenderers, added by reading the report's wording; Gabe included all three unprompted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W53WHx8mm2UHuLFAQWeF62
…'s file The model results were measured but never committed — the earlier command chain aborted on a failed copy, so the README append and commit silently never ran. Caught by verifying what was actually in the branch rather than assuming. incumbent parser 59% / 93% (96% single-contract, 44% multi-contract) gpt-5.6-luna 100% / 100% (~$4 for the whole backlog) gpt-5.6-terra 99% / 99% gpt-5.6-sol 100% / 100% claude-opus-5 100% / 100% (~$290, no better) Records the method rather than the script, which was not preserved, and states plainly what the numbers do NOT show: 100% precision means the models proposed nothing Alex did not have, so they matched his coverage rather than exceeding it — and two independent readers both under-covered the multi-contract packages. Also notes that the first run reported Sol at 22% because of a missing backoff in my own retry loop. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W53WHx8mm2UHuLFAQWeF62
Scored on Gabe's 5 solo documents (D11-D15, 25 companies Alex never saw): incumbent parser 68% recall / 81% precision all four hosted models 100% recall / 52-56% precision The low precision is the LABELS, not the models. All 23 "false positives" are verbatim in the source and every one carries a contract number Gabe never reached. Two traced to source: - D14 / RFP 10041763: "four (4) proposals were received from the following vendor(s)" lists exactly the four the model returned. Gabe labelled one of the document's four contracts. - D15 / RFP 10020367: a three-stage procurement — 8 pre-qualification submissions, 7 issued RFP documents, then "Five (5) proposals were received". The model returned exactly those five and EXCLUDED the two who pre-qualified and received documents but never proposed, applying the pre-qualified-vs-submitted rule correctly to an unseen case. This answers what the earlier round could not: against Alex's set the models matched his coverage without exceeding it, so nothing could be said about bids both humans missed. Here they recovered 23 such bids. Also records an intra-rater inconsistency: on D12 Gabe listed the City's typo "Airborne Imagine Inc." as a separate company alongside "Airborne Imaging Inc." and omitted Aeroquest Mapcon entirely, having labelled the same report correctly as D08. Consequence: precision against these labels does not measure extractor correctness. Only recall is sound; disagreements must be adjudicated against the source. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W53WHx8mm2UHuLFAQWeF62
… leaderboard General leaderboards aggregate capabilities your task does not use, so a price/performance frontier computed on one recommends the wrong model. The protocol uses the general index only to SAMPLE candidates, measures the real task against ground truth, then finds the published benchmark that orders models the way the task does and recomputes the frontier on that — buying a cheap reusable predictor with a one-time measurement. Documents the four steps, why it is economically right (direct measurement is more accurate than any proxy; the proxy exists only to narrow hundreds of models to a handful), and the failure modes — including the dangerous one, where a broken harness produces a false ordering and silently selects the wrong benchmark. Worked example from this repo's bid extraction, and it is explicit that step 3 COULD NOT BE COMPLETED here: the Intelligence Index spans 37.8-60.7 and produced a flat 99-100% line with a single outlier, and one failure is not an ordering. Completing it would require deliberately testing models expected to fail. The index's own weighting explains the mismatch: long-context reasoning is 6.25% of it and instruction-following another 6.25%, and this task is essentially only those two. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W53WHx8mm2UHuLFAQWeF62
…s best paid Full ladder against the human labels: every model beats the incumbent parser (59% recall), and the whole spread from free to $290 sits at 99-100% except DeepSeek V4 Flash at 74%. Price buys nothing above the free tier. nemotron-3-ultra:free 99% / 99% $0 gpt-5.6-luna 100% / 100% ~$4 whole backlog Nemotron's limits are operational rather than qualitative and are recorded as such: no structured-output support at all (it emits JSON by instruction, not constrained decoding), 20 req/min and 1,000/day, and free endpoints rotate out without notice. Recommended shape is Nemotron by default with Luna as automatic fallback — a 1-point difference, so the fallback costs pennies and removes a single point of failure. Also records what each benchmark got right. The Intelligence Index does not predict this task (long-context reasoning is 6.25% of it, instruction-following another 6.25%, and the task is essentially only those two). AA-LCR does order the one discriminating pair correctly, but only after a frontier bug was fixed: scoring a model's max-effort variant while pricing it at base rates had ranked DeepSeek above Nemotron, i.e. the bug was flattering the single model that fails the task. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W53WHx8mm2UHuLFAQWeF62
A procurement archive collects bids, awards and suppliers. Everything else is classified and then deliberately ignored — and this records the ignoring explicitly, per type, so "we don't extract X" is a decision rather than an omission nobody noticed. Four types extract: procurement award reports (full schema, contract-level not document-level, because one report can describe four contracts), meeting packages (split first, dedup on reference — 142 of 240 standalone TRCA reports also appear inside a package), minutes (decision date and body only), and agreements/MOU (conditional, refuse to a review queue rather than guess). Seven types extract nothing, each with its reason recorded, under one uniform escape hatch: a "nothing" document that turns out to contain a bid list was MISCLASSIFIED — re-route and log it. A nothing-verdict is a claim about the type, not permission to stop looking. Settled with Alex: - land/property out of scope (no solicitation, no bid, no supplier) - update/status reports contribute award REFERENCES for cross-checking only, never creating rows; on disagreement surface, never overwrite - prequalified/invited_to_tender are STORED, in their own table rather than in `bid`, since `bid` means "submitted a bid" and every count depends on that - minutes: decision date only, resolution text not retained Notes the consequence to accept knowingly: participation names flow into the supplier dimension, so supplier counts will include firms that never bid. Largest open item stays flagged: 1,902 documents (31%) do not classify, EP worst at 819 of 1,200. That is a classifier failure, not a document type, and those must route through procurement detection rather than be discarded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W53WHx8mm2UHuLFAQWeF62
…he LLMs Stage one of the two-stage design, measured against 125 human-labelled documents spanning every source, stratified and shuffled, source attribution withheld from the labeller. majority class 24% 0/22 procurement recall rules baseline 36% 12/22 char 3-5gram LinearSVC 65% CV 22/22 free, deterministic Nemotron 3 Ultra (free) 61-64% 19-22/22, VARIES BY RUN GPT-5.6 Luna (high) 67% 21/22 $0.009 GPT-5.2-Codex (xhigh) 62% 21/22 $1.37 Findings worth keeping: - A bag-of-words linear model matches the LLMs on the only metric that governs the design, with zero leakage into or out of the procurement class, and the features it learns are the archive's own documented procurement idiom. - Reasoning effort makes classification WORSE, monotonically, while costing 50% more. AA-LCR ranks those Luna variants in the opposite order — a long-context benchmark does not transfer to a 2,500-character header. - The most expensive model is the worst on every column. - LLM runs vary: Nemotron scored 22/22 once and 19/21 on repeat, same prompt and documents. A single run is not a measurement — which corrects a headline I reported earlier from n=1. - The two methods fail DIFFERENTLY: the SVM collapses to the majority class, the LLM to attachment_or_map, scoring 0/7 on empty_or_unreadable where the SVM got 5/7. Errors are complementary, not shared. - governance_finance <-> status_update confuses both symmetrically, which reads as a definitional boundary rather than weak models. Both extract nothing. - The taxonomy is right: free-text "Other" came back empty across all 125. - Classify-then-skip did not hide procurement data; the only non-procurement documents flagged as containing awards were 2 minutes, exactly as the schema predicts. Limits recorded: 2-fold CV on 125 documents, two classes never appeared at all (agenda, meeting_package), and the free tier throttles at CONC=6 in a way the harness reports as INVALID rather than as a zero score. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W53WHx8mm2UHuLFAQWeF62
… recall
Stage-one classification applied past the 125-document human sample to every
remaining document in background_pdf. Subagents reading FULL document text —
the earlier LLM measurements were all against a 2,500-char header, which is an
API constraint this method does not have.
Two independent haiku passes over all 5,750, a blind third fable vote on the
820 disagreements, and 120 of the human-labelled documents copied under fresh
opaque ids and mixed into the run so the DELIVERABLE — after judging, not a raw
vote — is scored end-to-end against known answers. Honeypots: 94% flag accuracy,
23/24 flag recall, 63% kind accuracy (kind tracks calibration and was never the
target; the flag is what gates extraction).
The finding worth keeping: MAJORITY VOTE IS THE WRONG RULE FOR THE FLAG, and
measurably so. It left 57 documents on the wrong side of it where at least one
reader had caught real award language — 27 with a procurement_award vote, some
naming a supplier and an amount the other two readers missed outright ("Award to
Carollo Engineers Canada Ltd./EXP Services Inc. $2,765,968.25; three suppliers
bid"). That is the archive's usual asymmetry: a false positive costs one wasted
extraction call, a false negative is never extracted and nothing downstream
notices. The flag is now OR-of-votes; kind stays majority-with-null-on-tie,
where a guessed label is worse than an honest gap and no such asymmetry applies.
Deliberately NOT ground truth and kept out of docs/ground-truth/ — labels-machine
.json is self-describing (provenance, per-vote detail, evidence quote per row) so
it can never be mistaken for the human set it was measured against.
One honeypot miss is a real schema question rather than a model error: a
Vendor-of-Record contract EXTENSION, called flag=false unanimously by both
passes. Whether an extension of an already-awarded arrangement should trip the
flag is Alex's call.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012TnxZz2vgmB3GXEyY7ERWy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Eleven commits of measurement work, all under
docs/. No file outsidedocs/is touched — nothing underscrapers/ordeploy/, so this cannot change what the nightly does.uv run pytest: 800 passed.Why it's one branch and not four
It grew past its name. The branch is
ground-truth-trca-labelsand only 7 of 27 files are TRCA — it accumulated four deliverables, because each one was the answer to a question the previous one raised. They cross-reference: the machine labels are scored against the ground-truth set and cite it by path. Splitting them now would either break those references or duplicate them. The 11 commits are the real split; each is a coherent finding with its own headline.What's in it
docs/ground-truth/trca-bid-labels/docs/ground-truth/document-classification/docs/machine-labels/document-classification/docs/protocols/The line count is 98% one file
136,150 insertions sounds enormous. 133,083 of them are JSON data and 130,549 are
labels-machine.jsonalone — a 3.6 MB machine-generated artifact. The reviewable surface is ~2,300 lines: 973 of markdown and 1,334 of one-off analysis Python. Review those; skip the JSON.Two things to know before reading
Machine labels are deliberately not ground truth, and are stored so they can't be mistaken for it. They live in
docs/machine-labels/, a sibling ofdocs/ground-truth/and never inside it. The JSON carriesprovenance: "machine", a warning string, and per-row vote detail. Where the two disagree, the human file is right until shown otherwise.The flag uses OR-of-votes, not majority — that's the finding worth keeping. Majority vote left 57 of 5,750 documents on the wrong side of
contains_bid_or_awarddespite at least one reader catching real award language; 27 had aprocurement_awardvote, some naming a supplier and an amount the other two readers missed outright ("Award to Carollo Engineers Canada Ltd./EXP Services Inc. $2,765,968.25; three suppliers bid"). That's this archive's usual asymmetry: a false positive costs one wasted extraction call, a false negative is never extracted and nothing downstream notices.kindstays majority-with-null — no equivalent asymmetry there, and a guessed label is worse than an honest gap.Known open, documented as limits rather than hidden
C088) is a Vendor-of-Record contract extension, called false unanimously by both passes. Whether an extension of an already-awarded arrangement should trip the flag is a schema question, not obviously a model error — left for a human call.🤖 Generated with Claude Code
https://claude.ai/code/session_01RiYypnPbDwFtp9zAp7tz6W