How far can deterministic, code-based checks push a document extraction pipeline before a human has to look, and exactly where do they stop working. This is the second study in this repository: a pipeline that transcribes private equity fund documents field by field, backed by a stack of hand-written guards and a hardcoded reference database, measured against nine vision-language models across a synthetic but structurally realistic corpus.
A frontier model reads a capital call notice beautifully. It gets the fund, the currency, the amounts, the dates, almost always right. The word "almost" is the whole problem: a probabilistic reader returns a number that is usually correct and never certified, and in a back office that reconciles other people's money, "usually" is not a state anyone signs off on. The common fix is a human in the loop on every document, and in this business that human stays.
The point is not to remove that human but to make the review faster and better aimed, concentrated where a human eye actually changes the outcome. This study asks a narrower question: given that the models are good but not certain, how much of that review can a pipeline of deterministic checks actually remove, so what remains is targeted rather than exhaustive, and what kind of error is left over that no check, however clever, can see? The answer turns out to depend entirely on which class of fault is in play, not on a single number.
The pipeline asks the model to do exactly one thing per document: transcribe what is printed. Every derivation lives in code. There is no currency conversion by the model, no summing of line items, no consolidation across documents. This is not a convenience, it moves entire classes of error out of the probabilistic component and into ordinary, unit-tested code.
An earlier version of this work asked the model to convert and consolidate, and one cheap model wrote a portfolio total one hundred euros short in ten runs out of twelve, silently, with an empty anomalies array. Move the conversion into code and that failure mode cannot occur, because the model never touches the arithmetic. The conversion code still has to be correct, and it is tested, so the error is not eliminated in some absolute sense; it is removed from the space of things a probabilistic reader can get wrong. What is left for the model is transcription, and what is left for the guards is checking that the transcription is self-consistent and consistent with a reference.
Fifty synthetic documents from eight fictional private equity funds (a fictional family office as the sole limited partner) over four quarters, three document types: capital call notices, distribution notices, and quarterly capital account statements.
The eight funds deliberately span a gradient of redundancy, since that gradient is what gives the guards something to check:
- Rich format (two funds): commitment, cumulative called before/after, unfunded, full breakdown, and drawn percentage all printed on the call; the statement carries a full roll-forward plus a transaction list.
- Standard format (three funds): a total and a breakdown are printed, but no running cumulative on the notices; the statement has a roll-forward with no transaction list.
- Minimal format (three funds): a one-page letter with none of that, and a statement reduced to a few KPIs. This is the floor of the ablation, where the guards have the least structure to work with and the most is left to a human.
Each document type is modeled on a real convention rather than invented from scratch: the ILPA Capital Call and Distribution Notice template, the ILPA Quarterly Reporting Standards roll-forward, and ISO 17442 (LEI) mod-97 check digits. The oracle is emitted by the same code that renders the documents, so it cannot drift from what is printed; errors are not injected, they are whatever the nine models actually produced, labeled automatically against the oracle. All seven fault classes are natural; two of them are too rare at the volume this corpus produces to measure reliably. Pure digit slips (18 instances) are too rare even to tabulate and are omitted from the results table below; order-of-magnitude errors are shown there but should be read as directional.
The campaign runs the corpus through three input conditions (native PDF, a first-tier scan, and a hardened second-tier scan) and nine models, for 4,050 extractions total. The documents and the funds are entirely synthetic and fictional; see Limits for what that does and does not let this study claim.
The stack is cumulative: schema checks (P0), then within-document invariants (P1), then cross-document consistency (P2), then checks against the reference database (P3). Every guard carries one of three classifications, published next to its scope clause rather than as a bare label:
- [A] sure alarm: designed so that a violation implies a real fault, with zero false alarms as a measured design target, never an unverified assertion.
- [C] complete on a named class: closes every fault in an explicit family, always stated with its scope (what it is blind to), for example "complete on any single-value error in the breakdown, blind to compensating errors."
- [H] heuristic: can fire on a correct document; its false-alarm rate is the real cost of using it.
| Stack | Guard | Class | Scope |
|---|---|---|---|
| P0 | format | [C] format | Complete on syntactic malformation (IBAN outside ISO 13616 structure, date outside ISO 8601, currency outside the enum, unit outside ones/thousands) on an extraction that already conforms to the schema; blind to any value that is syntactically valid but wrong, which is P1 to P3's job. |
| P1 | breakdown-sum | [C] | Complete on any single-value error within one breakdown line or the called-capital total, to the unit (notices are exact to the cent); blind to multiple compensating errors in the same family and to co-scaling. |
| P1 | due-sum | [C] | Complete on any single-value error within the breakdown, surcharges, or the total due, zero tolerance; blind to multiple compensations and co-scaling. |
| P1 | cumulative-step | [C] | Rich formats; safe absent an unregistered capital action, since one fund's equalization is consumed via the capital-action registry; conditional on that registry. |
| P1 | unfunded | [C] | Rich formats where both terms are printed; zero tolerance on notices, one printed unit on a statement denominated in thousands. |
| P1 | drawn-pct | [C] | Rich formats; tolerance is half the last printed digit's unit. |
| P1 | roll-forward | [C] | Statements with a full roll-forward; one printed unit of tolerance if the statement is in thousands, zero otherwise. |
| P1 | tvpi | [C] | Statements printing the TVPI multiple and its three components (closing NAV, distributed cumulative, paid-in); the multiple must equal (closing + distributed) / paid-in within one unit of its last printed digit. A second, independent equation over the cumulative columns. |
| P1 | dist-components | [C] | Itemized distribution notices; anecdotal at four documents and excluded from class-by-stack claims. |
| P1 | LEI mod-97 | [C] | Complete on any single-character error, a property of the ISO/IEC 7064 MOD 97-10 check; some two-character transpositions can pass. |
| P1 | dates-ordered | [A] | Exact lexical ISO 8601 comparison. |
| P1 | no-payment-zero | [A] | If a capital call prints zero called capital and zero surcharges, the total due must also be null or zero; any non-zero value is an internal contradiction, no oracle needed. |
| P2 | cumulative-chain | [C] | Rich formats; strict equality between the cumulative called after one call and the cumulative called before the next; a registered capital action never breaks this join, only the intra-document step. |
| P2 | NAV-chain | [C] | Rich and standard formats; one printed unit of tolerance if the fund reports in thousands. |
| P2 | distributed-step | [C] | The distribution twin of the NAV chain: distributed cumulative on one statement equals the previous statement's plus the period's distributions. The symmetric paid-in twin is deliberately absent: paid-in accumulates called capital plus any registered action principal while the period's contributions sum total due (called plus surcharges), so that equation is off by their gap and would fire on correct documents. |
| P2 | cumulative-monotone | [C] | Cumulative columns (paid-in, distributed) never decrease between consecutive statements; the only net available to minimal-format statements that print a cumulative without its period flow. Blind to errors that keep the column non-decreasing. |
| P2 | paid-in-vs-calls | [C] | Universal, since paid-in-cumulative is printed even in minimal format; a call whose called-capital field is omitted makes the pair inapplicable rather than a false expectation; a mistranscribed issue date can shift a call out of its true period, a documented residual limit found on one violation out of 103 tested. |
| P2 | commitment-const | [C] | On any fund printing commitment on at least two documents; tolerance of one printed unit if reported in thousands. |
| P2 | account-const | [C] | On any fund printing a payment block on at least two notices, independent of the database; compares IBAN, BIC, ABA, sort code, and account number after whitespace and case normalization (one shared canonical form across the labeler and every identifier guard: presentation spacing and case are never treated as transcription differences). |
| P2 | call-numbering | [C] | On calls with both a call number and an issue date; detects a missing notice or a numbering slip, not the two distinctly, since sorting itself runs on issue date. |
| P2 | transactions-match | [H] | Rich formats only; matches by date and amount within one unit; labels are not compared, so some matching noise is possible. |
| P3 | commitment | [C] | On any document printing commitment, compared to the registry, normalized to full units. |
| P3 | currency | [C] | On any document printing a currency; the only stack that closes the currency sub-case of format errors. |
| P3 | account-known | [A], anti-fraud | Exact match, after whitespace and case normalization, of each identifier present both in the extraction and in the fund's known account; does not judge an identifier the extraction never supplied. |
| P3 | investor-ref | [C] | On any document printing an investor reference. |
| P3 | lei-known | [A], anti-fraud | The printed LEI must match the fund's LEI in the registry outright; closes the checksum's one blind spot, the rare two-character transposition mod-97 passes. |
| P3 | units-known | [A] | A statement's units declaration ("ones" vs "thousands") must match how master data says this fund reports; the wrong declaration poisons every amount on the page by a factor of a thousand. |
| P3 | wire-format | [C] | The wire reference must match the fund's documented reference convention, carried in the registry as a pattern; without it the reference is a free-form string no equation, account, or checksum constrains. Its authority is exactly the registry's (see Limits). |
| P3 | fees | [C] | Recomputed from the registry's fee schedule; a mislabeled fee line becomes inapplicable, not a false alarm. |
| P3 | dates-fund-life | [A] | Issue date on or after the fund's vintage; due date respects the contractual notice period. |
| P3 | doc-expected | [A], structural | Compares the identity of documents actually received to the registry's calendar, independent of their content; the guarantee behind drop-one detection (see below), proven by tests, not by the campaign. |
| P3 | completeness | [C] | Closes the "null everywhere" hole: a field the registry declares mandatory for that vendor format and document type, rendered null by the extraction, is a violation; fields the format genuinely does not print stay out of scope, zero structural false alarm by construction. |
The honest way to report detection is per class. A single aggregate number would be dominated by wrong-picks, the largest class by far, and would hide that the classes differ enormously in how catchable they are. Prevalences below are specific to this corpus and its scan-degradation recipe; they do not generalize. What generalizes is the shape, which kinds of fault a deterministic stack closes and which it structurally cannot.
| Fault class | Faults | Escape the full stack | Captured | What closes it |
|---|---|---|---|---|
| F7 identifier (IBAN, LEI, reference) | 340 | 0 | 100.0% | Every identifier is either checksummed (LEI mod-97), matched against master data (account, LEI, investor ref), checked against the fund's documented reference convention, or required present by the completeness declaration. See the caveat below: this row is only as strong as the registry's coverage. |
| F4 order-of-magnitude (rare, directional) | 96 | 15 | 84.4% | The cross-document NAV chain, the roll-forward chains and the commitment-versus-registry check catch the "printed in thousands" co-scaling |
| F2 wrong-pick (semantic) | 1,134 | 280 | 75.3% [64.7%, 83.0%] | The printed-but-wrong amount usually breaks an equation; the ones that do not are the blind spot |
| F1 omission | 257 | 99 | 61.5% | Completeness against the format's mandatory fields, and the expected-document calendar |
| F6 format (currency, date, sign) | 60 | 30 | 50.0% | Currency- and units-versus-registry close some of it; a swapped date order partly survives |
| F5 hallucination | 168 | 118 | 29.8% | P1 to P3 catch a fabrication that breaks a sum or contradicts the reference; the ones that fit every check slip through |
Across all 2,073 non-anecdotal, natural faults from all nine models, the full stack catches 73.5% cumulatively, with a document-clustered bootstrap interval of [67.0%, 79.7%]. This is deliberately not the headline: more than half that pool is wrong-picks, so the aggregate mostly measures how well one equation catches one class, and it averages a fully closed class (identifiers) with a barely touched one (hallucinations). Read the per-class table, not the mean.
The identifier row deserves its caveat stated plainly: 100% means that in this corpus, every misread or omitted identifier ran into some piece of master-data machinery: a checksum, a known account, a known LEI, a declared reference convention, a mandatory-field declaration. All of it is machinery a real back office genuinely holds, but here the registry is authored alongside the corpus (see Limits), so the row measures what full master-data coverage buys, not what a typical registry has.
Capture is a map, not a mean. The per-field breakdown of amount misreads (regenerated in guards-report.md) is the clearest picture in the study, and it is what drove the stack's final shape:
| Field (amount misreads F2/F3/F4) | Faults | Captured | What holds it |
|---|---|---|---|
| transactions, commitment, call cumulatives, surcharges, total due | 123 | 100.0% | Full equations: the transaction ledger cross-check, commitment against the registry, the call-to-call cumulative chain |
| distributed cumulative | 78 | 100.0% | The roll-forward chain between statements; before that guard existed, this field's capture was 0/78, because only a presence check ever named it |
| paid-in cumulative | 57 | 100.0% | The delta against the calls actually issued in the period, plus the TVPI cross-equation |
| closing / opening NAV | 136 | 97.1% | The within-statement roll-forward and the statement-to-statement NAV chain; the residual sits at chain edges |
| distributions of the period | 116 | 91.4% | The roll-forward plus the distributed-cumulative chain |
| contributions, unrealized change, unfunded | 294 | 65.0% | One equation each; a misread that co-moves with another term can still balance |
| called capital | 437 | 59.3% | 88% on formats that print a breakdown to sum against; 6% on minimal letters that print a single unverifiable total |
| breakdown line items | 3 | 0.0% | One replicated digit slip, three runs of one model on one scan: twice the total was also omitted (no sum to check against), once the total was misread by the same amount, an internally coherent family the sum guard cannot fault |
| IRR | 4 | 0.0% | Nothing: IRR is not recomputable from anything printed, so no equation can exist |
The lesson is structural: capture is not a property of the stack, it is a property of the field. Give a field one full equation and it closes completely; give it a partial one and errors that co-move survive; give it nothing, as with IRR or a bare total on a one-page letter, and the stack is blind no matter how many guards it has. The distributed-cumulative row is the cleanest demonstration in the data: 78 misreads, zero caught, until the one chain equation that constrains it was written, after which all 78 are caught with zero false alarms.
Why not just run it three times and vote. When a model slips on a document and the same document is rerun, it makes the same slip 93.5% of the time (1,730 of 1,850 comparable occurrences across 657 groups, 95% Wilson interval [92.3%, 94.5%]). The error is a property of the checkpoint and the input, not noise, so voting across reruns fails on almost every slip: there is nothing to outvote when all runs agree on the wrong answer. But when the same fact is printed on two different documents (a NAV that closes one quarter and opens the next), the model replicates the slip only once in 38 comparable cases, 2.6%, with a wide Wilson interval of [0.5%, 13.5%]. Those 38 groups concentrate on three funds and three models (mostly Claude Haiku 4.5 and Claude Sonnet 5), so this is reported as a bound, not a point estimate; even the top of that interval sits roughly seven times below the bottom of the same-document interval. Structural redundancy beats statistical redundancy because it compares independent observations of the same fact, not the same observation repeated.
Specificity. The structural guards (equation and registry checks) have a false-alarm rate that is essentially zero by construction: a correctly transcribed document from a corpus whose invariants hold cannot violate an invariant. That is a property of the design, not a discovery, and it is reported as what it is, an audit of the normalization and comparison code: across 4,050 extractions it fired four times (two on the paid-in-versus-calls check, two on the fee check), each a genuine coarse-grained catch on a real misread, not a spurious fire. The number that actually measures whether vigilance costs noise is the false-alarm rate of the heuristic guards, the ones that can fire on a correct document:
| Heuristic guard | Disagreements raised | Detected / faults considered | False alarms | False-alarm rate | Detection rate |
|---|---|---|---|---|---|
| Majority vote, three same-model reruns | 571 | 416 / 880 | 155 | 27.1% | 47.3% |
| Cross-model agreement | 861 | 537 / 669 | 324 | 37.6% | 80.3% |
Scored on the same fields the guards cover, the heuristic checks are not drowning in noise: majority vote is wrong about a quarter of the times it fires (27% false alarms) and catches about half the faults, and cross-model agreement is wrong about a third of the time (38%) and catches four in five. The cross-model row is an agreement between two different budget models, not a majority vote across many: swapping reruns of one model for two different ones lifts detection, since different models make less correlated errors, but lifts the false alarms with it, and more models would not remove that, since vision models share failure modes on the hardest inputs and deciding which one is right needs the ground truth the guards avoid. So voting is a real tool, not a useless one. The narrower point is what makes the structural checks worth building: they fire on a correct document essentially never, by construction, while every heuristic flag, right or wrong, still needs a human or a ground truth to adjudicate. The structural checks are silent on correct documents but blind to semantic picks; the heuristic checks see the semantic pick but cannot certify it.
Triage. This is the acceleration made concrete, targeting the reviewer rather than replacing them. Ranking documents by how many guard violations they trip and reviewing the most-suspicious ones first: reviewing about 19.6% of documents surfaces roughly 80% of faults, and about 32.1% surfaces roughly 90%. The curve then plateaus near 91% capture from about 45% up to 55% of documents reviewed, since the remaining faults are the invisible ones from the section above, and reaching 95% requires reviewing about 73.1% of everything. This curve is computed with knowledge of where the faults are, so it is an optimistic bound on what a real triage queue would achieve.
Schema conformance. 98.8% of the 4,050 extractions parsed as schema-conformant on the first attempt (4,003 of 4,050). A bounded conformance retry, which reprompts for schema validity only and is separate from the guard-repair loop this study does not run, recovered 11 of the remaining 47; 36 (0.9%) stayed non-conformant, almost all from one weak model (pixtral-12b, 34 of 36).
What survives the full stack is not a whole fault class but a coherent subset of two: the wrong-picks and hallucinations that happen to pass every check. It is worth being precise, because the table invites overclaiming. Three in ten hallucinations are caught (50 of 168), for a concrete reason: a fabricated number that also breaks a sum, contradicts a cross-document chain, or fails to match the registry trips a guard like any other fault. What slips through sits where the guards have nothing to compare it against: a field that enters no equation, repeats on no other document, and matches nothing in the registry, or, less often, one whose wrong value still satisfies every equation that does apply. No guard fires, and not because the value was checked and found consistent, but because nothing constrains that field in the first place. Nothing is out of place, there is no redundancy to contradict it, and only semantics, or a human, catches a well-formed wrong answer. Wrong-picks are the larger part of this floor (280 of the 549 survivors), hallucinations the smaller (118). The field-capture map above says where they sit: on the fields with partial or no redundancy: a called capital on a one-page letter, an IRR nothing can recompute, a value that co-moves with its own total.
One extension targets the hallucination half directly. A hallucination is, by definition, a value that appears nowhere in the document, so checking that each extracted value is actually present in the source, grounding every field to the text it came from, attacks exactly this class and only this class (a wrong-pick is present, just misplaced). The difficulty is where the source text comes from. It stays a code-side guard only if presence is verified against a trusted non-model source, a PDF text layer or a reliable OCR pass; a bounding box returned by the model is the model checking itself, and a reader confident enough to invent a value is confident enough to invent its location, the correlated-error failure that voting already runs into. It is weakest exactly on the hardest scans, where the text layer is gone and OCR is least reliable, and it needs fuzzy, normalized matching to avoid firing on legitimate reformatting, which makes it a heuristic guard rather than a structural one. It would not touch the wrong-pick floor at all.
Almost everything above is a measured rate with a confidence interval and a corpus caveat. One result is not. Because the registry declares which documents are expected in each fund and period, a document that never arrives, or a call missing from a cumulative chain, is detected by construction rather than by any measurement campaign. This is proven by drop-one tests in CI, not by a sampled result: dropping one capital call from the observed set leaves a nonzero, call-sized residual in the paid-in chain (tests/v2/drop-one.test.ts), and a missing quarterly statement is flagged by the registry calendar alone, independent of any extraction (tests/v2/harness/guards/p3-registry.test.ts). A model-plus-vote pipeline has no way to notice a document it was never given; this is the one place the architecture offers a guarantee rather than a probability. There is a practitioner's lesson under it: a business rule, here the registry's calendar of which documents each period should contain, can pay off far more than any amount of parsing accuracy, since a document that simply never shows up is caught without a single character being read. Not "the model is more accurate," but "the pipeline knows what it has not seen."
- The documents are synthetic and the scans are clean-office degradations, not the hostile photocopies and phone captures of the real world.
- The prevalences (how many identifier errors, how many wrong-picks) are artifacts of this corpus and its degradation recipe; only the conditional catch rates and the completeness properties of the guards carry over, and even those should be read as directional given document-clustered confidence intervals. Only the method generalizes, not the numbers.
- The reference database is authored alongside the documents, so it is independent of any single issuer's claims but not epistemically independent of the study's author; in production its independence would come from being real master data, a stronger claim than can be made here. The identifier row's 100% leans on this the hardest: it includes a wire-reference format convention and per-fund declarations (LEI, reporting units, mandatory fields) that this registry knows completely, and a real registry might not.
- No economic headline is drawn from this study. The per-extraction API cost of the frontier tier ran about 8.7 times the budget tier here, but the cost that actually dominates is engineering the guards and the registry, and the cost of the errors that still escape, neither of which this study prices.
- This measures the transcription layer only, not the messier reality of bad source data and restatements, which is a different and harder problem.
This repository holds two studies: the first at the repository root, and this one (v2) under the v2/ paths (generator/v2, harness/v2, results/v2, tests/v2), which reuses a few low-level modules from the first (money formatting, the PDF shell and disclaimer, the model roster).
git clone https://github.com/PittsCraft/doc-extraction-evals
cd doc-extraction-evals
pnpm install
pnpm exec playwright install chromium # for PDF generation only
pnpm test # oracle, guards, labeling and scoring tests
pnpm typecheck
pnpm lint
pnpm generate:v2 # regenerate the v2 registry, oracle and PDFs
pnpm generate:scan:v2 # regenerate the two degraded-scan tiersRunning the campaign needs API keys for the models you want, either direct provider keys or a Vercel AI Gateway key (AI_GATEWAY_API_KEY). Copy .env.example to .env and fill what you have, then:
pnpm eval:v2 --model all # the full roster, all conditions, 3 runs each
pnpm coverage:v2 # gate-2 natural coverage per fault class
pnpm report:v2 # rebuild the guard inventory and results tablesThe real campaign cost was $36.33 across the nine pinned models, recomputed from the committed per-call costs: $6.81 over the six budget models (2,700 extractions, 2,747 model calls once conformance retries are counted) and $29.52 over the three frontier models (1,350 extractions). Every result figure here is regenerated into results/v2/guards-report.md and coverage.md by pnpm report:v2 and pnpm coverage:v2 from the committed model outputs under results/v2/, without spending anything further.
Nine models, split by role: six budget models chosen to generate complementary, attested fault profiles (family-of-slips behavior, systematic omissions, wrong-picks, transcription noise), and three frontier models that are correct on almost everything, used to measure the false-alarm rate of the guards on clean extractions.
| Model | Role | Input / output $ per MTok | Pinned | Gateway slug |
|---|---|---|---|---|
| gemini-2.5-flash-lite | error-generator | 0.10 / 0.40 | 2026-07-16 | google/gemini-2.5-flash-lite |
| pixtral-12b | error-generator | 0.15 / 0.15 | 2026-07-16 | mistral/pixtral-12b |
| gemini-3.1-flash-lite | error-generator | 0.25 / 1.50 | 2026-07-16 | google/gemini-3.1-flash-lite |
| gemini-2.5-flash | error-generator | 0.30 / 2.50 | 2026-07-14 | google/gemini-2.5-flash |
| gpt-5.4-mini | error-generator | 0.75 / 4.50 | 2026-07-16 | openai/gpt-5.4-mini |
| claude-haiku-4.5 | error-generator | 1.00 / 5.00 | 2026-07-14 | anthropic/claude-haiku-4.5 |
| gemini-3-flash | frontier | 0.50 / 3.00 | 2026-07-14 | google/gemini-3-flash |
| claude-sonnet-5 | frontier | 3.00 / 15.00 | 2026-07-14 | anthropic/claude-sonnet-5 |
| gpt-5.5 | frontier | 5.00 / 30.00 | 2026-07-14 | openai/gpt-5.5 |
Model versions are pinned and dated in harness/providers.ts; claude-haiku-4.5 resolves to claude-haiku-4-5-20251001 and gemini-3-flash to gemini-3-flash-preview on direct provider calls, everything else uses the id shown directly (pixtral-12b is gateway-only and has no direct-call path).
MIT. The documents are fictitious: every fund, manager, account number, and identifier is invented, and IBANs deliberately fail their checksum by design (see the guard stack above for why). Bank names and BICs are drawn from real public routing data for realism rather than invented, but the accounts behind them are not real. Each PDF carries a disclaimer.