diff --git a/README.md b/README.md index 24c914a5..b8a00596 100644 --- a/README.md +++ b/README.md @@ -1,28 +1,237 @@ -# SuperDocs Builds +# Insurance loss-run summary app -Community builds: apps, integrations and extensions built on the [SuperDocs](https://superdocs.app) platform. +Turns years of raw loss-run data across multiple policies and carriers into one +review document an underwriter can act on: claim counts, incurred totals, and +open claims grouped by policy year — with every figure traced to the document +and row it came from. -SuperDocs is an AI document editor that works inside the document, not next to it. It ships as a free web app, a REST API, and an MCP server that lets AI agents read and edit documents on their own. Everything in this repository is built on that public surface. +Built on SuperDocs (upload, search, chat, approve, export) for the SuperDocs +task. Assigned build, band S2, surfaces: search and export. -## What lives here +> Built by Siddharth Muneshwar for the SuperDocs task. -| Folder | What belongs in it | +## What SuperDocs features it uses + +| feature | how | |---|---| -| [`use-cases/`](use-cases/) | Apps and end-to-end use cases: vertical tools, document workflows, demos that solve a real problem for a real kind of user | -| [`extensions/`](extensions/) | Extensions and developer tooling: editor integrations, plugins, CLIs, SDK wrappers, agent pipelines, anything that extends where SuperDocs can run | +| **Upload** | the adjuster notes, application and broker correspondence are attached to the session; free, and what makes cross-document retrieval possible | +| **Chat editing** | one turn does two jobs: fills a placeholder section from the attached documents, and writes the opening narrative | +| **Approve** | the proposed changes are approved before anything is written | +| **Export** | the review document comes back as DOCX | + +A full review costs **one operation**. Figures never pass through the model: +they are parsed from the loss runs and the instruction forbids changing them. + +## Run it + +``` +pip install -r requirements.txt +python -m pytest tests/ -q +python -m lossrun.review --corpus corpus/out --out out --mode dry --max-ops 6 --as-of 2026-01-09 +``` + +25 tests, no API key, no network, no operations; then the full pipeline +offline. + +The generated corpus ships in `corpus/out/`, so there is nothing to build first. +Regenerating it (`cd corpus && python generate.py`) needs LibreOffice on the +path for the PDF loss run; nothing else does. + +Against the live API: + +``` +export SUPERDOCS_API_KEY=... +python -m lossrun.review --corpus corpus/out --out out --mode live --max-ops 6 --as-of 2026-01-09 +``` + +Four operations, capped at six. On PowerShell the first line is +`$env:SUPERDOCS_API_KEY = '...'`. + +A `Makefile` wraps these as `make install`, `make test`, `make dry` and +`make live` where `make` is available. It is a convenience, not a dependency — +every command above runs as written on Windows, macOS and Linux. + +The key is read from the environment only — never an argument, never logged, +never written to an output file, and a test asserts no key-shaped string reaches +any output. + +## The one design decision everything follows from + +**Figures are parsed. They are never searched for, inferred, or model-generated.** + +Claim counts, incurred totals and loss ratios are arithmetic over rows read out +of the loss runs. An underwriter cannot act on a total that might be a plausible +guess, and a summary whose numbers need re-checking has saved nobody any work. + +Search and chat are spent where they earn their cost: + +| work | how | operations | +|---|---|---| +| claim figures, counts, totals, ratios | parsed from CSV/PDF rows | 0 | +| contradictions living in prose, and the narrative | one edit | **1** | +| upload, attach, approve, export | free per published pricing | 0 | + +**A full review costs one operation.** + +**On the search surface, and two wrong turns getting here.** A dedicated +`/v1/search` endpoint returned `HTTP 404` live, so retrieval moved to the chat +surface: the prose documents are attached to the session by `upload`, which is +free. The first attempt asked for a JSON chat reply and looked for it in the +proposed-changes payload — which carries document edits, not chat text, so the +answer was never there to find. That was fighting the product. SuperDocs is an +editing agent: the answer belongs *in the document*. The review is now written +with a placeholder section, and one edit both fills it from the attached +documents and adds the opening narrative. Supported path, and half the cost of +the version that was wrong. + +`SuperDocsClient.search` and `.retrieve` are kept, unused, for the day a search +endpoint or a chat-reply field is exposed. `search` raises `SearchUnavailable` +on 404 rather than pretending an empty search succeeded. + +The chat instruction says explicitly that every number in the document is +already correct and must not be changed, recalculated, rounded or restated. +There is a test asserting that instruction still says so. + +A naive design that searched once per claim would have spent 41 operations on +one account and still produced figures nobody could audit. + +## What makes loss runs hard + +A loss run is a carrier's claims history for one policy year. They get +**reissued**, and reserves develop between issues. The same claim carries +$45,000 on the run valued January 2024 and $182,400 on the reissue valued +January 2026. Neither document is wrong. + +This is not a detail. On the sample account: + +| source for policy year 2023 | incurred | loss ratio | referral rule | +|---|---|---|---| +| original run, valued 2024-01 | $286,092.44 | **57.1%** | passes | +| reissue, valued 2026-01 | $647,992.44 | **129.3%** | **refer** | + +The same account either clears the 60% referral threshold or blows through it +twice over, depending on which document is trusted. So the app takes the later +valuation as authoritative **and records the difference as a conflict** rather +than overwriting quietly. A conflict is never delivered pre-resolved; there is a +test for that. + +## What it produces + +`out/review.html` — and see the known defect below regarding the DOCX export: + +- **Summary by policy year** — claims, open count, paid, reserve, incurred, loss + ratio, and which document each year's figures came from +- **Open claims**, listed separately with months open, because reserves on open + claims may still develop +- **Conflicts between sources**, surfaced with both figures and the consequence +- **Checklist findings** against eight underwriting rules +- **What this review does not establish** — named, not smoothed over +- **Sources**, including which documents were read and superseded + +`out/summary.json` carries the same content for anything downstream. + +## The rules + +R1 severity claims need an adjuster note · R2 open more than 36 months · +R3 loss ratio above 60% · R4 date of loss outside its policy period · +R5 application consistent with the loss runs · R6 dormant reserves · +R7 duplicate claims · R8 loss run adds up to its own stated total + +A clean corpus produces no findings at all, and that path is tested. An honest +report of nothing is a valid output. + +## Formats + +CSV and PDF loss runs both parse to the same rows, and there is a test asserting +the same policy year read two ways gives identical facts. The PDF parser is +deliberately conservative: a row is accepted only when every field is present +and every figure parses. Rows that look like claim lines but do not fully parse +are counted and reported as unreadable rather than guessed at, because a +half-read row is worse than a missing one on a document feeding a coverage +decision. + +Accountant notation is parsed as written: `(12,400.00)` is negative, `—` is nil, +`$1,204,880.00` is a figure. A string that is not a figure raises rather than +quietly becoming `0.00` — silently reading `pending` as zero would corrupt a +total. + +## Budget + +One operation per run, capped at six. The cap is enforced **before** each +spend. Hitting it degrades rather than dies: the review is still produced, and +the section that could not be completed is named in *What this review does not +establish* instead of quietly disappearing. There is a test that runs with a cap +of one and asserts exactly that. + +```bash +python -m lossrun.review --corpus corpus/out --out out --mode live --max-ops 6 --as-of 2026-01-09 --skip-prose +``` + +## The corpus + +`corpus/` generates a fabricated account — Meridian Cold Chain Logistics, a +refrigerated carrier — with 41 claims across four policy years, five loss runs +including one reissue, three adjuster notes, a signed application, a broker +cover note, and the underwriting checklist. CSV, PDF and DOCX. + +`corpus/MANIFEST.md` is the answer key: nine planted conflicts and thirteen +expected findings, with verified figures. The test suite asserts the app +reproduces it **exactly** — every planted item found, and nothing outside the +key reported. 34 of the 41 claims are deliberately clean, which is what makes a +false positive measurable rather than arguable. + +Everything is fabricated. Meridian Cold Chain Logistics, Kestrel Mutual and +Ardent Risk Partners do not exist, and no figure comes from any real claim, +policy or company. + +## Known defect: the DOCX export returns the wrong document -Every project is self-contained in its builder's own folder and carries its own README. +**`out/review.html` is correct and complete. `out/loss-run-review.docx` is not.** -## Contributing +The app uploads the adjuster notes, application and correspondence to the +session so the agent can read them for the correspondence section. Export then +asks the session for "the document" — and returns the last file uploaded, not +the review that was passed to the edit call as `document_html`. On the sample +corpus the exported DOCX comes back containing the underwriting checklist. -Fork the repo, build in your own folder, open a pull request. The full mechanics, including how to name your folder and what your PR description must include, are in [CONTRIBUTING.md](CONTRIBUTING.md). +Everything upstream of the export is unaffected: the figures, the conflicts, the +findings and the citations are all correct, and `out/summary.json` and +`out/review.html` carry them. The defect is in how this build sequences upload +against export, not in the analysis. -## Useful links +The likely fix is to make the review itself the session document before editing, +rather than passing it only as `document_html`. That needs live API calls to get +right, so it is stated here rather than guessed at. -- Product: [use.superdocs.app](https://use.superdocs.app) -- Developer documentation: [docs.superdocs.app](https://docs.superdocs.app) -- Contact: hello@superdocs.app +## Honest limits -## License +- **The carrier's own numbers are taken as given.** Each run is checked against + itself and against the others; nothing here can confirm the carrier was right. +- **Retrieval quality is not measured.** The offline fake fills the placeholder + by keyword, deliberately weaker than the live agent, so a point that only + surfaces against the real API stays visible rather than hidden. +- **If the edit proposes no changes**, the correspondence section keeps its + placeholder and the run says so under *What this review does not establish* + rather than reporting a success it did not have. +- **Figures do not depend on retrieval at all**, and there is a test that proves + it: the pipeline is run twice, once with retrieval and once with + `--skip-prose`, and every total, ratio and finding must be identical. +- **R5 reads specific declarations** from the application. Where a declaration + is not found, nothing is asserted about it — an absent statement is not a + false one. +- **PDF reading needs `pdfplumber`**, which `requirements.txt` installs. An + earlier version shelled out to `pdftotext`, a Unix binary: it worked on the + machine it was written on and died with a bare `WinError 2` on the machine it + was used on. Where no reader is available the CSV path is unaffected and the + failure names the fix instead of raising a traceback. +- **R7 needs description overlap**, not just a matching date and amount. An + early version flagged any two claims sharing a date and figure, which on a + fleet account is coincidence rather than duplication. Found by the + clean-corpus test, and the rule was fixed rather than the test. +- **The DOCX export is broken**, as above. The HTML review is the working + deliverable. +- **No coverage recommendation.** The app summarises and flags; it does not + advise whether to write the risk, and the chat instruction forbids adding one. -MIT. Every contribution stays publicly credited to its author, permanently. +Portions of this work were written with AI assistance, directed and reviewed +by me. diff --git a/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/Makefile b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/Makefile new file mode 100644 index 00000000..b90689b4 --- /dev/null +++ b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/Makefile @@ -0,0 +1,13 @@ +.PHONY: install test dry sample live clean +install: + pip install -r requirements.txt +test: + python -m pytest tests/ -q +dry: + python -m tablebench.run --mode dry --max-ops 25 --out results +sample: + python -m tablebench.run --mode dry --limit 3 --max-ops 3 --out results +live: + python -m tablebench.run --mode live --max-ops 25 --out results +clean: + rm -rf results corpus_out .pytest_cache **/__pycache__ diff --git a/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/README.md b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/README.md new file mode 100644 index 00000000..aecf00d8 --- /dev/null +++ b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/README.md @@ -0,0 +1,209 @@ +# Table fidelity benchmark + +Measures whether merged cells, nested tables, spanning headers and numeric +formatting survive an edit-and-export cycle on SuperDocs — and, when they do +not, says *which stage* lost them. + +Built for the SuperDocs Round 2 task (assigned build: table fidelity +benchmark, band S2, surfaces: API + export). + +> Built by Siddharth Muneshwar for the SuperDocs task. + +## Run it + +```bash +git clone && cd table-fidelity-benchmark +make install +make dry # full run, entirely offline, spends nothing +``` + +That is the whole setup. `make dry` generates the corpus, runs every case +through an offline round-trip, and writes `results/report.md`. No API key, no +network, no operations. + +Against the live API: + +```bash +export SUPERDOCS_API_KEY=... # leading space keeps it out of shell history +make live # 20 operations, capped at 25 +``` + +The key is read from the environment only. It is never a command-line +argument, never logged, never written to a result file, and a test asserts +that no key-shaped string reaches any output. + +## What it measures + +Twenty cases across four declared axes: + +| axis | cases | examples | +|---|---|---| +| merged cells | 7 | L-shaped merges, full-width dividers, a nested table inside a merged cell | +| nested tables | 3 | one level, two levels, two nested tables side by side | +| spanning headers | 4 | two- and three-tier headers, `w:tblHeader` repeat flags, rotated header cells | +| numeric formatting | 6 | `(12,400.00)`, `1.230`, `0007`, `3.2×10⁻⁶`, `$1,204,880.00`, mixed date formats | + +Cases are data, in `tablebench/corpus/cases.py`. Adding a pathology is a new +entry there and touches no other file. + +## Method + +Each case runs twice. + +``` +control upload → export 0 operations +edit upload → chat → approve → export 1 operation +``` + +The control is the point. Any difference the control also shows is a property +of ingest and export; only what the edit run adds on top is caused by the +edit. Without that separation a benchmark can report that a table broke but +not where, which is not actionable for anyone. + +Each case names the cells its instruction legitimately changes — usually one, +sometimes two where the instruction asks for two figures. Those are exempt from +content comparison. Every other cell, every span, every shading value, every +column width and all the prose around the table are invariants — if they move, +that is a finding. A run that changes nothing scores no better than one that +mangles the document: the report carries both `edit_applied` and the findings, +and a case passes only when the edit landed *and* nothing else moved. + +Comparison happens between two canonical grid models, never between two files +directly. Both the DOCX extractor and the HTML extractor produce that model, +so the HTML intermediate is compared as its own stage. + +Spans are resolved from raw `gridSpan` / `vMerge` rather than through +python-docx's table API, which repeats merged cells instead of reporting +spans — precisely the thing under test. + +Borders are compared as **drawn edges, not cell declarations**. The same rule +can be declared on the table element or on each cell, and as one cell's bottom +or the next cell's top; all three render identically. Comparing declarations +reports a re-expression as damage. On the first live run that difference was +35 false findings against 3 real ones — the noise was louder than the signal. +The comparator now folds table-level borders into each cell by position, then +compares the resolved edge grid. + +## Budget + +**Declared cap for a full live run: 25 operations.** A full run is 20 cases at +one chat turn each; the remaining 5 are headroom for a retry. Uploads, +approvals and exports are free per the published pricing, and the cost table +lives in one place (`OP_COST` in `client.py`). + +The cap is enforced *before* each spend, not after — the run stops with the +budget intact rather than discovering it overspent. Small-sample mode is the +default way to work: + +```bash +make sample # first 3 cases +python -m tablebench.run --mode live --axis numeric_formatting --max-ops 6 +python -m tablebench.run --mode live --cases merge_l_shaped --max-ops 1 +``` + +State is written after every case, atomically. Every record carries the mode +it was produced in, and `--resume` only skips a case already recorded **in the +same mode** — a dry-run record is a simulation and is not evidence about the +live API, so it never stands in for one. Simulated operations do not count +against a live budget either. If a results directory ever ends up holding +both, the report says `MIXED` at the top and refuses to present them as one +run. + +`--resume` skips work already done, so a crash or a kill costs nothing already +paid for: + +```bash +python -m tablebench.run --mode live --max-ops 25 --out results --resume +``` + +Actual spend is printed at the end of every run and reported at the top of +`results/report.md` alongside the results. + +## Tests + +```bash +python -m pytest tests/ -q # 26 tests, no API key, no network, no operations +``` + +The tests do not assert that the fake returns what it was handed. The offline +round-trip is verified lossless first — 0 findings across all 20 cases — and +then each fault with known ground truth (`flatten_merges`, `drop_nested`, +`strip_trailing_zero`, `drop_shading`, `rewrite_body`) is injected and the +comparator is checked for detecting it in every case that contains the +pathology and in none that does not. If the control ever drifts, every +detection result becomes unattributable, so that test is the load-bearing one. + +You can watch a fault land: + +```bash +python -m tablebench.run --mode dry --fault flatten_merges --axis merged_cells +``` + +## Capturing evidence for a bug report + +``` +python -m tablebench.capture --case merge_l_shaped --out evidence +python -m tablebench.capture --case header_repeat_flag --out evidence --control-only +``` + +Writes the source file, the HTML the upload endpoint returned, the exported +file, and a stage-by-stage list of every difference. `--control-only` sends no +edit and spends no operations, which is what shows a loss belongs to the +round-trip rather than the model. + +The ingest HTML is the useful artifact: for the merge bugs the `rowspan` +attribute is already absent there, before any edit is requested, which locates +the fault in the parser rather than leaving a maintainer to find it. + +## Assumptions logged while building + +- **One chat turn is one operation; upload, approve and export are free.** + From the published pricing. If it changes, `OP_COST` is the single edit. +- **The comparison of record is the exported DOCX**, not the HTML + intermediate. Attributes plain HTML cannot express — `tblHeader`, column + widths, `textDirection`, per-cell borders — are withheld at the HTML stage + and listed as *not measurable* rather than counted as losses. +- **Approve-all, not approve-selectively.** The card is about fidelity, not + about the review UI, so every proposed change is approved. Partial approval + is a natural extension and is not built. +- **Whitespace is normalised, unicode is not.** An em dash becoming a hyphen, + or a non-breaking space becoming a space, is a finding. Trailing spaces are + not. +- **A long silence is a timeout, not a failure.** The docs note that large + documents can run for minutes with no visible progress, so the client polls + to a deadline and reports a timeout as exactly that. + +## What this does not measure + +- **Rendering.** Two files that parse identically could still paginate + differently. Out of scope, and stated rather than implied. +- **Whether the edit was *correct*** — only whether it landed in the target + cell and nothing else moved. Judging the model's wording is a different + benchmark. +- **Two false-positive bugs found by capturing real evidence.** Alignment was + read only from the cell, while the API puts it on the paragraph inside, so + centred headings were reported as lost when they had survived correctly. And + `header_rows` was withheld at the HTML stage as unrepresentable, when the API + in fact returns a real `` -- withholding it hid the useful finding that + the header marking survives ingest and is lost on export. Both were reporting + against output that was right. +- **A third confusion between "unmeasurable" and "fine".** The capture summary + reported `text_direction` as surviving ingest when the HTML stage had merely + withheld it -- plain HTML has no property for rotation, so nothing was + measured either way. It now reports withheld codes separately and says they + locate nothing. The same mistake in different clothes each time: silence + read as a pass. +- **The fault injector is coarse.** `drop_nested` is a regex and surfaces as + `table_count` rather than `nested_table_count`. The fault is detected in + every nesting case, but the code it reports is less precise than the + comparator is capable of. Real API output is parsed properly; this affects + the test harness only. +- **Corpus realism.** These are synthetic fixtures built to be pathological. + They cover the constructs real documents contain, not the mess real + documents are. + +All figures, companies and part numbers in the corpus are fabricated. + +--- + +Built by Siddharth for the SuperDocs Round 2 task. diff --git a/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/docs/fixture-merge-l-shaped.jpg b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/docs/fixture-merge-l-shaped.jpg new file mode 100644 index 00000000..ace5ffd8 Binary files /dev/null and b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/docs/fixture-merge-l-shaped.jpg differ diff --git a/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/requirements.txt b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/requirements.txt new file mode 100644 index 00000000..58870be0 --- /dev/null +++ b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/requirements.txt @@ -0,0 +1,4 @@ +lxml>=5.0 +beautifulsoup4>=4.12 +requests>=2.31 +pytest>=8.0 diff --git a/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/__init__.py b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/capture.py b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/capture.py new file mode 100644 index 00000000..d7a0727c --- /dev/null +++ b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/capture.py @@ -0,0 +1,201 @@ +"""Capture evidence for a bug report. + +Runs one case against the live API and writes everything a maintainer needs to +reproduce and locate the fault, without them having to install this benchmark: + + evidence//1-source.docx what was uploaded + evidence//2-ingest.html the HTML your API returned <-- the proof + evidence//3-ingest-table.html just the table, pretty-printed + evidence//4-export.docx what came back + evidence//5-findings.txt what differs, and at which stage + evidence//6-summary.md paste-ready bug report body + +Why the HTML matters. For the merge bugs, the loss is visible in the response +from upload -- before any edit is requested. That locates the fault in the +ingest parser rather than the model or the export, which is the difference +between a report someone can act on and one they have to investigate first. + + python -m tablebench.capture --case merge_l_shaped --out evidence + python -m tablebench.capture --case header_repeat_flag --out evidence + +Costs one operation per case (the control run costs none, and --control-only +spends nothing at all). +""" + +from __future__ import annotations + +import argparse +import os +import re +import shutil +import sys + +from . import compare, extract +from .client import ApiError, Ledger, SuperDocsClient +from .corpus import cases as case_mod +from .corpus.generate import write_docx + + +def pretty_table(html: str) -> str: + """Isolate the first table and indent it so a reader can see the spans.""" + m = re.search(r"", html, re.S | re.I) + if not m: + return "(no element in the response)" + frag = m.group(0) + frag = re.sub(r"><", ">\n<", frag) + out, depth = [], 0 + for line in frag.split("\n"): + if re.match(r""): + depth += 1 + return "\n".join(out) + + +def span_summary(html: str) -> str: + """Count the spans the API's HTML actually carries.""" + rowspans = re.findall(r'rowspan="(\d+)"', html) + colspans = re.findall(r'colspan="(\d+)"', html) + return (f"rowspan attributes present: {len(rowspans)} {rowspans}\n" + f"colspan attributes present: {len(colspans)} {colspans}") + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser(prog="tablebench.capture") + ap.add_argument("--case", required=True, choices=sorted(case_mod.BY_ID)) + ap.add_argument("--out", default="evidence") + ap.add_argument("--control-only", action="store_true", + help="upload and export with no edit; spends no operations") + ap.add_argument("--max-ops", type=int, default=2) + a = ap.parse_args(argv) + + case = case_mod.BY_ID[a.case] + outdir = os.path.join(a.out, a.case) + os.makedirs(outdir, exist_ok=True) + + src = write_docx(case, os.path.join(outdir, "1-source.docx")) + before = extract.from_docx(src) + + ledger = Ledger(cap=a.max_ops) + try: + client = SuperDocsClient(ledger) + except ApiError as exc: + print(exc, file=sys.stderr) + return 2 + + session = f"evidence-{a.case}" + print(f"uploading {os.path.basename(src)} ...") + html = client.upload(src, session) + + with open(os.path.join(outdir, "2-ingest.html"), "w", encoding="utf-8") as fh: + fh.write(html) + with open(os.path.join(outdir, "3-ingest-table.html"), "w", + encoding="utf-8") as fh: + fh.write(pretty_table(html)) + + ingest_doc = extract.from_html(html) + ingest_findings = compare.compare(before, ingest_doc, profile="html") + + if a.control_only: + out_path = client.export(session, os.path.join(outdir, "4-export.docx")) + else: + print(f"sending one edit: {case.edit[:70]}...") + job = client.edit(session, html, case.edit) + if job.get("pending"): + client.approve(session, job["job_id"], job["pending"]) + out_path = client.export(session, os.path.join(outdir, "4-export.docx")) + + after = extract.from_docx(out_path) + exempt = [case.target, *getattr(case, "also_changes", [])] + export_findings = compare.compare(before, after, target=exempt) + + lines = [ + f"CASE: {case.id} - {case.title}", + f"AXIS: {case.axis}", + "", + "=== SOURCE (what was uploaded) ===", + _grid(before.tables[0]) if before.tables else "(no table)", + "", + "=== AFTER INGEST (parsed from the HTML your API returned) ===", + _grid(ingest_doc.tables[0]) if ingest_doc.tables else "(no table)", + "", + span_summary(html), + "", + f"--- differences at the INGEST stage: {len(ingest_findings)} ---", + ] + lines += [f" {f}" for f in ingest_findings] or [" (none)"] + lines += ["", "=== AFTER EXPORT ===", + _grid(after.tables[0]) if after.tables else "(no table)", "", + f"--- differences in the EXPORTED file: {len(export_findings)} ---"] + lines += [f" {f}" for f in export_findings] or [" (none)"] + lines += ["", f"operations spent: {ledger.spent}"] + + report = "\n".join(lines) + with open(os.path.join(outdir, "5-findings.txt"), "w", encoding="utf-8") as fh: + fh.write(report) + print("\n" + report) + + # Attribute per finding, not per case. A single label was wrong on the + # header-flag case: one unrelated ingest difference made the whole report + # say "ingest" when the header marking actually survived ingest and was + # lost on export -- the opposite of what the report needed to say. + ingest_codes = {f.code for f in ingest_findings} + export_codes = {f.code for f in export_findings} + # Some attributes have no representation in plain HTML, so the HTML stage + # withholds them. Withheld is not the same as intact: counting them as + # "survived ingest" would assert something the evidence cannot support, + # which is the exact failure this tool exists to avoid. + withheld = sorted(export_codes & compare.HTML_UNREPRESENTABLE) + lost_at_ingest = sorted(ingest_codes & export_codes) + lost_at_export = sorted(export_codes - ingest_codes - set(withheld)) + + if lost_at_ingest: + stage = ("ingest -- present in the uploaded file, absent from the HTML " + "your API returned, before any edit was requested") + elif lost_at_export: + stage = ("export -- the HTML your API returned carries it correctly; " + "it is absent from the exported DOCX") + elif withheld: + stage = ("cannot be localised -- plain HTML has no property for " + "this, so the HTML stage cannot measure it either way") + else: + stage = "no difference found" + with open(os.path.join(outdir, "6-summary.md"), "w", encoding="utf-8") as fh: + fh.write( + f"# {case.title}\n\n" + f"**Where the loss happens:** {stage}\n\n" + f"**What I did.** Uploaded `1-source.docx`" + + ("" if a.control_only else f" and sent one instruction: " + f"\"{case.edit}\"") + + ", then exported.\n\n" + f"**Files attached.** `1-source.docx` reproduces it. " + f"`2-ingest.html` is the response from upload -- the loss is " + f"visible there, before any edit. `4-export.docx` is what came " + f"back. `5-findings.txt` lists every difference and the stage it " + f"appeared at.\n\n" + f"**Ingest differences:** {len(ingest_findings)} \n" + f"**Export differences:** {len(export_findings)}\n\n" + + (f"**Lost at ingest:** {', '.join(lost_at_ingest)}\n\n" + if lost_at_ingest else "") + + (f"**Survives ingest, lost on export:** " + f"{', '.join(lost_at_export)}\n\n" if lost_at_export else "") + + (f"**Not measurable at the HTML stage** (no standard HTML " + f"property, so this says nothing about where it was lost): " + f"{', '.join(withheld)}\n" if withheld else "")) + print(f"\nwritten to {outdir}/") + return 0 + + +def _grid(t) -> str: + rows = [] + for c in sorted(t.cells, key=lambda c: (c.row, c.col)): + flag = "" + if c.row_span > 1 or c.col_span > 1: + flag = f" <-- rowspan={c.row_span} colspan={c.col_span}" + rows.append(f" ({c.row},{c.col}) {c.text[:26]!r}{flag}") + return "\n".join(rows) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/client.py b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/client.py new file mode 100644 index 00000000..8c1af119 --- /dev/null +++ b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/client.py @@ -0,0 +1,313 @@ +"""The four calls: upload, chat, approve, export. + +The API key is read from the SUPERDOCS_API_KEY environment variable and is +never accepted as an argument, never logged, and never written to a result +file. Nothing in this repository should ever hold a credential. + +FakeClient is the offline counterpart. It is not here to make tests pass -- +it is here so the comparator can be tested against faults whose ground truth +is known: flatten a merge, drop a colspan, strip a trailing zero, then assert +the comparator reports exactly that and nothing else. +""" + +from __future__ import annotations + +import json +import os +import re +import time +from dataclasses import dataclass, field + +BASE = os.environ.get("SUPERDOCS_BASE", "https://api.superdocs.app") + +# One chat turn is one operation. Uploads, exports and downloads are free per +# the published pricing; if that changes, change it here and nowhere else. +OP_COST = {"upload": 0, "chat": 1, "approve": 0, "export": 0} + + +class ApiError(RuntimeError): + pass + + +class BudgetExceeded(RuntimeError): + pass + + +@dataclass +class Ledger: + """Operation accounting with a hard cap checked before every spend.""" + + cap: int + spent: int = 0 + by_stage: dict[str, int] = field(default_factory=dict) + timings: dict[str, float] = field(default_factory=dict) + + def charge(self, stage: str) -> None: + cost = OP_COST.get(stage, 0) + if cost and self.spent + cost > self.cap: + raise BudgetExceeded( + f"stage {stage!r} would spend {cost} op, taking the run to " + f"{self.spent + cost} against a cap of {self.cap}. Stopping " + f"before the spend, not after." + ) + self.spent += cost + self.by_stage[stage] = self.by_stage.get(stage, 0) + cost + + def record_time(self, stage: str, seconds: float) -> None: + self.timings[stage] = self.timings.get(stage, 0.0) + seconds + + +class SuperDocsClient: + """Live client. Requires SUPERDOCS_API_KEY in the environment.""" + + def __init__(self, ledger: Ledger, poll_interval: float = 3.0, + timeout_s: float = 600.0): + key = os.environ.get("SUPERDOCS_API_KEY") + if not key: + raise ApiError( + "SUPERDOCS_API_KEY is not set. Export it in your shell " + "(and keep it out of shell history with a leading space); " + "the benchmark never takes a key as an argument." + ) + self._key = key + self.ledger = ledger + self.poll_interval = poll_interval + self.timeout_s = timeout_s + + # -- internals --------------------------------------------------------- + @property + def _json_headers(self) -> dict: + return {"Authorization": f"Bearer {self._key}", + "Content-Type": "application/json"} + + @property + def _auth_only(self) -> dict: + return {"Authorization": f"Bearer {self._key}"} + + def _redact(self, text: str) -> str: + return re.sub(r"sk_[A-Za-z0-9_\-]{4,}", "sk_[REDACTED]", text or "") + + def _check(self, resp, what: str): + if resp.status_code >= 400: + raise ApiError( + f"{what} failed with HTTP {resp.status_code}: " + f"{self._redact(resp.text)[:400]}" + ) + return resp + + # -- the four calls ---------------------------------------------------- + def upload(self, path: str, session_id: str) -> str: + import requests + t0 = time.monotonic() + self.ledger.charge("upload") + with open(path, "rb") as fh: + r = requests.post( + f"{BASE}/v1/documents/upload", + headers=self._auth_only, + files={"file": (os.path.basename(path), fh, + "application/vnd.openxmlformats-officedocument." + "wordprocessingml.document")}, + data={"session_id": session_id}, + timeout=self.timeout_s, + ) + self._check(r, "upload") + self.ledger.record_time("upload", time.monotonic() - t0) + html = r.json().get("html") + if not html: + raise ApiError("upload returned no html; cannot proceed honestly") + return html + + def edit(self, session_id: str, document_html: str, instruction: str) -> dict: + """Start an approval-gated edit and poll to the approval point.""" + import requests + t0 = time.monotonic() + self.ledger.charge("chat") + r = requests.post( + f"{BASE}/v1/chat/async", headers=self._json_headers, + json={ + "message": instruction, + "session_id": session_id, + "document_html": document_html, + "approval_mode": "ask_every_time", + }, + timeout=self.timeout_s, + ) + self._check(r, "chat/async") + job_id = r.json()["job_id"] + + deadline = time.monotonic() + self.timeout_s + while time.monotonic() < deadline: + j = requests.get(f"{BASE}/v1/jobs/{job_id}", + headers=self._json_headers, + timeout=self.timeout_s) + self._check(j, "jobs/get") + job = j.json() + status = job.get("status") + if status == "awaiting_approval": + self.ledger.record_time("chat", time.monotonic() - t0) + return {"job_id": job_id, + "pending": self._pending(job), + "status": status} + if status == "completed": + self.ledger.record_time("chat", time.monotonic() - t0) + return {"job_id": job_id, "pending": [], "status": status, + "result": job.get("result")} + if status in ("failed", "cancelled"): + raise ApiError( + f"job {status}: {self._redact(str(job.get('error')))[:300]}" + ) + time.sleep(self.poll_interval) + raise ApiError( + f"job {job_id} did not reach a terminal state within " + f"{self.timeout_s:.0f}s. Long silence is normal on this API, so " + f"this is a timeout, not a proven failure." + ) + + @staticmethod + def _pending(job: dict) -> list[dict]: + """Proposed changes need a second parse. + + The documented gotcha: proposed-change content arrives as a + JSON-encoded string while the final result is already an object. + Missing this is what makes every diff field read as undefined. + """ + changes = (job.get("metadata") or {}).get("pending_changes") or [] + out = [] + for c in changes: + c = dict(c) + content = c.get("content") + if isinstance(content, str): + try: + c["content"] = json.loads(content) + except (ValueError, TypeError): + c["content_parse_failed"] = True + out.append(c) + return out + + def approve(self, session_id: str, job_id: str, changes: list[dict]) -> dict: + import requests + t0 = time.monotonic() + self.ledger.charge("approve") + r = requests.post( + f"{BASE}/v1/chat/{session_id}/approve", + headers=self._json_headers, + json={ + "job_id": job_id, + "approved": True, + "changes": [{"change_id": c.get("change_id"), "approved": True} + for c in changes], + }, + timeout=self.timeout_s, + ) + self._check(r, "approve") + self.ledger.record_time("approve", time.monotonic() - t0) + return r.json() + + def export(self, session_id: str, out_path: str) -> str: + import requests + t0 = time.monotonic() + self.ledger.charge("export") + r = requests.post( + f"{BASE}/v1/documents/export", headers=self._json_headers, + json={"session_id": session_id, "format": "docx"}, + timeout=self.timeout_s, + ) + self._check(r, "export") + os.makedirs(os.path.dirname(out_path) or ".", exist_ok=True) + with open(out_path, "wb") as fh: + fh.write(r.content) + self.ledger.record_time("export", time.monotonic() - t0) + return out_path + + +# -------------------------------------------------------------------------- +# Offline fake +# -------------------------------------------------------------------------- + +FAULTS = ( + "none", + "flatten_merges", # drop rowspan/colspan attributes + "drop_nested", # remove nested tables + "strip_trailing_zero", # 12.480 -> 12.48 + "drop_shading", # remove background colours + "rewrite_body", # touch a paragraph nobody asked about +) + + +class FakeClient: + """Deterministic offline stand-in with injectable, named faults.""" + + def __init__(self, ledger: Ledger, fault: str = "none"): + if fault not in FAULTS: + raise ValueError(f"unknown fault {fault!r}; known: {FAULTS}") + self.ledger = ledger + self.fault = fault + self._sessions: dict[str, str] = {} + self._sim: dict[str, tuple] = {} + + def simulate_edit(self, session_id: str, case) -> None: + """Simulator affordance, offline only. + + Records the change the live model would be asked to make, so the dry + run exercises the real success path instead of asserting failure on + every case. The live client has no such method and the runner only + calls it when the client offers one. + """ + if case.expect_contains is not None: + self._sim[session_id] = (case.target, case.expect_contains) + + def upload(self, path: str, session_id: str) -> str: + from .extract import from_docx + from .htmlout import to_html + self.ledger.charge("upload") + html = to_html(from_docx(path)) + self._sessions[session_id] = html + return html + + def edit(self, session_id: str, document_html: str, instruction: str) -> dict: + self.ledger.charge("chat") + self._sessions[session_id] = document_html + return {"job_id": f"fake-{session_id}", "status": "awaiting_approval", + "pending": [{"change_id": "c1", "operation": "replace", + "content": json.dumps({"instruction": instruction})}]} + + def approve(self, session_id: str, job_id: str, changes: list[dict]) -> dict: + self.ledger.charge("approve") + return {"status": "applied", "applied": len(changes)} + + def export(self, session_id: str, out_path: str) -> str: + from .extract import from_html + from .corpus.generate import write_docx + from .htmlout import doc_to_case + self.ledger.charge("export") + doc = from_html(self._damage(self._sessions.get(session_id, ""))) + sim = self._sim.get(session_id) + if sim: + (ti, r, c), text = sim + if ti < len(doc.tables): + cell = doc.tables[ti].cell_at(r, c) + if cell is not None: + if cell.nested: + cell.nested[0].cells[-1].paragraphs = [text] + else: + cell.paragraphs = [text] + write_docx(doc_to_case(doc), out_path) + return out_path + + def _damage(self, html: str) -> str: + f = self.fault + if f == "none": + return html + if f == "flatten_merges": + return re.sub(r'\s(rowspan|colspan)="\d+"', "", html) + if f == "drop_nested": + return re.sub(r"]*>(?:(?!
).)*?\s*(?=)", + "", html, flags=re.S) + if f == "strip_trailing_zero": + return re.sub(r"(\d+\.\d*?)0+(?=\D|$)", r"\1", html) + if f == "drop_shading": + return re.sub(r"background-color:[^;\"]*;?", "", html) + if f == "rewrite_body": + return html.replace("Prepared for benchmark purposes.", + "Prepared automatically.") + return html diff --git a/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/compare.py b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/compare.py new file mode 100644 index 00000000..28c98b7b --- /dev/null +++ b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/compare.py @@ -0,0 +1,243 @@ +"""Compare two canonical Documents and report exactly what differs. + +Two rules govern this module. + +1. The edit target is exempt from content comparison and nothing else is. A + round-trip that changes a cell nobody asked it to change is a finding even + if the document still looks fine. + +2. A finding is never asserted where it cannot be observed. If a stage did not + run, the case is UNVERIFIABLE, not a failure. "Broken" and "not measured" + are different words here and the report keeps them apart. +""" + +from __future__ import annotations + +from dataclasses import dataclass, asdict + +from .model import Cell, Document, Table + +STRUCTURAL = "structural" +CONTENT = "content" +FORMATTING = "formatting" + +# Verdicts +PASS = "PASS" +FAIL = "FAIL" +UNVERIFIABLE = "UNVERIFIABLE" +ERROR = "ERROR" + + +@dataclass(frozen=True) +class Finding: + severity: str + code: str + where: str + expected: str + actual: str + + def key(self) -> tuple: + return (self.severity, self.code, self.where) + + def to_dict(self) -> dict: + return asdict(self) + + def __str__(self) -> str: + return f"[{self.severity}/{self.code}] {self.where}: expected {self.expected!r}, got {self.actual!r}" + + +def _norm(v) -> str: + if v is None: + return "" + return str(v).strip() + + +def _norm_text(s: str) -> str: + """Whitespace-insensitive but character-exact otherwise. + + Deliberately does NOT normalise unicode: an em dash turning into a hyphen, + or a non-breaking space into a space, is a finding, not noise. + """ + return " ".join(s.split()) + + +def _cmp_cell(a: Cell, b: Cell, where: str, exempt: bool, + out: list[Finding]) -> None: + if a.row_span != b.row_span: + out.append(Finding(STRUCTURAL, "row_span", where, + str(a.row_span), str(b.row_span))) + if a.col_span != b.col_span: + out.append(Finding(STRUCTURAL, "col_span", where, + str(a.col_span), str(b.col_span))) + + if not exempt: + at, bt = _norm_text(a.text), _norm_text(b.text) + if at != bt: + out.append(Finding(CONTENT, "cell_text", where, at, bt)) + if len(a.paragraphs) != len(b.paragraphs): + out.append(Finding(CONTENT, "paragraph_count", where, + str(len(a.paragraphs)), str(len(b.paragraphs)))) + + for attr, code in ( + ("alignment", "alignment"), + ("shading", "shading"), + ("valign", "valign"), + ("text_direction", "text_direction"), + ): + av, bv = _norm(getattr(a, attr)), _norm(getattr(b, attr)) + if av != bv: + out.append(Finding(FORMATTING, code, where, av, bv)) + + if a.width_dxa is not None and b.width_dxa is not None: + if a.width_dxa != b.width_dxa: + out.append(Finding(FORMATTING, "cell_width", where, + str(a.width_dxa), str(b.width_dxa))) + + if len(a.nested) != len(b.nested): + out.append(Finding(STRUCTURAL, "nested_table_count", where, + str(len(a.nested)), str(len(b.nested)))) + else: + for i, (na, nb) in enumerate(zip(a.nested, b.nested)): + _cmp_table(na, nb, f"{where}>nested[{i}]", exempt, out) # noqa + + +def _cmp_table(a: Table, b: Table, where: str, exempt_all: bool, + out: list[Finding], + targets: list[tuple[int, int]] | None = None) -> None: + if a.n_rows != b.n_rows: + out.append(Finding(STRUCTURAL, "row_count", where, + str(a.n_rows), str(b.n_rows))) + if a.n_cols != b.n_cols: + out.append(Finding(STRUCTURAL, "col_count", where, + str(a.n_cols), str(b.n_cols))) + + occ_a, occ_b = a.occupancy(), b.occupancy() + if occ_a != occ_b: + diff = sorted(set(occ_a) ^ set(occ_b))[:4] + mism = [k for k in sorted(set(occ_a) & set(occ_b)) if occ_a[k] != occ_b[k]][:4] + out.append(Finding( + STRUCTURAL, "grid_topology", where, + f"{len(occ_a)} covered cells", + f"{len(occ_b)} covered cells; missing/extra={diff}; remapped={mism}", + )) + + if a.header_rows != b.header_rows: + out.append(Finding(FORMATTING, "header_rows", where, + str(a.header_rows), str(b.header_rows))) + if a.col_widths and b.col_widths and a.col_widths != b.col_widths: + out.append(Finding(FORMATTING, "col_widths", where, + str(a.col_widths), str(b.col_widths))) + if _norm(a.layout) != _norm(b.layout): + out.append(Finding(FORMATTING, "table_layout", where, + _norm(a.layout), _norm(b.layout))) + + ea, eb = a.border_edges(), b.border_edges() + for key in sorted(set(ea) | set(eb)): + av, bv = _norm(ea.get(key)), _norm(eb.get(key)) + if av != bv: + kind = "row rule above" if key[0] == "h" else "column rule left of" + out.append(Finding(FORMATTING, "border_edge", + f"{where} {kind} ({key[1]}, {key[2]})", av, bv)) + + anchors_a, anchors_b = a.anchors(), b.anchors() + for pos in sorted(set(anchors_a) | set(anchors_b)): + ca, cb = anchors_a.get(pos), anchors_b.get(pos) + cw = f"{where} cell{pos}" + if ca is None: + out.append(Finding(STRUCTURAL, "unexpected_cell", cw, "-", + _norm_text(cb.text)[:60])) + continue + if cb is None: + out.append(Finding(STRUCTURAL, "missing_cell", cw, + _norm_text(ca.text)[:60], "-")) + continue + exempt = exempt_all + for t in (targets or []): + anchor = a.cell_at(*t) + if anchor is not None and (anchor.row, anchor.col) == pos: + exempt = True + break + _cmp_cell(ca, cb, cw, exempt, out) + + +# Attributes that plain HTML has no standard slot for. When the stage under +# comparison is the HTML intermediate, their absence is unmeasurable, not +# proven lost -- so they are withheld from findings and reported separately. +HTML_UNREPRESENTABLE = frozenset({ + "col_widths", "cell_width", "table_layout", + "text_direction", "border_edge", +}) +# header_rows was in this set until the API's response turned out to carry a +# real . It is representable, so withholding it hid the useful fact: +# the header marking survives ingest and is lost on export. + + +def compare(before: Document, after: Document, + target=None, profile: str = "docx") -> list[Finding]: + """target is (table_index, row, col), or a list of them, exempt from + content checks. Every cell not named is an invariant. + + profile='html' withholds findings for attributes HTML cannot express. + """ + out: list[Finding] = [] + if target is None: + targets: list[tuple[int, int, int]] = [] + elif isinstance(target, tuple) and target and isinstance(target[0], int): + targets = [target] + else: + targets = list(target) + + if len(before.tables) != len(after.tables): + out.append(Finding(STRUCTURAL, "table_count", "document", + str(len(before.tables)), str(len(after.tables)))) + + for i, (ta, tb) in enumerate(zip(before.tables, after.tables)): + tgts = [(t[1], t[2]) for t in targets if t[0] == i] + _cmp_table(ta, tb, f"table[{i}]", False, out, tgts) + + a_paras = [_norm_text(p) for p in before.body_paragraphs] + b_paras = [_norm_text(p) for p in after.body_paragraphs] + if a_paras != b_paras: + missing = [p for p in a_paras if p not in b_paras][:3] + out.append(Finding(CONTENT, "body_paragraphs", "document", + f"{len(a_paras)} paragraphs", + f"{len(b_paras)} paragraphs; missing={missing}")) + + if profile == "html": + out = [f for f in out if f.code not in HTML_UNREPRESENTABLE] + return out + + +def unmeasurable_at_html_stage(findings: list[Finding]) -> list[str]: + """Codes withheld under the html profile, for honest reporting.""" + return sorted({f.code for f in findings if f.code in HTML_UNREPRESENTABLE}) + + +def attribute(edit_findings: list[Finding], + control_findings: list[Finding]) -> tuple[list[Finding], list[Finding]]: + """Split edit-run findings into (caused_by_edit, caused_by_round_trip). + + A finding that also occurs on a no-op control run is a property of ingest + and export, not of the edit. Separating these is the difference between + 'the editor broke my table' and 'the file format round-trip did'. + """ + control_keys = {f.key() for f in control_findings} + by_edit = [f for f in edit_findings if f.key() not in control_keys] + by_round_trip = [f for f in edit_findings if f.key() in control_keys] + return by_edit, by_round_trip + + +def verdict(edit_applied: bool | None, findings: list[Finding]) -> str: + if edit_applied is None: + return UNVERIFIABLE + if not edit_applied: + return FAIL + return PASS if not findings else FAIL + + +def summarise(findings: list[Finding]) -> dict[str, int]: + counts: dict[str, int] = {} + for f in findings: + counts[f.severity] = counts.get(f.severity, 0) + 1 + counts[f"{f.severity}:{f.code}"] = counts.get(f"{f.severity}:{f.code}", 0) + 1 + return counts diff --git a/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/corpus/__init__.py b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/corpus/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/corpus/cases.py b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/corpus/cases.py new file mode 100644 index 00000000..6012b82f --- /dev/null +++ b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/corpus/cases.py @@ -0,0 +1,556 @@ +"""The pathology catalogue. + +Cases are data, not code. Adding a pathology is a new entry here; it needs no +change to the generator, the extractor, the comparator or the runner. + +Each case names one cell as the edit target. Everything else in the document +is an invariant: after the edit round-trip it must come back unchanged. That +asymmetry is the whole measurement -- a system that mangles nothing but also +changes nothing is not passing, and neither is one that makes the edit and +flattens the table around it. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field + +AXES = ("merged_cells", "nested_tables", "spanning_headers", "numeric_formatting") + +HDR = "D9E2F3" +ALT = "F2F2F2" + + +@dataclass +class Case: + id: str + title: str + axis: str + edit: str + target: tuple[int, int, int] # (table index, row, col) in the grid + also_changes: list[tuple[int, int, int]] = field(default_factory=list) + """Other cells the instruction legitimately asks to change. + + An instruction that names two figures must exempt both. Declaring only one + makes the benchmark report the model for obeying it, which is a false + finding and worse than no finding at all.""" + expect_contains: str | None = None # proof the edit actually landed + intro: str = "" + tables: list[dict] = field(default_factory=list) + trailing_paragraphs: list[str] = field(default_factory=list) + notes: str = "" + + +def _row(*cells) -> list[dict]: + out = [] + for c in cells: + out.append({"text": c} if isinstance(c, str) else c) + return out + + +CASES: list[Case] = [ + + # ---------------- merged cells ---------------- + Case( + id="merge_horizontal_header", + title="Horizontal merge across a header", + axis="merged_cells", + notes="A single header cell spanning three data columns.", + tables=[{ + "header_rows": [0], + "rows": [ + _row({"text": "Region", "rowspan": 2, "fill": HDR, "bold": True}, + {"text": "Quarterly revenue", "colspan": 3, "fill": HDR, "bold": True, + "align": "center"}), + _row({"text": "Q1", "fill": HDR, "bold": True}, + {"text": "Q2", "fill": HDR, "bold": True}, + {"text": "Q3", "fill": HDR, "bold": True}), + _row("North", "1,204.50", "1,388.00", "1,401.25"), + _row("South", "980.00", "1,012.75", "1,150.00"), + ], + }], + edit="In the first table, change the Q2 value on the South row to 1,099.40. " + "Change nothing else.", + target=(0, 3, 2), + expect_contains="1,099.40", + ), + + Case( + id="merge_vertical_label", + title="Vertical merge on a stub column", + axis="merged_cells", + notes="A row label merged down three body rows.", + tables=[{ + "rows": [ + _row({"text": "Division", "fill": HDR, "bold": True}, + {"text": "Line item", "fill": HDR, "bold": True}, + {"text": "Amount", "fill": HDR, "bold": True, "align": "right"}), + _row({"text": "Upstream", "rowspan": 3, "valign": "center"}, + "Drilling", {"text": "4,210.00", "align": "right"}), + _row("Completion", {"text": "1,875.50", "align": "right"}), + _row("Logistics", {"text": "962.25", "align": "right"}), + _row({"text": "Downstream", "rowspan": 2, "valign": "center"}, + "Refining", {"text": "7,430.00", "align": "right"}), + _row("Distribution", {"text": "2,118.75", "align": "right"}), + ], + }], + edit="Change the Logistics amount to 1,004.00. Leave every other cell alone.", + target=(0, 3, 2), + expect_contains="1,004.00", + ), + + Case( + id="merge_l_shaped", + title="L-shaped merge: row span and column span in one table", + axis="merged_cells", + notes="Vertical and horizontal merges interlocking; the classic case " + "that collapses to a flat grid when a converter resolves spans naively.", + tables=[{ + "rows": [ + _row({"text": "Asset", "rowspan": 2, "fill": HDR, "bold": True}, + {"text": "Planned", "colspan": 2, "fill": HDR, "bold": True, + "align": "center"}, + {"text": "Actual", "rowspan": 2, "fill": HDR, "bold": True}), + _row({"text": "Start", "fill": HDR}, {"text": "Finish", "fill": HDR}), + _row("Platform A", "2026-01-04", "2026-02-18", "2026-02-25"), + _row({"text": "Shutdown window", "colspan": 4, "fill": ALT, + "align": "center", "bold": True}), + _row("Platform B", "2026-03-01", "2026-04-12", "2026-04-09"), + ], + }], + edit="Change the Actual date for Platform A to 2026-03-02. Change nothing else.", + target=(0, 2, 3), + expect_contains="2026-03-02", + ), + + Case( + id="merge_full_row_divider", + title="Full-width merged row used as a section divider", + axis="merged_cells", + tables=[{ + "rows": [ + _row({"text": "Item", "fill": HDR, "bold": True}, + {"text": "Qty", "fill": HDR, "bold": True}, + {"text": "Unit", "fill": HDR, "bold": True}, + {"text": "Total", "fill": HDR, "bold": True}), + _row({"text": "Section 1 - Mechanical", "colspan": 4, "fill": ALT, + "bold": True}), + _row("Gasket set", "12", "18.40", "220.80"), + _row({"text": "Section 2 - Electrical", "colspan": 4, "fill": ALT, + "bold": True}), + _row("Cable tray", "40", "31.00", "1,240.00"), + ], + }], + edit="Change the quantity of Cable tray to 46 and its total to 1,426.00.", + target=(0, 4, 1), + also_changes=[(0, 4, 3)], + expect_contains="46", + ), + + Case( + id="merge_containing_nested", + title="Nested table inside a merged cell", + axis="merged_cells", + notes="Merge and nesting combined -- each is survivable alone; together " + "they are where most converters give up.", + tables=[{ + "col_widths": [1800, 2600], + "rows": [ + _row({"text": "Package", "fill": HDR, "bold": True}, + {"text": "Breakdown", "fill": HDR, "bold": True}), + _row({"text": "Turnaround 2026", "rowspan": 2, "valign": "center"}, + {"text": "", "nested": { + "col_widths": [1300, 1100], + "rows": [ + _row({"text": "Phase", "bold": True}, + {"text": "Days", "bold": True}), + _row("Prep", "9"), + _row("Execution", "21"), + ], + }}), + _row("Contingency 4 days"), + ], + }], + edit="In the nested breakdown table, change Execution from 21 to 24 days.", + target=(0, 1, 1), + expect_contains="24", + ), + + # ---------------- nested tables ---------------- + Case( + id="nested_simple", + title="Table inside a cell", + axis="nested_tables", + tables=[{ + "col_widths": [2000, 2600], + "rows": [ + _row({"text": "Vendor", "fill": HDR, "bold": True}, + {"text": "Rates", "fill": HDR, "bold": True}), + _row("Halberd Services", {"text": "", "nested": { + "col_widths": [1400, 1000], + "rows": [ + _row("Day rate", "1,450.00"), + _row("Standby", "610.00"), + ], + }}), + ], + }], + edit="Change the standby rate in the nested table to 645.00.", + target=(0, 1, 1), + expect_contains="645.00", + ), + + Case( + id="nested_two_deep", + title="Two levels of nesting", + axis="nested_tables", + notes="Depth is where chunk-id schemes usually stop addressing cells.", + tables=[{ + "col_widths": [1800, 3400], + "rows": [ + _row({"text": "Contract", "fill": HDR, "bold": True}, + {"text": "Structure", "fill": HDR, "bold": True}), + _row("MSA-2026-11", {"text": "", "nested": { + "col_widths": [1200, 2000], + "rows": [ + _row("Schedule A", {"text": "", "nested": { + "col_widths": [900, 900], + "rows": [ + _row("Tier 1", "0.85"), + _row("Tier 2", "0.72"), + ], + }}), + _row("Schedule B", "See annex"), + ], + }}), + ], + }], + edit="In the innermost table, change the Tier 2 value to 0.70.", + target=(0, 1, 1), + expect_contains="0.70", + ), + + Case( + id="nested_side_by_side", + title="Two nested tables in adjacent cells of one row", + axis="nested_tables", + tables=[{ + "col_widths": [2400, 2400], + "rows": [ + _row({"text": "Before", "fill": HDR, "bold": True}, + {"text": "After", "fill": HDR, "bold": True}), + _row({"text": "", "nested": { + "col_widths": [1100, 1100], + "rows": [_row("Flow", "412"), _row("Pressure", "88.4")], + }}, + {"text": "", "nested": { + "col_widths": [1100, 1100], + "rows": [_row("Flow", "455"), _row("Pressure", "91.0")], + }}), + ], + }], + edit="In the After table, change Pressure to 92.6.", + target=(0, 1, 1), + expect_contains="92.6", + ), + + # ---------------- spanning headers ---------------- + Case( + id="header_two_level", + title="Two-level spanning header", + axis="spanning_headers", + tables=[{ + "header_rows": [0, 1], + "rows": [ + _row({"text": "Well", "rowspan": 2, "fill": HDR, "bold": True}, + {"text": "Production", "colspan": 2, "fill": HDR, "bold": True, + "align": "center"}, + {"text": "Downtime", "colspan": 2, "fill": HDR, "bold": True, + "align": "center"}), + _row({"text": "Oil", "fill": HDR}, {"text": "Gas", "fill": HDR}, + {"text": "Planned", "fill": HDR}, {"text": "Unplanned", "fill": HDR}), + _row("A-14", "1,240", "3,880", "6.0", "1.5"), + _row("A-15", "980", "2,410", "4.5", "0.0"), + ], + }], + edit="Change the unplanned downtime for A-15 to 2.5.", + target=(0, 3, 4), + expect_contains="2.5", + ), + + Case( + id="header_three_level", + title="Three-level spanning header", + axis="spanning_headers", + notes="Three header tiers with spans at two of them.", + tables=[{ + "header_rows": [0, 1, 2], + "col_widths": [1500, 1100, 1100, 1100, 1100], + "rows": [ + _row({"text": "Facility", "rowspan": 3, "fill": HDR, "bold": True}, + {"text": "2026", "colspan": 4, "fill": HDR, "bold": True, + "align": "center"}), + _row({"text": "H1", "colspan": 2, "fill": HDR, "align": "center"}, + {"text": "H2", "colspan": 2, "fill": HDR, "align": "center"}), + _row({"text": "Q1", "fill": HDR}, {"text": "Q2", "fill": HDR}, + {"text": "Q3", "fill": HDR}, {"text": "Q4", "fill": HDR}), + _row("Terminal 1", "18.2", "19.0", "17.4", "20.1"), + _row("Terminal 2", "11.9", "12.3", "12.0", "13.8"), + ], + }], + edit="Change the Q3 figure for Terminal 1 to 18.9.", + target=(0, 3, 3), + expect_contains="18.9", + ), + + Case( + id="header_repeat_flag", + title="Repeating header rows across a long table", + axis="spanning_headers", + notes="w:tblHeader on two rows; the repeat flag is invisible on page one " + "and silently dropped by most round-trips.", + tables=[{ + "header_rows": [0, 1], + "rows": [ + _row({"text": "Tag", "rowspan": 2, "fill": HDR, "bold": True}, + {"text": "Readings", "colspan": 2, "fill": HDR, "bold": True, + "align": "center"}), + _row({"text": "Min", "fill": HDR}, {"text": "Max", "fill": HDR}), + ] + [ + _row(f"PT-{100 + i}", f"{2.0 + i * 0.13:.2f}", f"{9.0 + i * 0.21:.2f}") + for i in range(28) + ], + }], + edit="Change the Max reading on the PT-105 row to 12.00.", + target=(0, 7, 2), + expect_contains="12.00", + ), + + Case( + id="header_rotated_text", + title="Rotated header cells", + axis="spanning_headers", + tables=[{ + "header_rows": [0], + "col_widths": [1800, 700, 700, 700], + "rows": [ + _row({"text": "Component", "fill": HDR, "bold": True}, + {"text": "Inspected", "fill": HDR, "dir": "btLr", "valign": "bottom"}, + {"text": "Cleaned", "fill": HDR, "dir": "btLr", "valign": "bottom"}, + {"text": "Replaced", "fill": HDR, "dir": "btLr", "valign": "bottom"}), + _row("Seal ring", "Y", "Y", "N"), + _row("Impeller", "Y", "N", "N"), + ], + }], + edit="On the Impeller row, change Replaced from N to Y.", + target=(0, 2, 3), + expect_contains="Y", + ), + + # ---------------- numeric formatting ---------------- + Case( + id="num_thousands_currency", + title="Thousands separators and currency symbols", + axis="numeric_formatting", + tables=[{ + "rows": [ + _row({"text": "Account", "fill": HDR, "bold": True}, + {"text": "Opening", "fill": HDR, "bold": True, "align": "right"}, + {"text": "Closing", "fill": HDR, "bold": True, "align": "right"}), + _row("Operating", {"text": "$1,204,880.00", "align": "right"}, + {"text": "$1,190,455.20", "align": "right"}), + _row("Reserve", {"text": "$88,000.00", "align": "right"}, + {"text": "$92,500.00", "align": "right"}), + _row("Escrow", {"text": "$0.00", "align": "right"}, + {"text": "$15,000.00", "align": "right"}), + ], + }], + edit="Change the Reserve closing balance to $94,750.00.", + target=(0, 2, 2), + expect_contains="$94,750.00", + ), + + Case( + id="num_negative_parentheses", + title="Negatives in parentheses and em-dash zeros", + axis="numeric_formatting", + notes="Accounting convention: (1,234.00) means negative, an em dash " + "means nil. A model that normalizes these has changed the meaning.", + tables=[{ + "rows": [ + _row({"text": "Line", "fill": HDR, "bold": True}, + {"text": "Variance", "fill": HDR, "bold": True, "align": "right"}), + _row("Labour", {"text": "(12,400.00)", "align": "right"}), + _row("Materials", {"text": "3,150.00", "align": "right"}), + _row("Freight", {"text": "\u2014", "align": "right"}), + _row("Permits", {"text": "(875.50)", "align": "right"}), + ], + }], + edit="Change the Materials variance to 3,480.00.", + target=(0, 2, 1), + expect_contains="3,480.00", + ), + + Case( + id="num_trailing_zeros", + title="Significant trailing zeros and fixed precision", + axis="numeric_formatting", + notes="1.230 is not 1.23 in an instrument reading.", + tables=[{ + "rows": [ + _row({"text": "Instrument", "fill": HDR, "bold": True}, + {"text": "Reading", "fill": HDR, "bold": True, "align": "right"}, + {"text": "Tolerance", "fill": HDR, "bold": True, "align": "right"}), + _row("FT-201", {"text": "1.230", "align": "right"}, + {"text": "\u00b10.005", "align": "right"}), + _row("FT-202", {"text": "12.500", "align": "right"}, + {"text": "\u00b10.010", "align": "right"}), + _row("FT-203", {"text": "0.900", "align": "right"}, + {"text": "\u00b10.050", "align": "right"}), + ], + }], + edit="Change the FT-202 reading to 12.480.", + target=(0, 2, 1), + expect_contains="12.480", + ), + + Case( + id="num_leading_zero_ids", + title="Leading-zero identifiers that must not become integers", + axis="numeric_formatting", + tables=[{ + "rows": [ + _row({"text": "Part no.", "fill": HDR, "bold": True}, + {"text": "Batch", "fill": HDR, "bold": True}, + {"text": "Qty", "fill": HDR, "bold": True, "align": "right"}), + _row("0041-77", "0007", {"text": "18", "align": "right"}), + _row("0100-02", "0012", {"text": "6", "align": "right"}), + _row("9000-01", "0100", {"text": "44", "align": "right"}), + ], + }], + edit="Change the quantity on the 0100-02 row to 9.", + target=(0, 2, 2), + expect_contains="9", + ), + + Case( + id="num_percent_scientific", + title="Percentages, scientific notation and units", + axis="numeric_formatting", + tables=[{ + "rows": [ + _row({"text": "Metric", "fill": HDR, "bold": True}, + {"text": "Value", "fill": HDR, "bold": True, "align": "right"}), + _row("Availability", {"text": "99.87%", "align": "right"}), + _row("Leak rate", {"text": "3.2\u00d710\u207b\u2076 mbar\u00b7L/s", + "align": "right"}), + _row("Efficiency", {"text": "0.4%", "align": "right"}), + _row("Throughput", {"text": "1.4 \u00d7 10\u2076 m\u00b3/d", "align": "right"}), + ], + }], + edit="Change Availability to 99.92%.", + target=(0, 1, 1), + expect_contains="99.92%", + ), + + Case( + id="num_mixed_date_formats", + title="Mixed date and time formats in one column", + axis="numeric_formatting", + tables=[{ + "rows": [ + _row({"text": "Event", "fill": HDR, "bold": True}, + {"text": "Timestamp", "fill": HDR, "bold": True}), + _row("Trip", "2026-03-04 08:12:47Z"), + _row("Restart", "04/03/2026 09:05"), + _row("Report filed", "4 March 2026"), + ], + }], + edit="Change the Restart timestamp to 04/03/2026 09:40.", + target=(0, 2, 1), + expect_contains="09:40", + ), + + # ---------------- structure and invariance ---------------- + Case( + id="struct_wide_fixed_layout", + title="Wide table with fixed layout and explicit column widths", + axis="merged_cells", + notes="Column widths are the quietest thing to lose; nothing looks " + "broken, the table is just no longer the shape it was.", + tables=[{ + "layout": "fixed", + "col_widths": [1600, 700, 700, 700, 700, 700, 700, 900], + "header_rows": [0], + "rows": [ + _row({"text": "Line", "fill": HDR, "bold": True}, + *[{"text": m, "fill": HDR, "bold": True, "align": "center"} + for m in ("Jan", "Feb", "Mar", "Apr", "May", "Jun")], + {"text": "YTD", "fill": HDR, "bold": True, "align": "right"}), + _row("Volume", "412", "398", "444", "460", "451", "470", + {"text": "2,635", "align": "right"}), + _row("Downtime", "6.0", "2.5", "0.0", "1.5", "3.0", "0.5", + {"text": "13.5", "align": "right"}), + ], + }], + edit="Change the March volume to 448 and the YTD volume to 2,639.", + target=(0, 1, 3), + also_changes=[(0, 1, 7)], + expect_contains="448", + ), + + Case( + id="struct_shading_borders", + title="Alternating row shading with per-cell borders", + axis="merged_cells", + tables=[{ + "header_rows": [0], + "rows": [ + _row({"text": "Check", "fill": HDR, "bold": True}, + {"text": "Result", "fill": HDR, "bold": True}, + {"text": "Signed", "fill": HDR, "bold": True}), + _row({"text": "Isolation verified", "fill": ALT}, + {"text": "Pass", "fill": ALT, + "borders": {"bottom": ("18", "2E7D32")}}, + {"text": "RK", "fill": ALT}), + _row("Gas test", {"text": "Pass", + "borders": {"bottom": ("18", "2E7D32")}}, "RK"), + _row({"text": "Pressure hold", "fill": ALT}, + {"text": "Fail", "fill": ALT, + "borders": {"bottom": ("18", "C62828")}}, + {"text": "RK", "fill": ALT}), + ], + }], + edit="Change the Pressure hold result from Fail to Pass.", + target=(0, 3, 1), + expect_contains="Pass", + ), +] + + +# Every case gets the same surrounding prose. If a round-trip rewrites body +# text that was never mentioned, that is a finding too. +for _c in CASES: + _c.intro = _c.intro or ( + "The table below is a fixture. Only the cell named in the instruction " + "may change." + ) + if not _c.trailing_paragraphs: + _c.trailing_paragraphs = [ + "Prepared for benchmark purposes. All figures are fabricated.", + "Reference: TFB/" + _c.id.upper(), + ] + + +BY_ID = {c.id: c for c in CASES} + + +def select(ids: list[str] | None = None, axis: str | None = None, + limit: int | None = None) -> list[Case]: + out = CASES + if ids: + out = [BY_ID[i] for i in ids] + if axis: + out = [c for c in out if c.axis == axis] + if limit is not None: + out = out[:limit] + return out diff --git a/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/corpus/generate.py b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/corpus/generate.py new file mode 100644 index 00000000..2b47f447 --- /dev/null +++ b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/corpus/generate.py @@ -0,0 +1,215 @@ +"""Declarative table spec -> .docx. + +Written against raw WordprocessingML rather than python-docx because the +benchmark needs exact control over vMerge, gridSpan, tblHeader and +textDirection -- the very constructs under test. A generator that could not +express a pathology could not test for it. +""" + +from __future__ import annotations + +import os +import zipfile +from xml.sax.saxutils import escape + +W = "http://schemas.openxmlformats.org/wordprocessingml/2006/main" + +CONTENT_TYPES = """ + + + + + +""" + +ROOT_RELS = """ + + +""" + +DOC_RELS = """ + + +""" + +STYLES = f""" + + + + +""" + +DEFAULT_COL_DXA = 1800 + + +def _p(text: str, align: str | None = None, bold: bool = False) -> str: + ppr = f'' if align else "" + rpr = "" if bold else "" + body = "" + if text: + body = ( + f'{rpr}{escape(text)}' + ) + elif bold: + body = f"{rpr}" + return f"{ppr}{body}" + + +def _tc_pr(cell: dict, span: int, vmerge: str | None, width: int) -> str: + parts = [f''] + if span > 1: + parts.append(f'') + if vmerge == "restart": + parts.append('') + elif vmerge == "continue": + parts.append("") + if cell.get("fill"): + parts.append( + f'' + ) + if cell.get("valign"): + parts.append(f'') + if cell.get("dir"): + parts.append(f'') + if cell.get("borders"): + edges = "".join( + f'' + for e, (sz, col) in cell["borders"].items() + ) + parts.append(f"{edges}") + return "" + "".join(parts) + "" + + +def _cell_body(cell: dict) -> str: + out = [] + paras = cell.get("paragraphs") + if paras is None: + paras = [cell.get("text", "")] + align = cell.get("align") + bold = bool(cell.get("bold")) + for text in paras: + out.append(_p(text, align, bold)) + if cell.get("nested"): + out.append(_table_xml(cell["nested"])) + # OOXML requires a paragraph after a nested table inside a cell. + out.append(_p("")) + return "".join(out) + + +def _place(rows: list[list[dict]]): + """Resolve the grid. Returns (placements, n_cols). + + placements[r] is an ordered list of (cell, col, span, vmerge). + """ + occ: dict[tuple[int, int], dict] = {} + for r, row in enumerate(rows): + c = 0 + for cell in row: + while (r, c) in occ: + c += 1 + span = int(cell.get("colspan", 1)) + rspan = int(cell.get("rowspan", 1)) + cell["_r"], cell["_c"] = r, c + for rr in range(rspan): + for kk in range(span): + occ[(r + rr, c + kk)] = cell + c += span + + n_rows = max((cell["_r"] + int(cell.get("rowspan", 1)) + for row in rows for cell in row), default=0) + n_cols = max((cell["_c"] + int(cell.get("colspan", 1)) + for row in rows for cell in row), default=0) + + placements: list[list[tuple[dict, int, int, str | None]]] = [] + for r in range(n_rows): + line = [] + c = 0 + while c < n_cols: + cell = occ.get((r, c)) + if cell is None: + c += 1 + continue + if cell["_c"] != c: + c += 1 + continue + span = int(cell.get("colspan", 1)) + rspan = int(cell.get("rowspan", 1)) + if rspan > 1: + vmerge = "restart" if cell["_r"] == r else "continue" + else: + vmerge = None + line.append((cell, c, span, vmerge)) + c += span + placements.append(line) + return placements, n_cols + + +def _table_xml(spec: dict) -> str: + rows = spec["rows"] + placements, n_cols = _place(rows) + widths = spec.get("col_widths") or [DEFAULT_COL_DXA] * n_cols + total = sum(widths) + + grid = "".join(f'' for w in widths) + layout = spec.get("layout", "fixed") + tbl_pr = ( + "" + f'' + f'' + "" + '' + '' + '' + '' + '' + '' + "" + "" + ) + + header_rows = set(spec.get("header_rows", [])) + body = [] + for r, line in enumerate(placements): + tr_pr = "" if r in header_rows else "" + tcs = [] + for cell, col, span, vmerge in line: + width = sum(widths[col:col + span]) if col + span <= len(widths) else DEFAULT_COL_DXA + if vmerge == "continue": + inner = _p("") + else: + inner = _cell_body(cell) + tcs.append(f"{_tc_pr(cell, span, vmerge, width)}{inner}") + body.append(f"{tr_pr}{''.join(tcs)}") + + return f"{tbl_pr}{grid}{''.join(body)}" + + +def build_document_xml(case) -> str: + parts = [_p(case.title, bold=True)] + if case.intro: + parts.append(_p(case.intro)) + for spec in case.tables: + parts.append(_table_xml(spec)) + parts.append(_p("")) + for tail in case.trailing_paragraphs: + parts.append(_p(tail)) + sect = ( + '' + '' + ) + return ( + '' + f'{"".join(parts)}{sect}' + ) + + +def write_docx(case, out_path: str) -> str: + os.makedirs(os.path.dirname(out_path) or ".", exist_ok=True) + xml = build_document_xml(case) + with zipfile.ZipFile(out_path, "w", zipfile.ZIP_DEFLATED) as z: + z.writestr("[Content_Types].xml", CONTENT_TYPES) + z.writestr("_rels/.rels", ROOT_RELS) + z.writestr("word/document.xml", xml) + z.writestr("word/_rels/document.xml.rels", DOC_RELS) + z.writestr("word/styles.xml", STYLES) + return out_path diff --git a/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/extract.py b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/extract.py new file mode 100644 index 00000000..e93e1ea2 --- /dev/null +++ b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/extract.py @@ -0,0 +1,399 @@ +"""Extractors: DOCX -> Document, HTML -> Document. + +Both target the same canonical model so any two stages of the pipeline are +directly comparable. + +DOCX is parsed from raw WordprocessingML rather than through python-docx's +table API, because that API repeats merged cells instead of reporting spans, +which is precisely the thing under test here. +""" + +from __future__ import annotations + +import zipfile + +from lxml import etree + +from .model import Cell, Document, Table + +W = "http://schemas.openxmlformats.org/wordprocessingml/2006/main" +NS = {"w": W} + + +def _q(tag: str) -> str: + return f"{{{W}}}{tag}" + + +def _val(el, default=None): + if el is None: + return default + return el.get(_q("val"), default) + + +# -------------------------------------------------------------------------- +# DOCX +# -------------------------------------------------------------------------- + +def _para_text(p) -> str: + """Text of one w:p, preserving tabs and explicit breaks.""" + out: list[str] = [] + for node in p.iter(): + tag = etree.QName(node).localname + if tag == "t": + out.append(node.text or "") + elif tag == "tab": + out.append("\t") + elif tag == "br": + out.append("\n") + return "".join(out) + + +def _cell_paragraphs(tc) -> list[str]: + """Paragraphs directly in this cell -- not those inside a nested table.""" + return [_para_text(p) for p in tc.findall(f"{_q('p')}")] + + +def _border_spec(e) -> str: + return "{}/{}/{}".format( + e.get(_q("val"), ""), e.get(_q("sz"), ""), (e.get(_q("color")) or "").upper() + ) + + +def _table_borders(tbl) -> dict[str, str]: + """The table's own border declaration, by edge name.""" + tbl_pr = tbl.find(_q("tblPr")) + if tbl_pr is None: + return {} + bdr = tbl_pr.find(_q("tblBorders")) + if bdr is None: + return {} + return {etree.QName(e).localname: _border_spec(e) for e in bdr} + + +def _borders(pr) -> dict[str, str]: + """Borders declared on the cell itself. May be empty.""" + if pr is None: + return {} + bdr = pr.find(_q("tcBorders")) + if bdr is None: + return {} + out = {} + for edge in ("top", "left", "bottom", "right"): + e = bdr.find(_q(edge)) + if e is not None: + out[edge] = _border_spec(e) + return out + + +def _resolve_effective_borders(table: "Table", tbl_borders: dict[str, str]) -> None: + """Fold table-level borders into every cell, in place. + + A round-trip is free to move a border from the table element onto each + cell: the file changes, the rendered document does not. Comparing declared + borders would call that re-expression a loss and bury the real findings + under it. Comparing what each edge actually resolves to -- cell override + first, then the table's outer edge or inner rule by position -- compares + the document instead of the markup. + """ + if not tbl_borders and not any(c.borders for c in table.cells): + return + for c in table.cells: + eff = {} + at_top = c.row == 0 + at_left = c.col == 0 + at_bottom = c.row + c.row_span >= table.n_rows + at_right = c.col + c.col_span >= table.n_cols + for edge, outer, inner in ( + ("top", at_top, "insideH"), + ("bottom", at_bottom, "insideH"), + ("left", at_left, "insideV"), + ("right", at_right, "insideV"), + ): + spec = tbl_borders.get(edge) if outer else tbl_borders.get(inner) + if spec: + eff[edge] = spec + eff.update(c.borders_declared) # an explicit cell border always wins + c.borders = {k: v for k, v in eff.items() if v} + + +def _parse_tbl(tbl) -> Table: + table = Table() + tbl_borders = _table_borders(tbl) + + grid = tbl.find(_q("tblGrid")) + if grid is not None: + for gc in grid.findall(_q("gridCol")): + try: + table.col_widths.append(int(gc.get(_q("w"), "0"))) + except ValueError: + table.col_widths.append(0) + + tbl_pr = tbl.find(_q("tblPr")) + if tbl_pr is not None: + layout = tbl_pr.find(_q("tblLayout")) + if layout is not None: + table.layout = layout.get(_q("type")) + + occupied: dict[tuple[int, int], Cell] = {} + rows = tbl.findall(_q("tr")) + + for r, tr in enumerate(rows): + tr_pr = tr.find(_q("trPr")) + if tr_pr is not None and tr_pr.find(_q("tblHeader")) is not None: + table.header_rows.append(r) + + c = 0 + for tc in tr.findall(_q("tc")): + while (r, c) in occupied: + c += 1 + + pr = tc.find(_q("tcPr")) + span = 1 + vmerge = None + if pr is not None: + gs = pr.find(_q("gridSpan")) + if gs is not None: + span = int(gs.get(_q("val"), "1")) + vm = pr.find(_q("vMerge")) + if vm is not None: + vmerge = vm.get(_q("val"), "continue") + + if vmerge == "continue": + anchor = occupied.get((r - 1, c)) + if anchor is not None: + anchor.row_span = max(anchor.row_span, r - anchor.row + 1) + for k in range(span): + occupied[(r, c + k)] = anchor + c += span + continue + # No anchor above: a broken vMerge. Treat as its own cell and + # let the comparator report the topology difference honestly. + vmerge = None + + shading = None + valign = None + width = None + tdir = None + if pr is not None: + shd = pr.find(_q("shd")) + if shd is not None: + fill = shd.get(_q("fill")) + if fill and fill.lower() not in ("auto",): + shading = fill.upper() + va = pr.find(_q("vAlign")) + valign = _val(va) + tcw = pr.find(_q("tcW")) + if tcw is not None: + try: + width = int(tcw.get(_q("w"), "0")) + except ValueError: + width = None + td = pr.find(_q("textDirection")) + tdir = _val(td) + + paras = tc.findall(_q("p")) + alignment = None + for p in paras: + ppr = p.find(_q("pPr")) + if ppr is not None: + j = ppr.find(_q("jc")) + if j is not None: + alignment = _val(j) + break + + cell = Cell( + row=r, + col=c, + row_span=1, + col_span=span, + paragraphs=_cell_paragraphs(tc), + alignment=alignment, + shading=shading, + valign=valign, + width_dxa=width, + text_direction=tdir, + borders=_borders(pr), + borders_declared=_borders(pr), + nested=[_parse_tbl(t) for t in tc.findall(_q("tbl"))], + ) + table.cells.append(cell) + for k in range(span): + occupied[(r, c + k)] = cell + c += span + + table.n_rows = len(rows) + table.n_cols = max((c.col + c.col_span for c in table.cells), default=0) + if table.col_widths: + table.n_cols = max(table.n_cols, len(table.col_widths)) + _resolve_effective_borders(table, tbl_borders) + return table + + +def from_docx(path: str) -> Document: + with zipfile.ZipFile(path) as z: + xml = z.read("word/document.xml") + root = etree.fromstring(xml) + body = root.find(_q("body")) + doc = Document() + if body is None: + return doc + for child in body: + tag = etree.QName(child).localname + if tag == "tbl": + doc.tables.append(_parse_tbl(child)) + elif tag == "p": + txt = _para_text(child) + if txt.strip(): + doc.body_paragraphs.append(txt) + return doc + + +# -------------------------------------------------------------------------- +# HTML (the SuperDocs intermediate representation) +# -------------------------------------------------------------------------- + +def _style_map(el) -> dict[str, str]: + raw = el.get("style") or "" + out = {} + for part in raw.split(";"): + if ":" in part: + k, v = part.split(":", 1) + out[k.strip().lower()] = v.strip() + return out + + +def _html_cell_paragraphs(td) -> list[str]: + from bs4 import NavigableString + + blocks = td.find_all(["p", "div"], recursive=False) + if blocks: + return [b.get_text().strip() for b in blocks] + # Bare text in the cell, ignoring any nested table's text. + parts = [] + for node in td.children: + if isinstance(node, NavigableString): + parts.append(str(node)) + elif getattr(node, "name", None) != "table": + parts.append(node.get_text()) + text = "".join(parts).strip() + return [text] if text else [] + + +def _parse_html_table(tbl) -> Table: + table = Table() + occupied: dict[tuple[int, int], Cell] = {} + + rows = [tr for tr in tbl.find_all("tr") if tr.find_parent("table") is tbl] + for r, tr in enumerate(rows): + if tr.find_parent(["thead"]) is not None: + table.header_rows.append(r) + c = 0 + tds = [td for td in tr.find_all(["td", "th"]) if td.find_parent("tr") is tr] + for td in tds: + while (r, c) in occupied: + c += 1 + try: + colspan = int(td.get("colspan", 1)) + except ValueError: + colspan = 1 + try: + rowspan = int(td.get("rowspan", 1)) + except ValueError: + rowspan = 1 + + st = _style_map(td) + shading = st.get("background-color") or st.get("background") + if shading and shading.startswith("#"): + shading = shading[1:].upper() + elif shading: + shading = shading.upper() + + nested = [ + t for t in td.find_all("table") + if t.find_parent(["td", "th"]) is td + ] + + borders = {} + for k, v in td.attrs.items(): + if k.startswith("data-border-"): + borders[k[len("data-border-"):]] = v + try: + width = int(td.get("data-w")) if td.get("data-w") else None + except ValueError: + width = None + + # Alignment may sit on the cell or on the paragraph inside it. + # An earlier version read only the cell and reported every centred + # heading as lost -- a false finding against a product that had + # preserved it correctly. Check both. + align = st.get("text-align") + if not align: + first_p = td.find(["p", "div"], recursive=False) + if first_p is not None: + align = _style_map(first_p).get("text-align") + + cell = Cell( + row=r, + col=c, + row_span=rowspan, + col_span=colspan, + paragraphs=_html_cell_paragraphs(td), + alignment=align, + shading=shading, + valign=st.get("vertical-align"), + width_dxa=width, + text_direction=td.get("data-dir"), + borders=borders, + nested=[_parse_html_table(t) for t in nested], + ) + table.cells.append(cell) + for rr in range(rowspan): + for kk in range(colspan): + occupied[(r + rr, c + kk)] = cell + c += colspan + + hr = tbl.get("data-header-rows") + if hr: + table.header_rows = [int(x) for x in hr.split(",") if x.strip()] + elif tbl.find("thead") is not None: + # A real is how the API expresses header rows. Reading it means + # the HTML stage can distinguish "the header marking survived ingest and + # was lost on export" from "it never arrived" -- which is the whole + # value of a staged comparison. + thead_rows = [tr for tr in tbl.find_all("tr") + if tr.find_parent("thead") is not None] + all_rows = [tr for tr in tbl.find_all("tr") if tr.find_parent("table") is tbl] + table.header_rows = [all_rows.index(tr) for tr in thead_rows + if tr in all_rows] + table.layout = tbl.get("data-layout") + cg = tbl.find("colgroup") + if cg is not None and cg.find_parent("table") is tbl: + for col in cg.find_all("col"): + try: + table.col_widths.append(int(col.get("data-w", 0))) + except ValueError: + table.col_widths.append(0) + + table.n_rows = max((c.row + c.row_span for c in table.cells), default=0) + table.n_cols = max((c.col + c.col_span for c in table.cells), default=0) + if table.col_widths: + table.n_cols = max(table.n_cols, len(table.col_widths)) + return table + + +def from_html(html: str) -> Document: + from bs4 import BeautifulSoup + + soup = BeautifulSoup(html, "html.parser") + doc = Document() + for tbl in soup.find_all("table"): + if tbl.find_parent("table") is not None: + continue # nested tables are reached through their parent cell + doc.tables.append(_parse_html_table(tbl)) + for p in soup.find_all(["p", "h1", "h2", "h3"]): + if p.find_parent("table") is not None: + continue + txt = p.get_text().strip() + if txt: + doc.body_paragraphs.append(txt) + return doc diff --git a/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/htmlout.py b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/htmlout.py new file mode 100644 index 00000000..8bd0b9cd --- /dev/null +++ b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/htmlout.py @@ -0,0 +1,135 @@ +"""Model -> HTML, and model -> generator spec. + +Used only by the offline FakeClient, to give it a real round-trip +(docx -> model -> html -> model -> docx) rather than a stub that hands back +what it was given. A fake that returns the input unchanged would make every +test pass and prove nothing. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from xml.sax.saxutils import escape + +from .model import Cell, Document, Table + + +def _cell_html(c: Cell) -> str: + attrs = [] + if c.col_span > 1: + attrs.append(f'colspan="{c.col_span}"') + if c.row_span > 1: + attrs.append(f'rowspan="{c.row_span}"') + style = [] + if c.alignment: + style.append(f"text-align:{c.alignment}") + if c.shading: + style.append(f"background-color:#{c.shading}") + if c.valign: + style.append(f"vertical-align:{c.valign}") + if style: + attrs.append('style="' + ";".join(style) + '"') + if c.text_direction: + attrs.append(f'data-dir="{c.text_direction}"') + if c.width_dxa is not None: + attrs.append(f'data-w="{c.width_dxa}"') + for edge, spec in sorted(c.borders.items()): + attrs.append(f'data-border-{edge}="{spec}"') + open_tag = "" + + body = "".join(f"

{escape(p)}

" for p in c.paragraphs if p != "") + for nt in c.nested: + body += _table_html(nt) + return open_tag + body + "" + + +def _table_html(t: Table) -> str: + anchors = t.anchors() + rows_html = [] + for r in range(t.n_rows): + cells = [anchors[(rr, cc)] for (rr, cc) in sorted(anchors) if rr == r] + cells.sort(key=lambda c: c.col) + if not cells: + continue + rows_html.append("" + "".join(_cell_html(c) for c in cells) + "") + attrs = "" + if t.header_rows: + attrs += f' data-header-rows="{",".join(str(r) for r in t.header_rows)}"' + if t.layout: + attrs += f' data-layout="{t.layout}"' + cols = "" + if t.col_widths: + cols = "" + "".join( + f'' for w in t.col_widths) + "" + return f"" + cols + "".join(rows_html) + "" + + +def to_html(doc: Document) -> str: + parts = [f"

{escape(p)}

" for p in doc.body_paragraphs[:2]] + parts += [_table_html(t) for t in doc.tables] + parts += [f"

{escape(p)}

" for p in doc.body_paragraphs[2:]] + return "".join(parts) + + +# -------------------------------------------------------------------------- + +@dataclass +class _SyntheticCase: + """Duck-types the fields corpus.generate.write_docx reads off a Case.""" + + title: str = "" + intro: str = "" + tables: list[dict] = field(default_factory=list) + trailing_paragraphs: list[str] = field(default_factory=list) + + +def _table_to_spec(t: Table) -> dict: + anchors = t.anchors() + rows: list[list[dict]] = [] + for r in range(t.n_rows): + line = [c for (rr, _), c in sorted(anchors.items()) if rr == r] + line.sort(key=lambda c: c.col) + spec_row = [] + for c in line: + d: dict = {"paragraphs": list(c.paragraphs) or [""]} + if c.col_span > 1: + d["colspan"] = c.col_span + if c.row_span > 1: + d["rowspan"] = c.row_span + if c.alignment: + d["align"] = c.alignment + if c.shading: + d["fill"] = c.shading + if c.valign: + d["valign"] = c.valign + if c.text_direction: + d["dir"] = c.text_direction + if c.borders: + bd = {} + for edge, spec in c.borders.items(): + parts = spec.split("/") + if len(parts) == 3: + bd[edge] = (parts[1], parts[2]) + if bd: + d["borders"] = bd + if c.nested: + d["nested"] = _table_to_spec(c.nested[0]) + spec_row.append(d) + if spec_row: + rows.append(spec_row) + spec = {"rows": rows, "header_rows": list(t.header_rows)} + if t.col_widths: + spec["col_widths"] = list(t.col_widths) + if t.layout: + spec["layout"] = t.layout + return spec + + +def doc_to_case(doc: Document) -> _SyntheticCase: + bp = doc.body_paragraphs + return _SyntheticCase( + title=bp[0] if bp else "", + intro=bp[1] if len(bp) > 1 else "", + tables=[_table_to_spec(t) for t in doc.tables], + trailing_paragraphs=list(bp[2:]), + ) diff --git a/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/model.py b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/model.py new file mode 100644 index 00000000..df6cfacf --- /dev/null +++ b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/model.py @@ -0,0 +1,147 @@ +"""Canonical table model. + +One grid representation that the generator, the DOCX extractor and the HTML +extractor all produce. Comparison happens between two of these and never +between two file formats directly -- that is what lets the benchmark say +*where* fidelity was lost (ingest, edit, or export) instead of only that it was. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field, asdict +from typing import Any + + +@dataclass +class Cell: + """One anchor cell. Cells covered by a span are not repeated.""" + + row: int + col: int + row_span: int = 1 + col_span: int = 1 + paragraphs: list[str] = field(default_factory=list) + alignment: str | None = None + shading: str | None = None # fill colour, upper hex, no '#' + valign: str | None = None + width_dxa: int | None = None + text_direction: str | None = None + borders: dict[str, str] = field(default_factory=dict) + borders_declared: dict[str, str] = field(default_factory=dict) + """Borders written on the cell itself, before table-level inheritance. + + Kept apart from the resolved set because an interior rule is shared by two + cells: when they disagree, the one that declared it explicitly is the one + that was meant.""" + nested: list["Table"] = field(default_factory=list) + + @property + def text(self) -> str: + return "\n".join(self.paragraphs) + + @property + def is_merged(self) -> bool: + return self.row_span > 1 or self.col_span > 1 + + def to_dict(self) -> dict[str, Any]: + d = asdict(self) + d["nested"] = [t.to_dict() for t in self.nested] + return d + + +@dataclass +class Table: + n_rows: int = 0 + n_cols: int = 0 + cells: list[Cell] = field(default_factory=list) + col_widths: list[int] = field(default_factory=list) + layout: str | None = None # 'fixed' | 'autofit' + header_rows: list[int] = field(default_factory=list) # w:tblHeader rows + + def cell_at(self, row: int, col: int) -> Cell | None: + """The anchor covering (row, col), span-aware.""" + for c in self.cells: + if ( + c.row <= row < c.row + c.row_span + and c.col <= col < c.col + c.col_span + ): + return c + return None + + def occupancy(self) -> dict[tuple[int, int], tuple[int, int]]: + """(row, col) -> anchor coordinate. The topology fingerprint.""" + occ: dict[tuple[int, int], tuple[int, int]] = {} + for c in self.cells: + for r in range(c.row, c.row + c.row_span): + for k in range(c.col, c.col + c.col_span): + occ[(r, k)] = (c.row, c.col) + return occ + + def anchors(self) -> dict[tuple[int, int], Cell]: + return {(c.row, c.col): c for c in self.cells} + + def border_edges(self) -> dict[tuple[str, int, int], str]: + """Every drawn rule, keyed by its position in the grid. + + Interior edges are shared: the line between two cells can be declared + as one cell's bottom or the next cell's top, and the document looks + identical either way. Comparing cells would call that a difference. + Comparing edges compares what is actually drawn. + + ('h', r, c) is the horizontal rule above row r spanning column c. + ('v', r, c) is the vertical rule left of column c on row r. + """ + edges: dict[tuple[str, int, int], str] = {} + rank: dict[tuple[int, int, int], tuple[int, int]] = {} + + def claim(key, cell: "Cell", edge: str) -> None: + spec = cell.borders.get(edge) + if not spec: + return + # An explicitly declared border outranks an inherited one; between + # two of the same kind the heavier rule wins. Without a rule here + # the winner would depend on cell ordering, which would make the + # same document compare differently against itself. + declared = 1 if edge in cell.borders_declared else 0 + try: + weight = int(spec.split("/")[1] or 0) + except (IndexError, ValueError): + weight = 0 + score = (declared, weight) + if key not in edges or score > rank[key]: + edges[key] = spec + rank[key] = score + + for cell in self.cells: + for r in range(cell.row, cell.row + cell.row_span): + claim(("v", r, cell.col), cell, "left") + claim(("v", r, cell.col + cell.col_span), cell, "right") + for k in range(cell.col, cell.col + cell.col_span): + claim(("h", cell.row, k), cell, "top") + claim(("h", cell.row + cell.row_span, k), cell, "bottom") + return edges + + def to_dict(self) -> dict[str, Any]: + return { + "n_rows": self.n_rows, + "n_cols": self.n_cols, + "col_widths": self.col_widths, + "layout": self.layout, + "header_rows": self.header_rows, + "cells": [c.to_dict() for c in self.cells], + } + + +@dataclass +class Document: + """Just the tables. Prose between tables is captured as a flat list so the + invariance check can prove untouched body text did not move either.""" + + tables: list[Table] = field(default_factory=list) + body_paragraphs: list[str] = field(default_factory=list) + + def to_dict(self) -> dict[str, Any]: + return { + "tables": [t.to_dict() for t in self.tables], + "body_paragraphs": self.body_paragraphs, + } diff --git a/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/run.py b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/run.py new file mode 100644 index 00000000..ba8ca653 --- /dev/null +++ b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tablebench/run.py @@ -0,0 +1,327 @@ +"""Benchmark runner. + +Two runs per case: + + control upload -> export (no edit, 0 operations) + edit upload -> chat -> approve -> export (1 operation) + +The control is what makes the result attributable. Any difference the control +also shows is a property of ingest and export; only what the edit run adds is +caused by the edit. Without it a benchmark can say a table broke but not where. + +Resumable: state is written after every case, so a killed run resumes without +re-spending operations on cases already measured. +""" + +from __future__ import annotations + +import argparse +import json +import os +import sys +import time +import traceback + +from . import compare, extract +from .client import ApiError, BudgetExceeded, FakeClient, Ledger, SuperDocsClient +from .corpus import cases as case_mod +from .corpus.generate import write_docx + +DEFAULT_OUT = "results" + + +def _load_state(path: str) -> dict: + if os.path.exists(path): + with open(path) as fh: + return json.load(fh) + return {"cases": {}, "ops_spent": 0, "started": time.time()} + + +def _save_state(path: str, state: dict) -> None: + os.makedirs(os.path.dirname(path) or ".", exist_ok=True) + tmp = path + ".tmp" + with open(tmp, "w") as fh: + json.dump(state, fh, indent=2) + os.replace(tmp, path) # atomic: a crash mid-write cannot corrupt it + + +def run_case(case, client, workdir: str, mode: str = "dry") -> dict: + """One case, both runs. Returns a result record. + + The record carries the mode it was produced in. A dry record and a + live record are not interchangeable evidence, and nothing downstream + is allowed to treat them as if they were. + """ + src = write_docx(case, os.path.join(workdir, "corpus", f"{case.id}.docx")) + before = extract.from_docx(src) + rec: dict = {"id": case.id, "axis": case.axis, "title": case.title, + "mode": mode} + t0 = time.monotonic() + + # ---- control: no edit ------------------------------------------------- + ctrl_findings: list[compare.Finding] = [] + ctrl_sid = f"tfb-{case.id}-control" + try: + client.upload(src, ctrl_sid) + ctrl_out = client.export( + ctrl_sid, os.path.join(workdir, "out", f"{case.id}.control.docx")) + ctrl_findings = compare.compare(before, extract.from_docx(ctrl_out)) + rec["control"] = {"status": "ok", + "findings": [f.to_dict() for f in ctrl_findings]} + except Exception as exc: # noqa: BLE001 + rec["control"] = {"status": "error", "error": f"{type(exc).__name__}: {exc}"} + + # ---- edit run --------------------------------------------------------- + sid = f"tfb-{case.id}-edit" + try: + html = client.upload(src, sid) + html_doc = extract.from_html(html) + ingest = compare.compare(before, html_doc, profile="html") + rec["ingest_findings"] = [f.to_dict() for f in ingest] + rec["ingest_unmeasurable"] = compare.unmeasurable_at_html_stage( + compare.compare(before, html_doc)) + + if hasattr(client, "simulate_edit"): + client.simulate_edit(sid, case) + job = client.edit(sid, html, case.edit) + pending = job.get("pending") or [] + rec["proposed_changes"] = len(pending) + if pending: + client.approve(sid, job["job_id"], pending) + + out = client.export(sid, os.path.join(workdir, "out", f"{case.id}.edit.docx")) + after = extract.from_docx(out) + + exempt = [case.target, *getattr(case, "also_changes", [])] + findings = compare.compare(before, after, target=exempt) + by_edit, by_round_trip = compare.attribute(findings, ctrl_findings) + + edit_applied = None + if case.expect_contains is not None: + tgt_tbl = after.tables[case.target[0]] if len(after.tables) > case.target[0] else None + cell = tgt_tbl.cell_at(case.target[1], case.target[2]) if tgt_tbl else None + haystack = _all_text(after) + if cell is not None: + edit_applied = case.expect_contains in _cell_text_deep(cell) + if not edit_applied and case.expect_contains in haystack: + rec["note"] = ("expected value is present in the document " + "but not in the target cell") + else: + # The cell we were told to check no longer exists. That is a + # structural failure, not an unmeasurable one. + edit_applied = False + rec["note"] = "target cell absent from the exported grid" + + rec["findings"] = [f.to_dict() for f in findings] + rec["caused_by_edit"] = [f.to_dict() for f in by_edit] + rec["caused_by_round_trip"] = [f.to_dict() for f in by_round_trip] + rec["edit_applied"] = edit_applied + rec["verdict"] = compare.verdict(edit_applied, findings) + rec["summary"] = compare.summarise(findings) + except BudgetExceeded: + raise + except Exception as exc: # noqa: BLE001 + rec["verdict"] = compare.ERROR + rec["error"] = f"{type(exc).__name__}: {exc}" + rec["traceback"] = traceback.format_exc(limit=3) + + rec["seconds"] = round(time.monotonic() - t0, 2) + return rec + + +def _cell_text_deep(cell) -> str: + """A cell's own text plus everything in tables nested inside it.""" + parts = [cell.text] + for n in cell.nested: + for c in n.cells: + parts.append(_cell_text_deep(c)) + return "\n".join(parts) + + +def _all_text(doc) -> str: + parts = list(doc.body_paragraphs) + + def walk(t): + for c in t.cells: + parts.append(c.text) + for n in c.nested: + walk(n) + + for t in doc.tables: + walk(t) + return "\n".join(parts) + + +def render_report(state: dict, ledger: Ledger, mode: str) -> str: + recs = list(state["cases"].values()) + counts: dict[str, int] = {} + for r in recs: + counts[r.get("verdict", "?")] = counts.get(r.get("verdict", "?"), 0) + 1 + + modes: dict[str, int] = {} + for r in recs: + modes[r.get("mode", "unknown")] = modes.get(r.get("mode", "unknown"), 0) + 1 + mixed = len(modes) > 1 + mode_line = (f"- mode: **{mode}**" if not mixed else + "- mode: **MIXED** - " + + ", ".join(f"{k}: {v} cases" for k, v in sorted(modes.items()))) + + lines = [ + "# Table fidelity benchmark - results", + "", + mode_line, + f"- cases run: **{len(recs)}**", + f"- operations spent: **{ledger.spent}** (cap {ledger.cap})", + "- verdicts: " + ", ".join(f"{k} {v}" for k, v in sorted(counts.items())), + "", + ] + ([ + "> **These results are not one run.** The records below were " + "produced in more than one mode. A dry-mode record is a simulation " + "and is not evidence about the live API. Use a separate `--out` " + "directory per mode before quoting any of this.", + "", + ] if mixed else []) + [ + "Method: each case is uploaded, exported once with no edit (control), " + "then uploaded, edited through one approval-gated instruction, " + "approved and exported. Findings present in the control are attributed " + "to the format round-trip; the remainder are attributed to the edit. " + "The named target cell is exempt from content comparison; every other " + "cell is an invariant.", + "", + "| case | mode | axis | verdict | edit applied | by edit | by round-trip | s |", + "|---|---|---|---|---|---|---|---|", + ] + for r in sorted(recs, key=lambda x: (x.get("axis", ""), x["id"])): + applied = r.get("edit_applied") + applied_s = {True: "yes", False: "no", None: "unverifiable"}.get(applied, "?") + lines.append( + f"| `{r['id']}` | {r.get('mode','?')} | {r.get('axis','')} | " + f"{r.get('verdict','?')} | " + f"{applied_s} | {len(r.get('caused_by_edit', []))} | " + f"{len(r.get('caused_by_round_trip', []))} | {r.get('seconds','')} |" + ) + + lines += ["", "## Findings by case", ""] + for r in sorted(recs, key=lambda x: x["id"]): + if r.get("verdict") == compare.PASS and not r.get("caused_by_round_trip"): + continue + lines.append(f"### `{r['id']}` - {r.get('title','')}") + if r.get("error"): + lines.append(f"- run error: `{r['error']}`") + for label, key in (("caused by the edit", "caused_by_edit"), + ("caused by the round-trip", "caused_by_round_trip")): + items = r.get(key) or [] + if items: + lines.append(f"- **{label}** ({len(items)}):") + for f in items[:12]: + lines.append( + f" - `{f['severity']}/{f['code']}` {f['where']} - " + f"expected `{f['expected'][:70]}`, got `{f['actual'][:70]}`") + if len(items) > 12: + lines.append(f" - ... and {len(items) - 12} more") + if r.get("ingest_unmeasurable"): + lines.append("- not measurable at the HTML stage: " + + ", ".join(f"`{c}`" for c in r["ingest_unmeasurable"])) + if r.get("note"): + lines.append(f"- note: {r['note']}") + lines.append("") + + lines += ["## Timing by stage", ""] + for stage, secs in sorted(ledger.timings.items()): + lines.append(f"- {stage}: {secs:.1f}s") + lines += ["", "## Limits of this measurement", "", + "- Findings are reported against the exported DOCX. Attributes " + "that plain HTML cannot express are withheld at the HTML stage " + "and listed as not measurable rather than counted as losses.", + "- A case whose run errored is reported ERROR, never FAIL. " + "An unproven failure is not a failure.", + "- Rendering is not compared. Two files that parse identically " + "could still paginate differently; that is out of scope and " + "stated rather than implied."] + return "\n".join(lines) + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser(prog="tablebench") + ap.add_argument("--mode", choices=("dry", "live"), default="dry", + help="dry runs entirely offline and spends nothing") + ap.add_argument("--fault", default="none", + help="dry mode only: inject a named fidelity fault") + ap.add_argument("--limit", type=int, default=None, + help="small-sample mode: run only the first N cases") + ap.add_argument("--cases", default=None, help="comma-separated case ids") + ap.add_argument("--axis", default=None, choices=case_mod.AXES) + ap.add_argument("--max-ops", type=int, default=25, + help="hard operation cap; the run stops before exceeding it") + ap.add_argument("--out", default=DEFAULT_OUT) + ap.add_argument("--resume", action="store_true", + help="skip cases already recorded in the state file") + args = ap.parse_args(argv) + + selected = case_mod.select( + ids=args.cases.split(",") if args.cases else None, + axis=args.axis, limit=args.limit) + if not selected: + print("no cases selected", file=sys.stderr) + return 2 + + state_path = os.path.join(args.out, "state.json") + state = _load_state(state_path) if args.resume else { + "cases": {}, "ops_spent": 0, "started": time.time()} + + # Only spends made in THIS mode count against this cap. Simulated + # operations from a dry run are not money and must not reserve budget. + prior_spend = sum(r.get("ops", 0) for r in state.get("cases", {}).values() + if r.get("mode") == args.mode) + ledger = Ledger(cap=args.max_ops, spent=prior_spend) + if args.mode == "live": + client = SuperDocsClient(ledger) + else: + client = FakeClient(ledger, fault=args.fault) + + print(f"mode={args.mode} cases={len(selected)} cap={args.max_ops} ops") + if args.mode == "live": + est = sum(1 for _ in selected) + print(f"estimated spend: {est} operations " + f"(1 per case; control and export are free)") + + stopped = None + for case in selected: + prior = state["cases"].get(case.id) if args.resume else None + if prior is not None and prior.get("mode") == args.mode: + print(f" skip {case.id} (already recorded in {args.mode} mode)") + continue + if prior is not None: + print(f" rerun {case.id} (prior record was " + f"{prior.get('mode', 'unknown')} mode, not {args.mode})") + spent_before = ledger.spent + try: + rec = run_case(case, client, args.out, mode=args.mode) + except BudgetExceeded as exc: + stopped = str(exc) + print(f" STOP {exc}") + break + rec["ops"] = ledger.spent - spent_before + state["cases"][case.id] = rec + state["ops_spent"] = ledger.spent + _save_state(state_path, state) + print(f" {rec.get('verdict','?'):13s} {case.id:28s} " + f"edit={len(rec.get('caused_by_edit', []))} " + f"rt={len(rec.get('caused_by_round_trip', []))} " + f"ops={ledger.spent}") + + if stopped: + state["stopped_by_budget"] = stopped + _save_state(state_path, state) + + report = render_report(state, ledger, args.mode) + os.makedirs(args.out, exist_ok=True) + with open(os.path.join(args.out, "report.md"), "w") as fh: + fh.write(report) + print(f"\nreport: {os.path.join(args.out, 'report.md')} " + f"ops spent: {ledger.spent}/{ledger.cap}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tests/test_benchmark.py b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tests/test_benchmark.py new file mode 100644 index 00000000..ee4eaf67 --- /dev/null +++ b/extensions/SiddharthMuneshwar26/table-fidelity-benchmark/tests/test_benchmark.py @@ -0,0 +1,359 @@ +"""Tests. None require an API key and none cost an operation. + +These do not assert that the fake returns what it was told to return. Each one +injects a fault whose ground truth is known and asserts the comparator reports +that fault, only in the cases where the pathology exists, and nowhere else. +A benchmark whose measuring instrument is untested measures nothing. +""" + +from __future__ import annotations + +import json +import os +import re + +import pytest + +from tablebench import compare, extract +from tablebench.client import BudgetExceeded, FakeClient, Ledger +from tablebench.corpus import cases as case_mod +from tablebench.corpus.generate import write_docx +from tablebench.run import main as run_main, run_case + +# Cases that actually contain each pathology, so a detector can be checked for +# false negatives AND false positives. +HAS_MERGE = {"merge_horizontal_header", "merge_vertical_label", "merge_l_shaped", + "merge_full_row_divider", "merge_containing_nested", + "header_two_level", "header_three_level", "header_repeat_flag"} +HAS_NESTED = {"merge_containing_nested", "nested_simple", "nested_two_deep", + "nested_side_by_side"} + + +@pytest.fixture(scope="module") +def corpus(tmp_path_factory): + d = tmp_path_factory.mktemp("corpus") + return {c.id: write_docx(c, str(d / f"{c.id}.docx")) for c in case_mod.CASES} + + +# -------------------------------------------------------------------------- +# corpus and extraction +# -------------------------------------------------------------------------- + +def test_every_case_generates_a_readable_docx(corpus): + assert len(corpus) == len(case_mod.CASES) + for cid, path in corpus.items(): + assert os.path.getsize(path) > 0 + doc = extract.from_docx(path) + assert doc.tables, f"{cid} produced no table" + + +def test_spans_survive_extraction(corpus): + doc = extract.from_docx(corpus["merge_l_shaped"]) + t = doc.tables[0] + assert t.cell_at(0, 0).row_span == 2 # Asset, merged down + assert t.cell_at(0, 1).col_span == 2 # Planned, merged across + assert t.cell_at(3, 0).col_span == 4 # full-width divider + # A span is not the same cell repeated: anchors are unique. + assert t.cell_at(0, 1) is t.cell_at(0, 2) + + +def test_nested_tables_are_reached_and_not_flattened(corpus): + doc = extract.from_docx(corpus["nested_two_deep"]) + outer = doc.tables[0].cell_at(1, 1) + assert len(outer.nested) == 1 + inner = outer.nested[0].cell_at(0, 1) + assert len(inner.nested) == 1, "second level of nesting was lost" + assert "0.72" in inner.nested[0].cell_at(1, 1).text + + +def test_pathological_numbers_are_character_exact(corpus): + t = extract.from_docx(corpus["num_negative_parentheses"]).tables[0] + assert t.cell_at(1, 1).text == "(12,400.00)" + assert t.cell_at(3, 1).text == "\u2014", "em dash must not become a hyphen" + t2 = extract.from_docx(corpus["num_leading_zero_ids"]).tables[0] + assert t2.cell_at(1, 1).text == "0007", "leading zeros must not be dropped" + + +# -------------------------------------------------------------------------- +# the comparator +# -------------------------------------------------------------------------- + +def test_control_roundtrip_is_lossless(corpus): + """With no fault injected the pipeline must report nothing. + + If this drifts, every other detection result is unattributable noise. + """ + fc = FakeClient(Ledger(cap=999)) + for cid, path in corpus.items(): + before = extract.from_docx(path) + fc.upload(path, cid) + out = fc.export(cid, f"/tmp/tfb-control/{cid}.docx") + assert compare.compare(before, extract.from_docx(out)) == [], cid + + +@pytest.mark.parametrize("fault,applies_to", [ + ("flatten_merges", HAS_MERGE), + ("drop_nested", HAS_NESTED), +]) +def test_structural_faults_are_caught_where_they_apply(corpus, fault, applies_to): + fc = FakeClient(Ledger(cap=999), fault=fault) + detected = set() + for cid, path in corpus.items(): + before = extract.from_docx(path) + fc.upload(path, cid) + out = fc.export(cid, f"/tmp/tfb-{fault}/{cid}.docx") + if compare.compare(before, extract.from_docx(out)): + detected.add(cid) + missed = applies_to - detected + assert not missed, f"{fault} went undetected in {sorted(missed)}" + + +def test_numeric_fault_is_caught_and_named(corpus): + fc = FakeClient(Ledger(cap=999), fault="strip_trailing_zero") + path = corpus["num_trailing_zeros"] + before = extract.from_docx(path) + fc.upload(path, "n") + out = fc.export("n", "/tmp/tfb-num/x.docx") + findings = compare.compare(before, extract.from_docx(out)) + codes = {f.code for f in findings} + assert "cell_text" in codes + assert any("1.23" in f.actual for f in findings), \ + "the comparator saw a change but did not report the changed value" + + +def test_untouched_body_text_is_an_invariant(corpus): + fc = FakeClient(Ledger(cap=999), fault="rewrite_body") + path = corpus["nested_simple"] + before = extract.from_docx(path) + fc.upload(path, "b") + out = fc.export("b", "/tmp/tfb-body/x.docx") + findings = compare.compare(before, extract.from_docx(out)) + assert any(f.code == "body_paragraphs" for f in findings), \ + "prose outside the table changed and was not reported" + + +def test_target_cell_is_exempt_but_its_neighbours_are_not(corpus): + """The asymmetry that defines the measurement.""" + case = case_mod.BY_ID["merge_horizontal_header"] + before = extract.from_docx(corpus[case.id]) + after = extract.from_docx(corpus[case.id]) + + tgt = after.tables[0].cell_at(*case.target[1:]) + tgt.paragraphs = ["1,099.40"] + assert compare.compare(before, after, target=case.target) == [], \ + "changing the target cell must not be a finding" + + neighbour = after.tables[0].cell_at(case.target[1], case.target[2] - 1) + neighbour.paragraphs = ["tampered"] + findings = compare.compare(before, after, target=case.target) + assert [f.code for f in findings] == ["cell_text"], \ + "changing a neighbour must be a finding, and only that one" + + +def test_findings_are_attributed_between_edit_and_roundtrip(): + f1 = compare.Finding("content", "cell_text", "table[0] cell(1, 1)", "a", "b") + f2 = compare.Finding("structural", "col_span", "table[0] cell(0, 1)", "2", "1") + by_edit, by_rt = compare.attribute([f1, f2], control_findings=[f2]) + assert by_edit == [f1] and by_rt == [f2] + + +# -------------------------------------------------------------------------- +# honesty, budget, resumption +# -------------------------------------------------------------------------- + +def test_error_is_reported_as_error_not_as_failure(corpus): + class Broken(FakeClient): + def edit(self, *a, **k): + raise RuntimeError("simulated upstream 502") + + rec = run_case(case_mod.BY_ID["nested_simple"], + Broken(Ledger(cap=99)), "/tmp/tfb-err") + assert rec["verdict"] == compare.ERROR + assert rec["verdict"] != compare.FAIL + assert "502" in rec["error"] + + +def test_unverifiable_is_distinct_from_pass(): + assert compare.verdict(None, []) == compare.UNVERIFIABLE + assert compare.verdict(True, []) == compare.PASS + assert compare.verdict(False, []) == compare.FAIL + + +def test_budget_cap_stops_before_the_spend_not_after(): + led = Ledger(cap=2) + led.charge("chat") + led.charge("chat") + with pytest.raises(BudgetExceeded): + led.charge("chat") + assert led.spent == 2, "the ledger must not record a spend it refused" + + +def test_free_stages_do_not_consume_operations(): + led = Ledger(cap=1) + for stage in ("upload", "approve", "export"): + led.charge(stage) + assert led.spent == 0 + + +def test_run_is_resumable_after_being_killed(tmp_path): + out = str(tmp_path / "r") + run_main(["--mode", "dry", "--max-ops", "3", "--out", out]) + state = json.load(open(os.path.join(out, "state.json"))) + first = len(state["cases"]) + assert 0 < first < len(case_mod.CASES), "budget stop did not halt the run" + + run_main(["--mode", "dry", "--max-ops", "99", "--out", out, "--resume"]) + state2 = json.load(open(os.path.join(out, "state.json"))) + assert len(state2["cases"]) == len(case_mod.CASES) + for cid in list(state["cases"])[:first]: + assert state2["cases"][cid]["seconds"] == state["cases"][cid]["seconds"], \ + f"{cid} was re-run instead of resumed, re-spending its operation" + + +def test_proposed_changes_get_the_second_parse(): + """The documented integrator trap: content arrives JSON-encoded.""" + from tablebench.client import SuperDocsClient + job = {"metadata": {"pending_changes": [ + {"change_id": "c1", "content": json.dumps({"chunk_id": "p3", "html": "

x

"})}, + {"change_id": "c2", "content": "{not valid json"}, + ]}} + parsed = SuperDocsClient._pending(job) + assert parsed[0]["content"]["chunk_id"] == "p3", "second parse not applied" + assert parsed[1].get("content_parse_failed") is True, \ + "an unparseable change must be flagged, not silently passed through" + + +def test_no_credential_ever_reaches_an_output_file(tmp_path, monkeypatch): + monkeypatch.setenv("SUPERDOCS_API_KEY", "sk_live_THIS_MUST_NEVER_APPEAR") + out = str(tmp_path / "r") + run_main(["--mode", "dry", "--limit", "3", "--out", out]) + for root, _, files in os.walk(out): + for name in files: + if name.endswith((".json", ".md")): + body = open(os.path.join(root, name), encoding="utf-8").read() + assert not re.search(r"sk_[A-Za-z0-9_]{4,}", body), \ + f"a key-shaped string reached {name}" + + +def test_report_states_its_own_limits(tmp_path): + out = str(tmp_path / "r") + run_main(["--mode", "dry", "--limit", "2", "--out", out]) + report = open(os.path.join(out, "report.md")).read() + assert "Limits of this measurement" in report + assert "not measurable" in report or "never FAIL" in report + + +def test_a_dry_record_never_satisfies_a_live_resume(tmp_path): + """Regression: --resume once skipped live work because dry records held + those slots, and counted simulated operations against the live cap. A + simulation is not evidence about the API and must not stand in for it.""" + out = str(tmp_path / "r") + run_main(["--mode", "dry", "--limit", "3", "--out", out]) + path = os.path.join(out, "state.json") + state = json.load(open(path)) + assert {r["mode"] for r in state["cases"].values()} == {"dry"} + + # Pretend one case was measured live, then resume in dry mode. + victim = sorted(state["cases"])[0] + state["cases"][victim]["mode"] = "live" + state["cases"][victim]["seconds"] = -1.0 + json.dump(state, open(path, "w")) + + run_main(["--mode", "dry", "--limit", "3", "--out", out, "--resume"]) + after = json.load(open(path)) + assert after["cases"][victim]["seconds"] != -1.0, \ + "a live record was treated as a dry record and skipped" + assert after["cases"][victim]["mode"] == "dry" + + +def test_simulated_operations_do_not_reserve_live_budget(tmp_path): + out = str(tmp_path / "r") + run_main(["--mode", "dry", "--limit", "5", "--max-ops", "5", "--out", out]) + state = json.load(open(os.path.join(out, "state.json"))) + assert sum(r["ops"] for r in state["cases"].values()) == 5 + # Every one of those ops is dry, so a live run starts from zero. + live_prior = sum(r.get("ops", 0) for r in state["cases"].values() + if r.get("mode") == "live") + assert live_prior == 0 + + +def test_report_refuses_to_present_a_mixed_run_as_one(tmp_path): + out = str(tmp_path / "r") + run_main(["--mode", "dry", "--limit", "3", "--out", out]) + path = os.path.join(out, "state.json") + state = json.load(open(path)) + state["cases"][sorted(state["cases"])[0]]["mode"] = "live" + json.dump(state, open(path, "w")) + + run_main(["--mode", "dry", "--limit", "3", "--out", out, "--resume"]) + report = open(os.path.join(out, "report.md")).read() + if "MIXED" in report: + assert "not one run" in report + + +def test_an_instruction_naming_two_cells_exempts_both(corpus): + """Regression: a two-figure instruction with one declared target reported + the model for obeying it. A false finding is worse than no finding.""" + case = case_mod.BY_ID["merge_full_row_divider"] + assert case.also_changes, "this case's instruction changes two cells" + + before = extract.from_docx(corpus[case.id]) + after = extract.from_docx(corpus[case.id]) + after.tables[0].cell_at(*case.target[1:]).paragraphs = ["46"] + after.tables[0].cell_at(*case.also_changes[0][1:]).paragraphs = ["1,426.00"] + + exempt = [case.target, *case.also_changes] + assert compare.compare(before, after, target=exempt) == [], \ + "both cells named in the instruction must be exempt" + + # And a third cell is still an invariant. + after.tables[0].cell_at(2, 1).paragraphs = ["tampered"] + assert [f.code for f in compare.compare(before, after, target=exempt)] == \ + ["cell_text"] + + +def test_every_case_exempts_as_many_cells_as_its_instruction_changes(): + """A guard against the same mistake in a case added later.""" + for c in case_mod.CASES: + named = 1 + len(c.also_changes) + # crude but effective: count 'to ' clauses in the instruction + clauses = c.edit.lower().count(" to ") + assert clauses <= named, ( + f"{c.id}: instruction appears to change {clauses} cells but only " + f"{named} are exempt") + + +def test_shared_edges_resolve_the_same_way_every_time(corpus): + """Regression: an interior rule is written by both adjacent cells. Without + a precedence rule the winner depended on cell ordering, so a document + could compare unequal to itself and produce findings that were artifacts.""" + doc = extract.from_docx(corpus["struct_shading_borders"]) + t = doc.tables[0] + + edges = t.border_edges() + reversed_t = extract.from_docx(corpus["struct_shading_borders"]).tables[0] + reversed_t.cells.reverse() + assert reversed_t.border_edges() == edges, \ + "edge resolution depends on cell order" + + # The declared coloured rule must beat the inherited default it collides + # with, not the other way round. + assert edges[("h", 2, 1)] == "single/18/2E7D32" + assert edges[("h", 4, 1)] == "single/18/C62828" + + +def test_capture_does_not_call_a_withheld_measurement_a_survival(): + """Regression. The summary said text_direction 'survives ingest, lost on + export'. It does not survive ingest -- plain HTML has no property for it, + so the HTML stage withholds it and measures nothing either way. Reporting + withheld as intact asserts something the evidence cannot support, and it + contradicted the written bug report it was attached to.""" + from tablebench import capture, compare + assert "text_direction" in compare.HTML_UNREPRESENTABLE + assert "border_edge" in compare.HTML_UNREPRESENTABLE + # header_rows was withheld until the API turned out to return a real + # ; it is measurable and must not be withheld. + assert "header_rows" not in compare.HTML_UNREPRESENTABLE + src = open(capture.__file__, encoding="utf-8").read() + assert "HTML_UNREPRESENTABLE" in src, \ + "capture must exclude withheld codes from its stage attribution" diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/Makefile b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/Makefile new file mode 100644 index 00000000..b1c8e650 --- /dev/null +++ b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/Makefile @@ -0,0 +1,13 @@ +.PHONY: install corpus test dry live clean +install: + pip install -r requirements.txt +corpus: + cd corpus && python generate.py +test: + python -m pytest tests/ -q +dry: + python -m lossrun.review --corpus corpus/out --out out --mode dry --max-ops 6 --as-of 2026-01-09 +live: + python -m lossrun.review --corpus corpus/out --out out --mode live --max-ops 6 --as-of 2026-01-09 +clean: + rm -rf out .pytest_cache **/__pycache__ diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/README.md b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/README.md new file mode 100644 index 00000000..b8a00596 --- /dev/null +++ b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/README.md @@ -0,0 +1,237 @@ +# Insurance loss-run summary app + +Turns years of raw loss-run data across multiple policies and carriers into one +review document an underwriter can act on: claim counts, incurred totals, and +open claims grouped by policy year — with every figure traced to the document +and row it came from. + +Built on SuperDocs (upload, search, chat, approve, export) for the SuperDocs +task. Assigned build, band S2, surfaces: search and export. + +> Built by Siddharth Muneshwar for the SuperDocs task. + +## What SuperDocs features it uses + +| feature | how | +|---|---| +| **Upload** | the adjuster notes, application and broker correspondence are attached to the session; free, and what makes cross-document retrieval possible | +| **Chat editing** | one turn does two jobs: fills a placeholder section from the attached documents, and writes the opening narrative | +| **Approve** | the proposed changes are approved before anything is written | +| **Export** | the review document comes back as DOCX | + +A full review costs **one operation**. Figures never pass through the model: +they are parsed from the loss runs and the instruction forbids changing them. + +## Run it + +``` +pip install -r requirements.txt +python -m pytest tests/ -q +python -m lossrun.review --corpus corpus/out --out out --mode dry --max-ops 6 --as-of 2026-01-09 +``` + +25 tests, no API key, no network, no operations; then the full pipeline +offline. + +The generated corpus ships in `corpus/out/`, so there is nothing to build first. +Regenerating it (`cd corpus && python generate.py`) needs LibreOffice on the +path for the PDF loss run; nothing else does. + +Against the live API: + +``` +export SUPERDOCS_API_KEY=... +python -m lossrun.review --corpus corpus/out --out out --mode live --max-ops 6 --as-of 2026-01-09 +``` + +Four operations, capped at six. On PowerShell the first line is +`$env:SUPERDOCS_API_KEY = '...'`. + +A `Makefile` wraps these as `make install`, `make test`, `make dry` and +`make live` where `make` is available. It is a convenience, not a dependency — +every command above runs as written on Windows, macOS and Linux. + +The key is read from the environment only — never an argument, never logged, +never written to an output file, and a test asserts no key-shaped string reaches +any output. + +## The one design decision everything follows from + +**Figures are parsed. They are never searched for, inferred, or model-generated.** + +Claim counts, incurred totals and loss ratios are arithmetic over rows read out +of the loss runs. An underwriter cannot act on a total that might be a plausible +guess, and a summary whose numbers need re-checking has saved nobody any work. + +Search and chat are spent where they earn their cost: + +| work | how | operations | +|---|---|---| +| claim figures, counts, totals, ratios | parsed from CSV/PDF rows | 0 | +| contradictions living in prose, and the narrative | one edit | **1** | +| upload, attach, approve, export | free per published pricing | 0 | + +**A full review costs one operation.** + +**On the search surface, and two wrong turns getting here.** A dedicated +`/v1/search` endpoint returned `HTTP 404` live, so retrieval moved to the chat +surface: the prose documents are attached to the session by `upload`, which is +free. The first attempt asked for a JSON chat reply and looked for it in the +proposed-changes payload — which carries document edits, not chat text, so the +answer was never there to find. That was fighting the product. SuperDocs is an +editing agent: the answer belongs *in the document*. The review is now written +with a placeholder section, and one edit both fills it from the attached +documents and adds the opening narrative. Supported path, and half the cost of +the version that was wrong. + +`SuperDocsClient.search` and `.retrieve` are kept, unused, for the day a search +endpoint or a chat-reply field is exposed. `search` raises `SearchUnavailable` +on 404 rather than pretending an empty search succeeded. + +The chat instruction says explicitly that every number in the document is +already correct and must not be changed, recalculated, rounded or restated. +There is a test asserting that instruction still says so. + +A naive design that searched once per claim would have spent 41 operations on +one account and still produced figures nobody could audit. + +## What makes loss runs hard + +A loss run is a carrier's claims history for one policy year. They get +**reissued**, and reserves develop between issues. The same claim carries +$45,000 on the run valued January 2024 and $182,400 on the reissue valued +January 2026. Neither document is wrong. + +This is not a detail. On the sample account: + +| source for policy year 2023 | incurred | loss ratio | referral rule | +|---|---|---|---| +| original run, valued 2024-01 | $286,092.44 | **57.1%** | passes | +| reissue, valued 2026-01 | $647,992.44 | **129.3%** | **refer** | + +The same account either clears the 60% referral threshold or blows through it +twice over, depending on which document is trusted. So the app takes the later +valuation as authoritative **and records the difference as a conflict** rather +than overwriting quietly. A conflict is never delivered pre-resolved; there is a +test for that. + +## What it produces + +`out/review.html` — and see the known defect below regarding the DOCX export: + +- **Summary by policy year** — claims, open count, paid, reserve, incurred, loss + ratio, and which document each year's figures came from +- **Open claims**, listed separately with months open, because reserves on open + claims may still develop +- **Conflicts between sources**, surfaced with both figures and the consequence +- **Checklist findings** against eight underwriting rules +- **What this review does not establish** — named, not smoothed over +- **Sources**, including which documents were read and superseded + +`out/summary.json` carries the same content for anything downstream. + +## The rules + +R1 severity claims need an adjuster note · R2 open more than 36 months · +R3 loss ratio above 60% · R4 date of loss outside its policy period · +R5 application consistent with the loss runs · R6 dormant reserves · +R7 duplicate claims · R8 loss run adds up to its own stated total + +A clean corpus produces no findings at all, and that path is tested. An honest +report of nothing is a valid output. + +## Formats + +CSV and PDF loss runs both parse to the same rows, and there is a test asserting +the same policy year read two ways gives identical facts. The PDF parser is +deliberately conservative: a row is accepted only when every field is present +and every figure parses. Rows that look like claim lines but do not fully parse +are counted and reported as unreadable rather than guessed at, because a +half-read row is worse than a missing one on a document feeding a coverage +decision. + +Accountant notation is parsed as written: `(12,400.00)` is negative, `—` is nil, +`$1,204,880.00` is a figure. A string that is not a figure raises rather than +quietly becoming `0.00` — silently reading `pending` as zero would corrupt a +total. + +## Budget + +One operation per run, capped at six. The cap is enforced **before** each +spend. Hitting it degrades rather than dies: the review is still produced, and +the section that could not be completed is named in *What this review does not +establish* instead of quietly disappearing. There is a test that runs with a cap +of one and asserts exactly that. + +```bash +python -m lossrun.review --corpus corpus/out --out out --mode live --max-ops 6 --as-of 2026-01-09 --skip-prose +``` + +## The corpus + +`corpus/` generates a fabricated account — Meridian Cold Chain Logistics, a +refrigerated carrier — with 41 claims across four policy years, five loss runs +including one reissue, three adjuster notes, a signed application, a broker +cover note, and the underwriting checklist. CSV, PDF and DOCX. + +`corpus/MANIFEST.md` is the answer key: nine planted conflicts and thirteen +expected findings, with verified figures. The test suite asserts the app +reproduces it **exactly** — every planted item found, and nothing outside the +key reported. 34 of the 41 claims are deliberately clean, which is what makes a +false positive measurable rather than arguable. + +Everything is fabricated. Meridian Cold Chain Logistics, Kestrel Mutual and +Ardent Risk Partners do not exist, and no figure comes from any real claim, +policy or company. + +## Known defect: the DOCX export returns the wrong document + +**`out/review.html` is correct and complete. `out/loss-run-review.docx` is not.** + +The app uploads the adjuster notes, application and correspondence to the +session so the agent can read them for the correspondence section. Export then +asks the session for "the document" — and returns the last file uploaded, not +the review that was passed to the edit call as `document_html`. On the sample +corpus the exported DOCX comes back containing the underwriting checklist. + +Everything upstream of the export is unaffected: the figures, the conflicts, the +findings and the citations are all correct, and `out/summary.json` and +`out/review.html` carry them. The defect is in how this build sequences upload +against export, not in the analysis. + +The likely fix is to make the review itself the session document before editing, +rather than passing it only as `document_html`. That needs live API calls to get +right, so it is stated here rather than guessed at. + +## Honest limits + +- **The carrier's own numbers are taken as given.** Each run is checked against + itself and against the others; nothing here can confirm the carrier was right. +- **Retrieval quality is not measured.** The offline fake fills the placeholder + by keyword, deliberately weaker than the live agent, so a point that only + surfaces against the real API stays visible rather than hidden. +- **If the edit proposes no changes**, the correspondence section keeps its + placeholder and the run says so under *What this review does not establish* + rather than reporting a success it did not have. +- **Figures do not depend on retrieval at all**, and there is a test that proves + it: the pipeline is run twice, once with retrieval and once with + `--skip-prose`, and every total, ratio and finding must be identical. +- **R5 reads specific declarations** from the application. Where a declaration + is not found, nothing is asserted about it — an absent statement is not a + false one. +- **PDF reading needs `pdfplumber`**, which `requirements.txt` installs. An + earlier version shelled out to `pdftotext`, a Unix binary: it worked on the + machine it was written on and died with a bare `WinError 2` on the machine it + was used on. Where no reader is available the CSV path is unaffected and the + failure names the fix instead of raising a traceback. +- **R7 needs description overlap**, not just a matching date and amount. An + early version flagged any two claims sharing a date and figure, which on a + fleet account is coincidence rather than duplication. Found by the + clean-corpus test, and the rule was fixed rather than the test. +- **The DOCX export is broken**, as above. The HTML review is the working + deliverable. +- **No coverage recommendation.** The app summarises and flags; it does not + advise whether to write the risk, and the chat instruction forbids adding one. + +Portions of this work were written with AI assistance, directed and reviewed +by me. diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/MANIFEST.md b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/MANIFEST.md new file mode 100644 index 00000000..00542b9e --- /dev/null +++ b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/MANIFEST.md @@ -0,0 +1,130 @@ +# MANIFEST — ground truth for the synthetic loss-run corpus + +Everything in this corpus is fabricated. Meridian Cold Chain Logistics, Kestrel +Mutual Insurance and Ardent Risk Partners do not exist. No figure is drawn from +any real claim, policy or company. + +This file is the answer key. A system reading this corpus should surface the +conflicts and findings below, and should surface **nothing that is not on this +list**. Both halves matter: a corpus with a known-clean remainder is what makes +a false positive visible. + +Regenerate with `python generate.py`. The seed is fixed, so the corpus is +byte-stable. + +## Contents + +| file | format | role | +|---|---|---| +| `loss-runs/loss-run-2021-valued-2024-01.csv` | CSV | policy year 2021, 12 claims | +| `loss-runs/loss-run-2022-valued-2024-01.csv` | CSV | policy year 2022, 12 claims | +| `loss-runs/loss-run-2022-valued-2024-01.pdf` | PDF | same content, different format | +| `loss-runs/loss-run-2023-valued-2024-01-ORIGINAL.csv` | CSV | **superseded** 2023 run, 9 claims | +| `loss-runs/loss-run-2023-valued-2026-01-REISSUE.csv` | CSV | **authoritative** 2023 run, 10 claims | +| `loss-runs/loss-run-2024-valued-2026-01.csv` | CSV | policy year 2024, 7 claims | +| `adjuster-notes/*.docx` | DOCX | three claim file notes | +| `application/meridian-renewal-application-2026.docx` | DOCX | signed renewal application | +| `correspondence/ardent-cover-note-2026-01-28.docx` | DOCX | broker cover note | +| `rules/underwriting-checklist.docx` | DOCX | the eight rules to examine against | + +41 claims across four policy years. + +## The conflict that changes the decision + +Claim **KM-2023-0417** (jackknife on I-80) carries **$45,000.00** incurred on the +2023 run valued January 2024, and **$182,400.00** on the reissue valued January +2026. Neither document is wrong — the reserve developed after suit was filed, +and the adjuster note sets out the history. + +The consequence is not cosmetic: + +| source for policy year 2023 | incurred | earned premium | loss ratio | rule R3 | +|---|---|---|---|---| +| original run, valued 2024-01 | $286,092.44 | $501,200 | **57.1%** | passes | +| reissue, valued 2026-01 | $647,992.44 | $501,200 | **129.3%** | **refer** | + +The same account either clears the referral threshold or blows through it twice +over, depending on which document is trusted. A system that silently picks one +has made an underwriting decision without telling anyone. This is the conflict +to surface rather than resolve. + +## Planted conflicts + +**C1 — reserve development across a reissue.** KM-2023-0417, $45,000.00 → +$182,400.00. Adjuster note dated 2025-09-30 explains it and gives the reserve +history. Consequence above. + +**C2 — status contradicted by a later note.** KM-2022-0311 is `Closed` on every +loss run. The adjuster note dated 2024-03-12 states the claim reopened on +2024-02-19 with a re-established reserve of $34,000. The note explicitly says the +loss run was correct at its valuation date and is no longer correct. The loss +runs were never reissued to reflect it. + +**C3 — claim present in the reissue and absent from the original.** +KM-2023-0588, $224,500.00, date of loss 2024-02-27, reported 2024-06-14 — after +the original run was issued. Only document where it appears first: the reissue. + +**C4 — date of loss outside its stated policy period.** KM-2022-0402 has a date +of loss of 2023-04-11 but is filed under policy year 2022, which ran to +2023-04-01. Triggers R4. + +**C5 — one event under two claim numbers.** KM-2021-0219 and KM-2021-0224 both +describe a bridge strike on Route 6 on 2021-08-03, each at $27,450.00. The 2021 +total therefore double-counts $27,450.00. Triggers R7. + +**C6 — severity claim with no adjuster note.** KM-2024-0106, $149,000.00 +incurred, open. No note in the corpus. Triggers R1. + +**C7 — dormant reserve.** KM-2021-0333, $0.00 paid against a $96,000.00 reserve, +open 49 months at the January 2026 valuation. Triggers R2 and R6. + +**C8 — loss run that does not add up.** The original 2023 run states a total +incurred of $287,592.44; its rows sum to $286,092.44. A $1,500.00 discrepancy. +Real loss runs do this. Triggers R8. The other four runs are internally +consistent — check them and find nothing. + +**C9 — application contradicts the loss runs, three ways.** The signed +application dated 2026-01-22 states: + +| application says | loss runs show | +|---|---| +| 3 claims exceeding $50,000 in five years | **7** | +| no open claim exceeds $100,000 | **3** (KM-2023-0417, KM-2023-0588, KM-2024-0106) | +| all 2021 and 2022 claims are closed | **6 still open** | + +The broker's cover note explains why — the application was prepared in December, +before the reissued runs arrived. That explanation is in a different document +from the contradiction, which is the point. + +## Findings the checklist should produce + +Against the **authoritative** set (2021, 2022, 2023-reissue, 2024): + +| rule | expected findings | +|---|---| +| R1 adjuster note on ≥$100k | 1 — KM-2024-0106 | +| R2 open >36 months | 4 — KM-2021-0232, KM-2021-0252, KM-2021-0333, KM-2022-0237 | +| R3 loss ratio >60% | 1 — policy year 2023 at 129.3% | +| R4 policy period integrity | 1 — KM-2022-0402 | +| R5 application consistency | 3 — the rows in C9 | +| R6 dormant reserve >24 months | 1 — KM-2021-0333 | +| R7 duplicate claims | 1 — KM-2021-0219 / KM-2021-0224 | +| R8 loss run adds up | 1 — original 2023 run, $1,500.00 out | + +Loss ratios by year, for R3: 2021 **57.6%**, 2022 **41.2%**, 2023 **129.3%**, +2024 **33.3%**. Three of four pass, which is what makes the fourth mean +something. + +## Deliberately clean + +The other 34 claims carry no planted defect. Every one of the four policy years +except 2023 is internally consistent, under the loss-ratio threshold, and free +of period or duplicate problems. Anything a system reports outside the table +above is a false positive, and on this corpus that is measurable rather than a +matter of opinion. + +## A note on formats + +The 2022 loss run exists as both CSV and PDF with identical content. A system +that claims to handle mixed formats can be checked on it: the same policy year, +read two ways, must produce the same facts. diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/generate.py b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/generate.py new file mode 100644 index 00000000..a9d67b82 --- /dev/null +++ b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/generate.py @@ -0,0 +1,490 @@ +"""Synthetic insurance loss-run corpus. + +Everything here is fabricated. Meridian Cold Chain Logistics, Kestrel Mutual +and Ardent Risk Partners do not exist, and no figure is drawn from any real +claim, policy or company. + +Why loss runs. A loss run is a carrier's claims history for one policy year. +The thing that makes them hard in real life is that they get reissued: the same +claim shows one incurred amount in the run issued in 2024 and a much larger one +in the run issued in 2026, because reserves develop as a claim matures. Neither +document is wrong. They disagree, and somebody has to know which one they are +reading. That gives a document pile with native, non-contrived disagreement. + +The conflicts below are planted deliberately and are documented in MANIFEST.md. +That file is ground truth: a system that reads this corpus should find them, and +should find nothing that is not on the list. + +Run: python generate.py +""" + +from __future__ import annotations + +import csv +import os +import random +import subprocess +from dataclasses import dataclass, field + +OUT = "out" +SEED = 20260805 + +INSURED = "Meridian Cold Chain Logistics, Inc." +INSURED_SHORT = "Meridian Cold Chain" +CARRIER = "Kestrel Mutual Insurance Company" +CARRIER_SHORT = "Kestrel Mutual" +BROKER = "Ardent Risk Partners" +FEIN = "88-0143927" + +COVERAGES = { + "AL": "Auto Liability", + "APD": "Auto Physical Damage", + "CARGO": "Motor Truck Cargo", + "WC": "Workers Compensation", + "GL": "General Liability", + "PROP": "Property", +} + +POLICIES = { + 2021: ("KM-CA-2021-88417", "2021-04-01", "2022-04-01", 412_000), + 2022: ("KM-CA-2022-88417", "2022-04-01", "2023-04-01", 448_500), + 2023: ("KM-CA-2023-88417", "2023-04-01", "2024-04-01", 501_200), + 2024: ("KM-CA-2024-88417", "2024-04-01", "2025-04-01", 566_900), +} + + +@dataclass +class Claim: + claim_no: str + year: int + coverage: str + date_of_loss: str + date_reported: str + description: str + status: str # Open | Closed | Reopened + paid: float + reserve: float + claimant: str = "Third party" + note: str = "" + # Values as they appeared in the ORIGINAL run, when a later reissue differs + original: dict = field(default_factory=dict) + + @property + def incurred(self) -> float: + return round(self.paid + self.reserve, 2) + + +# --------------------------------------------------------------------------- +# Claims that carry a planted conflict. Explicit, never random. +# --------------------------------------------------------------------------- + +ANCHOR_CLAIMS: list[Claim] = [ + # C1 reserve development: modest at first issue, severe two years later + Claim("KM-2023-0417", 2023, "AL", "2023-11-08", "2023-11-09", + "Tractor-trailer jackknife on I-80, two vehicles involved, " + "bodily injury alleged by both occupants of the second vehicle.", + "Open", 12_400.00, 170_000.00, "Third party", + note="Severity reassessed after plaintiff filed suit.", + original={"paid": 8_150.00, "reserve": 36_850.00, "status": "Open"}), + + # C2 status flip: closed on the run, reopened per the adjuster note + Claim("KM-2022-0311", 2022, "WC", "2022-09-19", "2022-09-19", + "Warehouse associate lumbar strain, pallet jack incident.", + "Closed", 41_280.00, 0.00, "Employee", + note="Claimant returned with recurrence; see adjuster note.", + original={}), + + # C3 late-reported claim: absent from the original 2023 run entirely + Claim("KM-2023-0588", 2023, "CARGO", "2024-02-27", "2024-06-14", + "Refrigeration unit failure in transit, full load of pharmaceuticals " + "condemned on arrival.", + "Open", 0.00, 224_500.00, "Shipper", + note="Reported after the original loss run was issued.", + original={"absent": True}), + + # C4 date of loss falls outside the policy period it is filed under + Claim("KM-2022-0402", 2022, "GL", "2023-04-11", "2023-04-19", + "Slip and fall, visitor at the Fresno cross-dock.", + "Closed", 18_900.00, 0.00, "Third party", + note="Date of loss sits after the 2022 policy expiry."), + + # C5 same event carried under two claim numbers + Claim("KM-2021-0219", 2021, "APD", "2021-08-03", "2021-08-04", + "Reefer trailer sidewall damage, low bridge strike, Route 6.", + "Closed", 27_450.00, 0.00, "First party"), + Claim("KM-2021-0224", 2021, "APD", "2021-08-03", "2021-08-11", + "Trailer damage from bridge strike on Route 6 - unit 4471.", + "Closed", 27_450.00, 0.00, "First party", + note="Appears to duplicate KM-2021-0219."), + + # C6 large claim with no adjuster note on file + Claim("KM-2024-0106", 2024, "AL", "2024-06-22", "2024-06-23", + "Rear-end collision, company tractor, claimant hospitalised.", + "Open", 31_000.00, 118_000.00, "Third party"), + + # C7 long-open claim, zero paid against a standing reserve + Claim("KM-2021-0333", 2021, "GL", "2021-12-02", "2021-12-15", + "Alleged product contamination, retailer recall costs claimed.", + "Open", 0.00, 96_000.00, "Third party", + note="No payment activity since inception."), +] + +FILLER_DESCRIPTIONS = { + "AL": ["Minor collision in yard, no injuries", + "Sideswipe during lane change, property damage only", + "Backing incident at customer dock"], + "APD": ["Windshield replacement, road debris", + "Tractor door damage, parking lot", + "Tyre blowout, wheel arch damage"], + "CARGO": ["Partial load spoilage, temperature excursion", + "Carton crush damage in transit", + "Short delivery, three pallets unaccounted"], + "WC": ["Hand laceration, box cutter", + "Slip on wet dock plate, contusion", + "Repetitive strain, order picking"], + "GL": ["Damage to customer dock leveller", + "Minor property damage at delivery site"], + "PROP": ["Roof leak, Bakersfield warehouse", + "Freezer motor burnout, electrical surge"], +} + + +def build_filler(rng: random.Random) -> list[Claim]: + claims: list[Claim] = [] + per_year = {2021: 9, 2022: 10, 2023: 8, 2024: 6} + for year, n in per_year.items(): + _, start, _, _ = POLICIES[year] + for i in range(n): + cov = rng.choice(list(FILLER_DESCRIPTIONS)) + month = rng.randint(4, 12) if rng.random() < 0.7 else rng.randint(1, 3) + y = year if month >= 4 else year + 1 + dol = f"{y}-{month:02d}-{rng.randint(1, 28):02d}" + lag = rng.randint(0, 9) + rep_day = min(28, int(dol[-2:]) + lag) + drep = f"{dol[:-2]}{rep_day:02d}" + closed = rng.random() < (0.9 if year <= 2022 else 0.55) + size = rng.choice([1, 1, 1, 2, 2, 3]) + base = {1: (400, 4_000), 2: (4_000, 22_000), 3: (22_000, 65_000)}[size] + total = round(rng.uniform(*base), 2) + paid = total if closed else round(total * rng.uniform(0.05, 0.45), 2) + reserve = 0.0 if closed else round(total - paid, 2) + claims.append(Claim( + claim_no=f"KM-{year}-{200 + i * 7 + rng.randint(0, 4):04d}", + year=year, coverage=cov, date_of_loss=dol, date_reported=drep, + description=rng.choice(FILLER_DESCRIPTIONS[cov]), + status="Closed" if closed else "Open", + paid=paid, reserve=reserve, + claimant="Employee" if cov == "WC" else + ("First party" if cov in ("APD", "PROP") else "Third party"), + )) + return claims + + +# --------------------------------------------------------------------------- +# Loss run writers +# --------------------------------------------------------------------------- + +HEADER = ["Claim Number", "Policy Number", "Policy Year", "Coverage", + "Date of Loss", "Date Reported", "Status", "Claimant", + "Paid", "Reserve", "Incurred", "Description"] + + +def rows_for(claims: list[Claim], year: int, reissue: bool) -> list[list]: + pol = POLICIES[year][0] + out = [] + for c in sorted((c for c in claims if c.year == year), + key=lambda c: c.date_of_loss): + if not reissue and c.original.get("absent"): + continue + paid = c.paid + reserve = c.reserve + status = c.status + if not reissue and c.original: + paid = c.original.get("paid", paid) + reserve = c.original.get("reserve", reserve) + status = c.original.get("status", status) + out.append([ + c.claim_no, pol, year, COVERAGES[c.coverage], c.date_of_loss, + c.date_reported, status, c.claimant, + f"{paid:,.2f}", f"{reserve:,.2f}", f"{paid + reserve:,.2f}", + c.description, + ]) + return out + + +def write_loss_run(path: str, year: int, claims: list[Claim], issued: str, + reissue: bool, footer_error: float = 0.0) -> None: + _, start, end, premium = POLICIES[year] + rows = rows_for(claims, year, reissue) + total = sum(float(r[10].replace(",", "")) for r in rows) + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, "w", newline="", encoding="utf-8") as fh: + w = csv.writer(fh) + w.writerow([f"{CARRIER} - Loss Run Report"]) + w.writerow([f"Insured: {INSURED}"]) + w.writerow([f"FEIN: {FEIN}"]) + w.writerow([f"Policy Period: {start} to {end}"]) + w.writerow([f"Valuation Date: {issued}"]) + w.writerow([f"Report Type: {'REISSUE - supersedes prior' if reissue else 'Original'}"]) + w.writerow([]) + w.writerow(HEADER) + w.writerows(rows) + w.writerow([]) + # Real loss runs frequently carry a footer total that does not equal + # the sum of the rows above it. One year here reproduces that. + w.writerow(["", "", "", "", "", "", "", "TOTAL INCURRED", + "", "", f"{total + footer_error:,.2f}", ""]) + w.writerow(["", "", "", "", "", "", "", "CLAIM COUNT", + "", "", len(rows), ""]) + w.writerow(["", "", "", "", "", "", "", "EARNED PREMIUM", + "", "", f"{premium:,.2f}", ""]) + + +# --------------------------------------------------------------------------- +# DOCX writers +# --------------------------------------------------------------------------- + +def docx(path: str, blocks: list[tuple[str, str]]) -> None: + from docx import Document + from docx.shared import Pt + + d = Document() + d.styles["Normal"].font.name = "Calibri" + d.styles["Normal"].font.size = Pt(10.5) + for kind, text in blocks: + if kind == "h1": + d.add_heading(text, level=1) + elif kind == "h2": + d.add_heading(text, level=2) + elif kind == "b": + p = d.add_paragraph() + p.add_run(text).bold = True + else: + d.add_paragraph(text) + os.makedirs(os.path.dirname(path), exist_ok=True) + d.save(path) + + +def write_adjuster_notes() -> None: + docx(f"{OUT}/adjuster-notes/KM-2023-0417-adjuster-note.docx", [ + ("h1", "Claim file note"), + ("b", f"Claim: KM-2023-0417 Insured: {INSURED_SHORT} Coverage: Auto Liability"), + ("p", "Date of note: 2025-09-30. Adjuster: R. Okonjo, Kestrel Mutual."), + ("h2", "Position"), + ("p", "Reserve increased to $170,000 following receipt of the plaintiff's " + "demand and the treating physician's report. The earlier reserve of " + "$36,850 was set before suit was filed and reflected an assumption " + "of soft-tissue injury only. That assumption no longer holds."), + ("p", "Liability is not seriously in dispute. The insured unit crossed the " + "centre line. Exposure is a question of damages, not fault."), + ("h2", "Reserve history"), + ("p", "2023-11-15: $36,850 initial. 2024-08-02: $58,000. " + "2025-09-30: $170,000 current."), + ("h2", "Next steps"), + ("p", "Mediation scheduled Q2 2026. Recommend authority to $200,000."), + ]) + + docx(f"{OUT}/adjuster-notes/KM-2022-0311-adjuster-note.docx", [ + ("h1", "Claim file note"), + ("b", f"Claim: KM-2022-0311 Insured: {INSURED_SHORT} Coverage: Workers Compensation"), + ("p", "Date of note: 2024-03-12. Adjuster: L. Haddad, Kestrel Mutual."), + ("h2", "Position"), + ("p", "This file was closed on 2023-06-30 after the claimant returned to " + "full duty. The claimant has since reported a recurrence of the same " + "lumbar symptoms and has been placed on modified duty as of " + "2024-02-19. The file is reopened."), + ("p", "Note that the loss run issued in January 2024 shows this claim as " + "closed. That was correct at its valuation date and is no longer " + "correct."), + ("h2", "Revised position"), + ("p", "Reserve re-established at $34,000 for continued indemnity and " + "medical. Paid to date remains $41,280."), + ]) + + docx(f"{OUT}/adjuster-notes/KM-2023-0588-adjuster-note.docx", [ + ("h1", "Claim file note"), + ("b", f"Claim: KM-2023-0588 Insured: {INSURED_SHORT} Coverage: Motor Truck Cargo"), + ("p", "Date of note: 2024-07-01. Adjuster: R. Okonjo, Kestrel Mutual."), + ("h2", "Position"), + ("p", "Temperature excursion on a pharmaceutical load. The consignee " + "condemned the entire shipment on arrival. Salvage value is nil " + "because the product cannot re-enter the cold chain once excursion " + "is documented."), + ("p", "Date of loss 2024-02-27 falls within the 2023 policy period. The " + "claim was reported on 2024-06-14, after the original loss run for " + "that year had been issued, and therefore does not appear on it."), + ("h2", "Reserve"), + ("p", "$224,500, being the invoiced value of the load. Subrogation against " + "the reefer unit manufacturer is under review."), + ]) + + +def write_application() -> None: + docx(f"{OUT}/application/meridian-renewal-application-2026.docx", [ + ("h1", "Commercial insurance renewal application"), + ("b", f"Applicant: {INSURED}"), + ("p", f"FEIN: {FEIN}. Broker of record: {BROKER}. " + "Proposed effective date: 2026-04-01."), + ("h2", "Operations"), + ("p", "Refrigerated truckload and less-than-truckload carriage, with three " + "cross-dock facilities in California and one in Nevada. Power units: " + "64. Refrigerated trailers: 91. Employees: 212."), + ("h2", "Loss history - applicant statement"), + ("p", "The applicant reports 3 claims exceeding $50,000 in the last five " + "policy years, and confirms that no claim currently open exceeds " + "$100,000 in incurred value."), + ("p", "The applicant further reports that all claims from policy years " + "2021 and 2022 are closed."), + ("h2", "Prior carrier"), + ("p", f"{CARRIER}, continuously since 2021-04-01. No lapse in coverage. " + "No policy has been cancelled or non-renewed."), + ("h2", "Declaration"), + ("p", "The undersigned declares the statements above to be true to the " + "best of their knowledge. Signed: J. Ferreira, Chief Financial " + "Officer. Date: 2026-01-22."), + ]) + + +def write_broker_email() -> None: + docx(f"{OUT}/correspondence/ardent-cover-note-2026-01-28.docx", [ + ("h1", "Submission cover note"), + ("b", f"From: {BROKER} To: Underwriting Date: 2026-01-28"), + ("p", f"Please find attached the renewal submission for {INSURED_SHORT}, " + "effective 2026-04-01."), + ("p", "Enclosed: loss runs for policy years 2021 through 2024 valued " + "January 2026, a reissued 2023 loss run, three adjuster notes, and " + "the signed application."), + ("p", "Two points the underwriter should have in front of them. First, the " + "2023 loss run has been reissued and the incurred figures differ " + "materially from the version circulated in 2024; the reissue is the " + "one to work from. Second, the client's application was prepared in " + "December before the reissued runs arrived, so the loss summary in " + "it reflects the earlier figures."), + ("p", "The client is seeking terms at or below expiring. Happy to discuss."), + ("p", "Regards, D. Whitfield, Account Executive, " + BROKER), + ]) + + +def write_checklist() -> None: + docx(f"{OUT}/rules/underwriting-checklist.docx", [ + ("h1", "Casualty underwriting checklist - motor carrier accounts"), + ("p", "Applied to every submission before a coverage decision. Each rule " + "produces a finding or an explicit pass."), + ("h2", "R1 - Adjuster note required on severity claims"), + ("p", "Any claim with incurred value of $100,000 or more must have an " + "adjuster note in the file. A severity claim without a note cannot " + "be evaluated and must be referred."), + ("h2", "R2 - Stale open claims"), + ("p", "Any claim open more than 36 months from date of loss is flagged for " + "reserve adequacy review."), + ("h2", "R3 - Loss ratio referral"), + ("p", "Any policy year with incurred losses exceeding 60% of earned " + "premium is referred to a senior underwriter."), + ("h2", "R4 - Policy period integrity"), + ("p", "A claim whose date of loss falls outside the policy period it is " + "reported under must be queried with the carrier before the figures " + "are relied on."), + ("h2", "R5 - Application consistency"), + ("p", "The claim counts and severity thresholds stated on the application " + "must agree with the loss runs. Any disagreement is recorded and put " + "to the broker."), + ("h2", "R6 - Dormant reserves"), + ("p", "A claim with zero paid and a non-zero reserve, open more than 24 " + "months, is flagged for reserve review."), + ("h2", "R7 - Duplicate claims"), + ("p", "The same loss event must not be carried under two claim numbers. " + "Suspected duplicates are queried before totals are relied on."), + ("h2", "R8 - Loss run internal consistency"), + ("p", "The total incurred stated on a loss run must equal the sum of its " + "claim rows. A discrepancy is recorded against the document."), + ]) + + +def write_pdf_loss_run(year: int, claims: list[Claim], issued: str, + out_dir: str) -> None: + """The same policy year as a formatted PDF report. + + Carriers issue loss runs both ways. Converting the wide CSV directly gives + a truncated three-column page, which is not what a carrier PDF looks like; + this builds a proper landscape table with the columns an underwriter reads, + then renders it. + """ + from docx import Document + from docx.enum.section import WD_ORIENT + from docx.shared import Pt, Inches + + _, start, end, premium = POLICIES[year] + rows = rows_for(claims, year, reissue=False) + + d = Document() + sec = d.sections[0] + sec.orientation = WD_ORIENT.LANDSCAPE + sec.page_width, sec.page_height = sec.page_height, sec.page_width + sec.left_margin = sec.right_margin = Inches(0.5) + d.styles["Normal"].font.name = "Calibri" + d.styles["Normal"].font.size = Pt(8) + + d.add_paragraph(f"{CARRIER} - Loss Run Report").runs[0].bold = True + for line in (f"Insured: {INSURED}", f"FEIN: {FEIN}", + f"Policy Period: {start} to {end}", + f"Valuation Date: {issued}", "Report Type: Original"): + d.add_paragraph(line) + + cols = ["Claim Number", "Coverage", "Date of Loss", "Date Reported", + "Status", "Paid", "Reserve", "Incurred"] + idx = [0, 3, 4, 5, 6, 8, 9, 10] + t = d.add_table(rows=1, cols=len(cols)) + t.style = "Table Grid" + for i, c in enumerate(cols): + cell = t.rows[0].cells[i] + cell.text = c + cell.paragraphs[0].runs[0].bold = True + for r in rows: + cells = t.add_row().cells + for i, j in enumerate(idx): + cells[i].text = str(r[j]) + + total = sum(float(r[10].replace(",", "")) for r in rows) + d.add_paragraph("") + d.add_paragraph(f"TOTAL INCURRED: {total:,.2f}") + d.add_paragraph(f"CLAIM COUNT: {len(rows)}") + d.add_paragraph(f"EARNED PREMIUM: {premium:,.2f}") + + os.makedirs(out_dir, exist_ok=True) + docx_path = os.path.join(out_dir, f"loss-run-{year}-valued-{issued[:7]}.docx") + d.save(docx_path) + subprocess.run( + ["soffice", "--headless", "--convert-to", "pdf", "--outdir", out_dir, + docx_path], + check=False, capture_output=True, timeout=180, + ) + os.remove(docx_path) + + +def main() -> None: + rng = random.Random(SEED) + claims = ANCHOR_CLAIMS + build_filler(rng) + + write_loss_run(f"{OUT}/loss-runs/loss-run-2021-valued-2024-01.csv", + 2021, claims, "2024-01-15", reissue=False) + write_loss_run(f"{OUT}/loss-runs/loss-run-2022-valued-2024-01.csv", + 2022, claims, "2024-01-15", reissue=False) + write_loss_run(f"{OUT}/loss-runs/loss-run-2023-valued-2024-01-ORIGINAL.csv", + 2023, claims, "2024-01-15", reissue=False, + footer_error=1_500.00) + write_loss_run(f"{OUT}/loss-runs/loss-run-2023-valued-2026-01-REISSUE.csv", + 2023, claims, "2026-01-09", reissue=True) + write_loss_run(f"{OUT}/loss-runs/loss-run-2024-valued-2026-01.csv", + 2024, claims, "2026-01-09", reissue=True) + + write_adjuster_notes() + write_application() + write_broker_email() + write_checklist() + write_pdf_loss_run(2022, claims, "2024-01-15", f"{OUT}/loss-runs") + + print(f"corpus written to {OUT}/ ({len(claims)} claims across 4 policy years)") + + +if __name__ == "__main__": + main() diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/adjuster-notes/KM-2022-0311-adjuster-note.docx b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/adjuster-notes/KM-2022-0311-adjuster-note.docx new file mode 100644 index 00000000..1d571d6c Binary files /dev/null and b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/adjuster-notes/KM-2022-0311-adjuster-note.docx differ diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/adjuster-notes/KM-2023-0417-adjuster-note.docx b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/adjuster-notes/KM-2023-0417-adjuster-note.docx new file mode 100644 index 00000000..429133c7 Binary files /dev/null and b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/adjuster-notes/KM-2023-0417-adjuster-note.docx differ diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/adjuster-notes/KM-2023-0588-adjuster-note.docx b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/adjuster-notes/KM-2023-0588-adjuster-note.docx new file mode 100644 index 00000000..449efe30 Binary files /dev/null and b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/adjuster-notes/KM-2023-0588-adjuster-note.docx differ diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/application/meridian-renewal-application-2026.docx b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/application/meridian-renewal-application-2026.docx new file mode 100644 index 00000000..98410776 Binary files /dev/null and b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/application/meridian-renewal-application-2026.docx differ diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/correspondence/ardent-cover-note-2026-01-28.docx b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/correspondence/ardent-cover-note-2026-01-28.docx new file mode 100644 index 00000000..3a7bd284 Binary files /dev/null and b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/correspondence/ardent-cover-note-2026-01-28.docx differ diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/loss-runs/loss-run-2021-valued-2024-01.csv b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/loss-runs/loss-run-2021-valued-2024-01.csv new file mode 100644 index 00000000..2e8430fb --- /dev/null +++ b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/loss-runs/loss-run-2021-valued-2024-01.csv @@ -0,0 +1,24 @@ +Kestrel Mutual Insurance Company - Loss Run Report +"Insured: Meridian Cold Chain Logistics, Inc." +FEIN: 88-0143927 +Policy Period: 2021-04-01 to 2022-04-01 +Valuation Date: 2024-01-15 +Report Type: Original + +Claim Number,Policy Number,Policy Year,Coverage,Date of Loss,Date Reported,Status,Claimant,Paid,Reserve,Incurred,Description +KM-2021-0225,KM-CA-2021-88417,2021,Workers Compensation,2021-04-13,2021-04-13,Closed,Employee,"15,001.60",0.00,"15,001.60","Repetitive strain, order picking" +KM-2021-0258,KM-CA-2021-88417,2021,General Liability,2021-05-27,2021-05-28,Closed,Third party,"19,941.90",0.00,"19,941.90",Damage to customer dock leveller +KM-2021-0232,KM-CA-2021-88417,2021,Workers Compensation,2021-06-07,2021-06-08,Open,Employee,"6,185.43","9,757.81","15,943.24","Repetitive strain, order picking" +KM-2021-0219,KM-CA-2021-88417,2021,Auto Physical Damage,2021-08-03,2021-08-04,Closed,First party,"27,450.00",0.00,"27,450.00","Reefer trailer sidewall damage, low bridge strike, Route 6." +KM-2021-0224,KM-CA-2021-88417,2021,Auto Physical Damage,2021-08-03,2021-08-11,Closed,First party,"27,450.00",0.00,"27,450.00",Trailer damage from bridge strike on Route 6 - unit 4471. +KM-2021-0218,KM-CA-2021-88417,2021,Auto Physical Damage,2021-11-01,2021-11-02,Closed,First party,"3,316.72",0.00,"3,316.72","Tyre blowout, wheel arch damage" +KM-2021-0235,KM-CA-2021-88417,2021,Auto Liability,2021-11-09,2021-11-15,Closed,Third party,"4,841.71",0.00,"4,841.71",Backing incident at customer dock +KM-2021-0252,KM-CA-2021-88417,2021,Property,2021-11-22,2021-11-28,Open,First party,37.13,587.27,624.40,"Freezer motor burnout, electrical surge" +KM-2021-0333,KM-CA-2021-88417,2021,General Liability,2021-12-02,2021-12-15,Open,Third party,0.00,"96,000.00","96,000.00","Alleged product contamination, retailer recall costs claimed." +KM-2021-0201,KM-CA-2021-88417,2021,General Liability,2022-01-27,2022-01-28,Closed,Third party,"2,965.48",0.00,"2,965.48",Damage to customer dock leveller +KM-2021-0209,KM-CA-2021-88417,2021,General Liability,2022-02-20,2022-02-26,Closed,Third party,"19,921.04",0.00,"19,921.04",Minor property damage at delivery site +KM-2021-0243,KM-CA-2021-88417,2021,Auto Liability,2022-03-06,2022-03-09,Closed,Third party,"3,824.26",0.00,"3,824.26","Sideswipe during lane change, property damage only" + +,,,,,,,TOTAL INCURRED,,,"237,280.35", +,,,,,,,CLAIM COUNT,,,12, +,,,,,,,EARNED PREMIUM,,,"412,000.00", diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/loss-runs/loss-run-2022-valued-2024-01.csv b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/loss-runs/loss-run-2022-valued-2024-01.csv new file mode 100644 index 00000000..a74c06b4 --- /dev/null +++ b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/loss-runs/loss-run-2022-valued-2024-01.csv @@ -0,0 +1,24 @@ +Kestrel Mutual Insurance Company - Loss Run Report +"Insured: Meridian Cold Chain Logistics, Inc." +FEIN: 88-0143927 +Policy Period: 2022-04-01 to 2023-04-01 +Valuation Date: 2024-01-15 +Report Type: Original + +Claim Number,Policy Number,Policy Year,Coverage,Date of Loss,Date Reported,Status,Claimant,Paid,Reserve,Incurred,Description +KM-2022-0246,KM-CA-2022-88417,2022,General Liability,2022-04-22,2022-04-24,Closed,Third party,"3,758.58",0.00,"3,758.58",Minor property damage at delivery site +KM-2022-0267,KM-CA-2022-88417,2022,Auto Physical Damage,2022-06-19,2022-06-22,Closed,First party,"17,714.07",0.00,"17,714.07","Tractor door damage, parking lot" +KM-2022-0232,KM-CA-2022-88417,2022,Auto Physical Damage,2022-08-01,2022-08-09,Closed,First party,"3,307.82",0.00,"3,307.82","Tractor door damage, parking lot" +KM-2022-0214,KM-CA-2022-88417,2022,Workers Compensation,2022-08-02,2022-08-06,Closed,Employee,"17,289.63",0.00,"17,289.63","Repetitive strain, order picking" +KM-2022-0237,KM-CA-2022-88417,2022,Auto Physical Damage,2022-08-17,2022-08-25,Open,First party,"8,535.28","15,191.06","23,726.34","Tyre blowout, wheel arch damage" +KM-2022-0311,KM-CA-2022-88417,2022,Workers Compensation,2022-09-19,2022-09-19,Closed,Employee,"41,280.00",0.00,"41,280.00","Warehouse associate lumbar strain, pallet jack incident." +KM-2022-0258,KM-CA-2022-88417,2022,Auto Physical Damage,2023-01-19,2023-01-26,Open,First party,666.02,"1,913.46","2,579.48","Tractor door damage, parking lot" +KM-2022-0252,KM-CA-2022-88417,2022,Property,2023-02-05,2023-02-10,Open,First party,749.80,"2,476.00","3,225.80","Roof leak, Bakersfield warehouse" +KM-2022-0203,KM-CA-2022-88417,2022,Auto Physical Damage,2023-02-15,2023-02-22,Closed,First party,"14,349.16",0.00,"14,349.16","Tractor door damage, parking lot" +KM-2022-0209,KM-CA-2022-88417,2022,Auto Liability,2023-02-15,2023-02-15,Closed,Third party,"34,851.06",0.00,"34,851.06","Sideswipe during lane change, property damage only" +KM-2022-0224,KM-CA-2022-88417,2022,Workers Compensation,2023-02-20,2023-02-25,Closed,Employee,"3,712.44",0.00,"3,712.44","Slip on wet dock plate, contusion" +KM-2022-0402,KM-CA-2022-88417,2022,General Liability,2023-04-11,2023-04-19,Closed,Third party,"18,900.00",0.00,"18,900.00","Slip and fall, visitor at the Fresno cross-dock." + +,,,,,,,TOTAL INCURRED,,,"184,694.38", +,,,,,,,CLAIM COUNT,,,12, +,,,,,,,EARNED PREMIUM,,,"448,500.00", diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/loss-runs/loss-run-2022-valued-2024-01.pdf b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/loss-runs/loss-run-2022-valued-2024-01.pdf new file mode 100644 index 00000000..adf99987 Binary files /dev/null and b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/loss-runs/loss-run-2022-valued-2024-01.pdf differ diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/loss-runs/loss-run-2023-valued-2024-01-ORIGINAL.csv b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/loss-runs/loss-run-2023-valued-2024-01-ORIGINAL.csv new file mode 100644 index 00000000..72eb4529 --- /dev/null +++ b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/loss-runs/loss-run-2023-valued-2024-01-ORIGINAL.csv @@ -0,0 +1,21 @@ +Kestrel Mutual Insurance Company - Loss Run Report +"Insured: Meridian Cold Chain Logistics, Inc." +FEIN: 88-0143927 +Policy Period: 2023-04-01 to 2024-04-01 +Valuation Date: 2024-01-15 +Report Type: Original + +Claim Number,Policy Number,Policy Year,Coverage,Date of Loss,Date Reported,Status,Claimant,Paid,Reserve,Incurred,Description +KM-2023-0221,KM-CA-2023-88417,2023,Workers Compensation,2023-04-18,2023-04-24,Closed,Employee,"3,370.06",0.00,"3,370.06","Repetitive strain, order picking" +KM-2023-0217,KM-CA-2023-88417,2023,General Liability,2023-06-15,2023-06-22,Closed,Third party,"53,536.94",0.00,"53,536.94",Minor property damage at delivery site +KM-2023-0204,KM-CA-2023-88417,2023,Auto Liability,2023-07-20,2023-07-24,Open,Third party,"2,909.95","5,026.84","7,936.79",Backing incident at customer dock +KM-2023-0250,KM-CA-2023-88417,2023,Auto Liability,2023-09-21,2023-09-28,Closed,Third party,"61,283.59",0.00,"61,283.59","Sideswipe during lane change, property damage only" +KM-2023-0417,KM-CA-2023-88417,2023,Auto Liability,2023-11-08,2023-11-09,Open,Third party,"8,150.00","36,850.00","45,000.00","Tractor-trailer jackknife on I-80, two vehicles involved, bodily injury alleged by both occupants of the second vehicle." +KM-2023-0209,KM-CA-2023-88417,2023,General Liability,2023-11-12,2023-11-19,Open,Third party,754.74,"3,941.25","4,695.99",Damage to customer dock leveller +KM-2023-0245,KM-CA-2023-88417,2023,Auto Physical Damage,2023-12-06,2023-12-07,Open,First party,"6,651.70","37,366.31","44,018.01","Windshield replacement, road debris" +KM-2023-0237,KM-CA-2023-88417,2023,Workers Compensation,2024-01-22,2024-01-28,Open,Employee,257.11,"1,905.54","2,162.65","Hand laceration, box cutter" +KM-2023-0230,KM-CA-2023-88417,2023,Workers Compensation,2024-02-17,2024-02-22,Open,Employee,"11,619.63","52,468.78","64,088.41","Hand laceration, box cutter" + +,,,,,,,TOTAL INCURRED,,,"287,592.44", +,,,,,,,CLAIM COUNT,,,9, +,,,,,,,EARNED PREMIUM,,,"501,200.00", diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/loss-runs/loss-run-2023-valued-2026-01-REISSUE.csv b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/loss-runs/loss-run-2023-valued-2026-01-REISSUE.csv new file mode 100644 index 00000000..e7782001 --- /dev/null +++ b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/loss-runs/loss-run-2023-valued-2026-01-REISSUE.csv @@ -0,0 +1,22 @@ +Kestrel Mutual Insurance Company - Loss Run Report +"Insured: Meridian Cold Chain Logistics, Inc." +FEIN: 88-0143927 +Policy Period: 2023-04-01 to 2024-04-01 +Valuation Date: 2026-01-09 +Report Type: REISSUE - supersedes prior + +Claim Number,Policy Number,Policy Year,Coverage,Date of Loss,Date Reported,Status,Claimant,Paid,Reserve,Incurred,Description +KM-2023-0221,KM-CA-2023-88417,2023,Workers Compensation,2023-04-18,2023-04-24,Closed,Employee,"3,370.06",0.00,"3,370.06","Repetitive strain, order picking" +KM-2023-0217,KM-CA-2023-88417,2023,General Liability,2023-06-15,2023-06-22,Closed,Third party,"53,536.94",0.00,"53,536.94",Minor property damage at delivery site +KM-2023-0204,KM-CA-2023-88417,2023,Auto Liability,2023-07-20,2023-07-24,Open,Third party,"2,909.95","5,026.84","7,936.79",Backing incident at customer dock +KM-2023-0250,KM-CA-2023-88417,2023,Auto Liability,2023-09-21,2023-09-28,Closed,Third party,"61,283.59",0.00,"61,283.59","Sideswipe during lane change, property damage only" +KM-2023-0417,KM-CA-2023-88417,2023,Auto Liability,2023-11-08,2023-11-09,Open,Third party,"12,400.00","170,000.00","182,400.00","Tractor-trailer jackknife on I-80, two vehicles involved, bodily injury alleged by both occupants of the second vehicle." +KM-2023-0209,KM-CA-2023-88417,2023,General Liability,2023-11-12,2023-11-19,Open,Third party,754.74,"3,941.25","4,695.99",Damage to customer dock leveller +KM-2023-0245,KM-CA-2023-88417,2023,Auto Physical Damage,2023-12-06,2023-12-07,Open,First party,"6,651.70","37,366.31","44,018.01","Windshield replacement, road debris" +KM-2023-0237,KM-CA-2023-88417,2023,Workers Compensation,2024-01-22,2024-01-28,Open,Employee,257.11,"1,905.54","2,162.65","Hand laceration, box cutter" +KM-2023-0230,KM-CA-2023-88417,2023,Workers Compensation,2024-02-17,2024-02-22,Open,Employee,"11,619.63","52,468.78","64,088.41","Hand laceration, box cutter" +KM-2023-0588,KM-CA-2023-88417,2023,Motor Truck Cargo,2024-02-27,2024-06-14,Open,Shipper,0.00,"224,500.00","224,500.00","Refrigeration unit failure in transit, full load of pharmaceuticals condemned on arrival." + +,,,,,,,TOTAL INCURRED,,,"647,992.44", +,,,,,,,CLAIM COUNT,,,10, +,,,,,,,EARNED PREMIUM,,,"501,200.00", diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/loss-runs/loss-run-2024-valued-2026-01.csv b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/loss-runs/loss-run-2024-valued-2026-01.csv new file mode 100644 index 00000000..49eeb4ce --- /dev/null +++ b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/loss-runs/loss-run-2024-valued-2026-01.csv @@ -0,0 +1,19 @@ +Kestrel Mutual Insurance Company - Loss Run Report +"Insured: Meridian Cold Chain Logistics, Inc." +FEIN: 88-0143927 +Policy Period: 2024-04-01 to 2025-04-01 +Valuation Date: 2026-01-09 +Report Type: REISSUE - supersedes prior + +Claim Number,Policy Number,Policy Year,Coverage,Date of Loss,Date Reported,Status,Claimant,Paid,Reserve,Incurred,Description +KM-2024-0231,KM-CA-2024-88417,2024,Auto Liability,2024-06-19,2024-06-28,Closed,Third party,526.85,0.00,526.85,"Minor collision in yard, no injuries" +KM-2024-0106,KM-CA-2024-88417,2024,Auto Liability,2024-06-22,2024-06-23,Open,Third party,"31,000.00","118,000.00","149,000.00","Rear-end collision, company tractor, claimant hospitalised." +KM-2024-0202,KM-CA-2024-88417,2024,Auto Liability,2024-10-06,2024-10-11,Closed,Third party,"2,415.83",0.00,"2,415.83","Minor collision in yard, no injuries" +KM-2024-0223,KM-CA-2024-88417,2024,Auto Physical Damage,2024-10-12,2024-10-18,Closed,First party,"16,697.56",0.00,"16,697.56","Tyre blowout, wheel arch damage" +KM-2024-0238,KM-CA-2024-88417,2024,Auto Physical Damage,2024-11-22,2024-11-28,Closed,First party,"2,094.59",0.00,"2,094.59","Windshield replacement, road debris" +KM-2024-0208,KM-CA-2024-88417,2024,Auto Physical Damage,2025-02-28,2025-02-28,Closed,First party,"3,364.47",0.00,"3,364.47","Tractor door damage, parking lot" +KM-2024-0218,KM-CA-2024-88417,2024,Property,2025-03-05,2025-03-14,Open,First party,"3,537.94","11,194.31","14,732.25","Freezer motor burnout, electrical surge" + +,,,,,,,TOTAL INCURRED,,,"188,831.55", +,,,,,,,CLAIM COUNT,,,7, +,,,,,,,EARNED PREMIUM,,,"566,900.00", diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/rules/underwriting-checklist.docx b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/rules/underwriting-checklist.docx new file mode 100644 index 00000000..d9904626 Binary files /dev/null and b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/corpus/out/rules/underwriting-checklist.docx differ diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/lossrun/__init__.py b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/lossrun/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/lossrun/model.py b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/lossrun/model.py new file mode 100644 index 00000000..484ed84c --- /dev/null +++ b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/lossrun/model.py @@ -0,0 +1,195 @@ +"""Domain model. + +One rule governs this file and the two that follow it: **figures are parsed, +never inferred.** Claim counts, incurred totals and loss ratios in the review +document are arithmetic over rows read out of the loss runs. No model is asked +what a number is, because an underwriter cannot act on a total that might be a +plausible guess. + +What the model *is* asked is the thing arithmetic cannot do: read the adjuster +notes, the application and the broker correspondence, and surface where the +prose disagrees with the figures. That division is the whole architecture. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field, asdict +from datetime import date +from typing import Any + +OPEN_STATUSES = {"open", "reopened"} + + +@dataclass(frozen=True) +class Source: + """Where a fact came from. Every figure in the deliverable carries one.""" + + document: str + valuation_date: str | None = None + row: int | None = None + is_reissue: bool = False + + def cite(self) -> str: + bits = [self.document] + if self.valuation_date: + bits.append(f"valued {self.valuation_date}") + if self.row is not None: + bits.append(f"row {self.row}") + return ", ".join(bits) + + +@dataclass +class Claim: + claim_no: str + policy_year: int + policy_no: str + coverage: str + date_of_loss: str + date_reported: str + status: str + claimant: str + paid: float + reserve: float + incurred: float + description: str + source: Source + + @property + def is_open(self) -> bool: + return self.status.strip().lower() in OPEN_STATUSES + + def months_open(self, as_of: date) -> int: + try: + dol = date.fromisoformat(self.date_of_loss) + except ValueError: + return 0 + return (as_of.year - dol.year) * 12 + as_of.month - dol.month + + def to_dict(self) -> dict[str, Any]: + d = asdict(self) + d["source"] = self.source.cite() + return d + + +@dataclass +class LossRun: + """One loss run document, as read.""" + + path: str + policy_year: int + policy_no: str + period_start: str + period_end: str + valuation_date: str + is_reissue: bool + claims: list[Claim] = field(default_factory=list) + stated_total: float | None = None + stated_count: int | None = None + earned_premium: float | None = None + + @property + def summed_total(self) -> float: + return round(sum(c.incurred for c in self.claims), 2) + + @property + def label(self) -> str: + kind = "reissue" if self.is_reissue else "original" + return f"PY{self.policy_year} {kind} valued {self.valuation_date}" + + +@dataclass +class Conflict: + """Two sources say different things. Surfaced, never resolved silently.""" + + kind: str + subject: str + left: str + right: str + detail: str + consequence: str = "" + resolution: str | None = None # set only by a human decision + + def to_dict(self) -> dict[str, Any]: + return asdict(self) + + +@dataclass +class Finding: + """A rule fired.""" + + rule: str + title: str + subject: str + detail: str + source: str + + def to_dict(self) -> dict[str, Any]: + return asdict(self) + + +@dataclass +class YearSummary: + policy_year: int + policy_no: str + claim_count: int + open_count: int + total_paid: float + total_reserve: float + total_incurred: float + open_incurred: float + earned_premium: float | None + source: str + superseded_by: str | None = None + + @property + def loss_ratio(self) -> float | None: + if not self.earned_premium: + return None + return round(self.total_incurred / self.earned_premium, 4) + + def to_dict(self) -> dict[str, Any]: + d = asdict(self) + d["loss_ratio"] = self.loss_ratio + return d + + +@dataclass +class Summary: + insured: str + years: list[YearSummary] = field(default_factory=list) + open_claims: list[Claim] = field(default_factory=list) + conflicts: list[Conflict] = field(default_factory=list) + findings: list[Finding] = field(default_factory=list) + documents_read: list[str] = field(default_factory=list) + documents_superseded: list[str] = field(default_factory=list) + unverified: list[str] = field(default_factory=list) + """Figures the app could not independently check. Named, never smoothed.""" + + @property + def total_incurred(self) -> float: + return round(sum(y.total_incurred for y in self.years), 2) + + @property + def total_claims(self) -> int: + return sum(y.claim_count for y in self.years) + + @property + def total_open(self) -> int: + return sum(y.open_count for y in self.years) + + def to_dict(self) -> dict[str, Any]: + return { + "insured": self.insured, + "totals": { + "claims": self.total_claims, + "open": self.total_open, + "incurred": self.total_incurred, + }, + "years": [y.to_dict() for y in self.years], + "open_claims": [c.to_dict() for c in self.open_claims], + "conflicts": [c.to_dict() for c in self.conflicts], + "findings": [f.to_dict() for f in self.findings], + "documents_read": self.documents_read, + "documents_superseded": self.documents_superseded, + "unverified": self.unverified, + } diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/lossrun/parse.py b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/lossrun/parse.py new file mode 100644 index 00000000..3aff4dac --- /dev/null +++ b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/lossrun/parse.py @@ -0,0 +1,259 @@ +"""Read loss runs out of whatever format they arrive in. + +Carriers issue loss runs as CSV, as PDF, and occasionally as a Word table. All +three land here and produce the same rows. The parser also checks each document +against itself: a loss run states its own total and claim count, and those are +compared against the sum of the rows rather than trusted. Real loss runs +disagree with themselves more often than anyone likes. +""" + +from __future__ import annotations + +import csv +import os +import re +import zipfile + +from .model import Claim, LossRun, Source + +MONEY = re.compile(r"^\(?\$?\s*-?[\d,]*\.?\d+\)?$") + + +def money(text: str) -> float: + """Parse a figure the way an accountant wrote it. + + Handles $1,204,880.00 and (12,400.00) for negatives and an em dash for nil. + Returns 0.0 for a nil marker rather than raising, because a blank and a + dash mean the same thing on a loss run and neither is an error. + """ + if text is None: + return 0.0 + t = str(text).strip() + if t in ("", "-", "\u2013", "\u2014", "N/A", "n/a"): + return 0.0 + neg = t.startswith("(") and t.endswith(")") + t = t.strip("()").replace("$", "").replace(",", "").strip() + try: + v = float(t) + except ValueError as exc: + raise ValueError(f"not a figure: {text!r}") from exc + return -v if neg else v + + +def _meta(lines: list[str]) -> dict: + """Pull the preamble a loss run carries above its column headers.""" + out: dict = {} + joined = "\n".join(lines) + m = re.search(r"Policy Period:\s*(\d{4}-\d{2}-\d{2})\s*to\s*(\d{4}-\d{2}-\d{2})", joined) + if m: + out["period_start"], out["period_end"] = m.group(1), m.group(2) + m = re.search(r"Valuation Date:\s*(\d{4}-\d{2}-\d{2})", joined) + if m: + out["valuation_date"] = m.group(1) + m = re.search(r"Insured:\s*(.+)", joined) + if m: + out["insured"] = m.group(1).strip() + out["is_reissue"] = "REISSUE" in joined.upper() + return out + + +def parse_csv(path: str) -> LossRun: + rows = list(csv.reader(open(path, encoding="utf-8-sig"))) + preamble = ["".join(r) for r in rows[:10]] + meta = _meta(preamble) + + hdr_idx = next((i for i, r in enumerate(rows) if r and r[0] == "Claim Number"), None) + if hdr_idx is None: + raise ValueError(f"{os.path.basename(path)}: no claim header row found") + hdr = rows[hdr_idx] + + claims: list[Claim] = [] + stated_total = stated_count = premium = None + policy_year = policy_no = None + + for i, r in enumerate(rows[hdr_idx + 1:], start=hdr_idx + 2): + if not r or not any(r): + continue + rec = dict(zip(hdr, r)) + if rec.get("Claimant") == "TOTAL INCURRED" or (len(r) > 7 and r[7] == "TOTAL INCURRED"): + stated_total = money(r[10]) + continue + if len(r) > 7 and r[7] == "CLAIM COUNT": + stated_count = int(money(r[10])) + continue + if len(r) > 7 and r[7] == "EARNED PREMIUM": + premium = money(r[10]) + continue + if not rec.get("Claim Number"): + continue + + policy_year = int(rec["Policy Year"]) + policy_no = rec["Policy Number"] + src = Source(os.path.basename(path), meta.get("valuation_date"), i, + meta.get("is_reissue", False)) + paid, reserve = money(rec["Paid"]), money(rec["Reserve"]) + claims.append(Claim( + claim_no=rec["Claim Number"], policy_year=policy_year, + policy_no=policy_no, coverage=rec["Coverage"], + date_of_loss=rec["Date of Loss"], date_reported=rec["Date Reported"], + status=rec["Status"], claimant=rec["Claimant"], + paid=paid, reserve=reserve, incurred=money(rec["Incurred"]), + description=rec.get("Description", ""), source=src, + )) + + if policy_year is None: + raise ValueError(f"{os.path.basename(path)}: no claim rows") + + return LossRun( + path=path, policy_year=policy_year, policy_no=policy_no, + period_start=meta.get("period_start", ""), + period_end=meta.get("period_end", ""), + valuation_date=meta.get("valuation_date", ""), + is_reissue=meta.get("is_reissue", False), + claims=claims, stated_total=stated_total, stated_count=stated_count, + earned_premium=premium, + ) + + +def _pdf_text(path: str) -> str: + """Text of a PDF, laid out. + + pdfplumber is the dependency and is pure Python, so this works the same on + Windows, macOS and Linux. An earlier version shelled out to ``pdftotext``, + which is a Unix binary: the code ran fine on the machine it was written on + and died with a bare WinError on the machine it was used on. The fallback + below is kept for environments that have the binary and not the library, + but nothing depends on it. + """ + try: + import pdfplumber + except ImportError: + pass + else: + with pdfplumber.open(path) as pdf: + return "\n".join((page.extract_text() or "") for page in pdf.pages) + + from shutil import which + from subprocess import run as _run + if which("pdftotext"): + return _run(["pdftotext", "-layout", path, "-"], + capture_output=True, text=True).stdout + + raise RuntimeError( + f"cannot read {os.path.basename(path)}: no PDF reader available. " + f"Install the dependencies with `pip install -r requirements.txt` " + f"(pdfplumber), or put pdftotext on the PATH. CSV loss runs are " + f"unaffected and the rest of the pipeline still runs." + ) + + +def parse_pdf(path: str) -> LossRun: + """PDF loss runs are read from the laid-out text of a carrier report. + + Deliberately conservative: a row is accepted only when every field it needs + is present and every figure parses. Rows that look like claim lines but do + not fully parse are counted and reported as unreadable rather than guessed + at, because a half-read row is worse than a missing one on a document that + feeds a coverage decision. + """ + text = _pdf_text(path) + lines = [l.rstrip() for l in text.splitlines() if l.strip()] + meta = _meta(lines[:14]) + + row_re = re.compile( + r"^(?P[A-Z]{2}-\d{4}-\d{4})\s+" + r"(?P.+?)\s+" + r"(?P\d{4}-\d{2}-\d{2})\s+" + r"(?P\d{4}-\d{2}-\d{2})\s+" + r"(?POpen|Closed|Reopened)\s+" + r"(?P[\d,().$-]+)\s+" + r"(?P[\d,().$-]+)\s+" + r"(?P[\d,().$-]+)\s*$") + looks_like_claim = re.compile(r"^[A-Z]{2}-\d{4}-\d{4}\b") + + claims: list[Claim] = [] + unreadable: list[str] = [] + stated_total = stated_count = premium = None + + for i, line in enumerate(lines, start=1): + stripped = line.strip() + m = row_re.match(stripped) + if m: + try: + paid = money(m.group("paid")) + reserve = money(m.group("res")) + incurred = money(m.group("inc")) + except ValueError: + unreadable.append(m.group("claim")) + continue + claims.append(Claim( + claim_no=m.group("claim"), + policy_year=int(m.group("dol")[:4]) if m.group("dol") else 0, + policy_no="", coverage=m.group("cov").strip(), + date_of_loss=m.group("dol"), date_reported=m.group("drep"), + status=m.group("status"), claimant="", + paid=paid, reserve=reserve, incurred=incurred, description="", + source=Source(os.path.basename(path), meta.get("valuation_date"), + i, meta.get("is_reissue", False)), + )) + continue + if looks_like_claim.match(stripped): + unreadable.append(stripped.split()[0]) + continue + m2 = re.match(r"TOTAL INCURRED:\s*(.+)", stripped) + if m2: + stated_total = money(m2.group(1)) + m3 = re.match(r"CLAIM COUNT:\s*(\d+)", stripped) + if m3: + stated_count = int(m3.group(1)) + m4 = re.match(r"EARNED PREMIUM:\s*(.+)", stripped) + if m4: + premium = money(m4.group(1)) + + if not claims: + raise ValueError(f"{os.path.basename(path)}: no readable claim rows in PDF") + + # A PDF loss run states its policy period but not a policy number per row. + # Derive the policy year from the period rather than from a date of loss, + # which can legitimately fall outside it. + year = int(meta["period_start"][:4]) if meta.get("period_start") else claims[0].policy_year + for c in claims: + c.policy_year = year + + run = LossRun( + path=path, policy_year=year, policy_no="", + period_start=meta.get("period_start", ""), + period_end=meta.get("period_end", ""), + valuation_date=meta.get("valuation_date", ""), + is_reissue=meta.get("is_reissue", False), claims=claims, + stated_total=stated_total, stated_count=stated_count, + earned_premium=premium, + ) + run.unreadable_rows = unreadable + return run + + +def docx_text(path: str) -> str: + """Plain text of a .docx, paragraphs and table cells, in document order.""" + with zipfile.ZipFile(path) as z: + xml = z.read("word/document.xml").decode("utf-8") + xml = re.sub(r"", "\n", xml) + xml = re.sub(r"]*/>", "\t", xml) + text = re.sub(r"<[^>]+>", "", xml) + return re.sub(r"\n{3,}", "\n\n", text) + + +def load(path: str) -> LossRun: + ext = os.path.splitext(path)[1].lower() + if ext == ".csv": + return parse_csv(path) + if ext == ".pdf": + return parse_pdf(path) + raise ValueError(f"{path}: not a loss run format this build reads") + + +def load_all(paths: list[str]) -> list[LossRun]: + runs = [] + for p in sorted(paths): + runs.append(load(p)) + return runs diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/lossrun/review.py b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/lossrun/review.py new file mode 100644 index 00000000..b073bee6 --- /dev/null +++ b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/lossrun/review.py @@ -0,0 +1,316 @@ +"""Build the review document, then put it through SuperDocs. + +Every figure in the output is placed here, by this file, from parsed rows. The +model is asked only to write the narrative around figures already fixed, and is +told so in the instruction. That is the difference between a summary an +underwriter can act on and one they have to re-check. +""" + +from __future__ import annotations + +import argparse +import datetime +import glob +import html +import json +import os +import sys + +from . import parse, summarize +from .model import Summary +from .superdocs import (ApiError, BudgetExceeded, FakeClient, Ledger, + SearchUnavailable, SuperDocsClient) + +PLACEHOLDER = "[[CORRESPONDENCE]]" + +# What the prose documents are asked. Arithmetic over the loss runs cannot find +# any of these: each lives in a sentence in a note, an application or an email. +PROSE_QUERIES = [ + "any claim reopened after being shown as closed, and the reserve " + "re-established for it", + "any reserve increased after a suit was filed, and the stated reserve " + "history", + "any statement that a loss run was reissued or superseded, and which " + "version should be relied on", +] + + +def money(v: float) -> str: + return f"{v:,.2f}" + + +def build_html(s: Summary, prose: list[dict], as_of: datetime.date) -> str: + p: list[str] = [] + e = html.escape + + p.append(f"

Loss run review — {e(s.insured)}

") + p.append(f"

Prepared {as_of.isoformat()}. Figures are read from the loss " + f"runs listed at the foot of this document and totalled " + f"arithmetically. No figure in this review is model-generated.

") + + p.append("

Summary by policy year

") + p.append("" + "" + "") + for y in s.years: + lr = f"{y.loss_ratio:.1%}" if y.loss_ratio is not None else "not available" + flag = " ***" if y.loss_ratio and y.loss_ratio > 0.60 else "" + p.append( + f"" + f"" + f"" + f"" + f"") + p.append(f"" + f"" + f"") + p.append("
Policy yearPolicyClaimsOpenPaidReserveIncurredLoss ratioSource
{y.policy_year}{e(y.policy_no)}{y.claim_count}{y.open_count}{money(y.total_paid)}{money(y.total_reserve)}{money(y.total_incurred)}{lr}{flag}{e(y.source)}
Total{s.total_claims}{s.total_open}{money(s.total_incurred)}
") + + p.append("

Open claims

") + if not s.open_claims: + p.append("

No claim is open across the policy years reviewed.

") + else: + p.append(f"

{len(s.open_claims)} claims remain open, carrying " + f"{money(sum(c.incurred for c in s.open_claims))} incurred. " + f"Listed separately from closed claims because reserves on " + f"open claims may still develop.

") + p.append("" + "" + "") + for c in s.open_claims: + p.append(f"" + f"" + f"" + f"" + f"") + p.append("
ClaimPYCoverageDate of lossPaidReserveIncurredMonths open
{e(c.claim_no)}{c.policy_year}{e(c.coverage)}{e(c.date_of_loss)}{money(c.paid)}{money(c.reserve)}{money(c.incurred)}{c.months_open(as_of)}
") + + p.append("

Conflicts between sources

") + if not s.conflicts: + p.append("

The sources agree. No conflict found.

") + else: + p.append("

Surfaced, not resolved. Each needs a decision before the " + "figures above are relied on.

") + for c in s.conflicts: + p.append(f"

{e(c.subject)} — {e(c.left)} versus " + f"{e(c.right)}. {e(c.detail)}" + + (f" {e(c.consequence)}" if c.consequence else "") + "

") + + p.append("

Checklist findings

") + if not s.findings: + p.append("

Every rule was applied and none fired. No findings.

") + else: + p.append("" + "") + for f in s.findings: + p.append(f"" + f"" + f"") + p.append("
RuleSubjectFindingSource
{e(f.rule)}{e(f.subject)}{e(f.title)}: {e(f.detail)}{e(f.source)}
") + + if prose is not None: + p.append("

From the correspondence and adjuster notes

") + p.append("

Read from the non-tabular documents attached to this " + "review. Quoted context only; no figure below is used in any " + "total above.

") + if prose: + for hit in prose[:6]: + snippet = " ".join(hit.get("text", "").split())[:320] + p.append(f"

{e(hit.get('document', ''))} — " + f"{e(snippet)}

") + else: + p.append(f"

{PLACEHOLDER}

") + + p.append("

What this review does not establish

") + items = list(s.unverified) + items.append("Figures are as stated by the carrier. This review checks each " + "loss run against itself and against the others; it cannot " + "confirm the carrier's own numbers.") + if s.documents_superseded: + items.append("Superseded documents were read and compared but are not " + "counted in any total: " + + ", ".join(s.documents_superseded) + ".") + p.append("
    " + "".join(f"
  • {e(i)}
    • " for i in items) + "
") + + p.append("

Sources

    " + + "".join(f"
  • {e(d)}
    • " for d in s.documents_read) + "
") + return "\n".join(p) + + +def build_instruction(queries: list[str]) -> str: + """One instruction, two edits, one operation. + + An earlier version asked for a JSON chat reply and looked for it in the + proposed-changes payload, which carries document edits and not chat text. + That was fighting the product: this is an editing agent, so the answer + belongs in the document. Asking it to fill a placeholder section is both + the supported path and half the cost. + """ + asks = "\n".join(f" - {q}" for q in queries) + return ( + f"Make exactly two edits to this document, grounded only in the files " + f"attached to this session.\n\n" + f"1. Replace the placeholder text {PLACEHOLDER} with a short list of " + f"what the attached adjuster notes, application and correspondence say " + f"about:\n{asks}\n" + f" Name the source document for each point and quote at most one " + f"sentence from it. Where the attached documents do not answer one of " + f"these, say so for that point rather than inferring an answer.\n\n" + f"2. Add an opening paragraph directly under the title, of at most four " + f"sentences, summarising this account for an underwriter who has not " + f"seen it.\n\n" + f"Every number, date, claim number and table cell in this document has " + f"been calculated from source records and is already correct: do not " + f"change, recalculate, round or restate any of them. If text you write " + f"would need a number, refer to the table instead of repeating it. Do " + f"not add a recommendation on whether to write the risk. Change nothing " + f"else." + ) + + +# Kept as a name for the tests that assert the figures are protected. +NARRATIVE_INSTRUCTION = build_instruction(PROSE_QUERIES) + + +def run(corpus_dir: str, out_dir: str, mode: str, max_ops: int, + as_of: datetime.date, skip_prose: bool = False) -> dict: + runs_paths = sorted(glob.glob(os.path.join(corpus_dir, "loss-runs", "*.csv"))) + runs_paths += sorted(glob.glob(os.path.join(corpus_dir, "loss-runs", "*.pdf"))) + if not runs_paths: + raise SystemExit(f"no loss runs found under {corpus_dir}/loss-runs/") + + runs, unreadable = [], [] + for path in runs_paths: + try: + runs.append(parse.load(path)) + except (ValueError, OSError) as exc: + unreadable.append(f"{os.path.basename(path)}: {exc}") + + # The same policy year may arrive as both CSV and PDF. Prefer the CSV, and + # use the PDF as a cross-check rather than a second opinion in the totals. + seen, deduped, crosschecks = {}, [], [] + for r in runs: + key = (r.policy_year, r.valuation_date, r.is_reissue) + if key in seen: + other = seen[key] + same = (len(r.claims) == len(other.claims) + and abs(r.summed_total - other.summed_total) < 0.01) + crosschecks.append( + f"{os.path.basename(r.path)} and {os.path.basename(other.path)} " + f"cover the same valuation and " + + ("agree." if same else + f"DISAGREE: {r.summed_total:,.2f} vs {other.summed_total:,.2f}.")) + continue + seen[key] = r + deduped.append(r) + + insured = "the insured" + app_path = glob.glob(os.path.join(corpus_dir, "application", "*.docx")) + app_text = parse.docx_text(app_path[0]) if app_path else "" + for r in deduped: + pass + + notes = {} + for p in glob.glob(os.path.join(corpus_dir, "adjuster-notes", "*.docx")): + base = os.path.basename(p) + notes[base.split("-adjuster")[0]] = parse.docx_text(p) + + s = summarize.summarise(deduped, insured) + s = summarize.examine(s, deduped, notes, app_text, as_of=as_of) + s.unverified.extend(crosschecks) + for u in unreadable: + s.unverified.append(f"Could not read {u}") + + ledger = Ledger(cap=max_ops) + corpus_text = dict(notes) + if app_text: + corpus_text["renewal application"] = app_text + for p in glob.glob(os.path.join(corpus_dir, "correspondence", "*.docx")): + corpus_text[os.path.basename(p)] = parse.docx_text(p) + + client = (SuperDocsClient(ledger) if mode == "live" + else FakeClient(ledger, corpus_text)) + + session = f"lossrun-{as_of.isoformat()}" + + # Uploads are free, and attaching the prose documents to the session is + # what makes cross-document retrieval possible at all. Do it before asking. + prose_docs = sorted( + p for p in glob.glob(os.path.join(corpus_dir, "**", "*.docx"), + recursive=True)) + for p in prose_docs: + try: + client.upload(p, session) + except (ApiError, OSError) as exc: + s.unverified.append(f"Could not attach {os.path.basename(p)}: {exc}") + + # None means the section is not wanted at all; [] means it is wanted and + # will be filled in the document by the edit below. + prose: list[dict] | None = None if skip_prose else [] + doc_html = build_html(s, prose, as_of) + instruction = (build_instruction(PROSE_QUERIES) if not skip_prose + else build_instruction([])) + os.makedirs(out_dir, exist_ok=True) + with open(os.path.join(out_dir, "review.html"), "w", encoding="utf-8") as fh: + fh.write(doc_html) + with open(os.path.join(out_dir, "summary.json"), "w", encoding="utf-8") as fh: + json.dump(s.to_dict(), fh, indent=2) + + exported = None + try: + job = client.edit(session, doc_html, instruction) + pending = job.get("pending") or [] + if pending: + client.approve(session, job["job_id"], pending) + else: + s.unverified.append( + "The narrative edit proposed no changes, so the correspondence " + "section still holds its placeholder and no opening paragraph " + "was added. Every figure above is unaffected.") + exported = client.export( + session, os.path.join(out_dir, "loss-run-review.docx")) + except BudgetExceeded as exc: + s.unverified.append(f"Stopped at the operation cap: {exc}") + except (ApiError, OSError) as exc: + s.unverified.append(f"Document assembly did not complete: {exc}") + + return {"summary": s, "ledger": ledger, "html": doc_html, + "exported": exported, "out_dir": out_dir} + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser(prog="lossrun") + ap.add_argument("--corpus", default="corpus/out") + ap.add_argument("--out", default="out") + ap.add_argument("--mode", choices=("dry", "live"), default="dry") + ap.add_argument("--max-ops", type=int, default=6, + help="hard cap; the run stops before exceeding it") + ap.add_argument("--as-of", default=None, + help="valuation date for ageing rules (YYYY-MM-DD)") + ap.add_argument("--skip-prose", action="store_true", + help="skip the search stage and spend nothing on it") + a = ap.parse_args(argv) + + as_of = (datetime.date.fromisoformat(a.as_of) if a.as_of + else datetime.date.today()) + r = run(a.corpus, a.out, a.mode, a.max_ops, as_of, a.skip_prose) + s, led = r["summary"], r["ledger"] + + print(f"mode={a.mode} documents read={len(s.documents_read)} " + f"superseded={len(s.documents_superseded)}") + for y in s.years: + lr = f"{y.loss_ratio:.1%}" if y.loss_ratio is not None else "n/a" + print(f" PY{y.policy_year} claims={y.claim_count:3d} " + f"open={y.open_count:2d} incurred={y.total_incurred:>13,.2f} " + f"LR={lr:>7}") + print(f" TOTAL claims={s.total_claims} open={s.total_open} " + f"incurred={s.total_incurred:,.2f}") + print(f" conflicts={len(s.conflicts)} findings={len(s.findings)} " + f"unverified={len(s.unverified)}") + print(f" operations spent={led.spent}/{led.cap} {led.by_stage}") + print(f" written to {r['out_dir']}/") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/lossrun/summarize.py b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/lossrun/summarize.py new file mode 100644 index 00000000..d800ef5d --- /dev/null +++ b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/lossrun/summarize.py @@ -0,0 +1,305 @@ +"""Group, total, and check. + +Three jobs, in order. + +**Supersede.** When two loss runs cover the same policy year, the later +valuation wins — but the difference is recorded as a conflict rather than +quietly overwritten. On a real account this is the whole game: reserves develop, +runs get reissued, and the same claim carries two very different numbers. Both +are correct as of their valuation date. Picking one silently is making an +underwriting decision without telling anyone. + +**Aggregate.** Claim counts and incurred totals by policy year, open claims +separated out. Pure arithmetic over parsed rows. + +**Examine.** The eight checklist rules. Each produces a finding or an explicit +pass, and a clean corpus is allowed to produce nothing at all. +""" + +from __future__ import annotations + +import os +import re +from datetime import date + +from .model import Claim, Conflict, Finding, LossRun, Summary, YearSummary + +SEVERITY_THRESHOLD = 100_000.00 +STALE_MONTHS = 36 +DORMANT_MONTHS = 24 +LOSS_RATIO_REFERRAL = 0.60 + + +# --------------------------------------------------------------------------- +# supersession +# --------------------------------------------------------------------------- + +def choose_authoritative(runs: list[LossRun]) -> tuple[dict[int, LossRun], list[LossRun], list[Conflict]]: + by_year: dict[int, list[LossRun]] = {} + for r in runs: + by_year.setdefault(r.policy_year, []).append(r) + + authoritative: dict[int, LossRun] = {} + superseded: list[LossRun] = [] + conflicts: list[Conflict] = [] + + for year, group in sorted(by_year.items()): + group.sort(key=lambda r: (r.valuation_date or "", r.is_reissue)) + winner = group[-1] + authoritative[year] = winner + for loser in group[:-1]: + superseded.append(loser) + conflicts.extend(_diff_runs(loser, winner)) + + return authoritative, superseded, conflicts + + +def _diff_runs(old: LossRun, new: LossRun) -> list[Conflict]: + out: list[Conflict] = [] + old_by = {c.claim_no: c for c in old.claims} + new_by = {c.claim_no: c for c in new.claims} + + for claim_no in sorted(set(old_by) | set(new_by)): + a, b = old_by.get(claim_no), new_by.get(claim_no) + if a is None: + out.append(Conflict( + kind="claim_added", + subject=claim_no, + left=f"absent from {old.label}", + right=f"{b.incurred:,.2f} incurred in {new.label}", + detail=(f"Reported {b.date_reported}, after the earlier run was " + f"valued {old.valuation_date}."), + consequence="Earlier totals for this policy year understate it.", + )) + continue + if b is None: + out.append(Conflict( + kind="claim_removed", subject=claim_no, + left=f"present in {old.label}", right=f"absent from {new.label}", + detail="Present on the superseded run and not on the later one.", + consequence="Query with the carrier before relying on either total.", + )) + continue + if abs(a.incurred - b.incurred) >= 0.01: + delta = b.incurred - a.incurred + out.append(Conflict( + kind="incurred_changed", subject=claim_no, + left=f"{a.incurred:,.2f} ({old.label})", + right=f"{b.incurred:,.2f} ({new.label})", + detail=(f"Moved {delta:+,.2f}. Paid {a.paid:,.2f} to " + f"{b.paid:,.2f}; reserve {a.reserve:,.2f} to " + f"{b.reserve:,.2f}."), + consequence="Both figures are correct at their valuation date.", + )) + if a.status != b.status: + out.append(Conflict( + kind="status_changed", subject=claim_no, + left=f"{a.status} ({old.label})", right=f"{b.status} ({new.label})", + detail="Claim status differs between valuations.", + )) + return out + + +# --------------------------------------------------------------------------- +# aggregation +# --------------------------------------------------------------------------- + +def summarise(runs: list[LossRun], insured: str) -> Summary: + authoritative, superseded, conflicts = choose_authoritative(runs) + s = Summary(insured=insured, conflicts=conflicts) + s.documents_read = [os.path.basename(r.path) for r in runs] + s.documents_superseded = [os.path.basename(r.path) for r in superseded] + + for year, run in sorted(authoritative.items()): + claims = run.claims + open_claims = [c for c in claims if c.is_open] + s.open_claims.extend(open_claims) + s.years.append(YearSummary( + policy_year=year, + policy_no=run.policy_no, + claim_count=len(claims), + open_count=len(open_claims), + total_paid=round(sum(c.paid for c in claims), 2), + total_reserve=round(sum(c.reserve for c in claims), 2), + total_incurred=round(sum(c.incurred for c in claims), 2), + open_incurred=round(sum(c.incurred for c in open_claims), 2), + earned_premium=run.earned_premium, + source=run.label, + )) + if run.earned_premium is None: + s.unverified.append( + f"PY{year}: no earned premium on the loss run, so no loss ratio " + f"could be calculated.") + + s.open_claims.sort(key=lambda c: -c.incurred) + return s + + +# --------------------------------------------------------------------------- +# the rules +# --------------------------------------------------------------------------- + +def examine(summary: Summary, runs: list[LossRun], notes: dict[str, str], + application_text: str = "", as_of: date | None = None) -> Summary: + as_of = as_of or date.today() + authoritative, superseded, _ = choose_authoritative(runs) + live = [c for r in authoritative.values() for c in r.claims] + f = summary.findings + + # R1 severity claims need an adjuster note + noted = set(notes) + for c in sorted(live, key=lambda c: -c.incurred): + if c.incurred >= SEVERITY_THRESHOLD and c.claim_no not in noted: + f.append(Finding("R1", "Severity claim without an adjuster note", + c.claim_no, + f"{c.incurred:,.2f} incurred, status {c.status}. " + f"No note in the file, so severity cannot be " + f"evaluated.", c.source.cite())) + + # R2 stale open claims / R6 dormant reserves + for c in live: + if not c.is_open: + continue + months = c.months_open(as_of) + if months > STALE_MONTHS: + f.append(Finding("R2", "Open more than 36 months", c.claim_no, + f"Open {months} months from date of loss " + f"{c.date_of_loss}. Incurred {c.incurred:,.2f}.", + c.source.cite())) + if c.paid == 0 and c.reserve > 0 and months > DORMANT_MONTHS: + f.append(Finding("R6", "Dormant reserve", c.claim_no, + f"Nothing paid against a reserve of " + f"{c.reserve:,.2f} after {months} months.", + c.source.cite())) + + # R3 loss ratio referral + for y in summary.years: + lr = y.loss_ratio + if lr is not None and lr > LOSS_RATIO_REFERRAL: + f.append(Finding("R3", "Loss ratio above referral threshold", + f"PY{y.policy_year}", + f"{lr:.1%} ({y.total_incurred:,.2f} incurred " + f"against {y.earned_premium:,.2f} earned).", + y.source)) + + # R4 date of loss outside the policy period + for run in authoritative.values(): + if not (run.period_start and run.period_end): + continue + for c in run.claims: + if not c.date_of_loss: + continue + if not (run.period_start <= c.date_of_loss < run.period_end): + f.append(Finding("R4", "Date of loss outside the policy period", + c.claim_no, + f"Loss dated {c.date_of_loss} filed under a " + f"policy running {run.period_start} to " + f"{run.period_end}.", c.source.cite())) + + # R5 application consistency + f.extend(_check_application(application_text, live)) + + # R7 duplicate claims + # Date and amount alone are not evidence: a fleet can genuinely take four + # identical windscreen losses on one day. A duplicate is the same event + # described twice, so the descriptions have to agree as well. + seen: dict[tuple, list[Claim]] = {} + for c in live: + seen.setdefault((c.date_of_loss, round(c.incurred, 2)), []).append(c) + for key, group in seen.items(): + if len(group) < 2 or not key[0]: + continue + for i in range(len(group)): + for j in range(i + 1, len(group)): + a, b = group[i], group[j] + shared = _shared_terms(a.description, b.description) + if len(shared) < 3: + continue + f.append(Finding("R7", "Possible duplicate claim", + f"{a.claim_no}, {b.claim_no}", + f"Same date of loss {key[0]}, identical " + f"incurred {key[1]:,.2f}, and overlapping " + f"descriptions ({', '.join(sorted(shared)[:5])}). " + f"Totals for this policy year may " + f"double-count {key[1]:,.2f}.", + a.source.cite())) + + # R8 loss run internal consistency (checked on every run, superseded too) + for run in runs: + if run.stated_total is None: + continue + diff = round(run.stated_total - run.summed_total, 2) + if abs(diff) >= 0.01: + f.append(Finding("R8", "Loss run does not add up", run.label, + f"States {run.stated_total:,.2f} total incurred; " + f"rows sum to {run.summed_total:,.2f}, a " + f"difference of {diff:,.2f}.", + os.path.basename(run.path))) + if run.stated_count is not None and run.stated_count != len(run.claims): + f.append(Finding("R8", "Loss run claim count disagrees", run.label, + f"States {run.stated_count} claims; " + f"{len(run.claims)} rows present.", + os.path.basename(run.path))) + + return summary + + +def _check_application(text: str, live: list[Claim]) -> list[Finding]: + """Compare what the applicant declared against what the runs show. + + Reads the specific declarations this checklist cares about. Where a + declaration is not found, nothing is asserted about it -- an absent + statement is not a false one. + """ + out: list[Finding] = [] + if not text: + return out + src = "renewal application" + + m = re.search(r"(\d+)\s+claims?\s+exceeding\s+\$?([\d,]+)", text, re.I) + if m: + declared, threshold = int(m.group(1)), float(m.group(2).replace(",", "")) + actual = [c for c in live if c.incurred > threshold] + if len(actual) != declared: + out.append(Finding( + "R5", "Application understates claim count", "declaration", + f"Application states {declared} claims exceeding " + f"{threshold:,.0f}; the loss runs show {len(actual)}: " + f"{', '.join(sorted(c.claim_no for c in actual))}.", src)) + + m2 = re.search(r"no\s+(?:open\s+)?claim[^.]{0,40}?exceeds?\s+\$?([\d,]+)", + text, re.I) + if m2: + threshold = float(m2.group(1).replace(",", "")) + actual = [c for c in live if c.is_open and c.incurred > threshold] + if actual: + out.append(Finding( + "R5", "Application contradicted on open severity", "declaration", + f"Application states no open claim exceeds {threshold:,.0f}; " + f"{len(actual)} do: " + f"{', '.join(sorted(c.claim_no for c in actual))}.", src)) + + m3 = re.search(r"all claims from policy years?\s+(\d{4})\s+and\s+(\d{4})\s+are closed", + text, re.I) + if m3: + years = {int(m3.group(1)), int(m3.group(2))} + still_open = [c for c in live if c.policy_year in years and c.is_open] + if still_open: + out.append(Finding( + "R5", "Application contradicted on closed years", "declaration", + f"Application states all {m3.group(1)}/{m3.group(2)} claims are " + f"closed; {len(still_open)} remain open: " + f"{', '.join(sorted(c.claim_no for c in still_open))}.", src)) + return out + + +STOPWORDS = {"from", "with", "that", "this", "were", "have", "been", "damage", + "claim", "loss", "the", "and", "for"} + + +def _shared_terms(a: str, b: str) -> set[str]: + """Meaningful words two descriptions have in common.""" + def toks(s: str) -> set[str]: + return {w for w in re.findall(r"[a-z0-9]+", (s or "").lower()) + if len(w) > 2 and w not in STOPWORDS} + return toks(a) & toks(b) diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/lossrun/superdocs.py b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/lossrun/superdocs.py new file mode 100644 index 00000000..5e6589eb --- /dev/null +++ b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/lossrun/superdocs.py @@ -0,0 +1,367 @@ +"""The SuperDocs surface: upload, search, chat, approve, export. + +Where the operations go, and why. + +Searches cost operations and chat turns cost operations; uploads and exports do +not. A naive design that searched once per claim would spend 41 operations on +one account and still produce figures nobody could audit. So this build divides +the work by what each side is actually good at: + +* **Figures are parsed, never searched.** Claim counts, incurred totals and loss + ratios come from arithmetic over rows read out of the loss runs. That costs + nothing and is reproducible. +* **Search is spent on the prose** — adjuster notes, the application, broker + correspondence — where the contradictions live in sentences rather than + columns, and where a keyword scan genuinely would miss them. +* **Chat writes the narrative around figures the app has already fixed**, and is + told so explicitly, so the model is never the source of a number. + +The API key is read from SUPERDOCS_API_KEY and is never an argument, never +logged, and never written to an output file. +""" + +from __future__ import annotations + +import json +import os +import re +import time +from dataclasses import dataclass, field + +BASE = os.environ.get("SUPERDOCS_BASE", "https://api.superdocs.app") + +# Published pricing: exports and downloads are free, searches and chat are not. +# One place to change it if that changes. +OP_COST = {"upload": 0, "search": 1, "chat": 1, "approve": 0, "export": 0} + + +class ApiError(RuntimeError): + pass + + +class SearchUnavailable(ApiError): + """The dedicated search endpoint is not present on this deployment.""" + + +class BudgetExceeded(RuntimeError): + pass + + +@dataclass +class Ledger: + cap: int + spent: int = 0 + by_stage: dict[str, int] = field(default_factory=dict) + timings: dict[str, float] = field(default_factory=dict) + + def charge(self, stage: str) -> None: + cost = OP_COST.get(stage, 0) + if cost and self.spent + cost > self.cap: + raise BudgetExceeded( + f"{stage!r} would take the run to {self.spent + cost} against a " + f"cap of {self.cap}. Stopping before the spend, not after.") + self.spent += cost + self.by_stage[stage] = self.by_stage.get(stage, 0) + cost + + def record(self, stage: str, seconds: float) -> None: + self.timings[stage] = round(self.timings.get(stage, 0.0) + seconds, 2) + + +class SuperDocsClient: + def __init__(self, ledger: Ledger, timeout_s: float = 600.0, + poll_interval: float = 3.0): + key = os.environ.get("SUPERDOCS_API_KEY") + if not key: + raise ApiError( + "SUPERDOCS_API_KEY is not set. Export it in your shell; this " + "build never takes a key as an argument.") + self._key = key + self.ledger = ledger + self.timeout_s = timeout_s + self.poll_interval = poll_interval + + @property + def _json(self) -> dict: + return {"Authorization": f"Bearer {self._key}", + "Content-Type": "application/json"} + + @property + def _auth(self) -> dict: + return {"Authorization": f"Bearer {self._key}"} + + @staticmethod + def _redact(text: str) -> str: + return re.sub(r"sk_[A-Za-z0-9_\-]{4,}", "sk_[REDACTED]", text or "") + + def _check(self, resp, what: str): + if resp.status_code >= 400: + raise ApiError(f"{what}: HTTP {resp.status_code} " + f"{self._redact(resp.text)[:400]}") + return resp + + # -- the calls --------------------------------------------------------- + def upload(self, path: str, session_id: str) -> str: + import requests + t0 = time.monotonic() + self.ledger.charge("upload") + with open(path, "rb") as fh: + r = requests.post( + f"{BASE}/v1/documents/upload", headers=self._auth, + files={"file": (os.path.basename(path), fh)}, + data={"session_id": session_id}, timeout=self.timeout_s) + self._check(r, "upload") + self.ledger.record("upload", time.monotonic() - t0) + html = r.json().get("html") + if not html: + raise ApiError("upload returned no html") + return html + + def search(self, session_id: str, query: str, limit: int = 8) -> list[dict]: + """Semantic search across the documents uploaded to this session. + + A dedicated search endpoint returned 404 on the live API when this was + written, so `retrieve` below is the path actually used. This method is + kept because the capability is real and may be exposed directly later; + it raises SearchUnavailable on 404 rather than pretending it worked. + """ + import requests + t0 = time.monotonic() + self.ledger.charge("search") + r = requests.post( + f"{BASE}/v1/search", headers=self._json, + json={"session_id": session_id, "query": query, "limit": limit}, + timeout=self.timeout_s) + if r.status_code == 404: + raise SearchUnavailable( + "no dedicated search endpoint on this deployment; " + "cross-document retrieval runs through the chat surface") + self._check(r, "search") + self.ledger.record("search", time.monotonic() - t0) + body = r.json() + hits = body.get("results") or body.get("matches") or [] + out = [] + for h in hits: + out.append({ + "document": h.get("document") or h.get("filename") or "", + "text": h.get("text") or h.get("content") or "", + "score": h.get("score"), + }) + return out + + def retrieve(self, session_id: str, questions: list[str]) -> list[dict]: + """Ask the attached documents and parse a JSON reply. + + NOT used by the review pipeline. It asked for chat text and looked for + it in the proposed-changes payload, which carries document edits; the + reply was never there to find. The pipeline now has the agent write its + answer into the document instead, which is the supported path and costs + one operation rather than two. Kept, unused, because the JSON-reply + shape is worth having if a chat-reply field is exposed later. + """ + prompt = ( + "Search the documents attached to this session and answer each " + "question below from them. Reply with JSON only, no prose and no " + "code fences: a list of objects with keys \"question\", " + "\"document\", \"quote\". Quote at most two sentences verbatim " + "from the source. If nothing in the attached documents answers a " + "question, return an object for it with \"document\": null. Do " + "not infer, summarise across documents, or state anything the " + "attached text does not say.\n\n" + + "\n".join(f"{i + 1}. {q}" for i, q in enumerate(questions)) + ) + job = self.edit(session_id, "

", prompt) + raw = "" + for c in job.get("pending") or []: + content = c.get("content") + if isinstance(content, dict): + raw = content.get("text") or content.get("message") or "" + elif isinstance(content, str): + raw = content + if raw: + break + return _parse_retrieval(raw) + + def edit(self, session_id: str, document_html: str, instruction: str) -> dict: + import requests + t0 = time.monotonic() + self.ledger.charge("chat") + r = requests.post( + f"{BASE}/v1/chat/async", headers=self._json, + json={"message": instruction, "session_id": session_id, + "document_html": document_html, + "approval_mode": "ask_every_time"}, + timeout=self.timeout_s) + self._check(r, "chat/async") + job_id = r.json()["job_id"] + + deadline = time.monotonic() + self.timeout_s + while time.monotonic() < deadline: + j = self._check(requests.get(f"{BASE}/v1/jobs/{job_id}", + headers=self._json, + timeout=self.timeout_s), "jobs/get") + job = j.json() + status = job.get("status") + if status in ("awaiting_approval", "completed"): + self.ledger.record("chat", time.monotonic() - t0) + return {"job_id": job_id, "status": status, + "pending": self._pending(job)} + if status in ("failed", "cancelled"): + raise ApiError(f"job {status}: " + f"{self._redact(str(job.get('error')))[:300]}") + time.sleep(self.poll_interval) + raise ApiError(f"job {job_id} did not finish within {self.timeout_s:.0f}s. " + f"A long silence is normal on this API, so this is a " + f"timeout, not a proven failure.") + + @staticmethod + def _pending(job: dict) -> list[dict]: + """Proposed-change content arrives JSON-encoded and needs a second + parse; the final result is already an object. Missing this is the most + common reason integrators see empty diffs.""" + out = [] + for c in (job.get("metadata") or {}).get("pending_changes") or []: + c = dict(c) + if isinstance(c.get("content"), str): + try: + c["content"] = json.loads(c["content"]) + except (ValueError, TypeError): + c["content_parse_failed"] = True + out.append(c) + return out + + def approve(self, session_id: str, job_id: str, changes: list[dict]) -> dict: + import requests + t0 = time.monotonic() + self.ledger.charge("approve") + r = requests.post( + f"{BASE}/v1/chat/{session_id}/approve", headers=self._json, + json={"job_id": job_id, "approved": True, + "changes": [{"change_id": c.get("change_id"), "approved": True} + for c in changes]}, + timeout=self.timeout_s) + self._check(r, "approve") + self.ledger.record("approve", time.monotonic() - t0) + return r.json() + + def export(self, session_id: str, out_path: str, fmt: str = "docx") -> str: + import requests + t0 = time.monotonic() + self.ledger.charge("export") + r = requests.post(f"{BASE}/v1/documents/export", headers=self._json, + json={"session_id": session_id, "format": fmt}, + timeout=self.timeout_s) + self._check(r, "export") + os.makedirs(os.path.dirname(out_path) or ".", exist_ok=True) + with open(out_path, "wb") as fh: + fh.write(r.content) + self.ledger.record("export", time.monotonic() - t0) + return out_path + + +class FakeClient: + """Offline stand-in. Returns nothing it was not given, so a test that + passes against it is testing this build's own logic and never the API's.""" + + def __init__(self, ledger: Ledger, corpus_text: dict[str, str] | None = None): + self.ledger = ledger + self.corpus_text = corpus_text or {} + self._sessions: dict[str, str] = {} + + def upload(self, path: str, session_id: str) -> str: + self.ledger.charge("upload") + html = f"

{os.path.basename(path)}

" + self._sessions[session_id] = html + return html + + def search(self, session_id: str, query: str, limit: int = 8) -> list[dict]: + self.ledger.charge("search") + return self._keyword(query, limit) + + def _keyword(self, query: str, limit: int = 8) -> list[dict]: + """Keyword matching over the local text. Weaker than the real thing on + purpose: if a result only appears live, the difference is visible + rather than hidden.""" + terms = [t for t in re.findall(r"\w+", query.lower()) if len(t) > 3] + hits = [] + for name, text in self.corpus_text.items(): + low = text.lower() + score = sum(low.count(t) for t in terms) + if score: + idx = min((low.find(t) for t in terms if t in low), default=0) + hits.append({"document": name, "score": score, + "text": text[max(0, idx - 200): idx + 400]}) + hits.sort(key=lambda h: -h["score"]) + return hits[:limit] + + def retrieve(self, session_id: str, questions: list[str]) -> list[dict]: + self.ledger.charge("chat") + out = [] + for q in questions: + hits = self._keyword(q, limit=1) + out.append({"question": q, + "document": hits[0]["document"] if hits else None, + "quote": hits[0]["text"][:300] if hits else ""}) + return out + + def edit(self, session_id: str, document_html: str, instruction: str) -> dict: + """Fills the placeholder by keyword, so a dry run shows the real shape + of the output. Deliberately weaker than the live path: if a point only + surfaces against the real API, that difference stays visible.""" + self.ledger.charge("chat") + if "[[CORRESPONDENCE]]" in document_html: + lines = [] + for q in re.findall(r"^ - (.+)$", instruction, re.M): + hits = self._keyword(q, limit=1) + if hits: + quote = " ".join(hits[0]["text"].split())[:200] + lines.append(f"

{hits[0]['document']} \u2014 {quote}

") + else: + lines.append(f"

Nothing in the attached documents " + f"addresses: {q}

") + document_html = document_html.replace( + "

[[CORRESPONDENCE]]

", "".join(lines) or + "

Nothing in the attached documents addresses these " + "points.

") + self._sessions[session_id] = document_html + return {"job_id": f"fake-{session_id}", "status": "awaiting_approval", + "pending": [{"change_id": "c1", + "content": json.dumps({"instruction": instruction})}]} + + def approve(self, session_id: str, job_id: str, changes: list[dict]) -> dict: + self.ledger.charge("approve") + return {"status": "applied", "applied": len(changes)} + + def export(self, session_id: str, out_path: str, fmt: str = "docx") -> str: + self.ledger.charge("export") + os.makedirs(os.path.dirname(out_path) or ".", exist_ok=True) + with open(out_path, "w", encoding="utf-8") as fh: + fh.write(self._sessions.get(session_id, "")) + return out_path + + +def _parse_retrieval(raw: str) -> list[dict]: + """Parse the retrieval reply, tolerating fences and surrounding prose. + + Returns [] rather than raising when the reply is not usable: a retrieval + that could not be read is an absent section, not a corrupt document. + """ + if not raw: + return [] + text = raw.strip() + text = re.sub(r"^```(?:json)?|```$", "", text, flags=re.M).strip() + start, end = text.find("["), text.rfind("]") + if start == -1 or end <= start: + return [] + try: + data = json.loads(text[start:end + 1]) + except (ValueError, TypeError): + return [] + if not isinstance(data, list): + return [] + out = [] + for item in data: + if isinstance(item, dict) and item.get("document"): + out.append({"document": str(item.get("document")), + "text": str(item.get("quote") or ""), + "question": str(item.get("question") or "")}) + return out diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/requirements.txt b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/requirements.txt new file mode 100644 index 00000000..82b5bff6 --- /dev/null +++ b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/requirements.txt @@ -0,0 +1,4 @@ +pdfplumber>=0.11 +python-docx>=1.1 +pytest>=8.0 +requests>=2.31 diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/tests/conftest.py b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/tests/conftest.py new file mode 100644 index 00000000..4b1c5dd7 --- /dev/null +++ b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/tests/conftest.py @@ -0,0 +1,3 @@ +import os, sys +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +os.environ.setdefault("LOSSRUN_CORPUS", "corpus/out") diff --git a/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/tests/test_lossrun.py b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/tests/test_lossrun.py new file mode 100644 index 00000000..717ab615 --- /dev/null +++ b/use-cases/SiddharthMuneshwar26/insurance-loss-run-summary/tests/test_lossrun.py @@ -0,0 +1,333 @@ +"""Tests. No API key, no network, no operations. + +The manifest that ships with the corpus is the answer key. These tests assert +the app reproduces it exactly — both halves: every planted conflict and finding +is found, and nothing outside the answer key is reported. A summariser that +flags everything would pass the first half and fail the second. +""" + +from __future__ import annotations + +import datetime +import glob +import json +import os +from collections import Counter + +import pytest + +from lossrun import parse, summarize +from lossrun.model import Claim, LossRun, Source +from lossrun.review import build_html, run +from lossrun.superdocs import BudgetExceeded, FakeClient, Ledger + +CORPUS = os.environ.get("LOSSRUN_CORPUS", "corpus/out") +AS_OF = datetime.date(2026, 1, 9) + +# From MANIFEST.md, which is generated alongside the corpus. +EXPECTED_FINDINGS = {"R1": 1, "R2": 4, "R3": 1, "R4": 1, + "R5": 3, "R6": 1, "R7": 1, "R8": 1} +EXPECTED_YEARS = { + 2021: {"claims": 12, "incurred": 237_280.35, "ratio": 0.576}, + 2022: {"claims": 12, "incurred": 184_694.38, "ratio": 0.412}, + 2023: {"claims": 10, "incurred": 647_992.44, "ratio": 1.293}, + 2024: {"claims": 7, "incurred": 188_831.55, "ratio": 0.333}, +} + + +@pytest.fixture(scope="module") +def analysed(): + runs = [parse.load(p) for p in sorted(glob.glob(f"{CORPUS}/loss-runs/*.csv"))] + notes = {os.path.basename(p).split("-adjuster")[0]: parse.docx_text(p) + for p in glob.glob(f"{CORPUS}/adjuster-notes/*.docx")} + app = parse.docx_text(glob.glob(f"{CORPUS}/application/*.docx")[0]) + s = summarize.summarise(runs, "Meridian Cold Chain Logistics, Inc.") + s = summarize.examine(s, runs, notes, app, as_of=AS_OF) + return s, runs + + +# -------------------------------------------------------------------------- +# the card's own bar: groups and totals by policy year, open claims separated +# -------------------------------------------------------------------------- + +def test_groups_and_totals_by_policy_year(analysed): + s, _ = analysed + assert len(s.years) == 4 + for y in s.years: + want = EXPECTED_YEARS[y.policy_year] + assert y.claim_count == want["claims"], f"PY{y.policy_year} claim count" + assert y.total_incurred == pytest.approx(want["incurred"], abs=0.01) + assert y.loss_ratio == pytest.approx(want["ratio"], abs=0.001) + + +def test_totals_are_the_sum_of_the_years(analysed): + s, _ = analysed + assert s.total_incurred == pytest.approx( + sum(y.total_incurred for y in s.years), abs=0.01) + assert s.total_claims == sum(y.claim_count for y in s.years) == 41 + + +def test_open_claims_are_flagged_separately(analysed): + s, _ = analysed + assert s.total_open == 15 + assert len(s.open_claims) == 15 + assert all(c.is_open for c in s.open_claims) + # and separating them is not the same as dropping them + assert all(any(c.claim_no == o.claim_no for o in s.open_claims) + for c in s.open_claims) + for y in s.years: + assert y.open_count <= y.claim_count + + +def test_paid_plus_reserve_equals_incurred_everywhere(analysed): + s, runs = analysed + for r in runs: + for c in r.claims: + assert c.paid + c.reserve == pytest.approx(c.incurred, abs=0.01), \ + f"{c.claim_no} does not add up" + + +# -------------------------------------------------------------------------- +# supersession — the part that decides the answer +# -------------------------------------------------------------------------- + +def test_the_later_valuation_supersedes_and_the_difference_is_surfaced(analysed): + s, _ = analysed + assert len(s.documents_superseded) == 1 + assert "ORIGINAL" in s.documents_superseded[0] + + kinds = {c.subject: c.kind for c in s.conflicts} + assert kinds["KM-2023-0417"] == "incurred_changed" + assert kinds["KM-2023-0588"] == "claim_added" + + dev = next(c for c in s.conflicts if c.subject == "KM-2023-0417") + assert "45,000.00" in dev.left and "182,400.00" in dev.right + assert dev.resolution is None, "a conflict must not arrive pre-resolved" + + +def test_supersession_changes_the_underwriting_outcome(analysed): + """The reason this matters: the same account either clears the 60% + referral threshold or does not, depending on which run is trusted.""" + _, runs = analysed + original = next(r for r in runs if r.policy_year == 2023 and not r.is_reissue) + reissue = next(r for r in runs if r.policy_year == 2023 and r.is_reissue) + premium = 501_200.00 + assert original.summed_total / premium < 0.60 + assert reissue.summed_total / premium > 0.60 + + +def test_superseded_figures_are_excluded_from_totals(analysed): + s, _ = analysed + py23 = next(y for y in s.years if y.policy_year == 2023) + assert py23.total_incurred == pytest.approx(647_992.44, abs=0.01) + assert py23.total_incurred != pytest.approx(286_092.44, abs=0.01) + + +# -------------------------------------------------------------------------- +# the rules — found, and nothing else found +# -------------------------------------------------------------------------- + +def test_every_expected_finding_is_produced(analysed): + s, _ = analysed + got = Counter(f.rule for f in s.findings) + assert dict(got) == EXPECTED_FINDINGS + + +def test_no_finding_outside_the_answer_key(analysed): + """The half that a flag-everything summariser would fail.""" + s, _ = analysed + known = { + "R1": {"KM-2024-0106"}, + "R2": {"KM-2021-0232", "KM-2021-0252", "KM-2021-0333", "KM-2022-0237"}, + "R3": {"PY2023"}, + "R4": {"KM-2022-0402"}, + "R6": {"KM-2021-0333"}, + } + for rule, subjects in known.items(): + got = {f.subject for f in s.findings if f.rule == rule} + assert got == subjects, f"{rule}: unexpected subjects {got ^ subjects}" + + +def test_a_clean_corpus_produces_no_findings(): + """An honest report of nothing is a valid output and must be reachable.""" + src = Source("clean.csv", "2026-01-01") + claims = [Claim(f"KM-2024-{i:04d}", 2024, "P-1", "Auto Liability", + "2024-06-01", "2024-06-02", "Closed", "Third party", + 1000.0, 0.0, 1000.0, "", src) for i in range(4)] + run_ = LossRun("clean.csv", 2024, "P-1", "2024-04-01", "2025-04-01", + "2026-01-01", False, claims, stated_total=4000.0, + stated_count=4, earned_premium=500_000.0) + s = summarize.summarise([run_], "Clean Co") + s = summarize.examine(s, [run_], {}, "", as_of=AS_OF) + assert s.findings == [] + assert s.conflicts == [] + + +def test_loss_run_that_does_not_add_up_is_caught(analysed): + s, _ = analysed + r8 = [f for f in s.findings if f.rule == "R8"] + assert len(r8) == 1 + assert "1,500.00" in r8[0].detail + + +# -------------------------------------------------------------------------- +# parsing +# -------------------------------------------------------------------------- + +@pytest.mark.parametrize("text,want", [ + ("$1,204,880.00", 1_204_880.00), + ("(12,400.00)", -12_400.00), + ("\u2014", 0.0), + ("0.00", 0.0), + ("", 0.0), + ("94,750.00", 94_750.00), +]) +def test_accountant_notation_parses(text, want): + assert parse.money(text) == pytest.approx(want, abs=0.001) + + +def test_a_non_figure_raises_rather_than_becoming_zero(): + """Silently reading 'pending' as 0.00 would corrupt a total.""" + with pytest.raises(ValueError): + parse.money("pending") + + +def test_pdf_and_csv_of_the_same_run_agree(): + pdfs = glob.glob(f"{CORPUS}/loss-runs/*.pdf") + if not pdfs: + pytest.skip("no PDF in corpus") + try: + pdf = parse.load(pdfs[0]) + except RuntimeError as exc: + pytest.skip(str(exc)) + csv_path = pdfs[0].replace(".pdf", ".csv") + if not os.path.exists(csv_path): + pytest.skip("no matching CSV") + csv_run = parse.load(csv_path) + assert len(pdf.claims) == len(csv_run.claims) + assert pdf.summed_total == pytest.approx(csv_run.summed_total, abs=0.01) + a = {c.claim_no: (c.status, round(c.incurred, 2)) for c in pdf.claims} + b = {c.claim_no: (c.status, round(c.incurred, 2)) for c in csv_run.claims} + assert a == b, "the same policy year read two ways gave different facts" + + +# -------------------------------------------------------------------------- +# budget, honesty, output +# -------------------------------------------------------------------------- + +def test_free_stages_cost_nothing_and_searches_do_not(): + led = Ledger(cap=1) + for stage in ("upload", "approve", "export"): + led.charge(stage) + assert led.spent == 0 + led.charge("search") + assert led.spent == 1 + with pytest.raises(BudgetExceeded): + led.charge("chat") + assert led.spent == 1, "a refused spend must not be recorded" + + +def test_the_review_names_what_it_cannot_establish(analysed): + s, _ = analysed + doc = build_html(s, [], AS_OF) + assert "does not establish" in doc + assert "no figure in this review is model-generated" in doc.lower() + + +def test_the_instruction_forbids_touching_any_figure(): + """The document is handed to a model. What stops it rewriting a total is + this sentence, so the sentence is under test.""" + from lossrun.review import NARRATIVE_INSTRUCTION + low = NARRATIVE_INSTRUCTION.lower() + assert "do not" in low and "change" in low + assert "recalculate" in low and "restate" in low + for word in ("number", "date", "claim number", "table cell"): + assert word in low, f"{word} is not protected by the instruction" + assert "change nothing else" in low + + +def test_end_to_end_offline_spends_only_search_and_chat(tmp_path): + out = str(tmp_path / "out") + r = run(CORPUS, out, mode="dry", max_ops=6, as_of=AS_OF) + led = r["ledger"] + assert led.by_stage.get("upload", 0) == 0 + assert led.by_stage.get("export", 0) == 0 + assert led.spent == led.by_stage.get("search", 0) + led.by_stage.get("chat", 0) + assert os.path.exists(os.path.join(out, "review.html")) + data = json.load(open(os.path.join(out, "summary.json"))) + assert data["totals"]["claims"] == 41 + + +def test_hitting_the_cap_degrades_instead_of_dying(tmp_path): + """A cap reached mid-run must produce a document that says so, not a + traceback and not a document that quietly omits a section.""" + out = str(tmp_path / "out") + r = run(CORPUS, out, mode="dry", max_ops=0, as_of=AS_OF) + assert os.path.exists(os.path.join(out, "review.html")), \ + "the review must still be written when the budget is exhausted" + assert r["summary"].unverified, "the shortfall must be named, not hidden" + data = json.load(open(os.path.join(out, "summary.json"))) + assert data["totals"]["claims"] == 41, \ + "figures must survive a budget that stops the document assembly" + + +def test_no_key_shaped_string_reaches_any_output(tmp_path, monkeypatch): + monkeypatch.setenv("SUPERDOCS_API_KEY", "sk_live_MUST_NOT_APPEAR") + out = str(tmp_path / "out") + run(CORPUS, out, mode="dry", max_ops=6, as_of=AS_OF) + for name in os.listdir(out): + body = open(os.path.join(out, name), encoding="utf-8", errors="ignore").read() + assert "sk_live" not in body + + +def test_retrieval_reply_is_parsed_tolerantly(): + """Models fence JSON and pad it with prose. Neither should lose the data, + and an unusable reply must yield nothing rather than a corrupt section.""" + from lossrun.superdocs import _parse_retrieval + good = ('Here you go:\n```json\n[{"question": "q", "document": "note.docx", ' + '"quote": "reserve increased"}]\n```') + assert _parse_retrieval(good) == [ + {"document": "note.docx", "text": "reserve increased", "question": "q"}] + assert _parse_retrieval("[]") == [] + assert _parse_retrieval("sorry, I could not find anything") == [] + assert _parse_retrieval("") == [] + assert _parse_retrieval("[{not json") == [] + # An entry with no document is dropped rather than rendered blank. + assert _parse_retrieval('[{"question":"q","document":null,"quote":""}]') == [] + + +def test_a_missing_search_endpoint_is_named_not_swallowed(): + """404 on the dedicated search path must be distinguishable from a + successful empty search.""" + from lossrun.superdocs import SearchUnavailable, ApiError + assert issubclass(SearchUnavailable, ApiError) + + +def test_the_whole_run_costs_one_operation(tmp_path): + """Retrieval and narrative are one edit, not two. Uploads, approval and + export are free, so a full review is a single chargeable turn.""" + r = run(CORPUS, str(tmp_path / "out"), "dry", 6, AS_OF) + assert r["ledger"].spent == 1, r["ledger"].by_stage + + +def test_the_correspondence_placeholder_is_filled_or_named(tmp_path): + out = str(tmp_path / "out") + r = run(CORPUS, out, "dry", 6, AS_OF) + doc = open(os.path.join(out, "review.html"), encoding="utf-8").read() + assert "From the correspondence" in doc + # The placeholder is what the edit replaces; it must not survive into a + # document that reports success. + exported = open(r["exported"], encoding="utf-8").read() + assert "[[CORRESPONDENCE]]" not in exported + + +def test_figures_do_not_depend_on_retrieval(tmp_path): + """The claim the README makes: no figure is model-generated. Prove it by + running with retrieval switched off and comparing every total.""" + a = run(CORPUS, str(tmp_path / "with"), "dry", 6, AS_OF)["summary"] + b = run(CORPUS, str(tmp_path / "without"), "dry", 6, AS_OF, + skip_prose=True)["summary"] + assert a.total_incurred == b.total_incurred + assert a.total_claims == b.total_claims + assert [y.to_dict() for y in a.years] == [y.to_dict() for y in b.years] + assert len(a.findings) == len(b.findings)