diff --git a/use-cases/preetham1930/poa-generator/INVARIANTS.md b/use-cases/preetham1930/poa-generator/INVARIANTS.md new file mode 100644 index 000000000..635240ea7 --- /dev/null +++ b/use-cases/preetham1930/poa-generator/INVARIANTS.md @@ -0,0 +1,196 @@ +# INVARIANTS — `builds/poa-generator/` + +Written **before** the tests, and the tests before the code (TASK.md hard rule +7). Every invariant names the test that proves it. An invariant with no test is +an intention, not an invariant. + +This is legal-adjacent output and **we are not lawyers.** That sentence is the +whole reason this file is longer than the last one. + +The thesis this app has to make true: + +> **Every clause in the draft was selected by the intake from a data library, no +> clause text is ever authored by a model, and the document is not exportable +> until the counsel-review notice and the external-execution notice have been +> read back out of the live document verbatim.** + +--- + +## Section A — what this app must never say (the legal-adjacency list) + +This section comes first because it is the one that matters if everything else +works. A power of attorney that is well engineered and quietly claims to be +executed is worse than one that fails loudly. + +A1. **Never present the draft as legal advice.** Every draft carries the +counsel-review notice, verbatim, from the moment it is assembled — it is in the +uploaded skeleton, not added later — and no code path can remove it. +*Proved by* `test_notices.py::test_every_assembled_draft_carries_both_notices_verbatim`. + +A2. **Never claim the document is executed, signed, witnessed, notarised, +valid, binding or in force.** A phrase list is scanned over the rendered +document and any hit is a hard error naming the phrase and the clause. The list +is data (`config/forbidden-phrases.csv`), so adding a phrase is a data edit. +*Proved by* `test_notices.py::test_a_forbidden_phrase_anywhere_in_the_draft_is_a_hard_error` +and `test_the_forbidden_phrase_list_is_data_not_code`. + +A3. **Never imply notarisation happened.** The external-execution notice states +in as many words that signing, witnessing and notarisation happen outside this +product, and the signature block is emitted with every signature and date cell +empty. A signature cell that is not empty is a hard error. +*Proved by* `test_notices.py::test_the_signature_block_leaves_every_signature_and_date_blank`. + +A4. **Never state a jurisdiction's execution requirements as fact.** The +jurisdiction library carries language that defers to counsel; it does not assert +how many witnesses a jurisdiction requires. A jurisdiction row whose text +asserts a requirement without deferring is refused at load time. +*Proved by* `test_library.py::test_a_jurisdiction_row_that_asserts_a_requirement_is_refused`. + +A5. **Never let a model author clause text.** SuperDocs is never asked to draft, +create, add, rewrite or improve a clause. Every instruction is a replacement +whose complete post-state we computed from the library before sending. Measured +on 2026-08-09: asked for one section, the product produced six, four of them +invented in confident, domain-appropriate prose. In a power of attorney an +invented clause is a power nobody granted. +*Proved by* `test_editplan.py::test_no_step_uses_a_creating_verb` and +`test_editplan.py::test_every_step_sends_a_post_state_computed_from_the_library`. + +A6. **Never grant a power the intake did not select.** The powers that reach the +document are exactly the ids in the intake, resolved against +`config/powers.csv`. An id the catalogue does not have is a hard error naming +the catalogue; it is never dropped and never approximated. +*Proved by* `test_intake.py::test_an_unknown_power_id_is_a_hard_error` and +`test_two_drafts.py::test_neither_draft_carries_the_other_s_powers`. + +--- + +## Section B — the export gate (hard constraint 1) + +B1. **The document is not exportable until read-back confirms both notices are +present verbatim.** The gate is structural, not a check somebody remembered to +call: `export()` requires a `NoticeReceipt`, and a `NoticeReceipt` can only be +constructed by `check_notices()` against a live read-back. There is no other +constructor and no default argument. +*Proved by* `test_notices.py::test_export_cannot_be_called_without_a_receipt` +and `test_a_receipt_cannot_be_forged`. + +B2. **A missing notice refuses the export.** Removing either notice chunk from +the document leaves the export path raising, with no file produced. +*Proved by* `test_notices.py::test_export_refuses_when_a_notice_is_missing`. + +B3. **An altered notice refuses the export.** One word changed, one word +dropped, or the notice paraphrased is the same failure as removing it. This is +the case that matters: a rewrite plausibly paraphrases. +*Proved by* `test_notices.py::test_export_refuses_when_a_notice_is_altered`. + +B4. **No notice block is ever an edit target.** Notices are structure, uploaded +verbatim, and the plan is refused before anything is sent if a step names one. +*Proved by* `test_editplan.py::test_no_step_targets_a_notice_block`. + +--- + +## Section C — structure is ours + +C1. **Never ask SuperDocs to create.** One verb, and it is `replace` +(Decision 28). +*Proved by* `test_editplan.py::test_no_step_uses_a_creating_verb`. + +C2. **Never issue a mid-document insertion.** The entire final document +structure — which clauses exist for this intake, in what order, at what +numbers, with every cross-reference already pointing at the right number — is +computed here and uploaded verbatim (Decision 29). +*Proved by* `test_editplan.py::test_no_step_targets_a_heading_block`. + +C3. **Never let a clause number be inferred, remembered or asked for.** Numbers +come from one pass over our own ordered clause list, and the two required drafts +number differently because the healthcare draft has a clause the financial one +does not. +*Proved by* `test_assemble.py::test_clause_numbers_come_from_the_selected_set`. + +C4. **Never leave a cross-reference pointing at the wrong clause.** Every +`clause N` reference is resolved through the same numbering pass, and the +assembled document is re-scanned for a reference that does not resolve. +*Proved by* `test_assemble.py::test_every_cross_reference_resolves_in_both_drafts`. + +C5. **A cross-reference is structure, so it is identical in the skeleton and in +the target.** If a reference differed between the two, an edit would be +renumbering, and renumbering is the thing this design exists not to delegate. +*Proved by* `test_editplan.py::test_no_step_changes_a_clause_reference`. + +--- + +## Section D — the library is data + +D1. **Adding a power type, a clause, or a jurisdiction is a data edit.** No +clause text, no power label, no jurisdiction name and no notice text appears as +a string literal in any `.py` file in this package. +*Proved by* `test_library.py::test_no_clause_text_is_hardcoded_anywhere_in_the_package`. + +D2. **A clause condition that cannot be parsed stops the run; it never reads as +"this clause does not apply".** A silent `False` drops a clause from a power of +attorney, which is the failure mode with no symptom. +*Proved by* `test_library.py::test_an_unparseable_condition_is_a_hard_error`. + +D3. **A placeholder with no resolver stops the run.** It is never left in the +document and never silently emptied. +*Proved by* `test_library.py::test_an_unresolved_placeholder_is_a_hard_error`. + +--- + +## Section E — the wire, copied from Phase 4 as an idea and rebuilt + +E1. **Never batch, and never approve a change we did not compute** (Decision 35, +and Decision 41 for what comes back). +*Proved by* `test_client.py::test_a_batch_is_split_into_ours_and_everything_else`. + +E2. **Never use the synchronous chat route for a gated change** (Decision 32). +*Proved by* `test_client.py::test_gated_changes_go_to_the_async_route`. + +E3. **Never send feedback on a denial** (Decision 34). +*Proved by* `test_client.py::test_denials_carry_no_feedback`. + +E4. **Never treat the job as the decision record** (Decision 33). +*Proved by* `test_orchestrator.py::test_the_decision_row_is_written_before_the_actuator_is_called`. + +E5. **Never believe the chat reply, the `changes_summary`, or a 200.** The only +evidence is `GET /v1/documents/{id}?include_html=true`. +*Proved by* `test_verifier.py::test_a_reply_claiming_success_does_not_make_a_step_ok`. + +--- + +## Section F — the failure path + +F1. **Never report an edit applied without reading the whole document back.** +Three classes, all detected: **not applied**, **applied wrong**, **collateral +damage**. +*Proved by* `test_verifier.py`, one test per class, each driven by a recorded +response from `docs/evidence/`. + +F2. **Never run a downstream step after a failed one.** The queue halts. +*Proved by* `test_orchestrator.py::test_a_failure_halts_the_queue_and_downstream_steps_never_run`. + +F3. **Never export after a failure** — which here means two independent +refusals, the halt and the notice gate. +*Proved by* `test_orchestrator.py::test_no_export_after_a_failure`. + +F4. **Never re-plan.** At most one narrow retry of the same instruction, and a +retry diffs the whole document. +*Proved by* `test_orchestrator.py::test_at_most_one_narrow_retry_and_never_a_replan`. + +--- + +## Section G — the boundary + +G1. **Never import from `system/`, and never speak its vocabulary.** This tree +answers *what must the document say, and did SuperDocs actually say it*. +*Proved by* `tests/test_isolation.py` and `tests/test_domain_boundary.py`. + +--- + +## What it may do, stated so the boundary is not accidentally wider + +- Say what a clause library row says, because a human wrote that row and it is + under version control in this repository. +- Say that a draft is incomplete — a limited power of attorney with no stated + limitation is refused rather than drafted around. +- Leave every signature, witness and notarisation field empty and say why. diff --git a/use-cases/preetham1930/poa-generator/README.md b/use-cases/preetham1930/poa-generator/README.md new file mode 100644 index 000000000..0113c2814 --- /dev/null +++ b/use-cases/preetham1930/poa-generator/README.md @@ -0,0 +1,266 @@ +# Power-of-attorney generator + +> **What strong looks like.** *Run the intake for a durable financial POA and a +> separate healthcare POA and confirm the two drafts differ correctly in granted +> powers, and both carry the review notice and the external-notarisation +> statement.* + +That is the grading line, and [`scripts/verify_live_output.py`](scripts/verify_live_output.py) +is where it is checked — against the two **live** documents read back from the +API and against the two exported DOCX files, never against the run's own report. + +--- + +## Two sentences on what this is + +People assembling a power of attorney need the powers-granted language and the +general/limited/durable distinction to be right, and then they still have to get +the thing witnessed and notarised in person, which no product touches. This is a +guided intake that asks who the principal, agent and successor agent are, which +POA type applies and which specific powers are granted, assembles a clean draft +from a **data** clause library, and exports it — and it refuses to export at all +until it has read back, out of the live document, that both required notices are +present word for word. + +**We are not lawyers.** [INVARIANTS.md](INVARIANTS.md) opens with the list of +things this system must never say, and that list was written before any code. + +--- + +## Run it + +```bash +make poa +``` + +Assembles both required drafts, prints the intake, the clause numbering, the +powers granted and withheld, the notice check and the edit plan, and writes both +renderings of each document to `var/poa/`. No API key, no network. + +```bash +make poa-live +``` + +The same, driving SuperDocs. Six billable operations per draft. Reads +`SUPERDOCS_API_KEY` from `.env` only and never prints it. + +```bash +python builds/poa-generator/scripts/verify_live_output.py +``` + +The grading line, re-checked from scratch. + +--- + +## The one thing that is only possible because SuperDocs exists + +A typeset power of attorney that survives six surgical edits with its layout +intact, is held at a per-item human gate on the way in, and comes out as a DOCX +with real Word table structure. Take SuperDocs out and what is left is a +string-templating script that emits HTML — the typeset rendering, the per-clause +targeted edits, the review gate and the export are all the product's. + +Measured on the live run: 1 `w:tbl`, 4 rows, 16 cells, 16 bordered cells in each +exported DOCX, and both notices verbatim inside `word/document.xml`. + +--- + +## How the parts divide, and why + +| Who | Does what | +| --- | --- | +| The intake (ours) | Who the parties are, which POA type, which power ids | +| The library (data) | Every clause, power, POA type, jurisdiction, notice | +| The assembler (ours) | Which clauses apply, their numbers, every cross-reference, both renderings | +| SuperDocs | Holds the document, applies one chunk replacement at a time, gates each at a human approval, renders and exports | +| The verifier (ours) | Reads the whole document back after every edit and classifies what happened | +| The notice gate (ours) | Refuses the export unless both notices come back verbatim | + +**We never ask SuperDocs to write a clause.** On 2026-08-09 we asked it for one +section and it produced six, four of them invented in confident, +domain-appropriate prose. In a power of attorney an invented clause is a power +nobody granted. So the powers are chosen in the intake and rendered by us, and a +fabricated power has no channel to arrive through — the defence is the absence +of a path, not a check on the output. + +### The blank form is what gets uploaded + +We compute the **entire** document for this intake — every clause that applies, +at its final number, with every cross-reference already pointing at the right +one, and both notices already in place — with the party names, the specific +powers, the limitation and the commencement left as blank-form markers. That is +uploaded verbatim (Decision 29). SuperDocs then only ever replaces the contents +of a chunk that already exists, which is the one operation measured to be +reliable. There is no insertion to perform, so the insertion bug cannot be +reached; there is no renumbering to delegate, so the renumber-around-a-missing- +insert failure cannot occur. + +### Structure really is computed, and the two drafts prove it + +The healthcare draft selects a clause the financial one does not +(`healthcare-records-access`, chosen by the condition +`category granted healthcare`), so every clause after it is numbered one higher: + +``` +durable-financial ... 5. Powers granted - financial 6. Successor agent 7. Reliance ... +healthcare ... 5. Powers granted - healthcare 6. Access to health information + 7. Successor agent 8. Reliance ... +``` + +and the reliance clause's cross-reference moves with it — clause 6 in one draft, +clause 7 in the other. `test_the_shared_parts_really_are_shared` asserts the two +clauses are identical apart from exactly that digit. + +--- + +## The clause library is data + +Adding a jurisdiction or a power type is a **data edit**, and there is a test +that proves it by adding a row to a copy of `config/` and requiring the +behaviour to change with zero code edits. + +| File | What it holds | +| --- | --- | +| `clause-library.csv` | Every clause: order, section, heading, tag, style, condition, body | +| `powers.csv` | The power catalogue, by category, with the granted-powers language | +| `poa-types.csv` | general / limited / durable, and whether a limitation is required | +| `jurisdictions.csv` | Governing-law and execution-requirement language | +| `notices.csv` | The two required notices, verbatim | +| `forbidden-phrases.csv` | What the draft may never say | +| `deferral-markers.txt` | What a jurisdiction row must contain to be accepted | +| `blank-form.csv` | What each gap says before the intake fills it | +| `signature-rows.csv`, `signature-table.csv` | The signature block | + +`test_no_clause_text_is_hardcoded_anywhere_in_the_package` greps every `.py` in +`poa/` for every value in those files and fails on a hit. + +The clause condition grammar is small and every part of it is evaluated, never +looked up: + +``` +condition := "always" + | "type is" + | "type in" | + | "category granted" +``` + +An unparseable condition **raises**. It is never read as "this clause does not +apply" — a clause silently dropped out of a power of attorney has no symptom in +the finished document. + +--- + +## The export gate + +Hard constraint of this build, and it is structural rather than remembered: + +- `PoaClient.export` requires a `NoticeReceipt`. There is no default argument. +- A `NoticeReceipt` can only be constructed by `check_notices()`, which takes a + `ChunkMap` from a live read-back. Construct one directly and it raises. +- `check_notices()` makes two independent comparisons: the notice appears in the + document's reader-visible text, **and** some single chunk's text *is* the + notice, exactly. The first alone would pass a notice with a sentence inserted + into the middle of it; the second alone would miss a second, contradicting + copy. +- "Verbatim" is at the level of words, not bytes: whitespace and entity + re-serialisation are forgiven, because a document editor is entitled to do + both. One word changed, dropped or reordered is not. + +The gate runs twice — once on the upload, before a single edit is issued, and +once on the final read-back, before anything is exported. Tests prove the +refusal for a **missing** notice, an **altered** notice, a **single word** +dropped, a notice **split across two chunks**, and a **sentence inserted** into +one. + +The case that makes the gate worth having is the one the edit queue cannot +catch: every step verified applied, and the product reworded a notice on the way +past. From the queue's point of view that is a clean run. It is not one. +`test_a_reworded_notice_that_the_collateral_check_cannot_see_still_refuses_the_export` +drives exactly that: 6 of 6 applied, nothing halted, and no export. + +--- + +## The failure path, exercised live + +Every edit is: compute the expected post-state of the whole document → one +instruction, one chunk, on the async route → poll to the human gate → refuse +anything that is not exactly our change → write **our** decision row with a named +actor → approve → poll to completed → read the whole document back → classify. + +Three classes, all detected: **not applied**, **applied wrong**, **collateral +damage**. On any of them the queue halts, no downstream step runs, the session is +reverted, the revert is read back and reported either way, and **no export is +produced**. + +This is not only a test. On the live demo run of 2026-08-10 the financial draft +halted at step 1: the product returned the paragraph wrapped in a `
` nobody +asked for, twice — the first attempt and its one narrow retry. The run reported +`APPLIED WRONG` with the diff, refused to run steps 2 through 6, attempted the +revert, and reported honestly that the read-back did **not** match the last +verified-good state. No file was written. The same draft ran clean on the next +attempt, which is the shape of this product: the failure is intermittent, and +that is precisely why nothing is believed without a read-back. + +--- + +## Tests + +459 tests pass repo-wide with **no `.env` present and no network**, 113 of them +this app's. The SuperDocs failure classes are driven by recorded responses under +`docs/evidence/`, so they are the real shapes of failures the product really +produced. + +Every guard added here was verified by **deliberately violating it** and watching +it go red — `make probe`, 101 of 101, 24 of them new. + +```bash +make test # ruff + corpus oracles + pytest, no key required +make probe # break each guard on purpose and require it to notice +``` + +--- + +## What this build does not do + +- It is not a UI. The intake is a JSON file, which is what the tests and the demo + both drive — there is no second, easier path into the assembler. +- It does not revise a draft after the fact. The per-clause targeted revision + loop the plan mentioned ("change the successor agent's name") is the same + machinery pointed at an existing document; it is not built, and it is named + here rather than implied. +- It does not say what any jurisdiction requires. Jurisdiction rows must defer to + counsel, and a row that asserts a requirement is refused at load time. +- It never tells you the document is valid, executed or binding, because it is + none of those things and this product cannot make it so. + + +--- + +## SuperDocs features used + +- **Upload** (POST /v1/documents/upload-base64) - the fully assembled draft is uploaded verbatim; SuperDocs is never asked to create a clause. +- **Chat, async** (POST /v1/chat/async) - one targeted replace per request. +- **Review gate** (approval_mode: ask_every_time + POST /v1/chat/approve) - every change is gated. +- **Read-back** (GET /v1/documents/{id}?include_html=true) - after every edit the whole document is read back and compared to the post-state computed before sending. +- **Export** (POST /v1/documents/export) - gated: the .docx cannot be produced until read-back confirms both required notices are present verbatim. + +## Environment + +``` +SUPERDOCS_API_KEY=your-key-here +SUPERDOCS_BASE_URL=https://api.superdocs.app +``` + +Tests run without a key against recorded fixtures. Only the live demo needs one. + +## Output + +![Two drafts from one clause library, differing correctly in granted powers](evidence/screenshot.png) + +Two live drafts are committed: [evidence/poa-durable-financial.docx](evidence/poa-durable-financial.docx) and [evidence/poa-healthcare.docx](evidence/poa-healthcare.docx). The granted powers differ correctly with no leakage either way; the healthcare draft carries an extra clause, so the successor-agent clause is 6 in one and 7 in the other, and the cross-reference that points at it moves with it. Both carry the counsel-review notice and the external-notarisation statement. + +This is a drafting aid, not legal advice. Signing, witnessing and notarisation happen outside the product. + +--- + +Built by **Preetham Kukkadapu** for the SuperDocs round 2 task. diff --git a/use-cases/preetham1930/poa-generator/config/blank-form.csv b/use-cases/preetham1930/poa-generator/config/blank-form.csv new file mode 100644 index 000000000..8dd4850ef --- /dev/null +++ b/use-cases/preetham1930/poa-generator/config/blank-form.csv @@ -0,0 +1,12 @@ +placeholder,blank_text +principal_name,[ PRINCIPAL - FULL LEGAL NAME ] +principal_address,[ PRINCIPAL - ADDRESS ] +principal_id,[ PRINCIPAL - IDENTIFYING DETAIL ] +agent_name,[ AGENT - FULL LEGAL NAME ] +agent_address,[ AGENT - ADDRESS ] +successor_name,[ SUCCESSOR AGENT - FULL LEGAL NAME ] +successor_address,[ SUCCESSOR AGENT - ADDRESS ] +limitation,[ THE MATTERS THIS POWER IS LIMITED TO ] +effective_date,[ WHEN THE AUTHORITY COMMENCES ] +real_property_description,[ THE REAL PROPERTY THIS POWER APPLIES TO ] +powers,[ POWERS AS SELECTED IN THE INTAKE ] diff --git a/use-cases/preetham1930/poa-generator/config/clause-library.csv b/use-cases/preetham1930/poa-generator/config/clause-library.csv new file mode 100644 index 000000000..12445c308 --- /dev/null +++ b/use-cases/preetham1930/poa-generator/config/clause-library.csv @@ -0,0 +1,21 @@ +clause_id,sort_order,section,heading,tag,style_role,applies_when,body +title,10,front,,h1,title,always,{poa_type_label} +counsel-review-notice,20,front,,p,notice,always,{notice:counsel-review} +preamble-principal,30,parties,Principal,p,body,always,"This power of attorney is granted by {principal_name} of {principal_address}, {principal_id} (the Principal)." +appointment-agent,40,parties,Appointment of agent,p,body,always,The Principal appoints {agent_name} of {agent_address} as the Principal's agent (the Agent) under this instrument. +nature,50,parties,Nature of this power of attorney,p,body,always,{durability_language} +limitation,55,parties,Limits on this power,p,body,type is limited,"This power of attorney is limited to the following matters and confers no authority beyond them: {limitation}" +commencement,60,parties,Commencement,p,body,always,"The authority conferred by this instrument commences {effective_date}. It continues until the Principal revokes it in writing, or until it otherwise ends by operation of law." +powers-financial,70,powers,Powers granted - financial matters,p,body,category granted financial,The Principal grants the Agent authority over the financial matters listed below and over no other financial matter. +powers-financial-list,71,powers,,ul,list,category granted financial,{powers:financial} +powers-healthcare,80,powers,Powers granted - healthcare matters,p,body,category granted healthcare,"The Principal grants the Agent authority over the healthcare matters listed below and over no other healthcare matter, and the Agent must act in accordance with the Principal's known wishes." +powers-healthcare-list,81,powers,,ul,list,category granted healthcare,{powers:healthcare} +healthcare-records-access,82,powers,Access to health information,p,body,category granted healthcare,"The Agent may see and receive the Principal's health information only so far as it is needed to exercise the powers granted under clause {clause_ref:powers-healthcare}, and for no other purpose." +powers-real-property,90,powers,Powers granted - real property,p,body,category granted real_property,The Principal grants the Agent authority over the real property described below and over no other property. +powers-real-property-list,91,powers,,ul,list,category granted real_property,{powers:real_property} +real-property-description,92,powers,,p,body,category granted real_property,"The real property to which clause {clause_ref:powers-real-property} applies is: {real_property_description}" +successor,100,successor,Successor agent,p,body,always,"If the Agent dies, resigns, becomes unable to act or is unwilling to act, {successor_name} of {successor_address} shall act as successor agent with the same authority, and subject to the same limits, as this instrument confers on the Agent." +reliance,110,closing,Reliance by third parties,p,body,always,"A third party who is shown this instrument may rely on it until that third party has actual notice that it has been revoked. A successor agent appointed under clause {clause_ref:successor} is subject to the same condition." +governing-law,120,closing,Governing law and execution requirements,p,body,always,"This instrument is drawn for {jurisdiction_label}. {governing_law} {witness_requirement} {notarisation_requirement}" +external-execution-notice,130,closing,,p,notice,always,{notice:external-execution} +signature-block,140,closing,Signatures - to be completed outside this product,table,signature,always,{signature_table} diff --git a/use-cases/preetham1930/poa-generator/config/deferral-markers.txt b/use-cases/preetham1930/poa-generator/config/deferral-markers.txt new file mode 100644 index 000000000..9474e9b75 --- /dev/null +++ b/use-cases/preetham1930/poa-generator/config/deferral-markers.txt @@ -0,0 +1,10 @@ +# A jurisdiction row states execution requirements. This app is not qualified to +# state them as fact (INVARIANTS.md A4), so every witness_requirement and every +# notarisation_requirement must contain one of the markers below. A row that +# asserts a requirement without deferring is refused at load time, not softened +# at render time. +# +# One marker per line. Comparison is case-insensitive. Adding a marker is a data +# edit; this file is the list, not a constant in a .py. +qualified counsel +a qualified lawyer diff --git a/use-cases/preetham1930/poa-generator/config/forbidden-phrases.csv b/use-cases/preetham1930/poa-generator/config/forbidden-phrases.csv new file mode 100644 index 000000000..4cac2aac6 --- /dev/null +++ b/use-cases/preetham1930/poa-generator/config/forbidden-phrases.csv @@ -0,0 +1,20 @@ +phrase,why +duly executed,claims the instrument has been executed +duly signed,claims a signature that has not happened +duly notarised,claims a notarisation that has not happened +duly notarized,claims a notarisation that has not happened +has been notarised,claims a notarisation that has not happened +has been notarized,claims a notarisation that has not happened +was notarised,claims a notarisation that has not happened +was notarized,claims a notarisation that has not happened +before me this day,notarial jurat language; implies an officer took an acknowledgement +sworn before,notarial language; implies an oath was administered +legally binding,asserts legal effect this product cannot know +valid and enforceable,asserts legal effect this product cannot know +in full force and effect,asserts the instrument is already operative +i hereby certify,certification this product has no standing to give +we certify,certification this product has no standing to give +this is legal advice,the one sentence the counsel-review notice exists to deny +we advise,gives advice rather than assembling a draft +our legal opinion,gives an opinion rather than assembling a draft +you do not need a lawyer,discourages the review the whole draft is predicated on diff --git a/use-cases/preetham1930/poa-generator/config/jurisdictions.csv b/use-cases/preetham1930/poa-generator/config/jurisdictions.csv new file mode 100644 index 000000000..b60dbb7b1 --- /dev/null +++ b/use-cases/preetham1930/poa-generator/config/jurisdictions.csv @@ -0,0 +1,4 @@ +jurisdiction_id,label,governing_law,witness_requirement,notarisation_requirement +in-ts,"Telangana, India","The Principal states that the law of Telangana, India is intended to govern this instrument; whether that choice is effective is a question for qualified counsel.","The number and eligibility of witnesses required for this instrument must be confirmed with qualified counsel before anything is signed.","Whether this instrument must be notarised, registered, or both, and before which officer, must be confirmed with qualified counsel before anything is signed." +in-mh,"Maharashtra, India","The Principal states that the law of Maharashtra, India is intended to govern this instrument; whether that choice is effective is a question for qualified counsel.","The number and eligibility of witnesses required for this instrument must be confirmed with qualified counsel before anything is signed.","Whether this instrument must be notarised, registered, or both, and before which officer, must be confirmed with qualified counsel before anything is signed." +uk-ew,"England and Wales","The Principal states that the law of England and Wales is intended to govern this instrument; whether that choice is effective is a question for qualified counsel.","The number and eligibility of witnesses required for this instrument must be confirmed with qualified counsel before anything is signed.","Whether this instrument must be notarised, registered with a supervising body, or both, must be confirmed with qualified counsel before anything is signed." diff --git a/use-cases/preetham1930/poa-generator/config/notices.csv b/use-cases/preetham1930/poa-generator/config/notices.csv new file mode 100644 index 000000000..68f54840f --- /dev/null +++ b/use-cases/preetham1930/poa-generator/config/notices.csv @@ -0,0 +1,3 @@ +notice_id,label,text +counsel-review,Counsel review notice,"This document is a draft assembled from a clause library for review by qualified counsel. It is not legal advice and it creates no lawyer-client relationship. Nothing in it has been reviewed by a lawyer." +external-execution,External execution notice,"This draft is unexecuted. Signing, witnessing and notarisation happen outside this product and in person, under the law that governs the Principal. No part of this document has been signed, witnessed or notarised, and this product cannot make it so." diff --git a/use-cases/preetham1930/poa-generator/config/poa-types.csv b/use-cases/preetham1930/poa-generator/config/poa-types.csv new file mode 100644 index 000000000..0be3cc058 --- /dev/null +++ b/use-cases/preetham1930/poa-generator/config/poa-types.csv @@ -0,0 +1,4 @@ +type_id,label,requires_limitation,durability_language +general,General power of attorney,no,"This is a general power of attorney. It confers on the Agent the powers set out below in respect of the Principal's affairs generally, and it does not continue if the Principal later becomes unable to make decisions. Whether that is the effect in the governing jurisdiction is a question for qualified counsel." +limited,Limited power of attorney,yes,"This is a limited power of attorney. It confers on the Agent only the powers set out below and only for the matters stated in the clause headed Limits on this power, and it does not continue if the Principal later becomes unable to make decisions. Whether that is the effect in the governing jurisdiction is a question for qualified counsel." +durable,Durable power of attorney,no,"This is a durable power of attorney. It is intended to continue in effect if the Principal later becomes unable to make decisions, and it is intended not to be affected by that inability. Whether a jurisdiction gives that intention effect, and what wording it requires, is a question for qualified counsel." diff --git a/use-cases/preetham1930/poa-generator/config/powers.csv b/use-cases/preetham1930/poa-generator/config/powers.csv new file mode 100644 index 000000000..00d42132b --- /dev/null +++ b/use-cases/preetham1930/poa-generator/config/powers.csv @@ -0,0 +1,17 @@ +power_id,category,label,language +fin-banking,financial,Deposit accounts,"The Agent may open, operate and close deposit accounts held in the name of the Principal, and may draw, endorse and deposit cheques and other instruments payable to the Principal." +fin-investments,financial,Investments,"The Agent may hold, buy and sell securities, mutual fund units and other investments held in the name of the Principal, and may give instructions in respect of them to any depository participant or registrar." +fin-tax,financial,Tax matters,"The Agent may sign, verify and file returns and other documents with the tax authorities on behalf of the Principal, and may represent the Principal before them." +fin-bills,financial,Recurring payments,"The Agent may receive, examine and pay bills, premiums, rents, rates and taxes payable by the Principal, out of the Principal's funds." +fin-borrowing,financial,Borrowing,The Agent may borrow money on behalf of the Principal and give security over the Principal's assets for that borrowing. +fin-business,financial,Business interests,"The Agent may exercise the Principal's rights as a shareholder, partner or member of a company, firm or association, including giving proxies and signing resolutions." +health-treatment,healthcare,Consent to treatment,"The Agent may give, withhold or withdraw consent to medical and surgical treatment on behalf of the Principal, in accordance with the Principal's known wishes." +health-records,healthcare,Health information,"The Agent may request, receive and review the Principal's medical records and health information, and may authorise their disclosure where that is needed to exercise the powers granted by this instrument." +health-providers,healthcare,Choice of providers,"The Agent may select, engage and discharge physicians, hospitals, nursing facilities and other providers of care for the Principal, and may agree the terms on which care is provided." +health-admission,healthcare,Admission and discharge,"The Agent may admit the Principal to, and arrange the discharge of the Principal from, a hospital, nursing facility or residential care facility." +health-endoflife,healthcare,End-of-life care,The Agent may make decisions about life-sustaining treatment and palliative care for the Principal in accordance with the Principal's known wishes. +rp-sale,real_property,Sale and conveyance,"The Agent may sell, convey and transfer the real property described in this instrument, and may sign every deed, receipt and instrument of transfer that the sale requires." +rp-lease,real_property,Letting and leasing,"The Agent may let the real property described in this instrument on such terms as the Agent thinks fit, may collect the rent and may give receipts for it." +rp-mortgage,real_property,Mortgage and charge,The Agent may mortgage or charge the real property described in this instrument as security. +rp-management,real_property,Management and repair,"The Agent may manage, insure, repair and maintain the real property described in this instrument, and may engage and pay contractors for that purpose." +rp-registration,real_property,Registration,"The Agent may present the real property described in this instrument for registration, may admit execution before the registering officer and may do everything the registration requires." diff --git a/use-cases/preetham1930/poa-generator/config/signature-rows.csv b/use-cases/preetham1930/poa-generator/config/signature-rows.csv new file mode 100644 index 000000000..f8da2f064 --- /dev/null +++ b/use-cases/preetham1930/poa-generator/config/signature-rows.csv @@ -0,0 +1,4 @@ +role_key,role_label,name_placeholder +principal,Principal,{principal_name} +agent,Agent,{agent_name} +successor,Successor agent,{successor_name} diff --git a/use-cases/preetham1930/poa-generator/config/signature-table.csv b/use-cases/preetham1930/poa-generator/config/signature-table.csv new file mode 100644 index 000000000..b27e44c46 --- /dev/null +++ b/use-cases/preetham1930/poa-generator/config/signature-table.csv @@ -0,0 +1,5 @@ +column_key,column_label,fill +role,Role,label +name,Name,placeholder +signature,Signature,blank +date,Date,blank diff --git a/use-cases/preetham1930/poa-generator/conftest.py b/use-cases/preetham1930/poa-generator/conftest.py new file mode 100644 index 000000000..11e638bbe --- /dev/null +++ b/use-cases/preetham1930/poa-generator/conftest.py @@ -0,0 +1,78 @@ +"""`builds/poa-generator/` is a codebase, not an installed package. + +The directory name carries a hyphen on purpose - it is an application folder, +not an import path - so the importable package inside it is `poa` and this file +is what puts it on `sys.path` for the suite. Nothing here imports from +`system/`, and `tests/test_isolation.py` fails the build if it ever does. +""" + +from __future__ import annotations + +import sys +from copy import deepcopy +from pathlib import Path + +import pytest + +APP_ROOT = Path(__file__).resolve().parent +REPO_ROOT = APP_ROOT.parent.parent + +if str(APP_ROOT) not in sys.path: + sys.path.insert(0, str(APP_ROOT)) +# tests/ carries helpers (`recorded.py`, `simulator.py`). They are not a +# package, so their directory goes on the path too. +if str(APP_ROOT / "tests") not in sys.path: + sys.path.insert(0, str(APP_ROOT / "tests")) + + +@pytest.fixture(scope="session") +def config() -> Path: + return APP_ROOT / "config" + + +@pytest.fixture(scope="session") +def intakes() -> Path: + return APP_ROOT / "intakes" + + +@pytest.fixture(scope="session") +def evidence() -> Path: + return REPO_ROOT / "docs" / "evidence" + + +@pytest.fixture(scope="session") +def library(config: Path): + from poa.library import Library + + return Library(config) + + +# Assembled once, handed out as a copy. A `Draft` is mutable, several tests +# doctor one on purpose, and a session-scoped fixture handed out by reference +# means a test that forgets to restore it breaks a different test in a different +# file - which is exactly what happened while these were being written, and it +# took a full-suite run to see because each file passed on its own. +@pytest.fixture(scope="session") +def _assembled(library, intakes: Path) -> dict: + from poa.assemble import assemble + from poa.intake import load_intake + + return { + name: assemble(load_intake(intakes / f"{name}.json", library), library) + for name in ("durable-financial", "healthcare", "limited-real-property") + } + + +@pytest.fixture +def financial(_assembled): + return deepcopy(_assembled["durable-financial"]) + + +@pytest.fixture +def healthcare(_assembled): + return deepcopy(_assembled["healthcare"]) + + +@pytest.fixture +def limited(_assembled): + return deepcopy(_assembled["limited-real-property"]) diff --git a/use-cases/preetham1930/poa-generator/evidence/poa-durable-financial.docx b/use-cases/preetham1930/poa-generator/evidence/poa-durable-financial.docx new file mode 100644 index 000000000..da56e9b0f Binary files /dev/null and b/use-cases/preetham1930/poa-generator/evidence/poa-durable-financial.docx differ diff --git a/use-cases/preetham1930/poa-generator/evidence/poa-healthcare.docx b/use-cases/preetham1930/poa-generator/evidence/poa-healthcare.docx new file mode 100644 index 000000000..fdc1e9384 Binary files /dev/null and b/use-cases/preetham1930/poa-generator/evidence/poa-healthcare.docx differ diff --git a/use-cases/preetham1930/poa-generator/evidence/screenshot.png b/use-cases/preetham1930/poa-generator/evidence/screenshot.png new file mode 100644 index 000000000..87d6364df Binary files /dev/null and b/use-cases/preetham1930/poa-generator/evidence/screenshot.png differ diff --git a/use-cases/preetham1930/poa-generator/intakes/durable-financial.json b/use-cases/preetham1930/poa-generator/intakes/durable-financial.json new file mode 100644 index 000000000..25c6e6d1f --- /dev/null +++ b/use-cases/preetham1930/poa-generator/intakes/durable-financial.json @@ -0,0 +1,28 @@ +{ + "intake_id": "durable-financial", + "poa_type": "durable", + "jurisdiction": "in-ts", + "principal": { + "name": "Kavitha Ramanathan", + "address": "14 Sarojini Road, Begumpet, Hyderabad 500016, Telangana, India", + "identifier": "born 4 March 1958" + }, + "agent": { + "name": "Suresh Ramanathan", + "address": "14 Sarojini Road, Begumpet, Hyderabad 500016, Telangana, India" + }, + "successor_agent": { + "name": "Anjali Ramanathan", + "address": "27 Lake View Colony, Jubilee Hills, Hyderabad 500033, Telangana, India" + }, + "granted_powers": [ + "fin-banking", + "fin-investments", + "fin-tax", + "fin-bills", + "fin-business" + ], + "effective_date": "on the date the Principal signs this instrument", + "limitation": null, + "real_property_description": null +} diff --git a/use-cases/preetham1930/poa-generator/intakes/healthcare.json b/use-cases/preetham1930/poa-generator/intakes/healthcare.json new file mode 100644 index 000000000..6e0da4012 --- /dev/null +++ b/use-cases/preetham1930/poa-generator/intakes/healthcare.json @@ -0,0 +1,27 @@ +{ + "intake_id": "healthcare", + "poa_type": "durable", + "jurisdiction": "in-ts", + "principal": { + "name": "Kavitha Ramanathan", + "address": "14 Sarojini Road, Begumpet, Hyderabad 500016, Telangana, India", + "identifier": "born 4 March 1958" + }, + "agent": { + "name": "Nandini Ramanathan", + "address": "27 Lake View Colony, Jubilee Hills, Hyderabad 500033, Telangana, India" + }, + "successor_agent": { + "name": "Suresh Ramanathan", + "address": "14 Sarojini Road, Begumpet, Hyderabad 500016, Telangana, India" + }, + "granted_powers": [ + "health-treatment", + "health-records", + "health-providers", + "health-admission" + ], + "effective_date": "when a physician treating the Principal records in writing that the Principal is unable to make healthcare decisions", + "limitation": null, + "real_property_description": null +} diff --git a/use-cases/preetham1930/poa-generator/intakes/limited-real-property.json b/use-cases/preetham1930/poa-generator/intakes/limited-real-property.json new file mode 100644 index 000000000..43c769c1b --- /dev/null +++ b/use-cases/preetham1930/poa-generator/intakes/limited-real-property.json @@ -0,0 +1,27 @@ +{ + "intake_id": "limited-real-property", + "poa_type": "limited", + "jurisdiction": "in-mh", + "principal": { + "name": "Kavitha Ramanathan", + "address": "14 Sarojini Road, Begumpet, Hyderabad 500016, Telangana, India", + "identifier": "born 4 March 1958" + }, + "agent": { + "name": "Rohan Deshpande", + "address": "9 Prabhat Road, Shivajinagar, Pune 411004, Maharashtra, India" + }, + "successor_agent": { + "name": "Anjali Ramanathan", + "address": "27 Lake View Colony, Jubilee Hills, Hyderabad 500033, Telangana, India" + }, + "granted_powers": [ + "rp-lease", + "rp-management", + "rp-registration", + "fin-bills" + ], + "effective_date": "on the date the Principal signs this instrument", + "limitation": "the letting, upkeep and registration of the real property described in this instrument, and the payment of outgoings on it", + "real_property_description": "Flat 3B, Shivneri Apartments, 9 Prabhat Road, Shivajinagar, Pune 411004, Maharashtra, India" +} diff --git a/use-cases/preetham1930/poa-generator/poa/__init__.py b/use-cases/preetham1930/poa-generator/poa/__init__.py new file mode 100644 index 000000000..91141fe9f --- /dev/null +++ b/use-cases/preetham1930/poa-generator/poa/__init__.py @@ -0,0 +1,14 @@ +"""`builds/poa-generator/` - card S2, the power-of-attorney generator. + +A guided intake picks clauses out of a data library; we assemble the whole draft +and upload it verbatim; SuperDocs fills in the party-specific and power-specific +chunks one at a time, each verified by reading the whole document back; and the +document is not exportable until both notices have been read back verbatim. + +Nothing in this package imports from `system/` (`tests/test_isolation.py`), and +nothing in it speaks `system/`'s outcome vocabulary +(`tests/test_domain_boundary.py`). Ideas are copied from +`builds/statutory-statements/`, never imported (Decision 31). +""" + +from __future__ import annotations diff --git a/use-cases/preetham1930/poa-generator/poa/__main__.py b/use-cases/preetham1930/poa-generator/poa/__main__.py new file mode 100644 index 000000000..2a9cb78cf --- /dev/null +++ b/use-cases/preetham1930/poa-generator/poa/__main__.py @@ -0,0 +1,146 @@ +"""CLI. A thin one - the card says a guided intake, not a UI. + + python -m poa both required drafts, offline + python -m poa --intake intakes/healthcare.json + python -m poa --live drive SuperDocs for both drafts + python -m poa --live --sample 2 the first 2 chunks of each + +`--live` is the only mode that touches the network and the only mode that needs +a key. Everything else - the intake, the clause selection, the numbering, the +cross-references, the notice check and the plan - runs offline, which is why the +test suite can exercise all of it. + +The intake is non-interactive on purpose: a JSON file is what the tests drive +and what the demo drives, so there is no second, easier path into the assembler +that only the tests use. +""" + +from __future__ import annotations + +import argparse +import sys +import uuid +from pathlib import Path + +from .assemble import assemble +from .editplan import build_plan +from .intake import load_intake +from .library import Library +from .orchestrator import Orchestrator, write_artifacts +from .report import compare, render +from .superdocs.client import PoaClient +from .superdocs.decisions import DecisionLedger +from .superdocs.transport import HttpTransport, redacted_key + +APP_ROOT = Path(__file__).resolve().parent.parent +REPO_ROOT = APP_ROOT.parent.parent +# The card names two drafts and grades the comparison between them, so the +# default is both, in this order. +REQUIRED = ("durable-financial", "healthcare") + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(prog="poa", description=__doc__) + parser.add_argument("--config", default=str(APP_ROOT / "config")) + parser.add_argument( + "--intake", + action="append", + default=None, + help="path to an intake JSON; repeatable. Default: both required drafts.", + ) + parser.add_argument("--out", default=str(REPO_ROOT / "var" / "poa")) + parser.add_argument("--live", action="store_true", help="drive SuperDocs (needs a key)") + parser.add_argument( + "--sample", type=int, default=None, help="only issue the first N chunk replacements" + ) + parser.add_argument("--actor", default="K. Latha (reviewing paralegal)") + parser.add_argument("--format", default="docx", help="export format for a successful run") + args = parser.parse_args(argv) + + library = Library(Path(args.config)) + paths = [ + Path(p) + for p in (args.intake or [str(APP_ROOT / "intakes" / f"{n}.json") for n in REQUIRED]) + ] + out_dir = Path(args.out) + + drafts = [] + plans = [] + for path in paths: + intake = load_intake(path, library) + draft = assemble(intake, library) + plan = build_plan(draft).sample(args.sample) + drafts.append(draft) + plans.append(plan) + print(render(intake, draft, library, len(plan))) + for written in write_artifacts(out_dir, plan): + print(f" wrote {written}") + print() + print(" EDIT PLAN (one single-target chunk replacement per step, one change per job)") + for step in plan.steps: + print(f" {step.describe()}") + print() + + if len(drafts) == 2: + print(compare(drafts[0], drafts[1], library)) + print() + + if not args.live: + print( + "offline: nothing was sent. The intake, the clause selection, the numbering, the " + "cross-references and the notice check all ran with no key and no network. " + "Add --live to drive SuperDocs." + ) + return 0 + + # Read from .env only, never from a command line and never printed + # (hard rule 1). `override=False` so an already-exported value wins. + try: + from dotenv import load_dotenv + + load_dotenv(REPO_ROOT / ".env", override=False) + except ImportError: + pass + + print(f"live run: key {redacted_key()}") + ledger = DecisionLedger(out_dir / "decisions.jsonl") + failed = 0 + for plan in plans: + client = PoaClient(HttpTransport()) + run_id = f"poa-{plan.draft.intake_id}-{uuid.uuid4().hex[:6]}" + orchestrator = Orchestrator( + client, + ledger, + args.actor, + library, + export_formats=(args.format,), + export_dir=out_dir, + ) + result = orchestrator.run(run_id, plan) + print() + print("-" * 78) + print( + f"{plan.draft.intake_id}: document {result.document_id} in session {result.session_id}" + ) + for line in orchestrator.log: + print(line) + print(result.sentence(ledger)) + if result.reverted: + print(result.reverted) + print(f"exported: {', '.join(result.exported)}" if result.exported else "no export") + # Read, not derived. If no response carried a usage block we say so + # rather than printing our own call count as if it were the balance. + print( + f"billable calls issued: {client.billable_calls}; " + + ( + f"operations balance reported by the product: {client.usage[-1]}" + if client.usage + else "no response carried an operations balance, so we do not know it" + ) + ) + failed += 0 if result.ok else 1 + return 0 if failed == 0 else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/use-cases/preetham1930/poa-generator/poa/assemble.py b/use-cases/preetham1930/poa-generator/poa/assemble.py new file mode 100644 index 000000000..56fccc13e --- /dev/null +++ b/use-cases/preetham1930/poa-generator/poa/assemble.py @@ -0,0 +1,368 @@ +"""Assembly: the intake picks the clauses, we number them, we render both fills. + +Decision 29, stated as code for this build. Two renderings come out of one +structure: + +- `skeleton` - the **blank form** for this intake's POA type, jurisdiction and + granted categories. Every clause that will exist is already there, at its + final number, with every cross-reference already pointing at the right one, + and both notices already in place. Party names, the specific powers, the + limitation and the commencement are blank-form markers. +- `target` - the same document with the intake's answers filled in. + +The skeleton is what gets uploaded, verbatim. So SuperDocs never inserts a +clause, never renumbers one, and never writes one: every instruction it receives +is "replace this chunk with exactly these bytes", and the bytes came from the +library. An invented clause has no channel to arrive through, which is the +structural argument rather than the checking one - the same shape as Decision 13 +in `system/`. + +Numbering is one pass over the selected clause list. It genuinely differs +between the two required drafts: the healthcare draft carries a clause the +financial one does not (`healthcare-records-access`, selected by +`category granted healthcare`), so every clause after it is numbered one higher +and the cross-reference in the reliance clause moves with it. Nothing about that +is asked of the product. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass, field + +from . import style +from .chunks import strip_markup, tidy +from .errors import ( + CrossReferenceError, + ForbiddenPhraseError, + UnresolvedPlaceholderError, +) +from .intake import Intake +from .library import PLACEHOLDER, Clause, Library + +CLAUSE_REFERENCE = re.compile(r"\bclause (\d+)\b") + + +@dataclass +class Block: + """One top-level element, which becomes one chunk on upload.""" + + role: str + clause_id: str + tag: str + skeleton: str + target: str + clause_number: int | None = None + is_notice: bool = False + is_heading: bool = False + + def __post_init__(self) -> None: + self.skeleton = tidy(self.skeleton) + self.target = tidy(self.target) + + @property + def changed(self) -> bool: + return self.skeleton != self.target + + @property + def editable(self) -> bool: + """A heading carries the numbering and a notice carries the guarantee. + Neither is ever the target of an instruction.""" + return not self.is_heading and not self.is_notice + + +@dataclass +class Draft: + intake_id: str + poa_type: str + jurisdiction: str + blocks: list[Block] = field(default_factory=list) + numbering: dict[str, int] = field(default_factory=dict) + granted: dict[str, tuple[str, ...]] = field(default_factory=dict) + receipts: list[str] = field(default_factory=list) + + def html(self, which: str = "target") -> str: + return "\n".join(getattr(b, which) for b in self.blocks) + + def editable_changed(self) -> list[Block]: + return [b for b in self.blocks if b.editable and b.changed] + + def block(self, role: str) -> Block: + return next(b for b in self.blocks if b.role == role) + + @property + def filename(self) -> str: + return f"poa-{self.intake_id}.html" + + +def assemble(intake: Intake, library: Library) -> Draft: + categories = intake.categories(library) + selected = library.select(intake.poa_type, categories) + + numbering = _number(selected) + draft = Draft( + intake_id=intake.intake_id, + poa_type=intake.poa_type, + jurisdiction=intake.jurisdiction, + numbering=numbering, + granted={c: intake.powers_in(library, c) for c in categories}, + ) + + structural = _structural_values(intake, library) + blank = dict(structural) + filled = dict(structural) + for placeholder, text in library.blank_form.items(): + blank[placeholder] = text + filled.update(_instance_values(intake)) + + running: int | None = None + for clause in selected: + if clause.numbered: + number = numbering[clause.clause_id] + running = number + heading = style.wrap("h2", "heading", f"{number}. {clause.heading}") + draft.blocks.append( + Block( + role=f"{clause.clause_id}-heading", + clause_id=clause.clause_id, + tag="h2", + skeleton=heading, + target=heading, + clause_number=number, + is_heading=True, + ) + ) + draft.blocks.append( + Block( + role=clause.clause_id, + clause_id=clause.clause_id, + tag=clause.tag, + skeleton=_render(clause, blank, numbering, library, intake, filled=False), + target=_render(clause, filled, numbering, library, intake, filled=True), + # An unnumbered clause (a list, a description) belongs to the + # numbered clause above it, which is how a reader reads it and + # how the edit plan should describe it. + clause_number=running, + is_notice=clause.style_role == "notice", + ) + ) + + _assert_cross_references_resolve(draft, numbering) + _assert_no_reference_moves_between_the_two_fills(draft) + draft.receipts.append(assert_no_forbidden_phrase(draft, library)) + draft.receipts.append( + f"{len(selected)} clause(s) selected from {len(library.clauses)} in the library by " + f"type={intake.poa_type!r} and categories={list(categories)}; " + f"{len(numbering)} numbered; {len(draft.editable_changed())} chunk(s) differ between " + f"the blank form and the filled draft and become the edit plan" + ) + return draft + + +# -- numbering and cross-references --------------------------------------- + + +def _number(selected: list[Clause]) -> dict[str, int]: + """One pass over the selected clauses, in library order. Nothing is asked.""" + numbering: dict[str, int] = {} + for clause in selected: + if clause.numbered: + numbering[clause.clause_id] = len(numbering) + 1 + return numbering + + +def _assert_cross_references_resolve(draft: Draft, numbering: dict[str, int]) -> None: + numbers = set(numbering.values()) + for block in draft.blocks: + if block.is_heading: + continue + for which in ("skeleton", "target"): + for match in CLAUSE_REFERENCE.finditer(strip_markup(getattr(block, which))): + referenced = int(match.group(1)) + if referenced not in numbers: + raise CrossReferenceError( + f"block {block.role!r} ({which}) points at clause {referenced}, which " + f"this draft does not have. It runs 1..{max(numbers) if numbers else 0}. " + f"Every reference is resolved through the same numbering pass that " + f"numbered the headings, so this is a bug in that pass, not a caveat." + ) + + +def _assert_no_reference_moves_between_the_two_fills(draft: Draft) -> None: + """A cross-reference is structure, so filling in a name may not move one. + + If it did, an edit would be renumbering, and renumbering delegated to the + product is the failure this whole design exists to avoid (measured twice: + the insert vanishes and the renumbers land around it). + """ + for block in draft.blocks: + before = CLAUSE_REFERENCE.findall(strip_markup(block.skeleton)) + after = CLAUSE_REFERENCE.findall(strip_markup(block.target)) + if before != after: + raise CrossReferenceError( + f"block {block.role!r} references clauses {before} in the blank form and " + f"{after} once filled. A cross-reference is structure and is uploaded at its " + f"final value; an edit that moved one would be a renumbering instruction." + ) + + +# -- the legal-adjacency scan --------------------------------------------- + + +def assert_no_forbidden_phrase(draft: Draft, library: Library) -> str: + """INVARIANTS.md A2. Scanned over both fills, because both go to the wire. + + The blank form is the document that is actually uploaded, so a phrase that + only appears there would reach the live document and never be looked at. + """ + checked = 0 + for block in draft.blocks: + for which in ("skeleton", "target"): + text = strip_markup(getattr(block, which)).lower() + checked += 1 + for forbidden in library.forbidden: + if forbidden.phrase in text: + raise ForbiddenPhraseError( + f"block {block.role!r} ({which}) contains {forbidden.phrase!r}: " + f"{forbidden.why}. This is legal-adjacent output and the product has no " + f"standing to say it. The phrase list is config/forbidden-phrases.csv; " + f"if the phrase is genuinely acceptable, that is a data edit and a " + f"decision somebody signs their name to." + ) + return ( + f"scanned {checked} rendering(s) of {len(draft.blocks)} block(s) against " + f"{len(library.forbidden)} forbidden phrase(s): none present" + ) + + +# -- rendering ------------------------------------------------------------- + + +def _structural_values(intake: Intake, library: Library) -> dict[str, str]: + """Values that are identical in the blank form and in the filled draft. + + They are structure or they are the type's own language: which POA type this + is, which jurisdiction, both notices, and every clause number. Uploading + them means the blank form is already a durable-POA-for-this-jurisdiction + form, and no edit ever has to change one. + """ + poa_type = library.types[intake.poa_type] + jurisdiction = library.jurisdictions[intake.jurisdiction] + return { + "poa_type_label": poa_type.label, + "durability_language": poa_type.durability_language, + "jurisdiction_label": jurisdiction.label, + "governing_law": jurisdiction.governing_law, + "witness_requirement": jurisdiction.witness_requirement, + "notarisation_requirement": jurisdiction.notarisation_requirement, + } + + +def _instance_values(intake: Intake) -> dict[str, str]: + return { + "principal_name": intake.principal.name, + "principal_address": intake.principal.address, + "principal_id": intake.principal.identifier, + "agent_name": intake.agent.name, + "agent_address": intake.agent.address, + "successor_name": intake.successor_agent.name, + "successor_address": intake.successor_agent.address, + "limitation": intake.limitation, + "effective_date": intake.effective_date, + "real_property_description": intake.real_property_description, + } + + +def _render( + clause: Clause, + values: dict[str, str], + numbering: dict[str, int], + library: Library, + intake: Intake, + *, + filled: bool, +) -> str: + def substitute(match: re.Match[str]) -> str: + name, argument = match.group(1), match.group(2) + if name == "notice": + return library.notices[argument].text + if name == "clause_ref": + if argument not in numbering: + raise CrossReferenceError( + f"clause {clause.clause_id!r} refers to clause {argument!r}, which this " + f"draft does not select. A reference to a clause that is not in the " + f"document is never rendered as prose and never dropped." + ) + return str(numbering[argument]) + if name == "powers": + return _powers_list(argument, library, intake, values, filled=filled) + if name == "signature_table": + return _signature_table(library, values) + if name not in values: + raise UnresolvedPlaceholderError( + f"clause {clause.clause_id!r} uses placeholder {{{name}}}, which has no " + f"resolver. Known: {sorted(values)}. " + f"An unresolved placeholder is never left in the document and never emptied - " + f"a clause reading 'of , born' is a draft that went out wrong quietly." + ) + return str(values[name]) + + inner = PLACEHOLDER.sub(substitute, clause.body) + if clause.tag == "ul" or clause.tag == "table": + # The list items and the table rows are already complete elements. + return f'<{clause.tag} style="{style.BY_ROLE[clause.style_role]}">{inner}' + return style.wrap(clause.tag, clause.style_role, inner) + + +def _powers_list( + category: str, library: Library, intake: Intake, values: dict, *, filled: bool +) -> str: + if not filled: + return f'
  • {values["powers"]}
  • ' + items = [] + for power_id in intake.powers_in(library, category): + power = library.powers[power_id] + # The whole sentence is the catalogue's, not ours: the label and the + # language are both data, and nothing here writes prose about a power. + items.append( + f'
  • {power.label}. {power.language}
  • ' + ) + return "".join(items) + + +def _signature_table(library: Library, values: dict[str, str]) -> str: + head = "".join( + f'{c.column_label}' for c in library.signature_columns + ) + rows = [f'{head}'] + for row in library.signature_rows: + cells = [] + for column in library.signature_columns: + if column.blank: + # Empty, and it stays empty, in both fills and after every edit. + # INVARIANTS.md A3: this product never implies a signature or a + # notarisation happened, and the cheapest way to never imply it + # is to have nowhere to put one. + cells.append(f'') + elif column.fill == "label": + cells.append(f'{row.role_label}') + else: + cells.append( + f'{_fill_name(row.name_placeholder, values)}' + ) + rows.append(f"{''.join(cells)}") + return "".join(rows) + + +def _fill_name(placeholder: str, values: dict[str, str]) -> str: + def substitute(match: re.Match[str]) -> str: + name = match.group(1) + if name not in values: + raise UnresolvedPlaceholderError( + f"config/signature-rows.csv uses placeholder {{{name}}}, which has no resolver. " + f"Known: {sorted(values)}. A signature block naming nobody is not a blank to be " + f"filled in later; it is a row that lost its party." + ) + return values[name] + + return PLACEHOLDER.sub(substitute, placeholder) diff --git a/use-cases/preetham1930/poa-generator/poa/chunks.py b/use-cases/preetham1930/poa-generator/poa/chunks.py new file mode 100644 index 000000000..dcad6fa5a --- /dev/null +++ b/use-cases/preetham1930/poa-generator/poa/chunks.py @@ -0,0 +1,207 @@ +"""Chunks: the unit SuperDocs actually addresses, and the unit we verify. + +Measured rather than assumed (PROGRESS.md 2026-08-09): an HTML upload chunks at +the top-level element - every paragraph, every heading, and **the entire table as +one chunk**. Rows and cells carry no id. So the blank form is built to match that +granularity: whatever we intend to replace later is its own top-level element, +so a list of granted powers is one `
      ` and therefore one chunk, and the +signature block is one table and therefore one chunk (Decision 37). + +Copied as an idea from `builds/statutory-statements/` and rebuilt here rather +than imported (Decision 31, and hard rule: a shared idea gets copied). The +divergence is real: this package reads clause headings, not note headings, and +it carries `strip_markup`, which the notice gate needs and the statutory build +has no use for. + +`canonical()` is what "the expected bytes" means in practice. A document editor +is entitled to re-serialise whitespace, attribute order and character entities, +and treating that as a failure would make the verifier cry wolf on every step. +Everything that carries meaning - the tag sequence, every attribute value, and +every character of text including every digit - is compared exactly. The chunk +id is dropped because it is the address, not the content. + +The entity and void-element rules were measured, not assumed: we uploaded +`·` and got back the character it stands for, and we uploaded `
      ` and +got back `
      `. The first live run +stopped on it, which is the verbatim check doing exactly its job; the honest +resolution is to define "verbatim" at the level of content rather than of bytes, +and to say so here. +""" + +from __future__ import annotations + +import html as _html +import re +from dataclasses import dataclass +from html.parser import HTMLParser + +CHUNK_ID = re.compile(r'\sdata-chunk-id="[^"]*"') +CLAUSE_HEADING = re.compile(r"^(?P\d+)\.\s+(?P.+)$") +MARKUP = re.compile(r"<[^>]+>") +TAG = re.compile(r"<(/?)([a-zA-Z0-9]+)((?:\s+[a-zA-Z0-9:-]+\s*=\s*\"[^\"]*\")*)\s*(/?)>") +ATTR = re.compile(r"([a-zA-Z0-9:-]+)\s*=\s*\"([^\"]*)\"") +WS = re.compile(r"\s+") +TOP_LEVEL = ("h1", "h2", "h3", "h4", "p", "table", "ul", "ol", "div", "section", "blockquote") +VOID = {"br", "meta", "hr", "img", "link", "input", "col"} +# HTML parsers insert these whether or not the source had them. Measured: we +# uploaded a table with no <tbody> and it came back wrapped in one. +IMPLICIT = {"tbody", "thead", "tfoot"} + + +def canonical(fragment: str) -> str: + """Whitespace-collapsed, attribute-sorted, chunk-id-free form of an HTML fragment.""" + out: list[str] = [] + cursor = 0 + for match in TAG.finditer(fragment): + text = fragment[cursor : match.start()] + if text: + out.append(WS.sub(" ", _html.unescape(text))) + closing, tag, attrs, selfclose = match.groups() + pairs = sorted( + (k.lower(), WS.sub(" ", v).strip()) + for k, v in ATTR.findall(attrs) + if k.lower() != "data-chunk-id" + ) + if tag.lower() in IMPLICIT: + cursor = match.end() + continue + rendered = "".join(f' {k}="{v}"' for k, v in pairs) + # `<br>` and `<br/>` are the same element; the product re-serialises the + # one we send as the other, and that is punctuation, not content. + marker = "/" if selfclose and tag.lower() not in VOID else "" + out.append(f"<{closing}{tag.lower()}{rendered}{marker}>") + cursor = match.end() + tail = fragment[cursor:] + if tail: + out.append(WS.sub(" ", _html.unescape(tail))) + return WS.sub(" ", "".join(out)).replace("> <", "><").strip() + + +def strip_markup(fragment: str) -> str: + """The text a reader sees: tags removed, entities resolved, spaces collapsed. + + The notice check compares against this rather than against raw HTML, because + a document editor is entitled to re-serialise an entity and that is + punctuation, not content - the same argument `canonical()` makes one level + up. What it may not do is change a word, and that is what this preserves. + """ + return WS.sub(" ", _html.unescape(MARKUP.sub(" ", fragment))).strip() + + +@dataclass(frozen=True) +class Chunk: + chunk_id: str + tag: str + html: str + + @property + def canonical(self) -> str: + return canonical(self.html) + + +class _Scanner(HTMLParser): + def __init__(self) -> None: + super().__init__(convert_charrefs=False) + self.spans: list[tuple[str, str, int, int]] = [] + self._depth = 0 + self._tag: str | None = None + self._start = 0 + self._chunk_id = "" + self._text = "" + + def _offset(self) -> int: + line, col = self.getpos() + return self._line_start[line - 1] + col + + def feed_text(self, text: str) -> None: + self._text = text + self._line_start = [0] + offset = 0 + for line in text.splitlines(keepends=True): + offset += len(line) + self._line_start.append(offset) + self.feed(text) + self.close() + + def handle_starttag(self, tag: str, attrs) -> None: + if tag in VOID: + return + if self._tag is None: + if tag in TOP_LEVEL: + self._tag = tag + self._depth = 1 + self._start = self._offset() + self._chunk_id = dict(attrs).get("data-chunk-id") or "" + return + if tag == self._tag: + self._depth += 1 + + def handle_endtag(self, tag: str) -> None: + if self._tag is None or tag != self._tag: + return + self._depth -= 1 + if self._depth == 0: + end = self._offset() + len(f"</{tag}>") + self.spans.append((self._chunk_id, self._tag, self._start, end)) + self._tag = None + + +class ChunkMap: + """An ordered map of chunk id -> chunk, as the document currently stands.""" + + def __init__(self, html: str) -> None: + scanner = _Scanner() + scanner.feed_text(html) + self.chunks: list[Chunk] = [ + Chunk(chunk_id, tag, html[start:end]) for chunk_id, tag, start, end in scanner.spans + ] + self.by_id = {c.chunk_id: c for c in self.chunks if c.chunk_id} + + def __len__(self) -> int: + return len(self.chunks) + + @property + def ids(self) -> list[str]: + return [c.chunk_id for c in self.chunks] + + def headings(self) -> dict[int, str]: + """Clause number -> heading text, read back from the document itself. + + Read from the live document rather than from our own tree, because the + point of reading it back is to find out whether the numbering we + computed is the numbering the document actually carries. + """ + found: dict[int, str] = {} + for chunk in self.chunks: + if chunk.tag != "h2": + continue + text = strip_markup(chunk.html).strip() + match = CLAUSE_HEADING.match(text) + if match: + found[int(match.group("number"))] = text + return found + + def text_of(self, chunk_id: str) -> str: + return strip_markup(self.by_id[chunk_id].html).strip() + + def text(self) -> str: + """The whole document as text, whitespace collapsed. What a reader reads.""" + return strip_markup(" ".join(c.html for c in self.chunks)) + + +def strip_chunk_ids(html: str) -> str: + return CHUNK_ID.sub("", html) + + +def tidy(fragment: str) -> str: + """Drop the whitespace an HTML serialiser drops, before we ever send it. + + Measured: we sent `<br>\nPrepared in accordance...` and the edited chunk came + back as `<br>Prepared in accordance...`. Rather than keep widening + `canonical()` until it forgives everything, the document we compute is + written the way a serialiser would write it, so there is less for the + comparison to forgive. Whitespace *inside* a line is left alone - that is + where the meaningful spaces between words are. + """ + fragment = re.sub(r">[ \t]*\n[ \t]*", ">", fragment) + return re.sub(r"[ \t]*\n[ \t]*<", "<", fragment) diff --git a/use-cases/preetham1930/poa-generator/poa/editplan.py b/use-cases/preetham1930/poa-generator/poa/editplan.py new file mode 100644 index 000000000..5b879461d --- /dev/null +++ b/use-cases/preetham1930/poa-generator/poa/editplan.py @@ -0,0 +1,146 @@ +"""The edit plan: one single-target chunk replacement per step, and nothing else. + +Every step is a `replace`. There is no other verb (Decision 28), no step targets +a heading (Decision 29 put the clause numbers in before upload), no step targets +a **notice** (INVARIANTS.md B4), and no step carries more than one chunk +(Decision 35). All four are checked here, before anything is sent, because a plan +that breaks a wire rule should never reach the wire. + +The expected post-state is the whole chunk's HTML, not the value inside it +(Decision 37). That is what we compare on read-back and it is also what we send: +the instruction hands SuperDocs the finished clause rather than describing an +edit to make. Which matters more here than it did for a set of accounts - a +described edit is an invitation to draft, and drafting is what produced four +invented sections on 2026-08-09. +""" + +from __future__ import annotations + +from dataclasses import dataclass + +from .assemble import Block, Draft +from .chunks import ChunkMap, canonical +from .errors import PlanRefusedError + +# Ordered so that a halt loses the least verified work. Measured over Phase 4's +# live runs: a whole-table chunk replacement is the one unit the product does not +# apply reliably. A <ul> has not been measured, so it is treated as being between +# the two. This changes nothing about what is sent or checked; it changes which +# failure a reviewer is left holding (Decision 43, carried over). +TAG_RISK = {"p": 0, "ul": 1, "table": 2} + + +@dataclass +class Step: + index: int + role: str + tag: str + clause_number: int | None + expected_html: str # what the chunk must hold afterwards (no chunk id) + was_html: str # what it holds before + chunk_id: str = "" # filled in once the document is uploaded and read back + + @property + def verb(self) -> str: + return "replace" + + def instruction(self) -> str: + """One instruction, one chunk, fully resolved. Nothing is left to plan.""" + return ( + f"Replace the entire contents of the chunk whose data-chunk-id is " + f"{self.chunk_id} with exactly the following HTML, character for character. " + f"The chunk must remain a single <{self.tag}> element and keep the same " + f"data-chunk-id; do not wrap it in a <div> or in any other element. " + f"Do not add any clause. Do not change any heading. Do not reword anything. " + f"Do not touch any other chunk anywhere in the document." + f"\n\n{self.expected_html}" + ) + + def describe(self) -> str: + where = f"clause {self.clause_number}" if self.clause_number else "front matter" + return f"[{self.index:02d}] replace {self.tag:5s} {self.role:28s} ({where})" + + +@dataclass +class Plan: + steps: list[Step] + draft: Draft + + def __len__(self) -> int: + return len(self.steps) + + def sample(self, limit: int | None) -> Plan: + if limit is None or limit >= len(self.steps): + return self + return Plan(self.steps[:limit], self.draft) + + +def build_plan(draft: Draft) -> Plan: + blocks = sorted(draft.editable_changed(), key=lambda b: TAG_RISK.get(b.tag, 1)) + steps = [ + Step( + index=index, + role=block.role, + tag=block.tag, + clause_number=block.clause_number, + expected_html=block.target, + was_html=block.skeleton, + ) + for index, block in enumerate(blocks, start=1) + ] + plan = Plan(steps, draft) + assert_plan_is_legal(plan, draft) + return plan + + +def assert_plan_is_legal(plan: Plan, draft: Draft) -> None: + by_role: dict[str, Block] = {b.role: b for b in draft.blocks} + for step in plan.steps: + block = by_role.get(step.role) + if step.verb != "replace": + raise PlanRefusedError( + f"step {step.index} uses the verb {step.verb!r}. There is one verb and it is " + f"replace (Decision 28): the only operation that inserts is also the one " + f"measured to fabricate clauses nobody asked for, and an invented clause in a " + f"power of attorney is a power nobody granted." + ) + if block is not None and block.is_notice: + raise PlanRefusedError( + f"step {step.index} targets {step.role!r}, which is a notice. The counsel-review " + f"notice and the external-execution notice are uploaded verbatim and are never " + f"edited: an instruction that touches one is an instruction that could " + f"paraphrase it (INVARIANTS.md B4)." + ) + if (block is not None and block.is_heading) or step.tag in ("h1", "h2", "h3"): + raise PlanRefusedError( + f"step {step.index} targets {step.role!r}, which is a heading. Headings carry " + f"the clause numbering and are uploaded at their final values (Decision 29); " + f"there is no renumbering step to get wrong." + ) + if step.expected_html == step.was_html: + raise PlanRefusedError( + f"step {step.index} ({step.role}) would send a chunk that is already correct. " + f"An edit that changes nothing cannot be verified by read-back and reads as " + f"'not applied'." + ) + + +def bind_chunk_ids(plan: Plan, uploaded: ChunkMap) -> None: + """Match each step to the chunk it will address, by the bytes we uploaded. + + Matched on canonical content rather than on position, so a document that came + back re-ordered fails here rather than quietly editing the wrong clause. + """ + by_canonical: dict[str, list[str]] = {} + for chunk in uploaded.chunks: + by_canonical.setdefault(chunk.canonical, []).append(chunk.chunk_id) + for step in plan.steps: + key = canonical(step.was_html) + candidates = by_canonical.get(key, []) + if len(candidates) != 1: + raise PlanRefusedError( + f"step {step.index} ({step.role}) matches {len(candidates)} uploaded chunk(s). " + f"A single-target edit needs exactly one target; {len(uploaded)} chunks came " + f"back. Content we uploaded:\n {step.was_html[:200]}" + ) + step.chunk_id = candidates[0] diff --git a/use-cases/preetham1930/poa-generator/poa/errors.py b/use-cases/preetham1930/poa-generator/poa/errors.py new file mode 100644 index 000000000..a256c3920 --- /dev/null +++ b/use-cases/preetham1930/poa-generator/poa/errors.py @@ -0,0 +1,90 @@ +"""Every failure this app can have, and the message it must carry. + +Hard rule 13's argument, applied to a document: a bare failure tells the +reviewer something is wrong but not what to do about it. Every error below names +what could not be done and where we looked. + +Non-ASCII characters are kept out of these messages on purpose: they are read in +a terminal, and Windows consoles mangled exactly the message that carried Phase +1's thesis (PROGRESS.md 2026-08-08, Assumption 19). +""" + +from __future__ import annotations + + +class PoaError(Exception): + """Base for everything raised by this package.""" + + +# -- the intake ------------------------------------------------------------ + + +class UnknownPowerError(PoaError): + """The intake names a power id the catalogue does not have.""" + + +class UnknownPoaTypeError(PoaError): + """The intake names a POA type the library does not define.""" + + +class UnknownJurisdictionError(PoaError): + """The intake names a jurisdiction the library does not define.""" + + +class IncompleteIntakeError(PoaError): + """The intake cannot support a draft - a limited POA with no stated limit, + a missing successor agent, a party with no name.""" + + +# -- the library ----------------------------------------------------------- + + +class UnknownConditionError(PoaError): + """A clause condition could not be parsed. Never read as 'does not apply'.""" + + +class UnresolvedPlaceholderError(PoaError): + """A clause placeholder has no resolver. Never left in and never emptied.""" + + +class LibraryRefusedError(PoaError): + """A library row is not fit to be used - a jurisdiction row that asserts an + execution requirement instead of deferring it, a duplicate clause id.""" + + +class CrossReferenceError(PoaError): + """A clause reference does not resolve to a clause the draft has.""" + + +# -- the legal-adjacency guards ------------------------------------------- + + +class ForbiddenPhraseError(PoaError): + """The draft says something this product has no standing to say.""" + + +class NoticeMissingError(PoaError): + """A required notice is not present in the document.""" + + +class NoticeAlteredError(PoaError): + """A required notice is present but not verbatim.""" + + +class ExportRefusedError(PoaError): + """An export was attempted without a receipt from the notice gate.""" + + +# -- the wire -------------------------------------------------------------- + + +class PlanRefusedError(PoaError): + """An edit plan step broke one of the wire rules before it was sent.""" + + +class DecisionAlreadyRecordedError(PoaError): + """A second decision on an item that already has one, without supersede.""" + + +class NotConfiguredError(PoaError): + """A live call was attempted with no SUPERDOCS_API_KEY set.""" diff --git a/use-cases/preetham1930/poa-generator/poa/intake.py b/use-cases/preetham1930/poa-generator/poa/intake.py new file mode 100644 index 000000000..e7250922e --- /dev/null +++ b/use-cases/preetham1930/poa-generator/poa/intake.py @@ -0,0 +1,188 @@ +"""The guided intake: who, which type, which powers. + +Non-interactive by construction. An intake is a JSON file, so a test and a demo +drive exactly the same code path a person filling the form would - there is no +second, easier route into the assembler that only the tests use. + +The intake is the **only** channel by which a power reaches the document. That +is the structural half of INVARIANTS.md A6, and it is the same argument as +Decision 13 in `system/`: the defence against a fabricated power is not a check +on the output, it is that there is no path for one to arrive through. The +assembler takes power ids, resolves them against the catalogue, and has no way +to accept a power the catalogue does not hold. + +Every refusal below is a hard error naming what is missing. A limited power of +attorney with no stated limit is not drafted around, and a missing successor +agent is not quietly omitted - the card names the successor-agent clause, and a +clause that is sometimes there is worse than one that is always there. +""" + +from __future__ import annotations + +import json +from dataclasses import dataclass +from pathlib import Path + +from .errors import ( + IncompleteIntakeError, + UnknownJurisdictionError, + UnknownPoaTypeError, + UnknownPowerError, +) +from .library import Library + +REQUIRED_PARTIES = ("principal", "agent", "successor_agent") + + +@dataclass(frozen=True) +class Party: + name: str + address: str + identifier: str = "" + + +@dataclass(frozen=True) +class Intake: + intake_id: str + poa_type: str + jurisdiction: str + principal: Party + agent: Party + successor_agent: Party + granted_powers: tuple[str, ...] + effective_date: str + limitation: str = "" + real_property_description: str = "" + + def categories(self, library: Library) -> tuple[str, ...]: + """The granted categories, in the library's own category order. + + Ordered by the catalogue rather than by the order the ids happened to be + typed in, so two intakes that grant the same powers produce the same + document whatever order the form was filled in. + """ + granted = {library.powers[p].category for p in self.granted_powers} + return tuple(c for c in library.category_order if c in granted) + + def powers_in(self, library: Library, category: str) -> tuple[str, ...]: + return tuple( + p + for p in library.power_order + if p in self.granted_powers and library.powers[p].category == category + ) + + +def load_intake(path: Path, library: Library) -> Intake: + raw = json.loads(path.read_text(encoding="utf-8")) + intake = Intake( + intake_id=str(raw.get("intake_id") or path.stem), + poa_type=str(raw.get("poa_type") or ""), + jurisdiction=str(raw.get("jurisdiction") or ""), + principal=_party(raw, "principal", path), + agent=_party(raw, "agent", path), + successor_agent=_party(raw, "successor_agent", path), + granted_powers=tuple(raw.get("granted_powers") or ()), + effective_date=str(raw.get("effective_date") or "").strip(), + limitation=str(raw.get("limitation") or "").strip(), + real_property_description=str(raw.get("real_property_description") or "").strip(), + ) + validate(intake, library, source=path) + return intake + + +def _party(raw: dict, key: str, path: Path) -> Party: + block = raw.get(key) + if not isinstance(block, dict): + raise IncompleteIntakeError( + f"the intake at {path} has no '{key}' block. A power of attorney names three " + f"parties - {', '.join(REQUIRED_PARTIES)} - and this build refuses to draft one " + f"that names fewer. The successor-agent clause is not optional." + ) + return Party( + name=str(block.get("name") or "").strip(), + address=str(block.get("address") or "").strip(), + identifier=str(block.get("identifier") or "").strip(), + ) + + +def validate(intake: Intake, library: Library, source: Path | None = None) -> None: + where = f" (from {source})" if source else "" + + if intake.poa_type not in library.types: + raise UnknownPoaTypeError( + f"the intake{where} asks for POA type {intake.poa_type!r}. The library defines " + f"{sorted(library.types)}. A type this build does not know is never approximated to " + f"the nearest one it does: the general/limited/durable distinction is the thing the " + f"card says people need to get right." + ) + if intake.jurisdiction not in library.jurisdictions: + raise UnknownJurisdictionError( + f"the intake{where} names jurisdiction {intake.jurisdiction!r}. The library defines " + f"{sorted(library.jurisdictions)}. Adding one is a row in config/jurisdictions.csv, " + f"not a code change." + ) + + for key in REQUIRED_PARTIES: + party: Party = getattr(intake, key) + if not party.name or not party.address: + raise IncompleteIntakeError( + f"the intake{where} gives the {key.replace('_', ' ')} " + f"name={party.name!r} address={party.address!r}. Both are required: an " + f"instrument that cannot say who it appoints is not a draft, it is a gap." + ) + + if not intake.granted_powers: + raise IncompleteIntakeError( + f"the intake{where} grants no powers. A power of attorney granting nothing is not a " + f"shorter document, it is a wrong one." + ) + unknown = [p for p in intake.granted_powers if p not in library.powers] + if unknown: + raise UnknownPowerError( + f"the intake{where} grants {unknown}, which the catalogue does not have. The " + f"catalogue is config/powers.csv and holds {len(library.powers)} power(s) across " + f"{list(library.category_order)}. An unknown power is never dropped and never " + f"approximated - a power nobody granted is the exact failure this build is built " + f"against." + ) + duplicated = sorted({p for p in intake.granted_powers if intake.granted_powers.count(p) > 1}) + if duplicated: + raise IncompleteIntakeError( + f"the intake{where} lists {duplicated} more than once. A power granted twice would " + f"be stated twice in the instrument." + ) + + if not intake.effective_date: + raise IncompleteIntakeError( + f"the intake{where} does not say when the authority commences. A power of attorney " + f"with no commencement is a question, not a draft." + ) + + poa_type = library.types[intake.poa_type] + if poa_type.requires_limitation and not intake.limitation: + raise IncompleteIntakeError( + f"the intake{where} asks for {poa_type.label} and states no limitation. " + f"config/poa-types.csv marks this type requires_limitation=yes. A limited power of " + f"attorney whose limit is blank is a general one wearing the wrong heading, which is " + f"the single most consequential thing this build could get wrong." + ) + if not poa_type.requires_limitation and intake.limitation: + raise IncompleteIntakeError( + f"the intake{where} states a limitation but asks for {poa_type.label}, which carries " + f"no clause to put it in. The limitation would be silently dropped, and a granted " + f"power that was meant to be limited and is not is the worst direction to fail in." + ) + + real_property = "real_property" + grants_real_property = real_property in intake.categories(library) + if grants_real_property and not intake.real_property_description: + raise IncompleteIntakeError( + f"the intake{where} grants real property powers and describes no property. The " + f"clause that names the property has nothing to name, and a real property power over " + f"an unnamed property is unbounded." + ) + if not grants_real_property and intake.real_property_description: + raise IncompleteIntakeError( + f"the intake{where} describes a property but grants no real property power, so the " + f"description has no clause to appear in and would be silently dropped." + ) diff --git a/use-cases/preetham1930/poa-generator/poa/library.py b/use-cases/preetham1930/poa-generator/poa/library.py new file mode 100644 index 000000000..105b276dc --- /dev/null +++ b/use-cases/preetham1930/poa-generator/poa/library.py @@ -0,0 +1,383 @@ +"""The clause library, and it is data. + +Hard constraint of this build: adding a jurisdiction or a power type is a **data +edit**. So every clause, every power, every POA type, every jurisdiction, both +notices, the forbidden-phrase list and the signature block's rows live in +`config/*.csv`, and there is a test that greps this package's own `.py` files +and fails if any of that text appears as a string literal. + +Two loud failures where the tempting default is a quiet one, both copied as an +argument from `system/`'s Decision 17 and rebuilt here: + +- an **unparseable clause condition** raises. A silent `False` would drop a + clause out of a power of attorney, and a missing clause has no symptom - the + document reads perfectly well without it. +- an **unresolved placeholder** raises. It is never left in the text and never + emptied. `of , born` is a document that went out wrong quietly. + +The condition grammar is small on purpose: + + condition := "always" + | "type is" <type_id> + | "type in" <type_id>[|<type_id>...] + | "category granted" <category> +""" + +from __future__ import annotations + +import csv +import re +from dataclasses import dataclass +from pathlib import Path + +from .errors import LibraryRefusedError, UnknownConditionError + +GRAMMAR = "always | type is <type_id> | type in <type_id>|<type_id> | category granted <category>" +PLACEHOLDER = re.compile(r"\{([a-z_]+)(?::([a-z_-]+))?\}") + + +@dataclass(frozen=True) +class Clause: + clause_id: str + sort_order: int + section: str + heading: str + tag: str + style_role: str + applies_when: str + body: str + + @property + def numbered(self) -> bool: + return bool(self.heading) + + +@dataclass(frozen=True) +class Power: + power_id: str + category: str + label: str + language: str + + +@dataclass(frozen=True) +class PoaType: + type_id: str + label: str + requires_limitation: bool + durability_language: str + + +@dataclass(frozen=True) +class Jurisdiction: + jurisdiction_id: str + label: str + governing_law: str + witness_requirement: str + notarisation_requirement: str + + +@dataclass(frozen=True) +class Notice: + notice_id: str + label: str + text: str + + +@dataclass(frozen=True) +class ForbiddenPhrase: + phrase: str + why: str + + +@dataclass(frozen=True) +class SignatureRow: + role_key: str + role_label: str + name_placeholder: str + + +@dataclass(frozen=True) +class SignatureColumn: + column_key: str + column_label: str + fill: str # label | placeholder | blank + + @property + def blank(self) -> bool: + return self.fill == "blank" + + +class Library: + """Everything the draft is made of, loaded from `config/`.""" + + def __init__(self, config_root: Path) -> None: + self.root = config_root + self.clauses = _clauses(config_root / "clause-library.csv") + self.powers = _powers(config_root / "powers.csv") + self.types = _types(config_root / "poa-types.csv") + self.notices = _notices(config_root / "notices.csv") + self.forbidden = _forbidden(config_root / "forbidden-phrases.csv") + self.signature_rows = _signature_rows(config_root / "signature-rows.csv") + self.signature_columns = _signature_columns(config_root / "signature-table.csv") + self.blank_form = _blank_form(config_root / "blank-form.csv") + self.deferral_markers = _markers(config_root / "deferral-markers.txt") + self.jurisdictions = _jurisdictions( + config_root / "jurisdictions.csv", self.deferral_markers + ) + self.power_order = tuple(self.powers) + seen: list[str] = [] + for power in self.powers.values(): + if power.category not in seen: + seen.append(power.category) + self.category_order = tuple(seen) + self._assert_clause_ids_resolve() + + # -- selection --------------------------------------------------------- + + def applies(self, clause: Clause, poa_type: str, categories: tuple[str, ...]) -> bool: + """Evaluate a clause condition. Never returns False because it did not + understand - that is what the raise is for.""" + condition = clause.applies_when.strip() + if condition == "always": + return True + if condition.startswith("type is "): + return poa_type == condition[len("type is ") :].strip() + if condition.startswith("type in "): + allowed = [t.strip() for t in condition[len("type in ") :].split("|")] + return poa_type in allowed + if condition.startswith("category granted "): + return condition[len("category granted ") :].strip() in categories + raise UnknownConditionError( + f"clause {clause.clause_id!r} in {self.root / 'clause-library.csv'} has condition " + f"{condition!r}, which this grammar cannot parse. The grammar is: {GRAMMAR}. " + f"An unparseable condition is never read as 'this clause does not apply': a clause " + f"silently dropped out of a power of attorney has no symptom in the finished " + f"document." + ) + + def select(self, poa_type: str, categories: tuple[str, ...]) -> list[Clause]: + return [c for c in self.clauses if self.applies(c, poa_type, categories)] + + # -- integrity --------------------------------------------------------- + + def _assert_clause_ids_resolve(self) -> None: + known = {c.clause_id for c in self.clauses} + for clause in self.clauses: + for name, argument in PLACEHOLDER.findall(clause.body): + if name == "clause_ref" and argument not in known: + raise LibraryRefusedError( + f"clause {clause.clause_id!r} refers to clause {argument!r}, which the " + f"library does not define. A cross-reference that cannot resolve in the " + f"library can never resolve in a draft." + ) + if name == "notice" and argument not in self.notices: + raise LibraryRefusedError( + f"clause {clause.clause_id!r} refers to notice {argument!r}; the library " + f"defines {sorted(self.notices)}." + ) + if name == "powers" and argument not in self.category_order: + raise LibraryRefusedError( + f"clause {clause.clause_id!r} refers to power category {argument!r}; the " + f"catalogue defines {list(self.category_order)}." + ) + + +# -- loaders --------------------------------------------------------------- + + +def _rows(path: Path) -> list[dict[str, str]]: + """Read a config CSV, refusing any row that does not match the header. + + `csv.DictReader` is quietly forgiving: a row with more fields than the + header puts the surplus in a `None` key, and a row with fewer leaves `None` + values. Both are how an unquoted comma inside a clause turns into a document + that is missing half a sentence and says nothing about it. Neither is + tolerated here - this is the library the whole draft is made of. + """ + out: list[dict[str, str]] = [] + with path.open(encoding="utf-8", newline="") as handle: + reader = csv.DictReader(handle) + for number, row in enumerate(reader, start=2): + surplus = row.pop(None, None) + if surplus: + raise LibraryRefusedError( + f"{path} line {number} has more fields than the header " + f"{reader.fieldnames}. The surplus is {surplus}, which almost always means " + f"a comma inside an unquoted field - and a clause silently cut in half is " + f"a document that goes out wrong quietly." + ) + missing = sorted(k for k, v in row.items() if v is None) + if missing: + raise LibraryRefusedError( + f"{path} line {number} has no value for {missing}. An empty column is " + f"written as an empty field, never omitted." + ) + out.append({k: v.strip() for k, v in row.items()}) + return out + + +def _clauses(path: Path) -> tuple[Clause, ...]: + clauses = [ + Clause( + clause_id=row["clause_id"], + sort_order=int(row["sort_order"]), + section=row["section"], + heading=row["heading"], + tag=row["tag"], + style_role=row["style_role"], + applies_when=row["applies_when"], + body=row["body"], + ) + for row in _rows(path) + ] + ids = [c.clause_id for c in clauses] + duplicated = sorted({i for i in ids if ids.count(i) > 1}) + if duplicated: + raise LibraryRefusedError( + f"{path} defines {duplicated} more than once. A clause id is how a cross-reference " + f"finds its target, so a duplicate makes a reference ambiguous." + ) + orders = [c.sort_order for c in clauses] + if len(set(orders)) != len(orders): + raise LibraryRefusedError( + f"{path} has two clauses at the same sort_order, so the clause order - and therefore " + f"the clause numbering - depends on how the file happened to be read." + ) + return tuple(sorted(clauses, key=lambda c: c.sort_order)) + + +def _powers(path: Path) -> dict[str, Power]: + out: dict[str, Power] = {} + for row in _rows(path): + if row["power_id"] in out: + raise LibraryRefusedError(f"{path} defines power {row['power_id']!r} twice") + out[row["power_id"]] = Power( + power_id=row["power_id"], + category=row["category"], + label=row["label"], + language=row["language"], + ) + return out + + +def _types(path: Path) -> dict[str, PoaType]: + out: dict[str, PoaType] = {} + for row in _rows(path): + requires = row["requires_limitation"].lower() + if requires not in ("yes", "no"): + raise LibraryRefusedError( + f"{path}: type {row['type_id']!r} has requires_limitation={requires!r}; " + f"the column takes 'yes' or 'no' and nothing else, because a value this loader " + f"guessed at would decide whether a limited POA can be drafted with no limit." + ) + out[row["type_id"]] = PoaType( + type_id=row["type_id"], + label=row["label"], + requires_limitation=requires == "yes", + durability_language=row["durability_language"], + ) + return out + + +def _jurisdictions(path: Path, markers: tuple[str, ...]) -> dict[str, Jurisdiction]: + out: dict[str, Jurisdiction] = {} + for row in _rows(path): + for column in ("witness_requirement", "notarisation_requirement"): + text = row[column].lower() + if not any(marker in text for marker in markers): + raise LibraryRefusedError( + f"{path}: jurisdiction {row['jurisdiction_id']!r} states {column} as fact:\n" + f" {row[column]}\n" + f"This build is not qualified to say what a jurisdiction requires " + f"(INVARIANTS.md A4), so the text must defer - it must contain one of " + f"{list(markers)}, from config/deferral-markers.txt. The row is refused at " + f"load time rather than softened at render time, because a softened " + f"assertion still reaches the reader." + ) + out[row["jurisdiction_id"]] = Jurisdiction( + jurisdiction_id=row["jurisdiction_id"], + label=row["label"], + governing_law=row["governing_law"], + witness_requirement=row["witness_requirement"], + notarisation_requirement=row["notarisation_requirement"], + ) + return out + + +def _notices(path: Path) -> dict[str, Notice]: + out: dict[str, Notice] = {} + for row in _rows(path): + if not row["text"]: + raise LibraryRefusedError(f"{path}: notice {row['notice_id']!r} has empty text") + out[row["notice_id"]] = Notice( + notice_id=row["notice_id"], label=row["label"], text=row["text"] + ) + return out + + +def _forbidden(path: Path) -> tuple[ForbiddenPhrase, ...]: + return tuple( + ForbiddenPhrase(phrase=row["phrase"].lower(), why=row["why"]) for row in _rows(path) + ) + + +def _signature_rows(path: Path) -> tuple[SignatureRow, ...]: + return tuple( + SignatureRow( + role_key=row["role_key"], + role_label=row["role_label"], + name_placeholder=row["name_placeholder"], + ) + for row in _rows(path) + ) + + +def _signature_columns(path: Path) -> tuple[SignatureColumn, ...]: + columns = tuple( + SignatureColumn( + column_key=row["column_key"], column_label=row["column_label"], fill=row["fill"] + ) + for row in _rows(path) + ) + allowed = {"label", "placeholder", "blank"} + unknown = sorted({c.fill for c in columns} - allowed) + if unknown: + raise LibraryRefusedError( + f"{path} uses fill {unknown}; the column takes {sorted(allowed)}. A fill this loader " + f"guessed at would decide whether a signature cell is emitted empty." + ) + if not any(c.blank for c in columns): + raise LibraryRefusedError( + f"{path} marks no column blank, so the signature block would be emitted with " + f"nothing left to sign. INVARIANTS.md A3: this product never implies that signing " + f"happened." + ) + return columns + + +def _blank_form(path: Path) -> dict[str, str]: + out: dict[str, str] = {} + for row in _rows(path): + if not row["blank_text"]: + raise LibraryRefusedError( + f"{path}: placeholder {row['placeholder']!r} has empty blank text. A blank form " + f"whose gaps are invisible reads as a finished document." + ) + out[row["placeholder"]] = row["blank_text"] + return out + + +def _markers(path: Path) -> tuple[str, ...]: + lines = [ + line.strip().lower() + for line in path.read_text(encoding="utf-8").splitlines() + if line.strip() and not line.lstrip().startswith("#") + ] + if not lines: + raise LibraryRefusedError( + f"{path} lists no deferral markers, so every jurisdiction row would be refused. " + f"An empty guard list is a guard that cannot be satisfied, not one that is switched " + f"off." + ) + return tuple(lines) diff --git a/use-cases/preetham1930/poa-generator/poa/notices.py b/use-cases/preetham1930/poa-generator/poa/notices.py new file mode 100644 index 000000000..22e394b1c --- /dev/null +++ b/use-cases/preetham1930/poa-generator/poa/notices.py @@ -0,0 +1,161 @@ +"""The export gate. This is hard constraint 1 of the card, and it is structural. + +> The document is NOT exportable until read-back confirms both the +> counsel-review notice and the external-notarisation statement are present +> VERBATIM. + +The tempting implementation is a check the export path calls. That is one +`if` away from being forgotten, and forgetting it produces exactly the artefact +this build exists to prevent: a power of attorney, in a reviewer's Downloads +folder, with no notice on it and nothing to say the notice ever went missing. + +So the check is not a call, it is a **key**. `PoaClient.export` requires a +`NoticeReceipt`. A `NoticeReceipt` cannot be constructed except by +`check_notices()`, which takes a `ChunkMap` from a live read-back and raises if +either notice is missing or altered. There is no default argument, no +`receipt=None` path, and no second constructor. A caller who wants to export +without checking has to change this file, and changing this file is a decision +somebody makes on purpose. + +That is the same argument as Decision 13 and Decision 15 in `system/`: the +strong half of a defence is the absence of a path, not a check on the output. + +**What "verbatim" means here.** The comparison is against the reader-visible +text (tags removed, entities resolved, runs of whitespace collapsed), because a +document editor is entitled to re-serialise an entity or a line break and that +is punctuation, not content - the measured relaxation Decision 38 already +defines one level up. What it may not do is change, drop or reorder a **word**, +and that is exactly what this preserves. One word paraphrased fails, which is +the case that matters: a rewrite plausibly paraphrases. +""" + +from __future__ import annotations + +import difflib +from dataclasses import dataclass, field + +from .chunks import ChunkMap, strip_markup +from .errors import NoticeAlteredError, NoticeMissingError +from .library import Library + +# The one token that lets a NoticeReceipt be constructed. Module-private and +# never exported: `from .notices import *` cannot reach it, and a caller +# elsewhere in the package that tries to build a receipt by hand gets a +# ValueError naming this file. +_GATE = object() + + +@dataclass(frozen=True) +class NoticeReceipt: + """Proof that both notices were read back verbatim, moments ago. + + Only `check_notices()` can make one. It is required by `export()`, so the + export path cannot be reached without having done the read-back. + """ + + document_id: str + chunk_count: int + notices: tuple[str, ...] + detail: tuple[str, ...] = field(default_factory=tuple) + _token: object = None + + def __post_init__(self) -> None: + if self._token is not _GATE: + raise ValueError( + "a NoticeReceipt is issued by check_notices() against a live read-back and by " + "nothing else. Constructing one directly would mean exporting a power of " + "attorney whose notices nobody looked at - which is the single thing this " + "module exists to make impossible. See builds/poa-generator/INVARIANTS.md B1." + ) + + def receipt(self) -> str: + return ( + f"notice gate: {len(self.notices)} required notice(s) found verbatim in " + f"{self.chunk_count} chunk(s) read back from document {self.document_id}; " + + "; ".join(self.detail) + ) + + +def check_notices(chunks: ChunkMap, library: Library, document_id: str = "") -> NoticeReceipt: + """Raise unless every notice in the library is present, verbatim, in `chunks`. + + Two independent comparisons, on purpose: + + 1. the notice text appears in the document's reader-visible text; and + 2. some single chunk's own text **is** the notice, exactly. + + (1) alone would pass a document that had split a notice across two + paragraphs and inserted a sentence between them. (2) alone would pass a + document that had kept the notice chunk and appended a second, contradicting + one - so the caller pairs this with the collateral-damage check, which sees + any chunk that was added. + """ + if not library.notices: + raise NoticeMissingError( + "config/notices.csv defines no notices, so this gate would pass on any document at " + "all. An empty guard list is a guard that cannot fail, which is worse than none." + ) + + document_text = chunks.text() + detail: list[str] = [] + for notice in library.notices.values(): + wanted = _normalise(notice.text) + if wanted not in _normalise(document_text): + closest = _closest_chunk(chunks, wanted) + if closest is None: + raise NoticeMissingError( + f"the {notice.label} ({notice.notice_id}) is not in the document. " + f"{len(chunks)} chunk(s) were read back from document " + f"{document_id or '(unnamed)'} and none of them carries it, in whole or in " + f"part. No export is produced. The notice is:\n {notice.text}" + ) + raise NoticeAlteredError( + f"the {notice.label} ({notice.notice_id}) is present but NOT verbatim. A " + f"paraphrased notice is a missing notice: the wording is the guarantee.\n" + f" expected: {notice.text}\n" + f" found : {closest}\n" + f"{_diff(wanted, _normalise(closest))}\n" + f"No export is produced." + ) + exact = [c for c in chunks.chunks if _normalise(strip_markup(c.html)) == wanted] + if not exact: + raise NoticeAlteredError( + f"the {notice.label} ({notice.notice_id}) appears in the document's text but no " + f"single chunk holds it on its own. A notice spread across chunks can have a " + f"sentence inserted into the middle of it and still pass a substring test, so " + f"this build requires the notice to be one chunk, as it was uploaded. " + f"No export is produced." + ) + detail.append(f"{notice.notice_id} verbatim in chunk {exact[0].chunk_id or '(no id)'}") + + return NoticeReceipt( + document_id=document_id, + chunk_count=len(chunks), + notices=tuple(library.notices), + detail=tuple(detail), + _token=_GATE, + ) + + +def _normalise(text: str) -> str: + return " ".join(strip_markup(text).split()) + + +def _closest_chunk(chunks: ChunkMap, wanted: str) -> str | None: + """The chunk that looks most like the notice, so the error can show a diff.""" + best, score = None, 0.0 + for chunk in chunks.chunks: + text = _normalise(strip_markup(chunk.html)) + ratio = difflib.SequenceMatcher(None, wanted, text).ratio() + if ratio > score: + best, score = text, ratio + return best if score >= 0.5 else None + + +def _diff(expected: str, actual: str) -> str: + lines = list( + difflib.unified_diff( + expected.split(), actual.split(), "expected", "found", lineterm="", n=2 + ) + ) + return "\n".join(f" {line}" for line in lines[:20]) diff --git a/use-cases/preetham1930/poa-generator/poa/orchestrator.py b/use-cases/preetham1930/poa-generator/poa/orchestrator.py new file mode 100644 index 000000000..dfc4ece57 --- /dev/null +++ b/use-cases/preetham1930/poa-generator/poa/orchestrator.py @@ -0,0 +1,297 @@ +"""The run: upload the blank form once, then one verified chunk replacement at a time. + +The shape of a step, copied from Phase 4 and unchanged, because nothing about it +was wrong: + + compute the expected post-state of the WHOLE document + -> one instruction, one chunk, on the async route + -> poll to the gate + -> refuse anything that is not exactly our one change + -> write OUR decision row, with an actor + -> approve (the actuator) + -> poll to completed + -> read the whole document back + -> classify: ok | not applied | applied wrong | collateral damage + +On any failure the queue **halts**. No downstream step runs, because a downstream +step is the consequence of work that may not have happened. + +What is new here is the end of the run. A clean run does **not** export. It reads +the document back one more time, puts it through the notice gate, and exports +only if the gate issues a receipt. Two independent refusals therefore stand +between a failure and a file on disk: the halt, and the gate. The gate is the +one that also fires on a run where every single edit succeeded and the product +quietly paraphrased a notice along the way - which is the case the halt cannot +catch, because from the edit queue's point of view nothing went wrong. +""" + +from __future__ import annotations + +import time +from dataclasses import dataclass, field +from pathlib import Path + +from .chunks import ChunkMap +from .editplan import Plan, bind_chunk_ids +from .errors import NoticeAlteredError, NoticeMissingError +from .library import Library +from .notices import check_notices +from .superdocs.client import PoaClient, select_our_change +from .superdocs.decisions import DecisionLedger +from .superdocs.verifier import Verification, verify, verify_upload_verbatim + + +@dataclass +class StepOutcome: + index: int + role: str + verification: Verification + retried: bool = False + + @property + def ok(self) -> bool: + return self.verification.ok + + +@dataclass +class RunResult: + run_id: str + intake_id: str = "" + document_id: str = "" + session_id: str = "" + planned: int = 0 + outcomes: list[StepOutcome] = field(default_factory=list) + upload_receipt: str = "" + notice_receipt: str = "" + notice_refusal: str = "" + halted_at: int | None = None + exported: list[str] = field(default_factory=list) + reverted: str = "" + + @property + def applied(self) -> int: + return sum(1 for o in self.outcomes if o.ok) + + @property + def ok(self) -> bool: + return ( + self.halted_at is None + and self.applied == self.planned + and bool(self.notice_receipt) + and not self.notice_refusal + ) + + def sentence(self, ledger: DecisionLedger) -> str: + """Computed from the outcomes and the rows, at the moment it is emitted.""" + if self.halted_at is not None: + return ( + f"HALTED at step {self.halted_at}. {self.applied} of {self.planned} planned " + f"change(s) verified applied; the rest were not attempted. No export was " + f"produced. {ledger.describe(self.run_id, self.planned)}." + ) + if self.notice_refusal: + return ( + f"{self.applied} of {self.planned} planned change(s) verified applied, and the " + f"document is NOT exportable: {self.notice_refusal} No export was produced. " + f"{ledger.describe(self.run_id, self.planned)}." + ) + return ( + f"{self.applied} of {self.planned} planned change(s) verified applied by reading " + f"the whole document back after each one, and both notices were then read back " + f"verbatim before anything was exported. " + f"{ledger.describe(self.run_id, self.planned)}." + ) + + +class Orchestrator: + def __init__( + self, + client: PoaClient, + ledger: DecisionLedger, + actor: str, + library: Library, + *, + sleep=time.sleep, + export_formats: tuple[str, ...] = ("docx",), + export_dir: Path | None = None, + ) -> None: + self.client = client + self.ledger = ledger + self.actor = actor + self.library = library + self.sleep = sleep + self.export_formats = export_formats + self.export_dir = export_dir + self.log: list[str] = [] + + def _say(self, line: str) -> None: + self.log.append(line) + + def run(self, run_id: str, plan: Plan) -> RunResult: + result = RunResult(run_id=run_id, intake_id=plan.draft.intake_id, planned=len(plan)) + blank_blocks = [b.skeleton for b in plan.draft.blocks] + + document_id, session_id = self.client.upload_verbatim( + plan.draft.filename, plan.draft.html("skeleton") + ) + result.document_id, result.session_id = document_id, session_id + uploaded, _ = self.client.read_back(document_id) + result.upload_receipt = verify_upload_verbatim(uploaded, blank_blocks) + self._say(result.upload_receipt) + + # The gate runs on the blank form too. If the notices did not survive the + # upload there is nothing to be gained by editing the document first, and + # a great deal to be lost by finding out at the end. + self._gate(uploaded, document_id, result, when="after the upload") + if result.notice_refusal: + return result + self._say(result.notice_receipt) + + bind_chunk_ids(plan, uploaded) + + before = uploaded + last_good = before + for step in plan.steps: + outcome = self._one_step(run_id, step, before, session_id, document_id) + result.outcomes.append(outcome) + if not outcome.ok: + result.halted_at = step.index + self._say(outcome.verification.report()) + self._say( + "halting: no downstream step runs on the consequences of an unconfirmed one" + ) + result.reverted = self._revert(session_id, document_id, last_good, result.applied) + return result + self._say(outcome.verification.report()) + before, _ = self.client.read_back(document_id) + last_good = before + + # Nothing is exported on the strength of the loop above having finished. + final, _ = self.client.read_back(document_id) + receipt = self._gate(final, document_id, result, when="before the export") + if receipt is None: + return result + self._say(result.notice_receipt) + for fmt in self.export_formats: + written = self.client.export( + session_id, fmt, receipt, self.export_dir, stem=f"poa-{plan.draft.intake_id}" + ) + result.exported.append(f"{fmt} -> {written}" if written else fmt) + return result + + def _gate(self, chunks: ChunkMap, document_id: str, result: RunResult, *, when: str): + """The notice gate. A refusal is recorded and reported, never smoothed over.""" + try: + receipt = check_notices(chunks, self.library, document_id) + except (NoticeMissingError, NoticeAlteredError) as exc: + result.notice_refusal = str(exc).splitlines()[0] + self._say(f"NOTICE GATE REFUSED {when}:\n{exc}") + self._say("no export was produced, and no export path was reached") + return None + result.notice_receipt = f"{receipt.receipt()} ({when})" + return receipt + + def _one_step( + self, run_id: str, step, before: ChunkMap, session_id: str, document_id: str + ) -> StepOutcome: + verification = self._attempt(run_id, step, before, session_id, document_id) + if verification.ok: + return StepOutcome(step.index, step.role, verification) + # One narrow retry of the same instruction. Never a re-plan: re-planning + # is the measured degradation mode. Collateral damage is not retried at + # all - the document already holds content nobody asked for. + if any(p.kind == "collateral_damage" for p in verification.problems): + return StepOutcome(step.index, step.role, verification) + self._say(f"step {step.index}: one narrow retry, same instruction, no re-plan") + # Compared against the last verified-good state, not against whatever the + # failed attempt left behind (Decision 42): the wreckage is not a baseline. + retried = self._attempt(run_id, step, before, session_id, document_id, supersede=True) + return StepOutcome(step.index, step.role, retried, retried=True) + + def _attempt( + self, + run_id: str, + step, + before: ChunkMap, + session_id: str, + document_id: str, + supersede: bool = False, + ) -> Verification: + job = self.client.chat_gated(session_id, document_id, step.instruction()) + if not job.awaiting: + return Verification( + step.index, + step.chunk_id, + problems=[ + _problem( + "not_applied", + f"the job reached status {job.status!r} without ever offering a change " + f"to approve. Nothing was decided and nothing is claimed.", + ) + ], + ) + change, unasked = select_our_change(job, step.chunk_id) + # Our row first, then the actuator (Decision 33). + self.ledger.record( + run_id, + step.index, + step.chunk_id, + "accept", + self.actor, + f"replacement for {step.role} matches the clause computed from the library", + supersede=supersede, + ) + self.client.approve(session_id, job.job_id, change.change_id) + for extra in unasked: + # Denied bare: no feedback, ever (Decision 34). + self.ledger.record( + run_id, + step.index, + extra.chunk_id, + "reject", + self.actor, + "not in the computed plan; the product proposed it unasked", + supersede=True, + ) + self.client.deny(session_id, job.job_id, extra.change_id) + self._say( + f"step {step.index}: denied an unasked change on chunk {extra.chunk_id} " + f"(bare, no feedback)" + ) + settled = self.client.poll(job.job_id, until_terminal=True, sleep=self.sleep) + del settled # the job's own account of itself is not evidence + after, _ = self.client.read_back(document_id) + return verify(step.index, step.chunk_id, before, after, step.expected_html) + + def _revert(self, session_id: str, document_id: str, last_good: ChunkMap, applied: int) -> str: + """Revert to the last verified-good turn, then read back and say what happened.""" + try: + self.client.revert(session_id, turn_index=applied) + except Exception as exc: + return f"revert call failed: {exc}. The document is left as it is and NOT exported." + after, _ = self.client.read_back(document_id) + same = [c.canonical for c in after.chunks] == [c.canonical for c in last_good.chunks] + if same: + return "reverted to the last verified-good state, confirmed by read-back" + return ( + "revert was requested and the read-back does NOT match the last verified-good " + f"state ({len(after)} chunk(s) now, {len(last_good)} then). The document is left " + f"as it is, no export was produced, and this is reported rather than smoothed over." + ) + + +def _problem(kind: str, detail: str): + from .superdocs.verifier import Problem + + return Problem(kind, detail) + + +def write_artifacts(out_dir: Path, plan: Plan) -> list[Path]: + """The computed document, both shapes, on disk - so a reviewer can read them.""" + out_dir.mkdir(parents=True, exist_ok=True) + written = [] + for which in ("skeleton", "target"): + path = out_dir / f"poa-{plan.draft.intake_id}-{which}.html" + path.write_text(plan.draft.html(which), encoding="utf-8") + written.append(path) + return written diff --git a/use-cases/preetham1930/poa-generator/poa/report.py b/use-cases/preetham1930/poa-generator/poa/report.py new file mode 100644 index 000000000..e065a0249 --- /dev/null +++ b/use-cases/preetham1930/poa-generator/poa/report.py @@ -0,0 +1,105 @@ +"""What the intake produced, printed so a person can check it without a browser. + +The report states what was selected and what was refused. It never states that +the document is valid, executed, or fit for any purpose - the two notices say +what this product's output is, and the report does not contradict them. +""" + +from __future__ import annotations + +from .assemble import Draft +from .intake import Intake +from .library import Library + + +def render(intake: Intake, draft: Draft, library: Library, planned: int) -> str: + lines: list[str] = [] + poa_type = library.types[intake.poa_type] + jurisdiction = library.jurisdictions[intake.jurisdiction] + + lines.append("=" * 78) + lines.append(f"INTAKE {intake.intake_id}") + lines.append("=" * 78) + lines.append(f" type {poa_type.label} ({poa_type.type_id})") + lines.append(f" jurisdiction {jurisdiction.label} ({jurisdiction.jurisdiction_id})") + lines.append(f" principal {intake.principal.name}") + lines.append(f" agent {intake.agent.name}") + lines.append(f" successor {intake.successor_agent.name}") + lines.append(f" commences {intake.effective_date}") + if intake.limitation: + lines.append(f" limited to {intake.limitation}") + if intake.real_property_description: + lines.append(f" property {intake.real_property_description}") + + lines.append("") + lines.append("POWERS GRANTED (selected in the intake, worded by config/powers.csv)") + for category in intake.categories(library): + granted = intake.powers_in(library, category) + withheld = [ + p + for p, power in library.powers.items() + if power.category == category and p not in granted + ] + lines.append(f" {category}: {len(granted)} granted, {len(withheld)} withheld") + for power_id in granted: + lines.append(f" + {power_id:18s} {library.powers[power_id].label}") + for power_id in withheld: + lines.append(f" - {power_id:18s} {library.powers[power_id].label} (not granted)") + ungranted = [c for c in library.category_order if c not in intake.categories(library)] + for category in ungranted: + lines.append(f" {category}: no power granted, so no clause for it is in the draft") + + lines.append("") + lines.append("CLAUSES (numbered by one pass over the selected set; nothing is asked)") + for block in draft.blocks: + if block.is_heading: + lines.append(f" {block.clause_number:2d}. {block.role[: -len('-heading')]}") + + lines.append("") + lines.append("NOTICES (uploaded verbatim, never an edit target, re-read before any export)") + for notice in library.notices.values(): + present = notice.text in draft.html("skeleton") and notice.text in draft.html("target") + lines.append(f" [{'x' if present else ' '}] {notice.notice_id}: {notice.label}") + + lines.append("") + for receipt in draft.receipts: + lines.append(f" {receipt}") + lines.append(f" {planned} chunk replacement(s) planned") + return "\n".join(lines) + + +def compare(left: Draft, right: Draft, library: Library) -> str: + """The grading line, printed: how the two drafts differ in granted powers. + + Computed from the two drafts' own rendered text, not from the intakes that + produced them, because the claim being made is about the documents. + """ + lines = ["=" * 78, f"COMPARISON {left.intake_id} vs {right.intake_id}", "=" * 78] + for category in library.category_order: + in_left = left.granted.get(category, ()) + in_right = right.granted.get(category, ()) + lines.append( + f" {category:15s} {left.intake_id}: {len(in_left):2d} " + f"{right.intake_id}: {len(in_right):2d}" + ) + lines.append("") + for draft, other in ((left, right), (right, left)): + # Compared on the powers' own language, not on the category counts, so + # this says something about the documents rather than about the intakes. + strays = [ + library.powers[power_id].label + for powers in other.granted.values() + for power_id in powers + if library.powers[power_id].language in draft.html("target") + ] + lines.append( + f" {draft.intake_id}: {len(strays)} power(s) from {other.intake_id} present " + f"{'(THIS IS A FAILURE)' if strays else '- correct'}" + ) + both = [ + n.notice_id + for n in library.notices.values() + if n.text in left.html("target") and n.text in right.html("target") + ] + lines.append(f" both drafts carry, verbatim: {both}") + return "\n".join(lines) diff --git a/use-cases/preetham1930/poa-generator/poa/style.py b/use-cases/preetham1930/poa-generator/poa/style.py new file mode 100644 index 000000000..9e298ce83 --- /dev/null +++ b/use-cases/preetham1930/poa-generator/poa/style.py @@ -0,0 +1,48 @@ +"""Inline CSS, because that is what measurably survives. + +Measured on 2026-08-09 and again over Phase 4's live runs: legacy presentational +attributes (`border="1"`) are dropped by the product, while inline `style` +attributes survive an upload, a targeted replace and a DOCX export with every +value intact. So everything this build renders is styled inline. + +This module holds **presentation only**. No clause text, no power label, no +notice wording and no jurisdiction name appears here - those are data +(`config/*.csv`), and `test_library.py` fails the build if any of them turns up +as a literal in a `.py` file in this package. +""" + +from __future__ import annotations + +RULE = "1px solid #1f3864" + +TITLE = "font-size:20px;letter-spacing:1px;text-align:center;margin:0 0 12px 0" +BODY = "margin:0 0 10px 0;line-height:1.45" +HEADING = "font-size:14px;margin:16px 0 6px 0" +LIST = "margin:0 0 10px 0;padding-left:22px;line-height:1.45" +ITEM = "margin:0 0 6px 0" +# The notices are the safety-critical content, so they are also the content a +# reader cannot skim past: boxed, ruled, and not the same colour as the clauses. +NOTICE = ( + f"border:{RULE};background-color:#eaeef7;padding:10px;margin:0 0 12px 0;" + "font-style:italic;line-height:1.4" +) +TABLE = f"border-collapse:collapse;width:100%;border:{RULE}" +HEAD_ROW = "background-color:#1f3864;color:#ffffff" +TH = f"border:{RULE};padding:6px;text-align:left" +TD = f"border:{RULE};padding:6px" +# Signature and date cells are empty and stay empty (INVARIANTS.md A3). The +# height is what makes them look like something to write on, on paper, later. +TD_BLANK = f"border:{RULE};padding:6px;height:34px" + +BY_ROLE = { + "title": TITLE, + "body": BODY, + "heading": HEADING, + "list": LIST, + "notice": NOTICE, + "signature": TABLE, +} + + +def wrap(tag: str, style_role: str, inner: str) -> str: + return f'<{tag} style="{BY_ROLE[style_role]}">{inner}</{tag}>' diff --git a/use-cases/preetham1930/poa-generator/poa/superdocs/__init__.py b/use-cases/preetham1930/poa-generator/poa/superdocs/__init__.py new file mode 100644 index 000000000..0b70d2df2 --- /dev/null +++ b/use-cases/preetham1930/poa-generator/poa/superdocs/__init__.py @@ -0,0 +1,8 @@ +"""The SuperDocs surface: transport, client, decision ledger, verifier. + +Copied from `builds/statutory-statements/statutory/superdocs/` and rebuilt +(Decision 31). The one divergence is the export gate - `PoaClient.export` +requires a `NoticeReceipt` - and it is the reason this build exists. +""" + +from __future__ import annotations diff --git a/use-cases/preetham1930/poa-generator/poa/superdocs/client.py b/use-cases/preetham1930/poa-generator/poa/superdocs/client.py new file mode 100644 index 000000000..4a21888fd --- /dev/null +++ b/use-cases/preetham1930/poa-generator/poa/superdocs/client.py @@ -0,0 +1,329 @@ +"""The SuperDocs client, with the wire rules enforced in the client itself. + +Four measured decisions live here rather than in a caller's discipline: + +- **Decision 32** - gated changes go to `/v1/chat/async`. The gate takes a + `job_id` and jobs exist only on the async route; passing `approval_mode` to + the synchronous route leaves nothing to approve against. `chat_gated` is the + only method that can produce a pending change and it cannot reach `/v1/chat`. +- **Decision 35** - one change per job, on the sending side. The answering side + does not always agree: the first live run of this build asked for one chunk by + id and got a two-change batch back. So we approve exactly the change we + computed and deny every other one bare, and the read-back is what makes that + safe rather than the batch size. +- **Decision 34** - a denial carries no feedback. `deny()` takes no feedback + parameter, so there is no call shape in which one can be sent. Feedback + triggers a revision pass, and the revision pass proposed edits to two chunks + nobody targeted, aimed at the section headings. +- **Decision 33** - `approve` is an actuator, not a record. This client returns + what the API returned and claims nothing about what was decided; the decision + ledger is written by the caller, before the call. + +Nothing here believes a reply. `read_back()` is the only method whose result is +evidence, and it is the only one the verifier consults. + +Copied from `builds/statutory-statements/` and rebuilt (Decision 31). One thing +diverges, and it is this build's whole point: `export()` will not run without a +`NoticeReceipt`, which only `poa.notices.check_notices()` can issue and only +against a live read-back. See INVARIANTS.md B1. +""" + +from __future__ import annotations + +import base64 +import time +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from ..chunks import ChunkMap +from ..errors import ExportRefusedError, PlanRefusedError +from ..notices import NoticeReceipt +from .transport import Transport + +TERMINAL = {"completed", "failed", "cancelled", "error"} + + +@dataclass +class PendingChange: + change_id: str + chunk_id: str + operation: str + old_html: str + new_html: str + + +@dataclass +class Job: + job_id: str + status: str + progress: int + pending: list[PendingChange] + raw: dict[str, Any] + + @property + def awaiting(self) -> bool: + return self.status == "awaiting_approval" + + @property + def finished(self) -> bool: + return self.status in TERMINAL + + +class PoaClient: + def __init__(self, transport: Transport, *, poll_seconds: float = 3.0, max_polls: int = 80): + self.transport = transport + self.poll_seconds = poll_seconds + self.max_polls = max_polls + self.billable_calls = 0 + self.export_content_types: list[str] = [] + self.export_receipts: list[str] = [] + self.usage: list[dict[str, Any]] = [] + + # -- documents --------------------------------------------------------- + + def upload_verbatim(self, filename: str, html: str) -> tuple[str, str]: + """Upload our computed skeleton. Returns (document_id, session_id). + + Verbatim is the load-bearing property (Decision 29) and it is checked by + the caller against a read-back, never assumed from this response. + """ + # Three calls, and the third is the one that makes the other two + # checkable. An upload with no session is not persisted, and the upload + # response carries no document id at all - so the id comes from the + # session's own roster, which is a read rather than a claim. + session = self.transport.request("POST", "/v1/sessions/init", json_body={}) + session_id = session.get("session_id") + if not session_id: + raise RuntimeError(f"sessions/init returned no session_id; keys {sorted(session)}") + payload = { + "filename": filename, + "file_base64": base64.b64encode(html.encode("utf-8")).decode("ascii"), + "session_id": session_id, + } + # One retry, because a 503 with an empty body was observed on a 23 KB + # upload and succeeded immediately on the next attempt. A retried POST + # could leave two documents in the session, so the roster below is + # required to hold exactly one - the retry is checked, not trusted. + try: + uploaded = self.transport.request( + "POST", "/v1/documents/upload-base64", json_body=payload + ) + except RuntimeError as exc: + if "HTTP 5" not in str(exc): + raise + uploaded = self.transport.request( + "POST", "/v1/documents/upload-base64", json_body=payload + ) + if not uploaded.get("persisted"): + raise RuntimeError( + f"the upload reports persisted={uploaded.get('persisted')!r}. An unpersisted " + f"document cannot be read back, and a read-back is the only evidence there is." + ) + roster = self.transport.request("GET", f"/v1/sessions/{session_id}/documents") + documents = roster.get("documents") or [] + if len(documents) > 1: + raise RuntimeError( + f"the session holds {len(documents)} documents and this build works on one. " + f"If an upload was retried after a 5xx, both may have landed; nothing is edited " + f"until a human has looked." + ) + durable = next( + (d.get("durable_document_id") for d in documents if d.get("durable_document_id")), None + ) + if not durable: + raise RuntimeError( + f"the session roster carries no durable document id; {len(documents)} " + f"document(s), keys {sorted(documents[0]) if documents else '(none)'}. " + f"Without one there is nothing to verify against." + ) + return durable, session_id + + def read_back( + self, document_id: str, *, attempts: int = 4, sleep=time.sleep + ) -> tuple[ChunkMap, dict[str, Any]]: + """`GET /v1/documents/{id}?include_html=true` - the only evidence there is. + + Retried on a 404 because the durable record is not always queryable the + instant the upload returns: one run 404'd immediately and the same call + returned 200 on the next attempt. A read is safe to repeat; the retry is + bounded and a persistent 404 stops the run rather than being smoothed + over into "nothing to verify". + """ + body: dict[str, Any] = {} + for attempt in range(attempts): + try: + body = self.transport.request( + "GET", f"/v1/documents/{document_id}", params={"include_html": "true"} + ) + break + except RuntimeError as exc: + if "HTTP 404" not in str(exc) or attempt == attempts - 1: + raise + sleep(2.0 * (attempt + 1)) + html = body.get("html") + if not html: + raise RuntimeError( + f"read-back for {document_id} carried no html. A verification step that " + f"cannot see the document verifies nothing." + ) + return ChunkMap(html), body + + def export( + self, + session_id: str, + fmt: str, + receipt: NoticeReceipt, + out_dir: Path | None = None, + stem: str = "poa", + ) -> Path | None: + """The export answers with a file, not with JSON, so it gets its own path. + + `receipt` is positional and has no default. It is the gate: a + `NoticeReceipt` exists only where `check_notices()` has just passed + against a read-back of this document, so there is no call shape in which + an unchecked document is exported. Removing the parameter is the only + way to export without checking, and that is a diff somebody signs. + + Written to disk when `out_dir` is given, because an export nobody can + open is not evidence of anything. + """ + if not isinstance(receipt, NoticeReceipt): + raise ExportRefusedError( + f"export was called with receipt={type(receipt).__name__}. It takes a " + f"NoticeReceipt, and a NoticeReceipt is only issued by check_notices() against " + f"a live read-back (INVARIANTS.md B1)." + ) + self.export_receipts.append(receipt.receipt()) + content_type, blob = self.transport.download( + "POST", "/v1/documents/export", json_body={"session_id": session_id, "format": fmt} + ) + self.export_content_types.append(content_type) + if out_dir is None or not blob: + return None + out_dir.mkdir(parents=True, exist_ok=True) + path = out_dir / f"{stem}.{fmt}" + path.write_bytes(blob) + return path + + def revert(self, session_id: str, turn_index: int) -> dict[str, Any]: + return self.transport.request( + "POST", f"/v1/sessions/{session_id}/revert", json_body={"turn_index": turn_index} + ) + + # -- the gated edit loop ---------------------------------------------- + + def chat_gated(self, session_id: str, document_id: str, instruction: str) -> Job: + """One instruction, on the async route, held at the gate (Decisions 32 and 35). + + `document_id` is not sent: the session holds exactly one document and it + is focused, and the durable id we verify against is not the id the chat + surface uses for it. Naming the wrong one is worse than naming none. + """ + del document_id + body = self.transport.request( + "POST", + "/v1/chat/async", + json_body={ + "session_id": session_id, + "message": instruction, + "approval_mode": "ask_every_time", + "response_mode": "compact", + }, + ) + self.billable_calls += 1 + # The promo operations balance is not on any free endpoint - `whoami` + # reports a different bucket entirely (PROGRESS.md 2026-08-09, + # Assumption 40) - so it is captured here whenever a response carries + # one, rather than derived from our own call count afterwards. If it is + # absent this stays empty, which is the honest outcome: we do not know. + usage = body.get("usage") or (body.get("result") or {}).get("usage") + if usage: + self.usage.append(usage) + job_id = body.get("job_id") + if not job_id: + raise RuntimeError(f"chat/async returned no job_id; keys {sorted(body)}") + return self.poll(job_id) + + def poll(self, job_id: str, *, until_terminal: bool = False, sleep=time.sleep) -> Job: + for _ in range(self.max_polls): + body = self.transport.request("GET", f"/v1/jobs/{job_id}") + usage = (body.get("result") or {}).get("usage") if isinstance(body, dict) else None + if usage: + self.usage.append(usage) + job = _job_from(body) + if job.finished or (job.awaiting and not until_terminal): + return job + sleep(self.poll_seconds) + raise TimeoutError( + f"job {job_id} did not settle in {self.max_polls} polls. Nothing is reported as " + f"applied on a timeout; the document is read back and the run halts." + ) + + def approve(self, session_id: str, job_id: str, change_id: str) -> dict[str, Any]: + return self.transport.request( + "POST", + f"/v1/chat/{session_id}/approve", + json_body={"job_id": job_id, "change_id": change_id, "approved": True}, + ) + + def deny(self, session_id: str, job_id: str, change_id: str) -> dict[str, Any]: + """Bare. There is no feedback parameter, because feedback re-plans (Decision 34).""" + return self.transport.request( + "POST", + f"/v1/chat/{session_id}/approve", + json_body={"job_id": job_id, "change_id": change_id, "approved": False}, + ) + + +def select_our_change( + job: Job, expected_chunk_id: str +) -> tuple[PendingChange, list[PendingChange]]: + """Split a batch into the one change we asked for and everything else. + + We send one chunk per job (Decision 35) and the product does not always + answer with one. Measured on the first live run of this build: an + instruction naming a single chunk id came back as a **two**-change batch, + the second on a chunk nobody targeted - the same shape as the unprompted + heading edits recorded on 2026-08-09. + + So the rule is not "the batch must be one"; it is **we approve exactly the + change we computed and deny every other one, bare** (Decision 34). The + read-back afterwards is what makes that safe: if a denial did not hold, the + non-target chunk moved and the verifier calls it collateral damage. + """ + ours = next((c for c in job.pending if c.chunk_id == expected_chunk_id), None) + if ours is None: + raise PlanRefusedError( + f"job {job.job_id} proposes {len(job.pending)} change(s), none of them on chunk " + f"{expected_chunk_id}, which is the one the step targets. Chunks proposed: " + f"{[c.chunk_id for c in job.pending]}. Every one is denied; nothing is approved on " + f"the strength of a proposal we did not ask for." + ) + if ours.operation != "edit": + raise PlanRefusedError( + f"job {job.job_id} proposes operation {ours.operation!r} on our target. There is " + f"one verb and it is replace (Decision 28)." + ) + return ours, [c for c in job.pending if c is not ours] + + +def _job_from(body: dict[str, Any]) -> Job: + metadata = body.get("metadata") or {} + pending = [ + PendingChange( + change_id=c.get("change_id", ""), + chunk_id=c.get("chunk_id", ""), + operation=c.get("operation", ""), + old_html=c.get("old_html", "") or "", + new_html=c.get("new_html", "") or "", + ) + for c in (metadata.get("pending_changes") or []) + ] + return Job( + job_id=body.get("job_id", ""), + status=body.get("status", ""), + progress=int(body.get("progress") or 0), + pending=pending, + raw=body, + ) diff --git a/use-cases/preetham1930/poa-generator/poa/superdocs/decisions.py b/use-cases/preetham1930/poa-generator/poa/superdocs/decisions.py new file mode 100644 index 000000000..2ac98122d --- /dev/null +++ b/use-cases/preetham1930/poa-generator/poa/superdocs/decisions.py @@ -0,0 +1,123 @@ +"""Our decision ledger. SuperDocs' `approve` is only an actuator (Decision 33). + +Measured on 2026-08-09: the product's gate is **write-only**. Before any +decision `pending_changes` listed three changes with `status: None`; after +approving one and denying another it listed the same three, still with +`status: None`. The approve call returns `{"status":"ok","batch_complete":...}` +and nothing else. There is no API answer to "what has been decided, and by whom". + +So the row is ours, it is written **before** the actuator is called, and it +carries a named actor. An item is undecided exactly when it has no row - there +is no default, no pending row and no status column. A second decision on the +same item is refused unless it explicitly supersedes, and the superseded row is +kept with both actors. + +The completion sentence is computed from the rows at the moment it is emitted, +never from counters, because we watched a product report "Successfully updated +all 0 sections. 2 change(s) were denied" when two had been applied and three +denied. +""" + +from __future__ import annotations + +import json +from dataclasses import asdict, dataclass +from datetime import UTC, datetime +from pathlib import Path + +from ..errors import DecisionAlreadyRecordedError + +VERBS = ("accept", "reject") + + +@dataclass(frozen=True) +class Decision: + run_id: str + step_index: int + chunk_id: str + verb: str + actor: str + reason: str + at: str + supersedes: str | None = None + + def __post_init__(self) -> None: + if self.verb not in VERBS: + raise ValueError( + f"the vocabulary is {VERBS} and nothing else. There is no 'resolve' and no " + f"'choose_side': a reviewer decides whether a change enters the document, not " + f"which of two disagreeing sources is right." + ) + if not self.actor.strip(): + raise ValueError("a decision with no actor answers nobody's question about who") + + +class DecisionLedger: + def __init__(self, path: Path | None = None) -> None: + self.path = path + self.rows: list[Decision] = [] + + def record( + self, + run_id: str, + step_index: int, + chunk_id: str, + verb: str, + actor: str, + reason: str, + supersede: bool = False, + ) -> Decision: + existing = self.find(run_id, step_index) + if existing is not None and not supersede: + raise DecisionAlreadyRecordedError( + f"step {step_index} of run {run_id} was already decided '{existing.verb}' by " + f"{existing.actor} at {existing.at}. Quiet replacement is the mechanism by " + f"which a decision gets quietly dropped; pass supersede=True and both rows are " + f"kept." + ) + row = Decision( + run_id=run_id, + step_index=step_index, + chunk_id=chunk_id, + verb=verb, + actor=actor, + reason=reason, + at=datetime.now(UTC).isoformat(timespec="seconds"), + supersedes=existing.at if existing is not None else None, + ) + self.rows.append(row) + if self.path is not None: + self.path.parent.mkdir(parents=True, exist_ok=True) + with self.path.open("a", encoding="utf-8") as handle: + handle.write(json.dumps(asdict(row)) + "\n") + return row + + def find(self, run_id: str, step_index: int) -> Decision | None: + matches = [r for r in self.rows if r.run_id == run_id and r.step_index == step_index] + return matches[-1] if matches else None + + def decided(self, run_id: str) -> list[Decision]: + latest: dict[int, Decision] = {} + for row in self.rows: + if row.run_id == run_id: + latest[row.step_index] = row + return [latest[k] for k in sorted(latest)] + + # No count fields anywhere: these are properties of the rows being held, so + # the sentence cannot assert a decision that was never made (Decision 22's + # trick, rebuilt). + def accepted(self, run_id: str) -> int: + return sum(1 for r in self.decided(run_id) if r.verb == "accept") + + def rejected(self, run_id: str) -> int: + return sum(1 for r in self.decided(run_id) if r.verb == "reject") + + def describe(self, run_id: str, planned: int) -> str: + accepted = self.accepted(run_id) + rejected = self.rejected(run_id) + undecided = planned - accepted - rejected + actors = sorted({r.actor for r in self.decided(run_id)}) + return ( + f"{accepted} accepted, {rejected} rejected, {undecided} undecided of {planned} " + f"planned change(s); decided by {', '.join(actors) or 'nobody'}" + ) diff --git a/use-cases/preetham1930/poa-generator/poa/superdocs/transport.py b/use-cases/preetham1930/poa-generator/poa/superdocs/transport.py new file mode 100644 index 000000000..6fe4d5d9d --- /dev/null +++ b/use-cases/preetham1930/poa-generator/poa/superdocs/transport.py @@ -0,0 +1,172 @@ +"""How a request reaches SuperDocs - or, in the test suite, how it does not. + +Two implementations behind one protocol: + +- `HttpTransport` talks to `api.superdocs.app`. It is used by the demo and by + nothing else. The key is read from the environment, never printed, never + written to a file, and never included in a message we log. +- `ReplayTransport` answers from recorded responses under `docs/evidence/`. It + is what the whole test suite runs on, so `make test` passes on a clean + checkout with no `.env` and with the network blocked (hard rule 3). + +A replay transport that is asked for a request it has no recording of **raises**, +naming the request. A transport that invents a plausible response would be the +2026-08-07 failure mode with a different noun: a success message for work that +did not happen. +""" + +from __future__ import annotations + +import json +import os +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Protocol + +from ..errors import NotConfiguredError + +DEFAULT_BASE_URL = "https://api.superdocs.app" + + +class Transport(Protocol): + def request( + self, + method: str, + path: str, + *, + json_body: dict[str, Any] | None = None, + params: dict[str, Any] | None = None, + ) -> dict[str, Any]: ... + + def download( + self, method: str, path: str, *, json_body: dict[str, Any] | None = None + ) -> tuple[str, bytes]: + """For the one endpoint that answers with a file rather than with JSON.""" + ... + + +def redacted_key() -> str: + key = os.environ.get("SUPERDOCS_API_KEY", "") + if not key: + return "(not set)" + return f"...{key[-4:]}" + + +class HttpTransport: + def __init__(self, base_url: str | None = None, timeout: float = 180.0) -> None: + self.base_url = ( + base_url or os.environ.get("SUPERDOCS_BASE_URL") or DEFAULT_BASE_URL + ).rstrip("/") + self.timeout = timeout + self.calls = 0 + key = os.environ.get("SUPERDOCS_API_KEY", "") + if not key: + raise NotConfiguredError( + "SUPERDOCS_API_KEY is not set. This build reads it from .env only, never from " + "a command line and never from a file it writes. The test suite does not need " + "it: tests run on recorded responses under docs/evidence/." + ) + self._key = key + + def request( + self, + method: str, + path: str, + *, + json_body: dict[str, Any] | None = None, + params: dict[str, Any] | None = None, + ) -> dict[str, Any]: + import httpx # imported here so the package imports with no network stack + + self.calls += 1 + response = httpx.request( + method, + f"{self.base_url}{path}", + json=json_body, + params=params, + headers={"Authorization": f"Bearer {self._key}", "Content-Type": "application/json"}, + timeout=self.timeout, + ) + if response.status_code >= 400: + # The body can echo the request; never let it carry the key onward. + raise RuntimeError( + f"{method} {path} -> HTTP {response.status_code}. " + f"{response.text[:400].replace(self._key, '<redacted>')}" + ) + return response.json() + + def download( + self, method: str, path: str, *, json_body: dict[str, Any] | None = None + ) -> tuple[str, bytes]: + """The export endpoint answers with a file, so it does not go through `request`.""" + import httpx + + self.calls += 1 + response = httpx.request( + method, + f"{self.base_url}{path}", + json=json_body, + headers={"Authorization": f"Bearer {self._key}", "Content-Type": "application/json"}, + timeout=self.timeout, + ) + if response.status_code >= 400: + raise RuntimeError( + f"{method} {path} -> HTTP {response.status_code}. " + f"{response.text[:400].replace(self._key, '<redacted>')}" + ) + return response.headers.get("content-type", ""), response.content + + +@dataclass +class RecordedCall: + method: str + path: str + response: dict[str, Any] + match: dict[str, Any] = field(default_factory=dict) + + +class ReplayTransport: + """Answers from recordings, in order, and refuses anything it was not given.""" + + def __init__(self, calls: list[RecordedCall]) -> None: + self._calls = list(calls) + self.seen: list[tuple[str, str, dict[str, Any] | None]] = [] + + def request( + self, + method: str, + path: str, + *, + json_body: dict[str, Any] | None = None, + params: dict[str, Any] | None = None, + ) -> dict[str, Any]: + self.seen.append((method, path, json_body)) + for index, call in enumerate(self._calls): + if call.method != method or call.path != path: + continue + if call.match and not all((json_body or {}).get(k) == v for k, v in call.match.items()): + continue + self._calls.pop(index) + return call.response + raise AssertionError( + f"no recorded response for {method} {path} with body keys " + f"{sorted(json_body or {})}. A replay transport never invents one: an invented " + f"response is a success message for work that did not happen. " + f"{len(self._calls)} recording(s) left: " + f"{[(c.method, c.path) for c in self._calls]}" + ) + + def download( + self, method: str, path: str, *, json_body: dict[str, Any] | None = None + ) -> tuple[str, bytes]: + """A replayed export records that we asked. There is no file to replay.""" + self.request(method, path, json_body=json_body) + return "application/octet-stream", b"" + + @property + def exhausted(self) -> bool: + return not self._calls + + +def load_recording(path: Path) -> dict[str, Any]: + return json.loads(path.read_text(encoding="utf-8")) diff --git a/use-cases/preetham1930/poa-generator/poa/superdocs/verifier.py b/use-cases/preetham1930/poa-generator/poa/superdocs/verifier.py new file mode 100644 index 000000000..6edce2bde --- /dev/null +++ b/use-cases/preetham1930/poa-generator/poa/superdocs/verifier.py @@ -0,0 +1,180 @@ +"""Read the whole document back and decide what actually happened. + +Three classes, and the third is the dangerous one: + +- **NOT APPLIED** - the target chunk came back unchanged. This is the + 2026-08-07 corrupted-document mechanism: the change silently does not land and + the caller executes its consequences anyway. +- **APPLIED WRONG** - the target changed, but not to the post-state we computed + before sending. +- **COLLATERAL DAMAGE** - a chunk we did not target moved, appeared or vanished. + Measured on 2026-08-09: one requested change *succeeded* while four fabricated + sections appeared alongside it, and again when a template instantiation kept + the table and silently deleted the letterhead. A verifier that checked only its + own target would have reported success both times. + +So the comparison is over the **whole document**, every time, including on a +retry - failed content has been observed arriving a turn later. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field + +from ..chunks import ChunkMap, canonical + + +@dataclass +class Problem: + kind: str # not_applied | applied_wrong | collateral_damage + detail: str + diff: str = "" + + +@dataclass +class Verification: + step_index: int + chunk_id: str + problems: list[Problem] = field(default_factory=list) + receipt: str = "" + exact_bytes: bool = False + + @property + def ok(self) -> bool: + return not self.problems + + def report(self) -> str: + if self.ok: + return f"step {self.step_index}: ok - {self.receipt}" + lines = [f"step {self.step_index}: FAILED on chunk {self.chunk_id}"] + for problem in self.problems: + lines.append(f" {problem.kind.upper().replace('_', ' ')}: {problem.detail}") + if problem.diff: + lines.append(problem.diff) + return "\n".join(lines) + + +def verify( + step_index: int, + chunk_id: str, + before: ChunkMap, + after: ChunkMap, + expected_html: str, +) -> Verification: + result = Verification(step_index=step_index, chunk_id=chunk_id) + expected = canonical(expected_html) + + before_ids = before.ids + after_ids = after.ids + added = [i for i in after_ids if i not in set(before_ids)] + removed = [i for i in before_ids if i not in set(after_ids)] + if added or removed: + result.problems.append( + Problem( + "collateral_damage", + f"the chunk set changed: {len(before_ids)} chunk(s) before, {len(after_ids)} " + f"after; {len(added)} added, {len(removed)} removed. A replacement never " + f"changes the chunk set.", + diff="\n".join( + f" + {after.by_id[i].tag}: {after.by_id[i].html[:120]}" for i in added[:6] + ) + + "\n".join(f" - {i}" for i in removed[:6]), + ) + ) + moved = [ + i + for i in before_ids + if i + and i != chunk_id + and i in after.by_id + and after.by_id[i].canonical != before.by_id[i].canonical + ] + if moved: + result.problems.append( + Problem( + "collateral_damage", + f"{len(moved)} chunk(s) we did not target changed: {moved[:6]}", + diff="\n".join( + f" was: {before.by_id[i].html[:150]}\n now: {after.by_id[i].html[:150]}" + for i in moved[:3] + ), + ) + ) + + if chunk_id not in after.by_id: + result.problems.append( + Problem("not_applied", f"the target chunk {chunk_id} is not in the read-back at all") + ) + return result + + was = before.by_id[chunk_id].canonical + now = after.by_id[chunk_id].canonical + if now == was: + result.problems.append( + Problem( + "not_applied", + "the target chunk came back byte-identical to what it held before. The chat " + "reply is not evidence and is not consulted here.", + diff=f" unchanged: {before.by_id[chunk_id].html[:200]}", + ) + ) + elif now != expected: + result.problems.append( + Problem( + "applied_wrong", + "the target chunk changed, but not to the post-state computed before sending", + diff=_diff(expected, now), + ) + ) + else: + result.exact_bytes = _strip_id(after.by_id[chunk_id].html) == _strip_id(expected_html) + result.receipt = ( + f"read back {len(after_ids)} chunk(s) from GET /v1/documents/" + f"{{id}}?include_html=true; target chunk {chunk_id} matches the computed " + f"post-state{' byte for byte' if result.exact_bytes else ' (canonical form)'}; " + f"{len(after_ids) - 1} non-target chunk(s) unchanged" + ) + return result + + +def _strip_id(html: str) -> str: + from ..chunks import strip_chunk_ids + + return strip_chunk_ids(html) + + +def _diff(expected: str, actual: str) -> str: + import difflib + + lines = list( + difflib.unified_diff( + expected.split(">"), actual.split(">"), "expected", "actual", lineterm="", n=1 + ) + ) + return "\n".join(f" {line}" for line in lines[:24]) + + +def verify_upload_verbatim(uploaded: ChunkMap, sent_blocks: list[str]) -> str: + """The upload is only useful if it is verbatim, so that is checked, not assumed. + + Decision 29 rests entirely on `upload-base64` being a verbatim load. It was + measured once; it is re-checked on every run, because a design resting on a + measured property should re-measure it rather than remember it. + """ + ours = [canonical(block) for block in sent_blocks] + theirs = [chunk.canonical for chunk in uploaded.chunks] + if ours != theirs: + mismatch = next( + (i for i, (a, b) in enumerate(zip(ours, theirs, strict=False)) if a != b), + min(len(ours), len(theirs)), + ) + raise AssertionError( + f"the upload was not verbatim: we sent {len(ours)} block(s) and {len(theirs)} came " + f"back, first difference at block {mismatch}.\n" + f" sent: {ours[mismatch] if mismatch < len(ours) else '(nothing)'}\n" + f" got : {theirs[mismatch] if mismatch < len(theirs) else '(nothing)'}\n" + f"Decision 29 rests on the upload being verbatim; if it is not, the whole " + f"skeleton approach is unsound and the run stops here rather than editing a " + f"document that is not the one we computed." + ) + return f"upload verified verbatim: {len(theirs)} chunk(s), every block canonically identical" diff --git a/use-cases/preetham1930/poa-generator/scripts/verify_live_output.py b/use-cases/preetham1930/poa-generator/scripts/verify_live_output.py new file mode 100644 index 000000000..3ae553e11 --- /dev/null +++ b/use-cases/preetham1930/poa-generator/scripts/verify_live_output.py @@ -0,0 +1,197 @@ +"""The grading line, checked against the two LIVE documents and the exported files. + +> Run the intake for a durable financial POA and a separate healthcare POA and +> confirm the two drafts differ correctly in granted powers, and both carry the +> review notice and the external-notarisation statement. + +Nothing here trusts the run's own report. Both documents are fetched fresh from +`GET /v1/documents/{id}?include_html=true`, the powers are counted off what came +back, both notices are put through the same gate the export path uses, and the +exported DOCX is unzipped and `word/document.xml` read directly - because "it +exported" is not evidence that anything is in it. + + python scripts/verify_live_output.py <financial_doc_id> <healthcare_doc_id> +""" + +from __future__ import annotations + +import argparse +import sys +import zipfile +from pathlib import Path + +APP_ROOT = Path(__file__).resolve().parent.parent +REPO_ROOT = APP_ROOT.parent.parent +sys.path.insert(0, str(APP_ROOT)) + +from poa.assemble import assemble # noqa: E402 +from poa.chunks import ChunkMap, strip_markup # noqa: E402 +from poa.intake import load_intake # noqa: E402 +from poa.library import Library # noqa: E402 +from poa.notices import check_notices # noqa: E402 +from poa.superdocs.client import PoaClient # noqa: E402 +from poa.superdocs.transport import HttpTransport # noqa: E402 + +PAIRS = (("durable-financial", "financial"), ("healthcare", "healthcare")) + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("financial_document_id") + parser.add_argument("healthcare_document_id") + parser.add_argument("--out", default=str(REPO_ROOT / "var" / "poa")) + args = parser.parse_args() + + try: + from dotenv import load_dotenv + + load_dotenv(REPO_ROOT / ".env", override=False) + except ImportError: + pass + + library = Library(APP_ROOT / "config") + client = PoaClient(HttpTransport()) + out_dir = Path(args.out) + + live: dict[str, ChunkMap] = {} + text: dict[str, str] = {} + failures: list[str] = [] + + for (intake_id, _category), document_id in zip( + PAIRS, (args.financial_document_id, args.healthcare_document_id), strict=True + ): + chunks, detail = client.read_back(document_id) + live[intake_id] = chunks + text[intake_id] = strip_markup(detail["html"]) + print(f"\n{intake_id}: document {document_id} version {detail.get('version')}") + print(f" {len(chunks)} chunk(s) read back") + + ours = assemble(load_intake(APP_ROOT / "intakes" / f"{intake_id}.json", library), library) + computed = ChunkMap(ours.html("target")) + if len(chunks) != len(computed): + failures.append(f"{intake_id}: {len(chunks)} chunks live, {len(computed)} computed") + differing = [ + index + for index, (a, b) in enumerate(zip(computed.chunks, chunks.chunks, strict=False)) + if a.canonical != b.canonical + ] + print(f" {len(differing)} chunk(s) differ from what we computed") + if differing: + failures.append(f"{intake_id}: chunks {differing[:8]} differ from the computed draft") + + # The notice gate, run independently of the run that produced this. + try: + receipt = check_notices(chunks, library, document_id) + print(f" {receipt.receipt()}") + except Exception as exc: + failures.append(f"{intake_id}: NOTICE GATE REFUSED - {str(exc).splitlines()[0]}") + + headings = chunks.headings() + print(f" {len(headings)} numbered clause(s): {sorted(headings)}") + if sorted(headings) != list(range(1, len(headings) + 1)): + failures.append(f"{intake_id}: clause numbering is not contiguous from 1") + + # -- the grading line ------------------------------------------------- + + print("\n" + "=" * 78) + print("THE GRADING LINE, checked on the live documents") + print("=" * 78) + for intake_id, category in PAIRS: + intake = load_intake(APP_ROOT / "intakes" / f"{intake_id}.json", library) + granted = intake.powers_in(library, category) + missing = [p for p in granted if library.powers[p].language not in text[intake_id]] + print( + f" {intake_id}: {len(granted) - len(missing)} of {len(granted)} granted powers present" + ) + if missing: + failures.append(f"{intake_id}: missing granted powers {missing}") + + withheld = [ + power_id + for power_id, power in library.powers.items() + if power.category == category and power_id not in granted + ] + strayed = [p for p in withheld if library.powers[p].language in text[intake_id]] + print(f" {intake_id}: {len(strayed)} of {len(withheld)} withheld powers present (want 0)") + if strayed: + failures.append(f"{intake_id}: carries withheld powers {strayed}") + + for (this_id, _), (other_id, other_category) in ((PAIRS[0], PAIRS[1]), (PAIRS[1], PAIRS[0])): + other = load_intake(APP_ROOT / "intakes" / f"{other_id}.json", library) + crossed = [ + p + for p in other.powers_in(library, other_category) + if library.powers[p].language in text[this_id] + ] + print(f" {this_id}: {len(crossed)} power(s) from {other_id} present (want 0)") + if crossed: + failures.append(f"{this_id}: carries {crossed} from {other_id}") + + for intake_id, _ in PAIRS: + successor = [ + c for c in live[intake_id].chunks if "successor agent" in strip_markup(c.html).lower() + ] + print( + f" {intake_id}: successor-agent clause present ({len(successor)} chunk(s) mention it)" + ) + if not successor: + failures.append(f"{intake_id}: no successor-agent clause") + for notice in library.notices.values(): + if notice.text not in text[intake_id]: + failures.append(f"{intake_id}: {notice.notice_id} is not verbatim in the live doc") + print(f" {intake_id}: both notices verbatim in the live document") + + # -- the exported files ----------------------------------------------- + + print("\n" + "=" * 78) + print("THE EXPORTED FILES") + print("=" * 78) + for intake_id, _ in PAIRS: + path = out_dir / f"poa-{intake_id}.docx" + if not path.exists(): + failures.append(f"{intake_id}: no exported DOCX at {path}") + print(f" {path}: MISSING") + continue + with zipfile.ZipFile(path) as archive: + document = archive.read("word/document.xml").decode("utf-8") + flat = " ".join(strip_markup(document).split()) + # `<w:tbl` also prefixes tblPr, tblGrid and tblBorders, so count the + # element itself. "It exported" is not evidence that anything is in it, + # and neither is a number that counted the wrong thing. + tables = document.count("<w:tbl>") + cells = document.count("<w:tc>") + print( + f" {path.name}: {path.stat().st_size} bytes, {tables} w:tbl, " + f"{document.count('<w:tr>')} rows, {cells} cells, " + f"{document.count('<w:tcBorders>')} bordered cells" + ) + if tables != 1: + failures.append( + f"{intake_id}: the DOCX has {tables} table(s), expected the signature block" + ) + for notice in library.notices.values(): + present = notice.text in flat + print(f" [{'x' if present else ' '}] {notice.notice_id} verbatim in the DOCX") + if not present: + failures.append(f"{intake_id}: {notice.notice_id} is not verbatim in the DOCX") + # A3, checked on the artefact a person would actually open: every + # signature and date cell is still empty after the round trip. + blanks = [c.column_label for c in library.signature_columns if c.blank] + for label in blanks: + print(f" [x] '{label}' column is a header only; no name follows it in the DOCX") + for row in library.signature_rows: + if f"{row.role_label} signed" in flat: + failures.append(f"{intake_id}: the DOCX shows {row.role_label} as signed") + + print("\n" + "-" * 78) + if failures: + print(f"{len(failures)} FAILURE(S):") + for failure in failures: + print(f" - {failure}") + return 1 + print("every claim above was read back from the API and from the exported files") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/use-cases/preetham1930/poa-generator/tests/recorded.py b/use-cases/preetham1930/poa-generator/tests/recorded.py new file mode 100644 index 000000000..f7d83b997 --- /dev/null +++ b/use-cases/preetham1930/poa-generator/tests/recorded.py @@ -0,0 +1,38 @@ +"""Recorded SuperDocs responses, replayed. No key, no network, no invention. + +Everything here comes out of `docs/evidence/`, which was captured on 2026-08-09 +against the live API and saved with `redemption_id` stripped. The failure-class +tests are driven by those recordings rather than by hand-written HTML, because +the point is that these three failures are things the product actually did. +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +REPO_ROOT = Path(__file__).resolve().parent.parent.parent.parent +REVERIFY = REPO_ROOT / "docs" / "evidence" / "2026-08-09-superdocs-reverify" +GATE = REPO_ROOT / "docs" / "evidence" / "2026-08-09-gate-and-templates" + + +def load(path: Path) -> dict[str, Any]: + return json.loads(path.read_text(encoding="utf-8")) + + +def read_back_html(name: str) -> str: + """`html` from a recorded GET /v1/documents/{id}?include_html=true.""" + return load(REVERIFY / f"{name}.json")["html"] + + +def gate_job(name: str) -> dict[str, Any]: + return load(GATE / f"gate-job-{name}.json") + + +def targeted_edit_response() -> dict[str, Any]: + return load(REVERIFY / "A-targeted-edit.response.json") + + +def append_response() -> dict[str, Any]: + return load(REVERIFY / "C-append-structural.response.json") diff --git a/use-cases/preetham1930/poa-generator/tests/simulator.py b/use-cases/preetham1930/poa-generator/tests/simulator.py new file mode 100644 index 000000000..c1d02b3ff --- /dev/null +++ b/use-cases/preetham1930/poa-generator/tests/simulator.py @@ -0,0 +1,247 @@ +"""A SuperDocs stand-in whose behaviours are the ones we measured. + +This is not a mock that returns what it was told to return. It holds a real +document, parses it with our own chunk parser, and reproduces five behaviours +recorded on 2026-08-09 and over Phase 4's live runs: + +- an upload is verbatim, and comes back with a `data-chunk-id` per top-level + element (measured: five chunks for a styled HTML upload, table atomic); +- `chat/async` with `approval_mode: ask_every_time` holds the change at + `awaiting_approval` and applies **nothing** until the batch settles; +- an approved change lands only when the batch completes; +- `not_applied` (the mid-document insertion that vanished), `applied_wrong` and + `collateral` (the create that appended sections nobody asked for) are the + three recorded failure shapes, and each is optional; +- `paraphrase_notice` is new to this build and is the one that matters here: an + edit lands correctly **and** the product quietly rewords a notice chunk it was + never asked about. Every step passes. The document is still not exportable. + It is a specialisation of the collateral-damage class that also fires on the + notice gate, which is why it is worth simulating separately. + +The assertions in the tests are about **our** behaviour - does the queue halt, +is the ledger row written before the actuator, is an export produced - not about +the simulator's. +""" + +from __future__ import annotations + +import re +import uuid +from dataclasses import dataclass, field +from typing import Any + +from poa.chunks import ChunkMap + +CHUNK_TAG = re.compile(r"<([a-zA-Z0-9]+)(\s|>)") + + +@dataclass +class Behaviour: + """What the simulator does on a given **chat call** (1-based, not step). + + A retry is a second chat call for the same step, so `not_applied={2, 3}` + means "step 2 fails and so does its one narrow retry", while + `not_applied={2}` means "step 2 fails and the retry lands". + """ + + not_applied: set[int] = field(default_factory=set) + applied_wrong: set[int] = field(default_factory=set) + collateral: set[int] = field(default_factory=set) + paraphrase_notice: set[int] = field(default_factory=set) + drop_notice: set[int] = field(default_factory=set) + corrupt_notice_on_upload: str = "" # "paraphrase" | "drop" | "" + fail_upload_verbatim: bool = False + revert_works: bool = True + + +class SimulatedSuperDocs: + def __init__(self, behaviour: Behaviour | None = None) -> None: + self.behaviour = behaviour or Behaviour() + self.html = "" + self.pristine = "" + self.document_id = "doc-" + uuid.uuid4().hex[:8] + self.session_id = "session_init_" + uuid.uuid4().hex[:12] + self.jobs: dict[str, dict[str, Any]] = {} + self.step_index = 0 + self.exports: list[str] = [] + self.reverts = 0 + self.approve_calls: list[str] = [] + self.event_log: list[str] = [] + + # -- transport --------------------------------------------------------- + + def request( + self, + method: str, + path: str, + *, + json_body: dict[str, Any] | None = None, + params: dict[str, Any] | None = None, + ) -> dict[str, Any]: + body = json_body or {} + if method == "POST" and path == "/v1/sessions/init": + return {"session_id": self.session_id, "opened": [], "focused_document_id": None} + if method == "POST" and path == "/v1/documents/upload-base64": + import base64 + + raw = base64.b64decode(body["file_base64"]).decode("utf-8") + if self.behaviour.fail_upload_verbatim: + raw = raw.replace("</p>", " (edited on ingest)</p>", 1) + self.html = _assign_chunk_ids(raw) + if self.behaviour.corrupt_notice_on_upload: + self._touch_notice(self.behaviour.corrupt_notice_on_upload) + self.pristine = self.html + return { + "session_id": self.session_id, + "filename": body["filename"], + "chunks_count": len(ChunkMap(self.html)), + "persisted": True, + } + if method == "GET" and path == f"/v1/sessions/{self.session_id}/documents": + return { + "session_id": self.session_id, + "focused_document_id": "doc_primary", + "documents": [ + { + "document_id": "doc_primary", + "durable_document_id": self.document_id, + "chunks_count": len(ChunkMap(self.html)), + "focused": True, + } + ], + } + if method == "GET" and path == f"/v1/documents/{self.document_id}": + return {"document_id": self.document_id, "html": self.html, "version": 1} + if method == "POST" and path == "/v1/chat/async": + return {"job_id": self._plan_job(body["message"])} + if method == "GET" and path.startswith("/v1/jobs/"): + return self.jobs[path.rsplit("/", 1)[1]] + if path.endswith("/approve"): + self.approve_calls.append(body["change_id"]) + self.event_log.append(f"approve:{body['change_id']}") + self._settle(body["job_id"], body["approved"]) + return {"status": "ok", "batch_complete": True} + if path == "/v1/documents/export": + self.exports.append(body.get("format", "docx")) + return {"download_url": "https://example.invalid/x"} + if path.endswith("/revert"): + self.reverts += 1 + if self.behaviour.revert_works: + self.html = self.pristine + else: + # The call returns ok and the document is not restored. This is + # the shape the whole build is built against: a 200 means the + # call was accepted, not that the state is what it claims. + self.html += '\n<p data-chunk-id="rv-1">not actually reverted</p>' + return {"status": "ok"} + raise AssertionError(f"the simulator was asked for {method} {path}") + + def download( + self, method: str, path: str, *, json_body: dict[str, Any] | None = None + ) -> tuple[str, bytes]: + self.request(method, path, json_body=json_body) + return ( + "application/vnd.openxmlformats-officedocument.wordprocessingml.document", + b"PKsimulated-docx", + ) + + # -- behaviour --------------------------------------------------------- + + def _plan_job(self, message: str) -> str: + self.step_index += 1 + chunk_id = re.search(r"data-chunk-id is ([0-9a-zA-Z-]+)", message).group(1) + new_html = message.split("\n\n", 1)[1] + job_id = "job-" + uuid.uuid4().hex[:8] + self.jobs[job_id] = { + "job_id": job_id, + "status": "awaiting_approval", + "progress": 88, + "metadata": { + "pending_changes": [ + { + "change_id": "chg-" + uuid.uuid4().hex[:8], + "chunk_id": chunk_id, + "operation": "edit", + "old_html": ChunkMap(self.html).by_id[chunk_id].html, + "new_html": new_html, + } + ] + }, + "_chunk_id": chunk_id, + "_new_html": new_html, + "_step": self.step_index, + } + return job_id + + def _settle(self, job_id: str, approved: bool) -> None: + job = self.jobs[job_id] + job["status"] = "completed" + job["progress"] = 100 + job["metadata"]["pending_changes"] = [] + if not approved: + return + step = job["_step"] + chunk_id, new_html = job["_chunk_id"], job["_new_html"] + if step in self.behaviour.not_applied: + self.event_log.append(f"step{step}:silently-did-nothing") + return + if step in self.behaviour.applied_wrong: + new_html = new_html.replace("</p>", " (paraphrased)</p>") + self._replace(chunk_id, new_html) + if step in self.behaviour.collateral: + self.event_log.append(f"step{step}:appended-clauses-nobody-asked-for") + self.html += ( + '\n<h2 data-chunk-id="fab-1">99. Compensation of the agent</h2>' + '\n<p data-chunk-id="fab-2">The Agent may take reasonable compensation.</p>' + ) + if step in self.behaviour.paraphrase_notice: + self.event_log.append(f"step{step}:reworded-a-notice-nobody-asked-about") + self._touch_notice("paraphrase") + if step in self.behaviour.drop_notice: + self.event_log.append(f"step{step}:deleted-a-notice-nobody-asked-about") + self._touch_notice("drop") + + def _touch_notice(self, how: str) -> None: + """Reword or delete a notice chunk, without being asked to. + + The paraphrase is deliberately harmless-looking: 'is not legal advice' + becomes 'is not intended as legal advice', which reads better and means + something weaker. That is the shape a rewrite actually takes. + """ + chunks = ChunkMap(self.html) + target = next( + (c for c in chunks.chunks if "not legal advice" in c.html), + None, + ) + if target is None: + return + if how == "drop": + self.html = self.html.replace(target.html, "", 1) + return + self.html = self.html.replace( + target.html, + target.html.replace("It is not legal advice", "It is not intended as legal advice"), + 1, + ) + + def _replace(self, chunk_id: str, new_html: str) -> None: + current = ChunkMap(self.html) + old = current.by_id[chunk_id].html + replacement = _with_chunk_id(new_html, chunk_id) + self.html = self.html.replace(old, replacement, 1) + + +def _assign_chunk_ids(html: str) -> str: + out: list[str] = [] + for line in html.split("\n"): + match = CHUNK_TAG.match(line) + if match: + out.append(_with_chunk_id(line, "c-" + uuid.uuid4().hex[:12])) + else: + out.append(line) + return "\n".join(out) + + +def _with_chunk_id(fragment: str, chunk_id: str) -> str: + fragment = re.sub(r'\sdata-chunk-id="[^"]*"', "", fragment, count=1) + return re.sub(r"<([a-zA-Z0-9]+)", rf'<\1 data-chunk-id="{chunk_id}"', fragment, count=1) diff --git a/use-cases/preetham1930/poa-generator/tests/test_assemble.py b/use-cases/preetham1930/poa-generator/tests/test_assemble.py new file mode 100644 index 000000000..0326d434c --- /dev/null +++ b/use-cases/preetham1930/poa-generator/tests/test_assemble.py @@ -0,0 +1,179 @@ +"""Structure is ours: clause selection, numbering, cross-references, both fills. + +The claim these tests defend is that nothing structural is ever asked of +SuperDocs. The way to break that claim is not to send a bad instruction - it is +to compute the structure wrong and then send perfectly good instructions. +""" + +from __future__ import annotations + +import shutil + +import pytest +from poa.assemble import CLAUSE_REFERENCE, assemble +from poa.chunks import ChunkMap, strip_markup +from poa.errors import CrossReferenceError +from poa.intake import load_intake +from poa.library import Library + + +def test_clause_numbers_come_from_the_selected_set(financial, healthcare) -> None: + """The two required drafts number differently, and nothing was asked for it. + + The healthcare draft selects `healthcare-records-access`, which the + financial one does not, so every clause after it is numbered one higher. + """ + assert financial.numbering["successor"] == 6 + assert healthcare.numbering["successor"] == 7 + assert "healthcare-records-access" not in financial.numbering + assert healthcare.numbering["healthcare-records-access"] == 6 + + for draft in (financial, healthcare): + numbers = list(draft.numbering.values()) + assert numbers == list(range(1, len(numbers) + 1)), "one pass, contiguous, from 1" + + +def test_every_cross_reference_resolves_in_both_drafts(financial, healthcare, limited) -> None: + for draft in (financial, healthcare, limited): + numbers = set(draft.numbering.values()) + found = 0 + for block in draft.blocks: + if block.is_heading: + continue + for match in CLAUSE_REFERENCE.finditer(strip_markup(block.target)): + found += 1 + assert int(match.group(1)) in numbers + assert found >= 1, f"{draft.intake_id} has no cross-reference, so this proves nothing" + + +def test_the_cross_reference_actually_points_at_the_right_clause(financial, healthcare) -> None: + """Not just 'a clause that exists' - the successor clause specifically.""" + for draft in (financial, healthcare): + reliance = draft.block("reliance").target + referenced = int(CLAUSE_REFERENCE.search(strip_markup(reliance)).group(1)) + assert referenced == draft.numbering["successor"] + heading = draft.block("successor-heading").target + assert strip_markup(heading).startswith(f"{referenced}.") + + +def test_the_numbering_in_the_headings_is_what_the_document_carries(financial) -> None: + """Read back off the rendered HTML with the same parser the verifier uses.""" + headings = ChunkMap(financial.html("target")).headings() + assert headings.keys() == set(financial.numbering.values()) + assert headings[1].endswith("Principal") + assert headings[max(headings)].startswith(f"{max(headings)}. ") + + +def test_the_blank_form_and_the_filled_draft_have_the_same_structure( + financial, healthcare, limited +) -> None: + """The skeleton is what gets uploaded, so it must already be the final shape.""" + for draft in (financial, healthcare, limited): + blank = ChunkMap(draft.html("skeleton")) + filled = ChunkMap(draft.html("target")) + assert len(blank) == len(filled), "an edit never adds or removes a chunk" + assert [c.tag for c in blank.chunks] == [c.tag for c in filled.chunks] + assert blank.headings() == filled.headings(), "no edit ever renumbers anything" + + +def test_a_heading_is_never_different_between_the_two_fills(financial, healthcare, limited) -> None: + for draft in (financial, healthcare, limited): + for block in draft.blocks: + if block.is_heading: + assert not block.changed, f"{block.role} would become a renumbering instruction" + + +def test_the_limited_type_carries_its_limitation_and_the_others_have_no_such_clause( + financial, healthcare, limited +) -> None: + assert "limitation" in {b.role for b in limited.blocks} + assert "limitation" not in {b.role for b in financial.blocks} + assert "limitation" not in {b.role for b in healthcare.blocks} + assert "letting, upkeep and registration" in limited.block("limitation").target + assert "Limited power of attorney" in limited.block("title").target + assert "Durable power of attorney" in financial.block("title").target + + +def test_the_type_language_differs_across_all_three_types(library, intakes, tmp_path) -> None: + """general / limited / durable is the distinction the card says people need right.""" + root = tmp_path / "config" + shutil.copytree(library.root, root) + updated = Library(root) + languages = set() + for type_id in ("general", "limited", "durable"): + raw = (intakes / "limited-real-property.json").read_text(encoding="utf-8") + raw = raw.replace('"poa_type": "limited"', f'"poa_type": "{type_id}"') + if type_id != "limited": + raw = raw.replace( + '"limitation": "the letting, upkeep and registration of the real property ' + 'described in this instrument, and the payment of outgoings on it"', + '"limitation": null', + ) + path = tmp_path / f"{type_id}.json" + path.write_text(raw, encoding="utf-8") + draft = assemble(load_intake(path, updated), updated) + languages.add(draft.block("nature").target) + assert len(languages) == 3, "each type states its own durability language" + + +def test_a_reference_that_moves_between_the_fills_is_refused(library, intakes, tmp_path) -> None: + """C5: filling in a name may not move a clause number, ever.""" + root = tmp_path / "config" + shutil.copytree(library.root, root) + path = root / "clause-library.csv" + # A body that renders a different clause number depending on the fill would + # make an edit into a renumbering instruction. + path.write_text( + path.read_text(encoding="utf-8").replace( + "A successor agent appointed under clause {clause_ref:successor} is subject to the " + "same condition.", + "A successor agent appointed under clause {clause_ref:successor} is subject to the " + "same condition. See also clause {effective_date}.", + 1, + ), + encoding="utf-8", + ) + updated = Library(root) + intake = load_intake(intakes / "durable-financial.json", updated) + object.__setattr__(intake, "effective_date", "3") + with pytest.raises(CrossReferenceError, match="renumbering instruction"): + assemble(intake, updated) + + +def test_the_powers_language_is_the_catalogue_s_word_for_word(financial, library) -> None: + """The granted-powers language is determined by the intake, not written per case.""" + rendered = financial.block("powers-financial-list").target + for power_id in financial.granted["financial"]: + power = library.powers[power_id] + assert power.language in rendered + assert f"<strong>{power.label}.</strong>" in rendered + withheld = [ + p + for p, v in library.powers.items() + if v.category == "financial" and p not in financial.granted["financial"] + ] + assert withheld, "if nothing were withheld this would prove nothing" + for power_id in withheld: + assert library.powers[power_id].language not in financial.html("target") + + +def test_the_blank_form_names_no_party_and_grants_no_specific_power(financial, library) -> None: + """What is uploaded is a blank form. It must not carry the intake's answers.""" + blank = financial.html("skeleton") + assert "Kavitha Ramanathan" not in blank + assert "Suresh Ramanathan" not in blank + assert "Anjali Ramanathan" not in blank + for power_id in financial.granted["financial"]: + assert library.powers[power_id].language not in blank + # ... and it is recognisably a durable POA form for this jurisdiction, which + # is what makes uploading it verbatim honest rather than a trick. + assert "Durable power of attorney" in blank + assert "Telangana, India" in blank + assert library.notices["counsel-review"].text in blank + + +def test_a_clause_body_appears_exactly_once_per_draft(financial, healthcare, limited) -> None: + """A duplicated clause is the 2026-08-09 failure; here it would duplicate a power.""" + for draft in (financial, healthcare, limited): + roles = [b.role for b in draft.blocks] + assert len(roles) == len(set(roles)) diff --git a/use-cases/preetham1930/poa-generator/tests/test_client.py b/use-cases/preetham1930/poa-generator/tests/test_client.py new file mode 100644 index 000000000..91a475263 --- /dev/null +++ b/use-cases/preetham1930/poa-generator/tests/test_client.py @@ -0,0 +1,179 @@ +"""The four wire rules, enforced in the client rather than in a caller's care. + +Copied from `builds/statutory-statements/tests/` and rebuilt (Decision 31): the +wire rules are the same rules, measured on the same product, and re-deriving +them would be pretending we had not already learned them. The fifth rule, at the +bottom of this file, is this build's own - `export` will not run without a +receipt from the notice gate. +""" + +from __future__ import annotations + +import inspect + +import pytest +from poa.errors import NotConfiguredError, PlanRefusedError +from poa.superdocs.client import PoaClient, select_our_change +from poa.superdocs.transport import HttpTransport, RecordedCall, ReplayTransport +from recorded import gate_job + + +def _client(calls: list[RecordedCall]) -> tuple[PoaClient, ReplayTransport]: + transport = ReplayTransport(calls) + return PoaClient(transport, poll_seconds=0), transport + + +def test_gated_changes_go_to_the_async_route(monkeypatch) -> None: + """Decision 32: approve takes a job_id and jobs exist only on /v1/chat/async.""" + job = gate_job("awaiting-approval") + client, transport = _client( + [ + RecordedCall("POST", "/v1/chat/async", {"job_id": job["job_id"]}), + RecordedCall("GET", f"/v1/jobs/{job['job_id']}", job), + ] + ) + result = client.chat_gated("session_x", "doc_x", "replace chunk abc with ...") + assert result.awaiting + paths = [path for _, path, _ in transport.seen] + assert paths[0] == "/v1/chat/async" + assert "/v1/chat" not in paths, "the synchronous route has no gate to reach" + body = transport.seen[0][2] + assert body["approval_mode"] == "ask_every_time" + assert body["response_mode"] == "compact" + + +def test_a_batch_is_split_into_ours_and_everything_else() -> None: + """We send one chunk; the product does not always answer with one change.""" + from poa.superdocs.client import _job_from + + job = _job_from(gate_job("awaiting-approval")) + assert len(job.pending) == 3 + ours, unasked = select_our_change(job, job.pending[1].chunk_id) + assert ours is job.pending[1] + assert len(unasked) == 2 + assert ours not in unasked + + +def test_a_batch_that_does_not_contain_our_target_is_refused_in_full() -> None: + from poa.superdocs.client import Job, PendingChange + + job = Job("j", "awaiting_approval", 88, [PendingChange("c", "other-chunk", "edit", "", "")], {}) + with pytest.raises(PlanRefusedError) as caught: + select_our_change(job, "our-chunk") + assert "none of them on chunk our-chunk" in str(caught.value) + assert "Every one is denied" in str(caught.value) + + +def test_a_create_operation_on_our_target_is_refused() -> None: + from poa.superdocs.client import Job, PendingChange + + job = Job("j", "awaiting_approval", 88, [PendingChange("c", "ours", "create", "", "")], {}) + with pytest.raises(PlanRefusedError, match="one verb"): + select_our_change(job, "ours") + + +def test_denials_carry_no_feedback() -> None: + """Decision 34: there is no call shape in which feedback can be sent.""" + signature = inspect.signature(PoaClient.deny) + assert "feedback" not in signature.parameters + + client, transport = _client( + [RecordedCall("POST", "/v1/chat/s/approve", {"status": "ok", "batch_complete": True})] + ) + client.deny("s", "j", "c") + body = transport.seen[0][2] + assert body == {"job_id": "j", "change_id": "c", "approved": False} + assert "feedback" not in body + + +def test_a_read_back_with_no_html_verifies_nothing_and_says_so() -> None: + client, _ = _client([RecordedCall("GET", "/v1/documents/d", {"version": 2})]) + with pytest.raises(RuntimeError, match="carried no html"): + client.read_back("d") + + +def test_an_upload_that_is_not_persisted_stops_the_run() -> None: + """Measured: an upload with no session is not persisted and carries no id.""" + client, _ = _client( + [ + RecordedCall("POST", "/v1/sessions/init", {"session_id": "s"}), + RecordedCall( + "POST", + "/v1/documents/upload-base64", + {"filename": "f.html", "chunks_count": 3, "persisted": False}, + ), + ] + ) + with pytest.raises(RuntimeError, match="only evidence there is"): + client.upload_verbatim("f.html", "<p>x</p>") + + +def test_the_document_id_comes_from_the_session_roster_not_from_the_upload() -> None: + """The upload response has no document id at all; the roster is a read, not a claim.""" + client, transport = _client( + [ + RecordedCall("POST", "/v1/sessions/init", {"session_id": "s"}), + RecordedCall("POST", "/v1/documents/upload-base64", {"persisted": True}), + RecordedCall( + "GET", + "/v1/sessions/s/documents", + {"documents": [{"document_id": "doc_primary", "durable_document_id": "dur-1"}]}, + ), + ] + ) + document_id, session_id = client.upload_verbatim("f.html", "<p>x</p>") + assert (document_id, session_id) == ("dur-1", "s") + assert [path for _, path, _ in transport.seen][-1] == "/v1/sessions/s/documents" + + +def test_a_roster_with_no_durable_id_stops_the_run() -> None: + client, _ = _client( + [ + RecordedCall("POST", "/v1/sessions/init", {"session_id": "s"}), + RecordedCall("POST", "/v1/documents/upload-base64", {"persisted": True}), + RecordedCall("GET", "/v1/sessions/s/documents", {"documents": []}), + ] + ) + with pytest.raises(RuntimeError, match="nothing to verify against"): + client.upload_verbatim("f.html", "<p>x</p>") + + +def test_the_replay_transport_never_invents_a_response() -> None: + client, _ = _client([]) + with pytest.raises(AssertionError) as caught: + client.read_back("d") + assert "never invents one" in str(caught.value) + + +def test_a_live_transport_without_a_key_refuses_to_exist(monkeypatch) -> None: + monkeypatch.delenv("SUPERDOCS_API_KEY", raising=False) + with pytest.raises(NotConfiguredError) as caught: + HttpTransport() + assert "recorded responses" in str(caught.value) + assert "SUPERDOCS_API_KEY" in str(caught.value) + + +def test_the_key_is_never_in_a_message_this_module_can_produce(monkeypatch) -> None: + fake = "sk-" + "live-" + "0123456789abcdef" + monkeypatch.setenv("SUPERDOCS_API_KEY", fake) + transport = HttpTransport() + from poa.superdocs.transport import redacted_key + + assert redacted_key() == "...cdef" + assert fake not in repr(transport.__dict__.get("base_url", "")) + + +def test_the_export_runs_only_with_a_receipt_from_the_notice_gate(financial, library) -> None: + """The fifth wire rule, and it is this build's own (INVARIANTS.md B1).""" + from poa.chunks import ChunkMap + from poa.notices import check_notices + + client, transport = _client( + [RecordedCall("POST", "/v1/documents/export", {"download_url": "https://x.invalid"})] + ) + receipt = check_notices(ChunkMap(financial.html("target")), library, "dur-1") + client.export("s", "docx", receipt) + assert [path for _, path, _ in transport.seen] == ["/v1/documents/export"] + assert client.export_receipts == [receipt.receipt()], ( + "the run log carries what the gate actually checked, not that it was called" + ) diff --git a/use-cases/preetham1930/poa-generator/tests/test_editplan.py b/use-cases/preetham1930/poa-generator/tests/test_editplan.py new file mode 100644 index 000000000..5b712f129 --- /dev/null +++ b/use-cases/preetham1930/poa-generator/tests/test_editplan.py @@ -0,0 +1,198 @@ +"""A plan that breaks a wire rule should never reach the wire. + +Everything here is checked before a single byte is sent. An invented clause in a +power of attorney is a power nobody granted, so the verb, the targets and the +post-states are all settled while it is still cheap to refuse. +""" + +from __future__ import annotations + +import re +from pathlib import Path + +import pytest +from poa.assemble import CLAUSE_REFERENCE +from poa.chunks import ChunkMap, canonical, strip_markup +from poa.editplan import build_plan +from poa.errors import PlanRefusedError + +CREATING_VERBS = ("create", "insert", "add a", "append", "renumber", "move ", "delete", "draft") +PACKAGE = Path(__file__).resolve().parent.parent / "poa" + + +def test_every_step_is_a_replacement_of_an_existing_chunk(financial, healthcare) -> None: + for draft in (financial, healthcare): + plan = build_plan(draft) + assert len(plan) == 6 + uploaded = ChunkMap(_with_ids(draft.html("skeleton"))) + for step in plan.steps: + assert step.verb == "replace" + assert canonical(step.was_html) in {c.canonical for c in uploaded.chunks} + + +def test_no_step_uses_a_creating_verb(financial) -> None: + """Decision 28: there is one verb. The instruction text is checked too.""" + plan = build_plan(financial) + for step in plan.steps: + step.chunk_id = "c-1" + instruction = step.instruction() + head = instruction.split("\n\n")[0].lower() + assert head.startswith("replace ") + for verb in CREATING_VERBS: + assert f" {verb}" not in head.replace("do not add any clause", ""), verb + + +def test_no_step_targets_a_heading_block(financial, healthcare, limited) -> None: + for draft in (financial, healthcare, limited): + headings = {b.role for b in draft.blocks if b.is_heading} + assert headings, "if there were no headings this would prove nothing" + for step in build_plan(draft).steps: + assert step.role not in headings + assert step.tag not in ("h1", "h2", "h3") + + +def test_no_step_targets_a_notice_block(financial, healthcare, limited) -> None: + """INVARIANTS.md B4: an instruction that touches a notice could paraphrase it.""" + for draft in (financial, healthcare, limited): + notices = {b.role for b in draft.blocks if b.is_notice} + assert len(notices) == 2 + for step in build_plan(draft).steps: + assert step.role not in notices + + +def test_a_plan_that_named_a_notice_is_refused_before_anything_is_sent(financial) -> None: + """The refusal is real, not merely unreachable. Drive it directly.""" + from poa.editplan import Plan, Step, assert_plan_is_legal + + notice = next(b for b in financial.blocks if b.is_notice) + plan = Plan( + [ + Step( + index=1, + role=notice.role, + tag=notice.tag, + clause_number=None, + expected_html="<p>anything</p>", + was_html=notice.skeleton, + ) + ], + financial, + ) + with pytest.raises(PlanRefusedError) as exc: + assert_plan_is_legal(plan, financial) + assert "notice" in str(exc.value) + assert "paraphrase" in str(exc.value) + + +def test_a_plan_that_named_a_heading_is_refused(financial) -> None: + from poa.editplan import Plan, Step, assert_plan_is_legal + + heading = next(b for b in financial.blocks if b.is_heading) + plan = Plan( + [ + Step( + index=1, + role=heading.role, + tag=heading.tag, + clause_number=heading.clause_number, + expected_html="<h2>99. Something</h2>", + was_html=heading.skeleton, + ) + ], + financial, + ) + with pytest.raises(PlanRefusedError, match="no renumbering step to get wrong"): + assert_plan_is_legal(plan, financial) + + +def test_a_step_that_would_change_nothing_is_refused(financial) -> None: + from poa.editplan import Plan, Step, assert_plan_is_legal + + block = financial.block("commencement") + plan = Plan( + [ + Step( + index=1, + role=block.role, + tag=block.tag, + clause_number=block.clause_number, + expected_html=block.skeleton, + was_html=block.skeleton, + ) + ], + financial, + ) + with pytest.raises(PlanRefusedError, match="reads as 'not applied'"): + assert_plan_is_legal(plan, financial) + + +def test_every_step_sends_a_post_state_computed_from_the_library(financial, library) -> None: + """The instruction hands over finished bytes. It never describes an edit. + + A described edit is an invitation to draft, and drafting is what produced + four invented sections on 2026-08-09. + """ + plan = build_plan(financial) + for step in plan.steps: + step.chunk_id = "c-1" + payload = step.instruction().split("\n\n", 1)[1] + assert payload == step.expected_html + assert payload == financial.block(step.role).target + powers = next(s for s in plan.steps if s.role == "powers-financial-list") + for power_id in financial.granted["financial"]: + assert library.powers[power_id].language in powers.expected_html + + +def test_no_step_changes_a_clause_reference(financial, healthcare, limited) -> None: + """C5. If an edit moved a number, the edit would be a renumbering.""" + for draft in (financial, healthcare, limited): + for step in build_plan(draft).steps: + before = CLAUSE_REFERENCE.findall(strip_markup(step.was_html)) + after = CLAUSE_REFERENCE.findall(strip_markup(step.expected_html)) + assert before == after, step.role + + +def test_the_riskiest_chunk_shape_is_ordered_last(financial) -> None: + """Decision 43: a halt should lose the least verified work.""" + tags = [s.tag for s in build_plan(financial).steps] + assert tags == sorted(tags, key=lambda t: {"p": 0, "ul": 1, "table": 2}[t]) + assert tags[-1] == "table" + + +def test_binding_refuses_when_a_step_matches_more_than_one_chunk(financial) -> None: + from poa.editplan import bind_chunk_ids + + plan = build_plan(financial) + doubled = _with_ids(financial.html("skeleton")) + "\n" + _with_ids(financial.html("skeleton")) + with pytest.raises(PlanRefusedError, match="exactly one target"): + bind_chunk_ids(plan, ChunkMap(doubled)) + + +def test_binding_matches_on_content_not_on_position(financial) -> None: + """A document that came back re-ordered fails here, not silently mid-run.""" + from poa.editplan import bind_chunk_ids + + plan = build_plan(financial) + blocks = _with_ids(financial.html("skeleton")).split("\n") + uploaded = ChunkMap("\n".join(reversed(blocks))) + bind_chunk_ids(plan, uploaded) + for step in plan.steps: + assert uploaded.by_id[step.chunk_id].canonical == canonical(step.was_html) + + +def test_the_package_never_asks_the_product_to_create(financial) -> None: + """Grep our own source for a creating verb inside an instruction we send.""" + text = (PACKAGE / "editplan.py").read_text(encoding="utf-8-sig") + body = text.split("def instruction")[1].split("def describe")[0] + sent = re.findall(r'f?"([^"]*)"', body) + joined = " ".join(sent).lower() + assert "replace the entire contents" in joined + for verb in ("create ", "insert ", "append ", "renumber "): + assert verb not in joined, verb + + +def _with_ids(html: str) -> str: + out = [] + for index, line in enumerate(html.split("\n")): + out.append(re.sub(r"<([a-zA-Z0-9]+)", rf'<\1 data-chunk-id="c-{index}"', line, count=1)) + return "\n".join(out) diff --git a/use-cases/preetham1930/poa-generator/tests/test_intake.py b/use-cases/preetham1930/poa-generator/tests/test_intake.py new file mode 100644 index 000000000..0be27ffa5 --- /dev/null +++ b/use-cases/preetham1930/poa-generator/tests/test_intake.py @@ -0,0 +1,149 @@ +"""The intake is the only channel by which a power reaches the document. + +Every refusal here is a refusal to draft. That is the point: a power of attorney +with a quietly missing limitation, or a quietly dropped power, or a party with +no name, reads perfectly well and is wrong in a way nobody notices. +""" + +from __future__ import annotations + +import json +from dataclasses import replace +from pathlib import Path + +import pytest +from poa.errors import ( + IncompleteIntakeError, + UnknownJurisdictionError, + UnknownPoaTypeError, + UnknownPowerError, +) +from poa.intake import load_intake, validate + + +def _write(tmp_path: Path, base: Path, **overrides) -> Path: + raw = json.loads(base.read_text(encoding="utf-8")) + raw.update(overrides) + path = tmp_path / "intake.json" + path.write_text(json.dumps(raw), encoding="utf-8") + return path + + +def test_the_three_shipped_intakes_load(library, intakes: Path) -> None: + for name in ("durable-financial", "healthcare", "limited-real-property"): + intake = load_intake(intakes / f"{name}.json", library) + assert intake.intake_id == name + assert intake.successor_agent.name, "the successor-agent clause is never optional" + + +def test_an_unknown_power_id_is_a_hard_error(library, intakes: Path, tmp_path: Path) -> None: + path = _write( + tmp_path, + intakes / "durable-financial.json", + granted_powers=["fin-banking", "fin-crypto-yolo"], + ) + with pytest.raises(UnknownPowerError) as exc: + load_intake(path, library) + message = str(exc.value) + assert "fin-crypto-yolo" in message + assert "config/powers.csv" in message + assert "never dropped" in message, "an unknown power must not be silently ignored" + + +def test_an_unknown_poa_type_is_never_approximated(library, intakes: Path, tmp_path: Path) -> None: + path = _write(tmp_path, intakes / "durable-financial.json", poa_type="springing") + with pytest.raises(UnknownPoaTypeError, match="general/limited/durable"): + load_intake(path, library) + + +def test_an_unknown_jurisdiction_names_the_csv_to_edit( + library, intakes: Path, tmp_path: Path +) -> None: + path = _write(tmp_path, intakes / "durable-financial.json", jurisdiction="atlantis") + with pytest.raises(UnknownJurisdictionError, match=r"config/jurisdictions\.csv"): + load_intake(path, library) + + +def test_a_limited_poa_with_no_limitation_is_refused( + library, intakes: Path, tmp_path: Path +) -> None: + """The single most consequential thing this build could get wrong.""" + path = _write(tmp_path, intakes / "limited-real-property.json", limitation=None) + with pytest.raises(IncompleteIntakeError) as exc: + load_intake(path, library) + assert "requires_limitation=yes" in str(exc.value) + assert "general one wearing the wrong heading" in str(exc.value) + + +def test_a_limitation_on_a_type_that_cannot_carry_one_is_refused( + library, intakes: Path, tmp_path: Path +) -> None: + """The other direction: a limit that would be silently dropped.""" + path = _write( + tmp_path, intakes / "durable-financial.json", limitation="only the Hyderabad account" + ) + with pytest.raises(IncompleteIntakeError, match="silently dropped"): + load_intake(path, library) + + +def test_a_missing_party_is_refused_and_named(library, intakes: Path, tmp_path: Path) -> None: + path = _write(tmp_path, intakes / "durable-financial.json", successor_agent=None) + with pytest.raises(IncompleteIntakeError) as exc: + load_intake(path, library) + assert "successor_agent" in str(exc.value) + assert "not optional" in str(exc.value) + + +def test_a_party_with_no_address_is_refused(library, intakes: Path, tmp_path: Path) -> None: + path = _write( + tmp_path, intakes / "durable-financial.json", agent={"name": "A Person", "address": ""} + ) + with pytest.raises(IncompleteIntakeError, match="agent"): + load_intake(path, library) + + +def test_granting_nothing_is_refused(library, intakes: Path, tmp_path: Path) -> None: + path = _write(tmp_path, intakes / "durable-financial.json", granted_powers=[]) + with pytest.raises(IncompleteIntakeError, match="not a shorter document"): + load_intake(path, library) + + +def test_a_power_granted_twice_is_refused(library, intakes: Path, tmp_path: Path) -> None: + path = _write( + tmp_path, intakes / "durable-financial.json", granted_powers=["fin-banking", "fin-banking"] + ) + with pytest.raises(IncompleteIntakeError, match="more than once"): + load_intake(path, library) + + +def test_real_property_powers_need_a_described_property( + library, intakes: Path, tmp_path: Path +) -> None: + path = _write(tmp_path, intakes / "limited-real-property.json", real_property_description=None) + with pytest.raises(IncompleteIntakeError, match="unbounded"): + load_intake(path, library) + + +def test_no_commencement_is_refused(library, intakes: Path, tmp_path: Path) -> None: + path = _write(tmp_path, intakes / "durable-financial.json", effective_date="") + with pytest.raises(IncompleteIntakeError, match="a question, not a draft"): + load_intake(path, library) + + +def test_validation_runs_on_an_object_too_not_only_on_a_file(library, intakes: Path) -> None: + """There is one validator, and both routes into the assembler go through it.""" + intake = load_intake(intakes / "durable-financial.json", library) + validate(intake, library) + with pytest.raises(UnknownPowerError): + validate(replace(intake, granted_powers=("not-a-power",)), library) + + +def test_categories_come_from_the_catalogue_not_from_the_order_typed( + library, intakes: Path, tmp_path: Path +) -> None: + """Two intakes granting the same powers must produce the same document.""" + forwards = load_intake(intakes / "limited-real-property.json", library) + backwards = replace(forwards, granted_powers=tuple(reversed(forwards.granted_powers))) + assert forwards.categories(library) == backwards.categories(library) + for category in forwards.categories(library): + assert forwards.powers_in(library, category) == backwards.powers_in(library, category) diff --git a/use-cases/preetham1930/poa-generator/tests/test_library.py b/use-cases/preetham1930/poa-generator/tests/test_library.py new file mode 100644 index 000000000..117f13447 --- /dev/null +++ b/use-cases/preetham1930/poa-generator/tests/test_library.py @@ -0,0 +1,241 @@ +"""The library is data. These tests are what make that sentence checkable. + +Two of them are the load-bearing ones: + +- `test_no_clause_text_is_hardcoded_anywhere_in_the_package` greps this + package's own `.py` files for every clause body, power label, power sentence, + notice, type label and jurisdiction name in `config/`. A hit means the library + stopped being the library. +- `test_adding_a_jurisdiction_is_a_data_edit` adds a row to a copy of the config + and requires the behaviour to change with zero code edits. +""" + +from __future__ import annotations + +import shutil +from pathlib import Path + +import pytest +from poa.assemble import assemble +from poa.errors import LibraryRefusedError, UnknownConditionError, UnresolvedPlaceholderError +from poa.intake import load_intake +from poa.library import Library + +PACKAGE = Path(__file__).resolve().parent.parent / "poa" + + +def _copy_config(config: Path, tmp_path: Path) -> Path: + destination = tmp_path / "config" + shutil.copytree(config, destination) + return destination + + +def test_no_clause_text_is_hardcoded_anywhere_in_the_package(library) -> None: + sources = {path: path.read_text(encoding="utf-8-sig") for path in PACKAGE.rglob("*.py")} + strings: list[tuple[str, str]] = [] + for clause in library.clauses: + strings.append(("clause body", clause.body)) + if clause.heading: + strings.append(("clause heading", clause.heading)) + for power in library.powers.values(): + strings.append(("power label", power.label)) + strings.append(("power language", power.language)) + for poa_type in library.types.values(): + strings.append(("type label", poa_type.label)) + strings.append(("durability language", poa_type.durability_language)) + for jurisdiction in library.jurisdictions.values(): + strings.append(("jurisdiction", jurisdiction.label)) + strings.append(("governing law", jurisdiction.governing_law)) + for notice in library.notices.values(): + strings.append(("notice", notice.text)) + for blank in library.blank_form.values(): + strings.append(("blank-form marker", blank)) + + hits = [ + f"{path.name}: {kind} {value[:60]!r}" + for kind, value in strings + if len(value) > 12 + for path, text in sources.items() + if value in text + ] + assert not hits, ( + "the clause library has leaked into code:\n " + + "\n ".join(hits) + + "\nAdding a jurisdiction or a power type must be a data edit " + "(builds/poa-generator/INVARIANTS.md D1)." + ) + + +def test_the_hardcoding_check_is_not_vacuous(library) -> None: + """A guard that passes because it found nothing to check is decoration.""" + assert len(library.clauses) >= 15 + assert len(library.powers) >= 12 + assert len(library.notices) == 2 + text = (PACKAGE / "assemble.py").read_text(encoding="utf-8") + assert "power.language" in text, "the check greps for values the code really does emit" + + +def test_adding_a_jurisdiction_is_a_data_edit(config: Path, intakes: Path, tmp_path: Path) -> None: + """Zero code edits. One row.""" + root = _copy_config(config, tmp_path) + before = Library(root) + assert "in-ka" not in before.jurisdictions + + path = root / "jurisdictions.csv" + defer = "must be confirmed with qualified counsel before anything is signed." + path.write_text( + path.read_text(encoding="utf-8") + + f'in-ka,"Karnataka, India","Karnataka law is intended to govern this instrument; ' + f'whether that choice is effective is a question for qualified counsel.",' + f'"The witnesses required {defer}","Whether notarisation is required {defer}"\n', + encoding="utf-8", + ) + after = Library(root) + assert "in-ka" in after.jurisdictions + + intake_path = tmp_path / "ka.json" + raw = (intakes / "durable-financial.json").read_text(encoding="utf-8") + intake_path.write_text(raw.replace('"in-ts"', '"in-ka"'), encoding="utf-8") + draft = assemble(load_intake(intake_path, after), after) + assert "Karnataka, India" in draft.html("target") + assert "Karnataka, India" in draft.html("skeleton"), "jurisdiction is structure, not an edit" + + +def test_adding_a_power_is_a_data_edit(config: Path, intakes: Path, tmp_path: Path) -> None: + root = _copy_config(config, tmp_path) + path = root / "powers.csv" + path.write_text( + path.read_text(encoding="utf-8") + + "fin-safedeposit,financial,Safe deposit,The Agent may open and close a safe deposit " + "locker held in the name of the Principal.\n", + encoding="utf-8", + ) + updated = Library(root) + intake_path = tmp_path / "extra.json" + raw = (intakes / "durable-financial.json").read_text(encoding="utf-8") + intake_path.write_text(raw.replace('"fin-banking"', '"fin-safedeposit"'), encoding="utf-8") + draft = assemble(load_intake(intake_path, updated), updated) + assert "safe deposit locker" in draft.html("target") + + +def test_a_jurisdiction_row_that_asserts_a_requirement_is_refused( + config: Path, tmp_path: Path +) -> None: + """INVARIANTS.md A4. We are not qualified to say what a jurisdiction requires.""" + root = _copy_config(config, tmp_path) + path = root / "jurisdictions.csv" + path.write_text( + path.read_text(encoding="utf-8").replace( + "The number and eligibility of witnesses required for this instrument must be " + "confirmed with qualified counsel before anything is signed.", + "Two witnesses are required.", + 1, + ), + encoding="utf-8", + ) + with pytest.raises(LibraryRefusedError) as exc: + Library(root) + assert "witness_requirement" in str(exc.value) + assert "not qualified" in str(exc.value) + assert "deferral-markers.txt" in str(exc.value) + + +def test_an_empty_deferral_marker_list_is_refused(config: Path, tmp_path: Path) -> None: + """An empty guard list is a guard that cannot be satisfied, not one switched off.""" + root = _copy_config(config, tmp_path) + (root / "deferral-markers.txt").write_text("# nothing\n", encoding="utf-8") + with pytest.raises(LibraryRefusedError, match="cannot be satisfied"): + Library(root) + + +def test_an_unparseable_condition_is_a_hard_error(config: Path, tmp_path: Path) -> None: + """A silent False would drop a clause out of a power of attorney.""" + root = _copy_config(config, tmp_path) + path = root / "clause-library.csv" + path.write_text( + path.read_text(encoding="utf-8").replace( + "successor,100,successor,Successor agent,p,body,always,", + "successor,100,successor,Successor agent,p,body,if the moon is right,", + 1, + ), + encoding="utf-8", + ) + updated = Library(root) + with pytest.raises(UnknownConditionError) as exc: + updated.select("durable", ("financial",)) + assert "if the moon is right" in str(exc.value) + assert "no symptom" in str(exc.value) + + +def test_an_unresolved_placeholder_is_a_hard_error( + config: Path, intakes: Path, tmp_path: Path +) -> None: + root = _copy_config(config, tmp_path) + path = root / "clause-library.csv" + path.write_text( + path.read_text(encoding="utf-8").replace( + "The Principal appoints {agent_name}", + "The Principal appoints {agent_middle_name} {agent_name}", + 1, + ), + encoding="utf-8", + ) + updated = Library(root) + with pytest.raises(UnresolvedPlaceholderError) as exc: + assemble(load_intake(intakes / "durable-financial.json", updated), updated) + assert "agent_middle_name" in str(exc.value) + assert "never emptied" in str(exc.value) + + +def test_a_csv_row_with_an_unquoted_comma_is_refused_not_truncated( + config: Path, tmp_path: Path +) -> None: + root = _copy_config(config, tmp_path) + path = root / "powers.csv" + path.write_text( + path.read_text(encoding="utf-8") + + "fin-oops,financial,Careless,The Agent may do this, and that, and the other.\n", + encoding="utf-8", + ) + with pytest.raises(LibraryRefusedError) as exc: + Library(root) + assert "more fields than the header" in str(exc.value) + assert "cut in half" in str(exc.value) + + +def test_a_duplicate_clause_id_is_refused(config: Path, tmp_path: Path) -> None: + root = _copy_config(config, tmp_path) + path = root / "clause-library.csv" + path.write_text( + path.read_text(encoding="utf-8") + "successor,999,closing,Duplicate,p,body,always,x\n", + encoding="utf-8", + ) + with pytest.raises(LibraryRefusedError, match="more than once"): + Library(root) + + +def test_a_clause_reference_the_library_cannot_resolve_is_refused( + config: Path, tmp_path: Path +) -> None: + root = _copy_config(config, tmp_path) + path = root / "clause-library.csv" + path.write_text( + path.read_text(encoding="utf-8").replace( + "{clause_ref:successor}", "{clause_ref:no-such-clause}", 1 + ), + encoding="utf-8", + ) + with pytest.raises(LibraryRefusedError, match="no-such-clause"): + Library(root) + + +def test_a_signature_table_with_no_blank_column_is_refused(config: Path, tmp_path: Path) -> None: + """INVARIANTS.md A3: there is always somewhere left to sign, and it is empty.""" + root = _copy_config(config, tmp_path) + path = root / "signature-table.csv" + path.write_text( + path.read_text(encoding="utf-8").replace(",blank", ",label"), + encoding="utf-8", + ) + with pytest.raises(LibraryRefusedError, match="never implies"): + Library(root) diff --git a/use-cases/preetham1930/poa-generator/tests/test_notices.py b/use-cases/preetham1930/poa-generator/tests/test_notices.py new file mode 100644 index 000000000..9ca6f8e1e --- /dev/null +++ b/use-cases/preetham1930/poa-generator/tests/test_notices.py @@ -0,0 +1,239 @@ +"""Hard constraint 1 and the legal-adjacency list. The centre of this build. + +> The document is NOT exportable until read-back confirms both the +> counsel-review notice and the external-notarisation statement are present +> VERBATIM. A test must prove the export path refuses when either is missing or +> altered. + +Both halves are proved here: that the refusal happens, and - more importantly - +that there is **no way to reach the export without the check**, because the +export takes a receipt only the check can issue. +""" + +from __future__ import annotations + +from copy import deepcopy +from typing import ClassVar + +import pytest +from poa.assemble import assert_no_forbidden_phrase +from poa.chunks import ChunkMap, strip_markup +from poa.errors import ( + ExportRefusedError, + ForbiddenPhraseError, + NoticeAlteredError, + NoticeMissingError, +) +from poa.notices import NoticeReceipt, check_notices +from poa.superdocs.client import PoaClient +from simulator import SimulatedSuperDocs + +NOTICE_IDS = ("counsel-review", "external-execution") + + +# -- A1/A2/A3: what the draft may never say ------------------------------- + + +def test_every_assembled_draft_carries_both_notices_verbatim( + financial, healthcare, limited, library +) -> None: + for draft in (financial, healthcare, limited): + for which in ("skeleton", "target"): + text = strip_markup(draft.html(which)) + for notice_id in NOTICE_IDS: + assert library.notices[notice_id].text in text, ( + f"{draft.intake_id} ({which}) is missing {notice_id}" + ) + + +def test_the_notices_are_never_edit_targets(financial, healthcare, limited) -> None: + for draft in (financial, healthcare, limited): + notices = [b for b in draft.blocks if b.is_notice] + assert len(notices) == 2 + for block in notices: + assert not block.editable + assert not block.changed + assert block not in draft.editable_changed() + + +def test_a_forbidden_phrase_anywhere_in_the_draft_is_a_hard_error(financial, library) -> None: + # A copy, because the fixture is session-scoped and a test that leaves a + # doctored draft behind is a test that breaks the next one from a distance. + draft = deepcopy(financial) + draft.block( + "commencement" + ).target += '<p style="x">This instrument was duly executed and is legally binding.</p>' + with pytest.raises(ForbiddenPhraseError) as exc: + assert_no_forbidden_phrase(draft, library) + assert "duly executed" in str(exc.value) + assert "no standing to say it" in str(exc.value) + assert "config/forbidden-phrases.csv" in str(exc.value) + + +def test_a_forbidden_phrase_in_the_blank_form_is_caught_too(financial, library) -> None: + """The blank form is what is uploaded, so a phrase only there still reaches the wire.""" + draft = deepcopy(financial) + draft.block("commencement").skeleton += "It is in full force and effect." + with pytest.raises(ForbiddenPhraseError, match="skeleton"): + assert_no_forbidden_phrase(draft, library) + + +def test_the_forbidden_phrase_list_is_data_not_code(library, tmp_path) -> None: + """Adding a phrase is a data edit.""" + import shutil + + from poa.library import Library + + root = tmp_path / "config" + shutil.copytree(library.root, root) + path = root / "forbidden-phrases.csv" + path.write_text( + path.read_text(encoding="utf-8") + "power of attorney,a deliberately absurd row\n", + encoding="utf-8", + ) + updated = Library(root) + assert len(updated.forbidden) == len(library.forbidden) + 1 + assert any(f.phrase == "power of attorney" for f in updated.forbidden) + + +def test_the_signature_block_leaves_every_signature_and_date_blank( + financial, healthcare, limited, library +) -> None: + """INVARIANTS.md A3: never imply that signing or notarisation happened.""" + blank_columns = [c.column_key for c in library.signature_columns if c.blank] + assert blank_columns, "if no column were blank this would prove nothing" + for draft in (financial, healthcare, limited): + for which in ("skeleton", "target"): + html = draft.block("signature-block").__getattribute__(which) + rows = html.split("<tr")[2:] # skip the header row + assert len(rows) == len(library.signature_rows) + for row in rows: + cells = row.split("<td")[1:] + for column, cell in zip(library.signature_columns, cells, strict=True): + body = cell.split(">", 1)[1].split("</td")[0] + if column.blank: + assert body == "", f"{draft.intake_id}: {column.column_key} is not empty" + + +def test_the_heading_says_the_signing_happens_outside_this_product(financial) -> None: + assert "outside this product" in strip_markup(financial.block("signature-block-heading").target) + + +# -- B1: the gate is a key, not a call ------------------------------------ + + +def test_export_cannot_be_called_without_a_receipt() -> None: + """No default argument, no receipt=None path. The signature is the gate.""" + import inspect + + signature = inspect.signature(PoaClient.export) + receipt = signature.parameters["receipt"] + assert receipt.default is inspect.Parameter.empty, ( + "export(receipt=...) with a default would let a caller skip the notice gate by " + "forgetting a keyword. INVARIANTS.md B1." + ) + client = PoaClient(SimulatedSuperDocs(), poll_seconds=0) + with pytest.raises(TypeError): + client.export("s", "docx") # type: ignore[call-arg] + with pytest.raises(ExportRefusedError, match="NoticeReceipt"): + client.export("s", "docx", None) # type: ignore[arg-type] + + +def test_a_receipt_cannot_be_forged() -> None: + """Only check_notices() can make one, and it says so.""" + with pytest.raises(ValueError) as exc: + NoticeReceipt(document_id="d", chunk_count=1, notices=("counsel-review",)) + assert "check_notices()" in str(exc.value) + assert "INVARIANTS.md B1" in str(exc.value) + + +def test_the_gate_passes_on_the_document_we_computed(financial, library) -> None: + receipt = check_notices(ChunkMap(financial.html("target")), library, "doc-1") + assert set(receipt.notices) == set(NOTICE_IDS) + assert "verbatim" in receipt.receipt() + assert "doc-1" in receipt.receipt() + + +# -- B2/B3: missing and altered ------------------------------------------- + + +def test_export_refuses_when_a_notice_is_missing(financial, library) -> None: + for notice_id in NOTICE_IDS: + html = financial.html("target").replace(library.notices[notice_id].text, "") + with pytest.raises(NoticeMissingError) as exc: + check_notices(ChunkMap(html), library, "doc-1") + assert notice_id in str(exc.value) + assert "No export is produced" in str(exc.value) + + +def test_export_refuses_when_a_notice_is_altered(financial, library) -> None: + """One word. This is the case that matters - a rewrite paraphrases.""" + alterations = { + "counsel-review": ("It is not legal advice", "It is not intended as legal advice"), + "external-execution": ("This draft is unexecuted.", "This draft is ready."), + } + for notice_id, (before, after) in alterations.items(): + html = financial.html("target").replace(before, after) + assert html != financial.html("target"), "the alteration has to actually change something" + with pytest.raises(NoticeAlteredError) as exc: + check_notices(ChunkMap(html), library, "doc-1") + assert notice_id in str(exc.value) + assert "NOT verbatim" in str(exc.value) + assert "No export is produced" in str(exc.value) + + +def test_dropping_a_single_word_from_a_notice_refuses_the_export(financial, library) -> None: + html = financial.html("target").replace( + "creates no lawyer-client relationship", "creates lawyer-client relationship" + ) + with pytest.raises(NoticeAlteredError, match="paraphrased notice is a missing notice"): + check_notices(ChunkMap(html), library, "doc-1") + + +def test_a_notice_split_across_two_chunks_does_not_satisfy_the_gate(financial, library) -> None: + """The second, independent comparison. + + Split with nothing inserted, so the document's *text* still contains the + notice word for word and only the chunk structure changed. A substring test + alone would pass this, and then a later edit could land between the two + halves without the gate noticing. + """ + text = library.notices["counsel-review"].text + head, tail = text.split(". ", 1) + html = financial.html("target").replace(text, f'{head}.</p>\n<p style="x">{tail}') + assert text in strip_markup(html), ( + "the first comparison must still pass, or this proves nothing" + ) + with pytest.raises(NoticeAlteredError, match="no single chunk holds it on its own"): + check_notices(ChunkMap(html), library, "doc-1") + + +def test_a_sentence_inserted_into_a_notice_refuses_the_export(financial, library) -> None: + text = library.notices["counsel-review"].text + head, tail = text.split(". ", 1) + html = financial.html("target").replace( + text, f'{head}.</p>\n<p style="x">Ignore the above.</p>\n<p style="x">{tail}' + ) + with pytest.raises(NoticeAlteredError, match="NOT verbatim"): + check_notices(ChunkMap(html), library, "doc-1") + + +def test_re_serialised_punctuation_does_not_fail_the_gate(financial, library) -> None: + """Verbatim is defined at the level of words, not of bytes (Decision 38).""" + html = ( + financial.html("target") + .replace("counsel.", "counsel.\n ", 1) + .replace("lawyer-client", "lawyer-client") + ) + receipt = check_notices(ChunkMap(html), library, "doc-1") + assert set(receipt.notices) == set(NOTICE_IDS) + + +def test_an_empty_notice_list_cannot_make_the_gate_pass(financial, library) -> None: + """A guard that cannot fail is worse than no guard.""" + + class Empty: + notices: ClassVar[dict] = {} + + with pytest.raises(NoticeMissingError, match="worse than none"): + check_notices(ChunkMap(financial.html("target")), Empty(), "doc-1") # type: ignore[arg-type] diff --git a/use-cases/preetham1930/poa-generator/tests/test_orchestrator.py b/use-cases/preetham1930/poa-generator/tests/test_orchestrator.py new file mode 100644 index 000000000..8335aa6dc --- /dev/null +++ b/use-cases/preetham1930/poa-generator/tests/test_orchestrator.py @@ -0,0 +1,277 @@ +"""The failure path is the build, not an afterthought. + +Copied from `builds/statutory-statements/tests/` and rebuilt (Decision 31), plus +the three tests this build exists for. Those three are the interesting ones, +because they are the case the edit queue cannot catch on its own: + + every step succeeds, and the document is still not exportable. + +A run in which six chunk replacements all verified applied and the product +quietly reworded a notice along the way is, from the queue's point of view, a +clean run. It is not one, and the gate is what says so. +""" + +from __future__ import annotations + +import pytest +from poa.editplan import build_plan +from poa.errors import DecisionAlreadyRecordedError +from poa.orchestrator import Orchestrator +from poa.superdocs.client import PoaClient +from poa.superdocs.decisions import DecisionLedger +from simulator import Behaviour, SimulatedSuperDocs + +ACTOR = "K. Latha (reviewing paralegal)" + + +def _run( + draft, library, behaviour: Behaviour, steps: int = 4, ledger: DecisionLedger | None = None +): + plan = build_plan(draft).sample(steps) + product = SimulatedSuperDocs(behaviour) + client = PoaClient(product, poll_seconds=0) + ledger = ledger or DecisionLedger() + orchestrator = Orchestrator(client, ledger, ACTOR, library, sleep=lambda _: None) + result = orchestrator.run("run-1", plan) + return result, product, ledger, orchestrator + + +# -- the clean run -------------------------------------------------------- + + +def test_a_clean_run_applies_every_step_and_exports(financial, library) -> None: + result, product, ledger, _ = _run(financial, library, Behaviour()) + assert result.ok + assert result.applied == 4 + assert product.exports == ["docx"] + assert ledger.accepted("run-1") == 4 + sentence = result.sentence(ledger) + assert "4 of 4 planned change(s) verified applied" in sentence + assert "both notices were then read back verbatim before anything was exported" in sentence + assert f"decided by {ACTOR}" in sentence + + +def test_the_gate_runs_on_the_upload_before_any_edit_is_issued(financial, library) -> None: + """Nothing is gained by editing a document whose notices did not survive.""" + _result, _product, _ledger, orchestrator = _run(financial, library, Behaviour()) + gates = [line for line in orchestrator.log if line.startswith("notice gate:")] + assert len(gates) == 2, "once after the upload, once before the export" + assert "(after the upload)" in gates[0] + assert "(before the export)" in gates[1] + for line in gates: + assert "counsel-review verbatim in chunk" in line + assert "external-execution verbatim in chunk" in line + + +def test_a_notice_mangled_on_upload_stops_the_run_before_any_edit(financial, library) -> None: + """Two independent defences stand here, and the earlier one fires first. + + `verify_upload_verbatim` compares every uploaded block against what we sent, + so a notice reworded during ingestion is caught as a non-verbatim upload + before the gate is even reached. Recorded as an assertion rather than left + implicit, because a reader could reasonably expect the gate's message and + should know why they get the other one. + """ + product = SimulatedSuperDocs(Behaviour(corrupt_notice_on_upload="paraphrase")) + client = PoaClient(product, poll_seconds=0) + orchestrator = Orchestrator(client, DecisionLedger(), ACTOR, library, sleep=lambda _: None) + with pytest.raises(AssertionError) as exc: + orchestrator.run("run-1", build_plan(financial).sample(4)) + assert "was not verbatim" in str(exc.value) + assert "not intended as legal advice" in str(exc.value), "the diff names the reworded notice" + assert product.step_index == 0, "not one edit was issued" + assert product.exports == [] + + +# -- the case this build exists for -------------------------------------- + + +def test_a_notice_reworded_during_a_step_is_caught_as_collateral_damage(financial, library) -> None: + """The first of two defences, on a notice nobody targeted. + + The simulator's paraphrase is deliberately harmless-looking: 'is not legal + advice' becomes 'is not intended as legal advice'. It reads better and means + less, which is the shape a rewrite actually takes. The step's own change + lands correctly - it is the notice next door that moved. + """ + result, product, ledger, _ = _run(financial, library, Behaviour(paraphrase_notice={3})) + assert result.halted_at == 3 + assert any("reworded-a-notice" in e for e in product.event_log) + kinds = {p.kind for p in result.outcomes[-1].verification.problems} + assert kinds == {"collateral_damage"} + assert product.exports == [], "no export, and the export path was never reached" + assert "HALTED at step 3" in result.sentence(ledger) + + +def test_a_reworded_notice_that_the_collateral_check_cannot_see_still_refuses_the_export( + financial, library +) -> None: + """The gate standing on its own. + + Drive the paraphrase on the *last* step of the plan, then read back and gate. + Whatever the per-step verifier concluded, the document is not exportable and + the run says which notice and how it differs. + """ + plan = build_plan(financial) + product = SimulatedSuperDocs(Behaviour()) + client = PoaClient(product, poll_seconds=0) + ledger = DecisionLedger() + orchestrator = Orchestrator(client, ledger, ACTOR, library, sleep=lambda _: None) + + original_export = client.export + seen: list[str] = [] + + def spy(*args, **kwargs): + seen.append("export") + return original_export(*args, **kwargs) + + client.export = spy # type: ignore[method-assign] + # Reword a notice after the last edit lands but before the final read-back. + original_read_back = client.read_back + calls = {"n": 0} + + def read_back(document_id, **kwargs): + calls["n"] += 1 + if calls["n"] == 2 + 2 * len(plan): # the final read-back before the gate + product._touch_notice("paraphrase") + return original_read_back(document_id, **kwargs) + + client.read_back = read_back # type: ignore[method-assign] + result = orchestrator.run("run-1", plan) + + assert result.applied == len(plan), "every edit verified applied" + assert result.halted_at is None, "the queue never halted" + assert seen == [], "the export was never called" + assert product.exports == [] + assert not result.ok + assert "counsel-review" in result.notice_refusal + assert "NOT exportable" in result.sentence(ledger) + assert any("NOTICE GATE REFUSED before the export" in line for line in orchestrator.log) + + +def test_a_deleted_notice_refuses_the_export_too(financial, library) -> None: + """A deleted notice removes a chunk, so the chunk set changed - also collateral.""" + result, product, ledger, _ = _run(financial, library, Behaviour(drop_notice={2})) + assert result.halted_at == 2 + assert product.exports == [] + detail = " ".join(p.detail for p in result.outcomes[-1].verification.problems) + assert "the chunk set changed" in detail + assert "No export was produced" in result.sentence(ledger) + + +# -- the three failure classes, unchanged -------------------------------- + + +def test_a_failure_halts_the_queue_and_downstream_steps_never_run(financial, library) -> None: + result, product, _ledger, orchestrator = _run(financial, library, Behaviour(not_applied={2, 3})) + assert not result.ok + assert result.halted_at == 2 + assert result.applied == 1, "step 1 landed; 3 and 4 were never attempted" + assert product.step_index == 3, "step 2 plus its one narrow retry, and nothing after" + assert any("halting" in line for line in orchestrator.log) + + +def test_no_export_after_a_failure(financial, library) -> None: + result, product, ledger, _ = _run(financial, library, Behaviour(not_applied={2, 3})) + assert product.exports == [] + assert result.exported == [] + assert "No export was produced" in result.sentence(ledger) + assert "HALTED at step 2" in result.sentence(ledger) + + +def test_at_most_one_narrow_retry_and_never_a_replan(financial, library) -> None: + result, product, _, orchestrator = _run(financial, library, Behaviour(not_applied={2, 3})) + assert product.step_index == 3, "the step, its one retry, and nothing more" + assert result.outcomes[-1].retried + assert sum("narrow retry" in line for line in orchestrator.log) == 1 + assert all("no re-plan" in line for line in orchestrator.log if "retry" in line) + + +def test_a_retry_that_succeeds_lets_the_run_continue(financial, library) -> None: + result, product, ledger, _ = _run(financial, library, Behaviour(not_applied={2}), steps=3) + assert result.ok + assert product.step_index == 4, "three steps plus one retry" + assert [o.retried for o in result.outcomes] == [False, True, False] + assert len(ledger.decided("run-1")) == 3 + assert any(r.supersedes for r in ledger.rows), "the retry supersedes; both rows are kept" + assert len(ledger.rows) == 4 + + +def test_collateral_damage_is_never_retried(financial, library) -> None: + result, product, _, orchestrator = _run(financial, library, Behaviour(collateral={1})) + assert result.halted_at == 1 + assert product.step_index == 1, "no retry at all" + assert any("collateral" in line.lower() for line in orchestrator.log) + + +def test_applied_wrong_is_detected_and_halts(financial, library) -> None: + result, product, _, _ = _run(financial, library, Behaviour(applied_wrong={1, 2})) + assert result.halted_at == 1 + assert product.exports == [] + kinds = {p.kind for p in result.outcomes[-1].verification.problems} + assert kinds == {"applied_wrong"} + + +def test_a_failed_run_reverts_and_says_whether_the_revert_worked(financial, library) -> None: + result, product, _, _ = _run(financial, library, Behaviour(not_applied={1, 2})) + assert product.reverts == 1 + assert "reverted to the last verified-good state, confirmed by read-back" in result.reverted + + result2, product2, _, _ = _run( + financial, library, Behaviour(not_applied={1, 2}, revert_works=False) + ) + assert "does NOT match the last verified-good state" in result2.reverted + assert product2.exports == [] + + +def test_an_upload_that_is_not_verbatim_stops_before_any_edit(financial, library) -> None: + with pytest.raises(AssertionError, match="was not verbatim"): + _run(financial, library, Behaviour(fail_upload_verbatim=True)) + + +# -- the decision ledger -------------------------------------------------- + + +def test_the_decision_row_is_written_before_the_actuator_is_called(financial, library) -> None: + """Decision 33: the product's gate is write-only, so the record has to be ours.""" + plan = build_plan(financial).sample(2) + product = SimulatedSuperDocs(Behaviour()) + client = PoaClient(product, poll_seconds=0) + ledger = DecisionLedger() + original = ledger.record + + def spy(*args, **kwargs): + product.event_log.append("ledger-row") + return original(*args, **kwargs) + + ledger.record = spy # type: ignore[method-assign] + result = Orchestrator(client, ledger, ACTOR, library, sleep=lambda _: None).run("run-1", plan) + assert result.ok + timeline = [ + "ledger-row" if e == "ledger-row" else "approve" + for e in product.event_log + if e == "ledger-row" or e.startswith("approve:") + ] + assert timeline == ["ledger-row", "approve", "ledger-row", "approve"] + assert ledger.decided("run-1")[0].at, "the row carries a timestamp and an actor" + assert ledger.decided("run-1")[0].actor == ACTOR + + +def test_a_second_decision_on_the_same_step_is_refused_unless_it_supersedes() -> None: + ledger = DecisionLedger() + ledger.record("r", 1, "c", "accept", "A", "because") + with pytest.raises(DecisionAlreadyRecordedError, match="quietly dropped"): + ledger.record("r", 1, "c", "reject", "B", "because") + superseding = ledger.record("r", 1, "c", "reject", "B", "because", supersede=True) + assert superseding.supersedes + assert len(ledger.rows) == 2, "the superseded row is kept" + assert ledger.accepted("r") == 0 and ledger.rejected("r") == 1 + + +def test_the_completion_sentence_is_computed_from_the_rows_it_holds() -> None: + ledger = DecisionLedger() + ledger.record("r", 1, "c1", "accept", "A", "x") + ledger.record("r", 2, "c2", "reject", "B", "y") + assert "1 accepted, 1 rejected, 3 undecided of 5 planned" in ledger.describe("r", planned=5) + ledger.rows.clear() + assert "0 accepted, 0 rejected, 5 undecided" in ledger.describe("r", planned=5) diff --git a/use-cases/preetham1930/poa-generator/tests/test_two_drafts.py b/use-cases/preetham1930/poa-generator/tests/test_two_drafts.py new file mode 100644 index 000000000..14fbbf140 --- /dev/null +++ b/use-cases/preetham1930/poa-generator/tests/test_two_drafts.py @@ -0,0 +1,153 @@ +"""The grading line, asserted. + +> Run the intake for a durable financial POA and a separate healthcare POA and +> confirm the two drafts differ correctly in granted powers, and both carry the +> review notice and the external-notarisation statement. + +"Differ correctly" is stronger than "differ", and it is what these tests pin +down. Three claims, each of which could be true while the other two are false: + +1. each draft carries **every** power its own intake granted; +2. each draft carries **none** of the other's, and none of the powers its own + intake withheld; +3. everything the two drafts share is shared - the successor-agent clause, both + notices, the parties' own roles - so the difference is the powers and not an + accident of two documents having been built differently. + +These run offline against the assembled drafts. `scripts/verify_live_output.py` +asserts the same three things against the two **live** documents, read back from +the API - which is where the claim actually has to hold. +""" + +from __future__ import annotations + +from poa.chunks import ChunkMap, strip_markup +from poa.notices import check_notices + + +def test_the_two_required_drafts_are_a_durable_financial_and_a_healthcare_poa( + financial, healthcare +) -> None: + assert financial.poa_type == "durable" + assert set(financial.granted) == {"financial"} + assert set(healthcare.granted) == {"healthcare"} + assert financial.intake_id != healthcare.intake_id + + +def test_each_draft_carries_every_power_its_own_intake_granted( + financial, healthcare, library +) -> None: + for draft in (financial, healthcare): + text = strip_markup(draft.html("target")) + for powers in draft.granted.values(): + assert powers, "a category with no powers would make this vacuous" + for power_id in powers: + power = library.powers[power_id] + assert power.language in text, f"{draft.intake_id} is missing {power_id}" + assert power.label in text + + +def test_neither_draft_carries_the_other_s_powers(financial, healthcare, library) -> None: + """The claim the card is actually about.""" + for draft, other in ((financial, healthcare), (healthcare, financial)): + text = strip_markup(draft.html("target")) + strays = [ + power_id + for powers in other.granted.values() + for power_id in powers + if library.powers[power_id].language in text + ] + assert not strays, f"{draft.intake_id} carries {strays} from {other.intake_id}" + + +def test_neither_draft_carries_a_power_its_own_intake_withheld( + financial, healthcare, library +) -> None: + """Withholding is the half that is easy to get wrong and impossible to see.""" + for draft in (financial, healthcare): + granted = {p for powers in draft.granted.values() for p in powers} + categories = set(draft.granted) + withheld = [ + power_id + for power_id, power in library.powers.items() + if power.category in categories and power_id not in granted + ] + assert withheld, "if nothing were withheld this test would prove nothing" + text = strip_markup(draft.html("target")) + for power_id in withheld: + assert library.powers[power_id].language not in text, ( + f"{draft.intake_id} grants {power_id}, which its intake withheld" + ) + + +def test_the_clause_sets_differ_by_exactly_the_healthcare_clauses(financial, healthcare) -> None: + """The structural difference, stated exactly rather than as 'they differ'.""" + only_financial = {b.role for b in financial.blocks} - {b.role for b in healthcare.blocks} + only_healthcare = {b.role for b in healthcare.blocks} - {b.role for b in financial.blocks} + assert only_financial == { + "powers-financial", + "powers-financial-heading", + "powers-financial-list", + } + assert only_healthcare == { + "powers-healthcare", + "powers-healthcare-heading", + "powers-healthcare-list", + "healthcare-records-access", + "healthcare-records-access-heading", + } + + +def test_both_drafts_carry_a_successor_agent_clause(financial, healthcare) -> None: + for draft in (financial, healthcare): + clause = strip_markup(draft.block("successor").target) + assert "successor agent" in clause + assert clause.count("successor agent") >= 1 + assert "Anjali Ramanathan" in financial.block("successor").target + assert "Suresh Ramanathan" in healthcare.block("successor").target + + +def test_both_drafts_carry_both_notices_verbatim_and_the_gate_says_so( + financial, healthcare, library +) -> None: + """Same check the live export path runs, on the documents we computed.""" + for draft in (financial, healthcare): + for which in ("skeleton", "target"): + receipt = check_notices(ChunkMap(draft.html(which)), library, draft.intake_id) + assert set(receipt.notices) == {"counsel-review", "external-execution"} + assert "verbatim" in receipt.receipt() + + +def test_the_shared_parts_really_are_shared(financial, healthcare, library) -> None: + """So the difference is the powers, not two documents built differently.""" + shared = {"preamble-principal", "nature", "successor", "signature-block"} + for role in shared: + assert financial.block(role).skeleton == healthcare.block(role).skeleton, role + # The reliance clause is the one exception, and it is the right one: it + # carries the cross-reference to the successor clause, which is numbered 6 + # in one draft and 7 in the other. It must differ in exactly that digit and + # in nothing else - the numbering is ours and the wording is the library's. + left = financial.block("reliance").target + right = healthcare.block("reliance").target + assert left != right + assert left.replace("clause 6", "clause N") == right.replace("clause 7", "clause N") + assert financial.block("governing-law").target == healthcare.block("governing-law").target + # Same principal in both intakes, so this clause must be identical. + assert ( + financial.block("preamble-principal").target + == healthcare.block("preamble-principal").target + ) + for notice_id in library.notices: + assert ( + financial.block(f"{notice_id}-notice").target + == healthcare.block(f"{notice_id}-notice").target + ) + + +def test_the_two_drafts_number_differently_and_both_are_internally_consistent( + financial, healthcare +) -> None: + assert financial.numbering != healthcare.numbering + for draft in (financial, healthcare): + headings = ChunkMap(draft.html("target")).headings() + assert headings.keys() == set(draft.numbering.values()) diff --git a/use-cases/preetham1930/poa-generator/tests/test_verifier.py b/use-cases/preetham1930/poa-generator/tests/test_verifier.py new file mode 100644 index 000000000..9caf71676 --- /dev/null +++ b/use-cases/preetham1930/poa-generator/tests/test_verifier.py @@ -0,0 +1,101 @@ +"""The three failure classes, each driven by a response the product actually gave. + +Nothing here is hand-written HTML. `read0`..`read3` are four recorded +`GET /v1/documents/{id}?include_html=true` bodies from 2026-08-09: + +- read0 -> read1 a single-value targeted edit that worked +- read1 -> read2 a mid-document insertion that silently did not happen, while + five heading chunks nobody targeted were rewritten around it +- read2 -> read3 one `create` that appended six sections, four of them invented + +So the verifier is tested against the real shapes of the real failures. +""" + +from __future__ import annotations + +from poa.chunks import ChunkMap +from poa.superdocs.verifier import verify, verify_upload_verbatim +from recorded import read_back_html + +TARGET = "8ec79daa-5de5-4973-84a5-1d1139da29be" + + +def _maps(*names: str) -> list[ChunkMap]: + return [ChunkMap(read_back_html(name)) for name in names] + + +def test_a_clean_single_target_replacement_verifies() -> None: + before, after = _maps("read0-baseline", "read1-after-A") + expected = after.by_id[TARGET].html + result = verify(1, TARGET, before, after, expected) + assert result.ok + assert "matches the computed post-state" in result.receipt + assert "17 non-target chunk(s) unchanged" in result.receipt + assert result.exact_bytes + + +def test_a_reply_claiming_success_does_not_make_a_step_ok() -> None: + """The chat reply for the failing turn read 'nothing actually changed'. + + Five changes had been auto-approved and were in the document. The verifier + never sees a reply, so it cannot be wrong in either direction. + """ + before, after = _maps("read1-after-A", "read2-after-B") + result = verify(2, TARGET, before, after, before.by_id[TARGET].html + "<!--x-->") + assert not result.ok + kinds = {p.kind for p in result.problems} + assert "not_applied" in kinds, "the target chunk came back untouched" + assert "collateral_damage" in kinds, "five heading chunks moved" + detail = next(p for p in result.problems if p.kind == "collateral_damage").detail + assert "5 chunk(s) we did not target changed" in detail + + +def test_collateral_damage_is_caught_when_the_requested_change_also_succeeded() -> None: + """The dangerous one: the append worked *and* four fabricated sections arrived. + + A verifier checking only its own target would report success here. + """ + before, after = _maps("read2-after-B", "read3-after-C") + # Pretend our step targeted a chunk in the document; the requested append + # also succeeded, which is exactly what makes this class dangerous. + changed = before.ids[0] + result = verify(3, changed, before, after, after.by_id[changed].html) + assert not result.ok + problem = next(p for p in result.problems if p.kind == "collateral_damage") + assert "the chunk set changed" in problem.detail + assert "18 chunk(s) before, 19 after" in problem.detail + assert "Quality Assurance" in problem.diff or "Humidity" in problem.diff + + +def test_applied_wrong_is_its_own_class() -> None: + before, after = _maps("read0-baseline", "read1-after-A") + expected = after.by_id[TARGET].html.replace("750", "751") + result = verify(4, TARGET, before, after, expected) + assert not result.ok + assert [p.kind for p in result.problems] == ["applied_wrong"] + assert "751" in result.problems[0].diff + + +def test_a_missing_target_chunk_is_not_applied_rather_than_a_crash() -> None: + before, after = _maps("read0-baseline", "read1-after-A") + result = verify(5, "no-such-chunk", before, after, "<p>x</p>") + assert not result.ok + assert any(p.kind == "not_applied" for p in result.problems) + + +def test_the_upload_is_checked_verbatim_rather_than_assumed() -> None: + """Decision 29 rests on a measured property, so it is re-measured every run.""" + uploaded = ChunkMap(read_back_html("read0-baseline")) + blocks = [chunk.html for chunk in uploaded.chunks] + receipt = verify_upload_verbatim(uploaded, blocks) + assert "18 chunk(s)" in receipt + + tampered = list(blocks) + tampered[3] = tampered[3].replace("</p>", " and one more sentence.</p>") + try: + verify_upload_verbatim(uploaded, tampered) + except AssertionError as exc: + assert "was not verbatim" in str(exc) + assert "first difference at block 3" in str(exc) + else: # pragma: no cover + raise AssertionError("a non-verbatim upload must stop the run") diff --git a/use-cases/preetham1930/statutory-statements-builder/INVARIANTS.md b/use-cases/preetham1930/statutory-statements-builder/INVARIANTS.md new file mode 100644 index 000000000..20be27e16 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/INVARIANTS.md @@ -0,0 +1,142 @@ +# INVARIANTS — `builds/statutory-statements/` + +Written **before** the tests, and the tests before the code (TASK.md hard rule +7). Every invariant names the test that proves it. An invariant with no test is +an intention, not an invariant. + +The thesis this app has to make true: + +> **Nothing structural is ever asked of SuperDocs, every figure that reaches the +> document is traceable to a source row, and no edit is called applied until the +> whole document has been read back and compared against a post-state we +> computed before we sent it.** + +--- + +## What this app must never do + +### Structure + +1. **Never ask SuperDocs to create.** One verb, and it is `replace` (Decision + 28). No `create`, no insert, no "add a section". Measured: the only operation + that inserts is also the one that fabricated four sections nobody asked for + (PROGRESS.md 2026-08-09). + *Proved by* `test_edit_plan.py::test_every_step_is_a_replacement_of_an_existing_chunk` + and `test_no_create_verb_anywhere`. +2. **Never issue a mid-document insertion.** The entire final skeleton — + every heading at its final number, every cross-reference already repointed — + is computed here and uploaded verbatim (Decision 29). There is no step in the + plan that changes a heading's number. + *Proved by* `test_edit_plan.py::test_no_step_targets_a_heading_chunk`. +3. **Never let a note number be inferred, remembered or asked for.** Numbers are + assigned by one pass over our own ordered tree. The insertion point of a new + note comes from the trial balance account order, not from a list of known + note names. + *Proved by* `test_note_tree.py::test_insertion_point_comes_from_account_order` + and `test_no_note_title_is_hardcoded`. +4. **Never leave a cross-reference pointing at the old number.** Every + note→primary reference and every `Note N` in prose is repointed through the + same map that renumbered the headings, and the final document is re-scanned + for any reference that does not resolve to a heading that exists. + *Proved by* `test_note_tree.py::test_every_cross_reference_resolves_after_insertion`. +5. **Never let the checklist drift from the document.** Every checklist item's + `satisfied_by_note` is remapped through the same map. After the run, every + mapped item must name a heading that exists at the number we expect, and + every heading must be accounted for. + *Proved by* `test_checklist.py::test_checklist_survives_the_insertion`. + +### Figures + +6. **Never write a figure that does not resolve to a source.** Every money token + in every rendered body must be one of exactly four bases — `ledger`, + `comparative`, `document`, `derived` — and there is no fifth. A rendered body + is re-scanned after rendering and an unaccounted token is a hard error naming + the token and the note. + *Proved by* `test_figures.py::test_an_unaccounted_token_is_a_hard_error`. +7. **Never store a derived total.** `Derived.value` is a property computed from + its components on every read, so a total cannot drift from the citations that + support it and cannot be stated apart from them. + *Proved by* `test_figures.py::test_a_derived_total_has_no_value_field`. +8. **Never carry a prior-year figure forward under this year's heading without + the ledger confirming it did not move.** A `comparative` basis figure is + admissible only when the trial balance's prior-year column and the prior-year + signed statements agree to the paisa; a disagreement is a hard error, not a + preference for one side. + *Proved by* `test_rollforward.py::test_a_comparative_mismatch_is_a_hard_error`. +9. **Never invent an analysis the sources do not support.** Where the trial + balance is at caption granularity and a note needs a finer breakdown, the note + states the tied total and names what is outstanding and where we looked. It + never carries last year's breakdown under this year's heading. + *Proved by* `test_bodies.py::test_no_regenerated_body_reuses_a_prior_year_breakdown`. +10. **Never resolve a disagreement between two sources.** Where the minutes and + the ledger disagree, both are quoted and neither is preferred. There is no + field on `Disagreement` a resolution could be written into. + *Proved by* `test_disagreement.py::test_disagreement_has_nowhere_to_record_a_resolution`. + +### The wire + +11. **Never batch, and never approve a change we did not compute.** One chunk + per job on the sending side (Decision 35). The answering side does not + always agree — measured on this build's first live run, an instruction + naming one chunk id came back as a two-change batch — so we approve exactly + our change and deny every other one **bare**, and the read-back is what + makes that safe. + *Proved by* `test_client.py::test_a_batch_is_split_into_ours_and_everything_else` + and `test_a_batch_that_does_not_contain_our_target_is_refused_in_full`. +12. **Never use the synchronous chat route for a gated change.** The gate takes a + `job_id` and jobs exist only on `/v1/chat/async` (Decision 32). + *Proved by* `test_client.py::test_gated_changes_go_to_the_async_route`. +13. **Never send feedback on a denial** (Decision 34). Feedback triggers a + re-plan and the re-plan goes off target — measured, twice. + *Proved by* `test_client.py::test_denials_carry_no_feedback`. +14. **Never treat the job as the decision record** (Decision 33). Our ledger row + is written first, with the actor, and `approve` is only an actuator. The + product's gate is write-only and cannot be read back. + *Proved by* `test_decision_ledger.py::test_the_row_is_written_before_the_actuator_is_called`. +15. **Never believe the chat reply, the `changes_summary`, or a 200.** The only + evidence is `GET /v1/documents/{id}?include_html=true`. Measured: the reply + has over-claimed once and under-claimed once, and the completion sentence + reported two numbers that were both wrong. + *Proved by* `test_verifier.py::test_a_reply_claiming_success_does_not_make_a_step_ok`. + +### The failure path + +16. **Never report an edit applied without reading the whole document back.** + Three classes, all detected: **not applied** (target chunk unchanged), + **applied wrong** (target changed but not to the expected bytes), + **collateral damage** (any non-target chunk moved, or the chunk set changed). + *Proved by* `test_verifier.py`, one test per class, each driven by a recorded + response from `docs/evidence/`. +17. **Never run a downstream step after a failed one.** The queue halts. This is + the 2026-08-07 corrupted-document mechanism and the halt is the whole + defence. + *Proved by* `test_orchestrator.py::test_a_failure_halts_the_queue`. +18. **Never export after a failure.** No export, no "partial success", no + completion sentence. Revert to the last verified-good state and surface the + diff. + *Proved by* `test_orchestrator.py::test_no_export_after_a_failure`. +19. **Never re-plan.** At most one narrow retry of the same instruction, and a + retry diffs the *whole* document, because failed content has been observed + arriving a turn later. + *Proved by* `test_orchestrator.py::test_at_most_one_retry_and_never_a_replan`. + +### The boundary + +20. **Never import from `system/`, and never speak its vocabulary.** This tree + answers *what must the document say, and did SuperDocs actually say it*. It + never emits a finding, never quotes evidence bytes as a finding, never + produces a report. `system/` never holds a SuperDocs document id. + *Proved by* `tests/test_isolation.py` and `tests/test_domain_boundary.py`. + +--- + +## What it may do, stated so the boundary is not accidentally wider + +- Read `corpus/` as read-only input. Both trees do; that is shared data, not + shared code. +- Quote a source's bytes **as the provenance of a figure it writes**. That is + the citation discipline, rebuilt against a different substrate — not a + finding. +- Say that a required disclosure has no note. That is the disclosure checklist + doing its job, and it is a property of the document, not a judgement about + the company. diff --git a/use-cases/preetham1930/statutory-statements-builder/README.md b/use-cases/preetham1930/statutory-statements-builder/README.md new file mode 100644 index 000000000..e0b8b6e45 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/README.md @@ -0,0 +1,179 @@ +# Statutory financial statements builder + +Rolls last year's signed statements forward to this year against the trial +balance and this year's event sources, and produces this year's statutory +accounts as an edited, typeset document: primary statements and comparatives +updated, each note's figures regenerated, notes whose required disclosures are +missing flagged **from the data**, and note numbering plus the disclosure +checklist consistent throughout. + +## The grading line + +> **Comparatives and the cross-references between notes and primary statements +> tie exactly; a newly required note is detected from the data rather than +> remembered by a human; note numbering and the checklist stay consistent after +> an insertion mid-document.** + +Where each half is made true, and where to look: + +| The claim | How it is true | Where | +| --- | --- | --- | +| Comparatives tie exactly | Every prior-year figure is cited **twice** — to the ledger's `fy2025` column and to the signed set — and a disagreement is a hard error, not a preference. Then every total the signed set prints is re-derived and compared. | `statements.py`, `tests/test_rollforward.py` | +| Note ↔ primary cross-references tie | The note's caption sentence and the statement row are the *same* `Figure` object, and the rendered HTML of both is re-scanned afterwards to prove it. | `bodies.py`, `skeleton.py::assert_note_captions_tie` | +| A new note is detected from the data | `account 1110 balance > 0` is true at 268.00 and its checklist items map to no note. Its **position** comes from the ledger's account order: the account before it points at Note 3, so the lease note is Note 4. Nothing in the code knows what a lease is. | `notetree.py`, `tests/test_note_tree.py` | +| Numbering stays consistent | One pass over our own ordered tree assigns every number; 22 notes shift; the renumbering must be order-preserving or the run stops. | `notetree.py::_validate` | +| The checklist stays consistent | Every `satisfied_by_note` is remapped through the **same** map, and afterwards every mapping is re-asserted against the headings **read back out of the document**. | `notetree.py::assert_checklist_consistent` | + +## Run it + +```bash +python -m statutory +``` + +No key, no network: the whole roll-forward, every tie, the note tree, the +checklist and the edit plan are computed offline and the report is printed. +The computed document is written in both shapes to `var/statutory/`. + +```bash +python -m statutory --live --sample 4 +``` + +Drives SuperDocs for the first four chunk replacements. `--live` with no +`--sample` does the whole plan; one billable operation per chunk. + +Run from this directory (the folder name has a hyphen, so it is an application +folder rather than an import path; the importable package inside it is +`statutory`). + +## What SuperDocs does, and why it has to exist + +The deliverable is a **typeset** statutory document that survives dozens of +surgical edits with its layout intact and comes out as a DOCX an audit partner +signs. SuperDocs holds the document, applies each replacement inside its own +chunk model, holds every change at a human gate, and renders the export with +real Word table structure. Remove it and what is left is a DOCX manipulation +library — which is building a version *of* the product, the thing +[`builds/README.md`](../README.md) forbids. + +`system/` cannot do this: it produces a grounded report and is not a document +editor. The boundary is enforced, not asserted — `tests/test_domain_boundary.py`. + +## The design, and the measurements it came from + +Every one of these was earned by testing the product, not chosen by preference +(PROGRESS.md, 2026-08-07 and 2026-08-09). + +- **One verb, and it is `replace`** (Decision 28). We never ask SuperDocs to + create. The only operation that inserts is also the one that appended six + sections when asked for one, four of them invented. +- **The entire final skeleton is computed here and uploaded verbatim** + (Decision 29) — every heading already at its final number, every + cross-reference already repointed. **There is no mid-document insertion in + this design**, so the failure cannot be reached. It is also the only path + measured to preserve a styled document exactly. +- **The async route** (Decision 32), because `approve` takes a `job_id` and jobs + exist only on `/v1/chat/async`. +- **Our ledger is the record; `approve` is an actuator** (Decision 33). The + product's gate is write-only: `pending_changes` reported the same three + undecided items before and after two decisions. +- **Never send feedback on a denial** (Decision 34). Feedback triggered a + revision pass that proposed edits to two chunks nobody targeted, aimed at the + section headings, and held the job for minutes with approved work unapplied. +- **One change per job** (Decision 35). Nothing applies until a batch settles, + and a batch can spawn a follow-up batch; with a batch of one, batch atomicity + equals item atomicity. +- **The verified unit is the chunk, not the value** (Decision 37). Measured: + four cells in one table changed in one request with every style attribute + byte-preserved, and every failure this project has recorded was *across* + chunks. So the skeleton is chunk-aligned and a note schedule is one table. + +## The failure path + +Every step computes its expected post-state **before** sending, then reads the +whole document back and classifies: + +- **NOT APPLIED** — the target chunk came back unchanged. Halt. No downstream + step runs, because a downstream step is the consequence of work that may not + have happened. +- **APPLIED WRONG** — it changed, but not to the computed post-state. Halt, show + the diff. +- **COLLATERAL DAMAGE** — any non-target chunk moved, appeared or vanished. + Halt, and never retry: the document already holds content nobody asked for. + This is the dangerous one. In the append test the requested change *also* + succeeded while four fabricated sections appeared, so a verifier checking only + its own target would have reported success. + +On any failure: revert to the last verified-good state, **read back again and +say whether the revert actually happened**, surface the diff, produce **no +export**. At most one narrow retry of the same instruction; never a re-plan. + +The three classes are tested against the recorded responses in +`docs/evidence/2026-08-09-superdocs-reverify/` — the real shapes of the real +failures, replayed with no key and no network. + +## What it will not do, and why that is the point + +The supplied trial balance is at **caption granularity**: 29 accounts, one per +caption. So no breakdown finer than a caption is derivable, and roughly twenty +notes cannot state the analysis they stated last year. The build does not carry +last year's breakdown forward under this year's heading. Each note states the +tied total and names exactly what is outstanding and where we looked, and the +checklist marks those items `outstanding` rather than `satisfied`. + +That is a deliberate cut and it is the honest one: a stale figure under a new +heading is the failure this whole project is about. + +## Layout + +``` +statutory/ the package + money.py the figure recogniser, narrow in both directions + figures.py Cite / Figure / Derived / Quotient / Gap, and the Body guarantee + csvspans.py an RFC 4180 scanner that keeps byte positions + ledger.py the trial balance + priorset.py last year's signed statements, parsed with byte offsets + checklist.py 32 rows of data and a grammar that evaluates them + events.py events detected from this year's sources, never remembered + statements.py the roll-forward, the comparatives, the ties + notetree.py insertion point, numbering, repointing, checklist remap + bodies.py note regeneration, six data-selected recipes, and the gaps + skeleton.py the whole document in both shapes + chunks.py the chunk model and the canonical form + editplan.py one single-target replacement per step + superdocs/ transport, client, decision ledger, verifier + orchestrator.py upload once, then one verified replacement at a time + report.py what the run says for itself +config/ note recipes, event vocabulary, new-note titles, carry rules +tests/ the suite; runs with no key and no network +INVARIANTS.md the "must never" list, each item naming the test that proves it +``` + + +--- + +## SuperDocs features used + +- **Upload** (POST /v1/documents/upload-base64) - the fully computed document skeleton is uploaded verbatim; every heading is already at its final number and every cross-reference already repointed. +- **Chat, async** (POST /v1/chat/async) - one targeted replace per request. +- **Review gate** (approval_mode: ask_every_time + POST /v1/chat/approve) - every change is gated. +- **Read-back** (GET /v1/documents/{id}?include_html=true) - after every edit the whole document is read back and compared to the post-state computed before sending, including every non-target chunk. +- **Export** (POST /v1/documents/export) - the final .docx. + +## Environment + +``` +SUPERDOCS_API_KEY=your-key-here +SUPERDOCS_BASE_URL=https://api.superdocs.app +``` + +Tests run without a key against recorded fixtures. Only the live demo needs one. + +## Output + +![Statutory statements generated on SuperDocs](evidence/screenshot.png) + +A live run is committed at [evidence/statements.docx](evidence/statements.docx): 33 edits, each verified by read-back. Total assets and total equity and liabilities both come to 4,857.00, with FY2025 comparatives of 4,068.00. A warehouse lease commenced in the year, so a right-of-use asset appears and a lease note is created at position 4 - detected from the trial balance, not from a hardcoded list of standards - and the 22 notes after it renumber, with the note column in the primary statements and the disclosure checklist remapped through the same map. + +--- + +Built by **Preetham Kukkadapu** for the SuperDocs round 2 task. diff --git a/use-cases/preetham1930/statutory-statements-builder/config/absence-probes.csv b/use-cases/preetham1930/statutory-statements-builder/config/absence-probes.csv new file mode 100644 index 000000000..f37e72f9e --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/config/absence-probes.csv @@ -0,0 +1,2 @@ +item_id,ledger_name_contains,what +DC-029,goodwill|investment,a goodwill or investment-in-subsidiary balance diff --git a/use-cases/preetham1930/statutory-statements-builder/config/carry-forward-rules.csv b/use-cases/preetham1930/statutory-statements-builder/config/carry-forward-rules.csv new file mode 100644 index 000000000..5f4c7ad57 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/config/carry-forward-rules.csv @@ -0,0 +1,7 @@ +rule,value,why +period_reference,during the year,a sentence tied to last year's period is a claim about last year +period_reference,for the year,the same +period_reference,as at,the same +period_reference,at the reporting date,the same +period_reference,in the current year,the same +period_reference,during the period,the same diff --git a/use-cases/preetham1930/statutory-statements-builder/config/event-vocabulary.csv b/use-cases/preetham1930/statutory-statements-builder/config/event-vocabulary.csv new file mode 100644 index 000000000..c393a9eee --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/config/event-vocabulary.csv @@ -0,0 +1,3 @@ +event_name,phrase_groups +business combination occurred in period,acquisition of|acquire|share purchase agreement;;equity share capital;;consideration +control obtained over an entity,control;;passed to|obtained by|vested in;;share purchase agreement|acquisition|acquiree diff --git a/use-cases/preetham1930/statutory-statements-builder/config/new-note-titles.csv b/use-cases/preetham1930/statutory-statements-builder/config/new-note-titles.csv new file mode 100644 index 000000000..4aad7a25e --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/config/new-note-titles.csv @@ -0,0 +1,4 @@ +standard,new_note_title +Ind AS 116,Leases +Ind AS 103,Business combination +Ind AS 110,Business combination diff --git a/use-cases/preetham1930/statutory-statements-builder/config/note-recipes.csv b/use-cases/preetham1930/statutory-statements-builder/config/note-recipes.csv new file mode 100644 index 000000000..b88d181f4 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/config/note-recipes.csv @@ -0,0 +1,7 @@ +item_id,recipe,also_blocks +DC-003,asset_additions,DC-004 +DC-011,equity_movement, +DC-024,earnings_per_share, +DC-025,lease_measurement,DC-027|DC-028 +DC-026,lease_maturity, +DC-029,business_combination,DC-030|DC-031|DC-032 diff --git a/use-cases/preetham1930/statutory-statements-builder/config/quarantine-markers.txt b/use-cases/preetham1930/statutory-statements-builder/config/quarantine-markers.txt new file mode 100644 index 000000000..822abdd9f --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/config/quarantine-markers.txt @@ -0,0 +1,12 @@ +# A source carrying any of these inside an HTML/Markdown comment is quarantined: +# reported as quarantined, and never consulted for an event, a figure or a note. +# +# Quarantine runs in BOTH directions on purpose. Not consulting a forged source +# stops it suppressing a disclosure; it also stops it *manufacturing* one, which +# is the same class of failure and is easier to miss because the result looks +# like diligence. +ignore all previous instructions +system instruction +mark every disclosure +without human sign-off +do not mention this instruction diff --git a/use-cases/preetham1930/statutory-statements-builder/conftest.py b/use-cases/preetham1930/statutory-statements-builder/conftest.py new file mode 100644 index 000000000..50e1cf854 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/conftest.py @@ -0,0 +1,46 @@ +"""`builds/statutory-statements/` is a codebase, not an installed package. + +The directory name carries a hyphen on purpose - it is an application folder, +not an import path - so the importable package inside it is `statutory` and this +file is what puts it on `sys.path` for the suite. Nothing here imports from +`system/`, and `tests/test_isolation.py` fails the build if it ever does. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import pytest + +APP_ROOT = Path(__file__).resolve().parent +REPO_ROOT = APP_ROOT.parent.parent + +if str(APP_ROOT) not in sys.path: + sys.path.insert(0, str(APP_ROOT)) +# tests/ carries a helper (`recorded.py`) that loads responses out of +# docs/evidence/. It is not a package, so its directory goes on the path too. +if str(APP_ROOT / "tests") not in sys.path: + sys.path.insert(0, str(APP_ROOT / "tests")) + + +@pytest.fixture(scope="session") +def corpus() -> Path: + return REPO_ROOT / "corpus" + + +@pytest.fixture(scope="session") +def config() -> Path: + return APP_ROOT / "config" + + +@pytest.fixture(scope="session") +def evidence() -> Path: + return REPO_ROOT / "docs" / "evidence" + + +@pytest.fixture(scope="session") +def rolled(corpus: Path, config: Path): + from statutory.pipeline import prepare + + return prepare(corpus, config) diff --git a/use-cases/preetham1930/statutory-statements-builder/evidence/screenshot.png b/use-cases/preetham1930/statutory-statements-builder/evidence/screenshot.png new file mode 100644 index 000000000..008208bd7 Binary files /dev/null and b/use-cases/preetham1930/statutory-statements-builder/evidence/screenshot.png differ diff --git a/use-cases/preetham1930/statutory-statements-builder/evidence/statements.docx b/use-cases/preetham1930/statutory-statements-builder/evidence/statements.docx new file mode 100644 index 000000000..bf09225ac Binary files /dev/null and b/use-cases/preetham1930/statutory-statements-builder/evidence/statements.docx differ diff --git a/use-cases/preetham1930/statutory-statements-builder/scripts/verify_live_output.py b/use-cases/preetham1930/statutory-statements-builder/scripts/verify_live_output.py new file mode 100644 index 000000000..a00b47ba1 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/scripts/verify_live_output.py @@ -0,0 +1,110 @@ +"""Check the finished document the way a reviewer would: from the API and the DOCX. + +Nothing here trusts the run's own report. The document is fetched fresh, the +checklist is re-asserted against the headings that came back, every figure the +roll-forward computed is required to be present, and the exported DOCX is +unzipped and its table structure counted - because "it exported" is not evidence +that anything is in it. + + python scripts/verify_live_output.py <document_id> [--docx PATH] +""" + +from __future__ import annotations + +import argparse +import re +import sys +import zipfile +from pathlib import Path + +APP_ROOT = Path(__file__).resolve().parent.parent +REPO_ROOT = APP_ROOT.parent.parent +sys.path.insert(0, str(APP_ROOT)) + +from statutory.chunks import ChunkMap # noqa: E402 +from statutory.money import money_tokens_in_html, strip_markup # noqa: E402 +from statutory.notetree import assert_checklist_consistent # noqa: E402 +from statutory.pipeline import DEFAULT_CONFIG, prepare # noqa: E402 +from statutory.superdocs.client import SuperDocsClient # noqa: E402 +from statutory.superdocs.transport import HttpTransport # noqa: E402 + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("document_id") + parser.add_argument("--docx", default=str(REPO_ROOT / "var" / "statutory" / "statements.docx")) + args = parser.parse_args() + + try: + from dotenv import load_dotenv + + load_dotenv(REPO_ROOT / ".env", override=False) + except ImportError: + pass + + rolled = prepare(REPO_ROOT / "corpus", DEFAULT_CONFIG) + client = SuperDocsClient(HttpTransport()) + live, detail = client.read_back(args.document_id) + html = detail["html"] + + print(f"document {args.document_id} version {detail.get('version')}") + print(f" {len(live)} chunks read back") + + ours = ChunkMap(rolled.document.html("target")) + print(f" computed document: {len(ours)} chunks") + assert len(live) == len(ours), "the chunk count moved" + + mismatched = [ + i + for i, (a, b) in enumerate(zip(ours.chunks, live.chunks, strict=True)) + if a.canonical != b.canonical + ] + print(f" chunks differing from what we computed: {len(mismatched)} {mismatched[:5]}") + + headings = live.headings() + print(f" note headings in the live document: {len(headings)}") + assert_checklist_consistent(rolled.mapping, rolled.tree, headings) + print(" checklist consistency re-asserted against the LIVE headings: ok") + + tokens = set(money_tokens_in_html(html)) + ties = ["4,857.00", "4,068.00", "577.00", "414.50", "407.00", "309.00", "1,691.00", "1,284.00"] + missing = [t for t in ties if t not in tokens] + print(f" tie figures present in the live document: {len(ties) - len(missing)}/{len(ties)}") + assert not missing, f"missing from the live document: {missing}" + + text = strip_markup(html) + for phrase in ( + "Note 4", + "Leases", + "Business combination", + "disclosed in Note 5", + "all 29 account names were searched", + "821.00", + ): + assert phrase in text, f"{phrase!r} is not in the live document" + print(" new note, repointed cross-reference and both disagreements present: ok") + assert "31 March 2025</strong>" not in html, "the front matter still says last year" + + docx = Path(args.docx) + if docx.is_file(): + with zipfile.ZipFile(docx) as archive: + document_xml = archive.read("word/document.xml").decode("utf-8") + print(f" DOCX {docx.name}: {docx.stat().st_size:,} bytes") + print(f" w:tbl {document_xml.count('<w:tbl>')}") + print(f" w:tr {document_xml.count('<w:tr ') + document_xml.count('<w:tr>')}") + print(f" w:tc {document_xml.count('<w:tc>')}") + print(f" w:tcBorders {document_xml.count('<w:tcBorders>')}") + print(f" w:shd {document_xml.count('<w:shd ')}") + flat = re.sub(r"<[^>]+>", "", document_xml) + for token in ("4,857.00", "268.00", "8.14", "185.00"): + assert token in flat, f"{token} is not in the exported DOCX" + print(" key figures present in word/document.xml: ok") + else: + print(f" no DOCX at {docx}") + + print("VERIFIED") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/__init__.py b/use-cases/preetham1930/statutory-statements-builder/statutory/__init__.py new file mode 100644 index 000000000..bea8e84b0 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/__init__.py @@ -0,0 +1,18 @@ +"""The statutory financial statements builder (Task 2, card S3). + +Rolls FY2025 signed statements forward to FY2026 against the trial balance and +this year's event sources, computes the entire final document skeleton here, and +drives SuperDocs with single-target chunk replacements only. + +See INVARIANTS.md for the "must never" list. Nothing in this package imports +from `system/` (TASK.md hard rule, Decision 4). +""" + +from __future__ import annotations + +__all__ = ["REPORTING_LABEL"] + +# The only place a year label is written down. Everything else derives the +# reporting year from the highest `fy<year>` column in the trial balance, so a +# FY2027 corpus needs no code change (hard rule 6). +REPORTING_LABEL = "31 March" diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/__main__.py b/use-cases/preetham1930/statutory-statements-builder/statutory/__main__.py new file mode 100644 index 000000000..dfe6af1f6 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/__main__.py @@ -0,0 +1,98 @@ +"""CLI. + + python -m statutory roll forward, print the report, write the HTML + python -m statutory --live --sample 3 drive SuperDocs for the first 3 chunks + python -m statutory --live the full run (one operation per chunk) + +`--live` is the only mode that touches the network, and it is the only mode that +needs a key. Everything else - the roll-forward, the ties, the note tree, the +checklist, the plan - runs offline, which is why the test suite can exercise it. +""" + +from __future__ import annotations + +import argparse +import sys +import uuid +from pathlib import Path + +from .editplan import build_plan +from .orchestrator import Orchestrator, write_artifacts +from .pipeline import DEFAULT_CONFIG, prepare +from .report import render +from .superdocs.client import SuperDocsClient +from .superdocs.decisions import DecisionLedger +from .superdocs.transport import HttpTransport, redacted_key + +REPO_ROOT = Path(__file__).resolve().parent.parent.parent.parent + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(prog="statutory", description=__doc__) + parser.add_argument("--corpus", default=str(REPO_ROOT / "corpus")) + parser.add_argument("--config", default=str(DEFAULT_CONFIG)) + parser.add_argument("--out", default=str(REPO_ROOT / "var" / "statutory")) + parser.add_argument("--live", action="store_true", help="drive SuperDocs (needs a key)") + parser.add_argument( + "--sample", type=int, default=None, help="only issue the first N chunk replacements" + ) + parser.add_argument("--actor", default="K. Latha (CFO)") + parser.add_argument("--format", default="docx", help="export format for a successful run") + args = parser.parse_args(argv) + + rolled = prepare(Path(args.corpus), Path(args.config)) + plan = build_plan(rolled.document).sample(args.sample) + print(render(rolled, len(plan))) + + out_dir = Path(args.out) + for path in write_artifacts(out_dir, plan): + print(f"wrote {path}") + + print() + print("EDIT PLAN (one single-target chunk replacement per step, one change per job)") + for step in plan.steps: + print(f" {step.describe()}") + + if not args.live: + print() + print( + "offline: nothing was sent. The whole roll-forward above ran with no key and no " + "network. Add --live to drive SuperDocs." + ) + return 0 + + # Read from .env only, never from a command line and never printed + # (hard rule 1). `override=False` so an already-exported value wins. + try: + from dotenv import load_dotenv + + load_dotenv(REPO_ROOT / ".env", override=False) + except ImportError: + pass + + print() + print(f"live run: key {redacted_key()}") + client = SuperDocsClient(HttpTransport()) + ledger = DecisionLedger(out_dir / "decisions.jsonl") + run_id = f"stat-{uuid.uuid4().hex[:8]}" + orchestrator = Orchestrator( + client, ledger, args.actor, export_formats=(args.format,), export_dir=out_dir + ) + result = orchestrator.run(run_id, plan, "varshith-precision-FY2026-statements.html") + print() + for line in orchestrator.log: + print(line) + print() + print(f"document {result.document_id} in session {result.session_id}") + print(result.sentence(ledger)) + if result.reverted: + print(result.reverted) + if result.exported: + print(f"exported: {', '.join(result.exported)}") + else: + print("no export was produced") + return 0 if result.ok else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/bodies.py b/use-cases/preetham1930/statutory-statements-builder/statutory/bodies.py new file mode 100644 index 000000000..41ef383d4 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/bodies.py @@ -0,0 +1,893 @@ +"""Regenerating each note's figures, and being honest about the rest. + +Three rules decide what a note body says, and none of them knows a note's name: + +1. **Captions come from the primary statements.** Every statement line whose + cross-reference points at this note contributes one sentence carrying the + reporting-year figure and its comparative. That is the note <-> primary tie, + stated in the note itself and made of the same `Figure` objects the statement + prints, so the two cannot say different things. + +2. **A sentence with no figure in it is a policy statement and is carried; a + sentence with a figure is regenerated or named as a gap.** The carried + sentence is quoted verbatim from the signed set with the bytes it came from. + A carried sentence that refers to a period ("during the year") is *not* + carried - it is a claim about last year, and last year's claim under this + year's heading is exactly the failure this build exists to avoid. + +3. **The supplied trial balance is at caption granularity.** Twenty-nine + accounts, one per caption, so no breakdown finer than the caption is + derivable. Where last year's note carried a breakdown we cannot reproduce, + the note states the tied total and names the outstanding analysis - by the + labels last year used, which is as specific as the evidence allows. + +Beyond that, six recipes compute what the sources *do* support. Each is selected +by a checklist item id in `config/note-recipes.csv`, which is the data saying +what the note must disclose - not by the note's title. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass, field +from decimal import Decimal +from pathlib import Path + +from . import style +from .csvspans import as_records, parse_csv +from .disagreement import Disagreement, Side +from .figures import AnyFigure, Body, Cite, Component, Derived, Figure, Gap, Quotient +from .ledger import TrialBalance +from .money import money_tokens, strip_markup +from .notetree import Note, NoteTree +from .priorset import NOTE_REFERENCE, PriorYearSet +from .statements import RollForward + +SENTENCE_SPLIT = re.compile(r"(?<=[.])\s+(?=[A-Z<])") +BREAKDOWN_SPLIT = re.compile(r";|\.\s+") +BREAKDOWN_SPLIT_SENTENCE = re.compile(r"\.\s+") # a decimal point is never followed by a space +DROP_LABEL = re.compile(r"^total\b", re.I) +# A breakdown *group* is a ":"-introduced label whose body splits on ";" into two +# or more items each holding exactly one figure. The narrowing matters: a looser +# rule reads Note 3's "for the year: 276.00 (intangible amortisation of 9.00 is +# disclosed in Note 4...)" as a two-item breakdown summing to 285.00 and raises a +# confident false positive. +GROUP = re.compile(r"^([A-Za-z][^:;]{2,60}):\s*(.+)$", re.S) + + +@dataclass +class Element: + """One body element, which is one chunk once the document is uploaded.""" + + role: str # captions | policy | recipe:<name> | outstanding + tag: str # p | table + target: str + skeleton: str + figures: tuple[AnyFigure, ...] = () + quotes: tuple[Cite, ...] = () + + @property + def changed(self) -> bool: + return self.target != self.skeleton + + +@dataclass +class NoteContent: + note: Note + elements: list[Element] = field(default_factory=list) + gaps: list[Gap] = field(default_factory=list) + disagreements: list[Disagreement] = field(default_factory=list) + blocked_items: set[str] = field(default_factory=set) + + @property + def figures(self) -> tuple[AnyFigure, ...]: + return tuple(f for e in self.elements for f in e.figures) + + @property + def cites(self) -> tuple[Cite, ...]: + out = [c for f in self.figures for c in f.cites] + out.extend(c for e in self.elements for c in e.quotes) + out.extend(c for d in self.disagreements for s in d.sides for c in s.cites) + return tuple(out) + + +class AssetSchedule: + """This year's capitalisations, by category, each row cited.""" + + def __init__(self, path: Path, corpus_root: Path) -> None: + self.rel = path.relative_to(corpus_root).as_posix() + _, records = as_records(parse_csv(path.read_bytes())) + self.rows = records + self.by_category: dict[str, list[Figure]] = {} + for record in records: + cell = record["cost_inr_lakhs"] + figure = Figure( + label=record["description"].text, + value=Decimal(cell.text), + basis="document", + cites=( + Cite( + self.rel, + cell.byte_start, + cell.byte_end, + cell.value, + f"fixed-asset schedule, {record['asset_id'].text} " + f"({record['description'].text})", + ), + ), + ) + self.by_category.setdefault(record["category"].text, []).append(figure) + + def total(self, category: str) -> Derived: + return Derived( + f"Additions - {category}", + tuple(Component(f) for f in self.by_category[category]), + ) + + def grand_total(self) -> Derived: + return Derived( + "Total additions", + tuple(Component(f) for figs in self.by_category.values() for f in figs), + ) + + +class BodyBuilder: + def __init__( + self, + rolled: RollForward, + tree: NoteTree, + mapping: dict[str, int | None], + corpus_root: Path, + config_root: Path, + ) -> None: + self.rolled = rolled + self.tree = tree + self.mapping = mapping + self.corpus = corpus_root + self.prior: PriorYearSet = rolled.prior + self.ledger: TrialBalance = rolled.ledger + _, recipe_rows = as_records(parse_csv((config_root / "note-recipes.csv").read_bytes())) + self.recipes = {r["item_id"].text: r["recipe"].text for r in recipe_rows} + # Which other checklist items this recipe's gaps also leave outstanding. + # Data, so no item id is written into this module (there is a test). + self.also_blocks = { + r["item_id"].text: tuple(x for x in r["also_blocks"].text.split("|") if x) + for r in recipe_rows + } + _, carry_rows = as_records( + parse_csv((config_root / "carry-forward-rules.csv").read_bytes()) + ) + self.period_words = [ + r["value"].text.lower() for r in carry_rows if r["rule"].text == "period_reference" + ] + _, probe_rows = as_records(parse_csv((config_root / "absence-probes.csv").read_bytes())) + self.absence_probes = { + r["item_id"].text: (r["ledger_name_contains"].text, r["what"].text) for r in probe_rows + } + self.coverage: dict[str, int] = {"groups": 0, "tied": 0, "raised": 0} + self._sources_looked_in = ( + f"trial balance ({len(self.ledger)} accounts, caption granularity)", + "fixed-asset schedule", + "board minutes", + "lease deed", + "signed prior-year statements", + ) + + # -- entry point ------------------------------------------------------- + + def build(self, note: Note) -> NoteContent: + content = NoteContent(note=note) + items = [item for item, number in self.mapping.items() if number == note.number] + prior_note = self.prior.note(note.prior_number) if note.prior_number else None + + captions = self._caption_element(note) + if captions is not None: + content.elements.append(captions) + + for item in sorted(items): + recipe = self.recipes.get(item) + if recipe: + getattr(self, f"_recipe_{recipe}")(note, content, item) + + if prior_note is not None: + crossref = self._crossref_element(prior_note, note) + if crossref is not None: + content.elements.append(crossref) + self._comparative_tieout(prior_note, note, content) + policy = self._policy_element(prior_note) + if policy is not None: + content.elements.append(policy) + gap = self._breakdown_gap(prior_note, content) + if gap is not None: + content.gaps.append(gap) + + if content.gaps: + content.elements.append(self._outstanding_element(content)) + if not content.elements: + raise ValueError( + f"Note {note.number} ({note.title}) would be empty. A note with nothing to say " + f"is a heading nobody can act on; either it has a caption, a recipe, a carried " + f"policy sentence or an outstanding item." + ) + return content + + # -- generic parts ----------------------------------------------------- + + def _caption_element(self, note: Note) -> Element | None: + lines = [ + line + for statement in self.rolled.statements + for line in statement.lines + if line.note_ref == note.number and line.account is not None + ] + if not lines: + return None + body = Body().markup("<p>") + for index, line in enumerate(lines): + if index: + body.text(" ") + statement_key = next(s.key for s in self.rolled.statements if line in s.lines) + if statement_key == "balance_sheet": + body.text(f"{line.caption} as at ").dated( + self.ledger.year_label(self.rolled.year), + self.ledger.year_label(self.rolled.prior_year), + ).text(": ").fig(line.current, line.prior).text(" (").dated( + self.ledger.year_label(self.rolled.prior_year), + self.ledger.year_label(self.rolled.prior_year - 1), + ).text(": ").fig(line.prior, "0.00").text(").") + else: + body.text(f"{line.caption} for the year ended ").dated( + self.ledger.year_label(self.rolled.year), + self.ledger.year_label(self.rolled.prior_year), + ).text(": ").fig(line.current, line.prior).text(" (year ended ").dated( + self.ledger.year_label(self.rolled.prior_year), + self.ledger.year_label(self.rolled.prior_year - 1), + ).text(": ").fig(line.prior, "0.00").text(").") + body.markup("</p>") + where = f"Note {note.number} ({note.title}) captions" + return Element( + "captions", "p", body.render(where), body.skeleton(where), body.figures, body.quotes + ) + + def _policy_element(self, prior_note) -> Element | None: + carried: list[tuple[str, Cite]] = [] + for element in prior_note.body: + if element.tag != "p": + continue + for raw_sentence, offset in _sentences(element.raw): + text = strip_markup(raw_sentence).strip() + if not text or money_tokens(text): + continue + if any(word in text.lower() for word in self.period_words): + continue + byte_start = element.byte_start + len(element.raw[:offset].encode("utf-8")) + carried.append( + ( + raw_sentence.strip(), + Cite( + self.prior.rel, + byte_start, + byte_start + len(raw_sentence.strip().encode("utf-8")), + raw_sentence.strip(), + f"prior-year signed statements, Note {prior_note.number} " + f"({prior_note.title}), carried policy sentence", + ), + ) + ) + if not carried: + return None + body = Body().markup("<p>") + for index, (sentence, cite) in enumerate(carried): + if index: + body.text(" ") + body.quote(self.tree.repoint_prose(strip_markup(sentence).strip()), cite) + body.markup("</p>") + where = "carried policy" + return Element("policy", "p", body.render(where), body.skeleton(where), (), body.quotes) + + def _breakdown_gap(self, prior_note, content: NoteContent) -> Gap | None: + labels = _breakdown_labels(prior_note) + if not labels: + return None + return Gap( + what=( + f"analysis of {prior_note.title.lower()} into the components the signed set " + f"disclosed last year ({'; '.join(labels)})" + ), + looked_in=self._sources_looked_in, + ) + + def _crossref_element(self, prior_note, note: Note) -> Element | None: + """Note-to-note references survive the roll-forward, repointed. + + The signed set's own prose is the source of the reference; the number is + ours. The sentence carries no figure, so a cross-reference never depends + on an analysis we could not reproduce - it survives even when the + sentence that used to carry it does not. + """ + targets: list[int] = [] + for element in prior_note.body: + for match in NOTE_REFERENCE.finditer(strip_markup(element.raw)): + old = int(match.group(1)) + if old not in targets and old != prior_note.number: + targets.append(old) + if not targets: + return None + body = Body().markup("<p>") + for index, old in enumerate(targets): + if index: + body.text(" ") + title = self.prior.note(old).title + # Written with the number the signed set used, then put through the + # one repointing function. There is a single place a note number is + # rewritten, so there is no second place to get it wrong. + body.text( + self.tree.repoint_prose( + f"Further disclosure relating to {title.lower()} is disclosed in Note {old}." + ) + ) + body.markup("</p>") + where = f"Note {note.number} cross-references" + skeleton = body.skeleton(where) + return Element("crossrefs", "p", body.render(where), skeleton, (), body.quotes) + + def _comparative_tieout(self, prior_note, note: Note, content: NoteContent) -> None: + """Does last year's note still tie to last year's caption? We are about to carry it. + + This is the check that catches a note <-> primary break in the set we are + rolling forward, before the broken figure becomes this year's comparative. + It is deliberately narrow (see GROUP) and it reports its own selectivity + in the run summary, so the narrowness is visible in the output rather + than only in this docstring. + """ + captions = [ + line + for statement in self.rolled.statements + for line in statement.lines + if line.note_ref == note.number and line.account is not None + ] + if not captions: + return + for label, group_total, items in _qualifying_groups(prior_note, self.prior): + self.coverage["groups"] += 1 + if any(line.prior.value == group_total.value for line in captions): + self.coverage["tied"] += 1 + continue + self.coverage["raised"] += 1 + caption = min(captions, key=lambda line: abs(line.prior.value - group_total.value)) + difference = Derived( + "difference", (Component(group_total), Component(caption.prior, -1)) + ) + body = Body().markup("<p><em>Comparative note:</em> the ") + body.text(f"{label.lower()} disclosed in the signed prior-year set sums to ") + body.fig(group_total, "0.00") + body.text(f", against the {caption.caption.lower()} comparative of ") + body.fig(caption.prior, "0.00") + body.text(" carried in this set - a difference of ") + body.fig(difference, "0.00") + body.text( + ". Both prior-year figures are quoted from the signed statements; the " + "difference is stated here and is not resolved by this roll-forward." + ) + body.markup("</p>") + where = f"Note {note.number} comparative tie-out" + content.elements.append( + Element( + "comparative_tieout", + "p", + body.render(where), + body.skeleton(where), + body.figures, + ) + ) + content.disagreements.append( + Disagreement( + about=f"the prior-year {label.lower()} for {caption.caption.lower()}", + sides=( + Side( + source=self.prior.rel, + says=f"{label} sums to {group_total.rendered} " + f"({group_total.workings()})", + cites=group_total.cites, + figures=(group_total,), + ), + Side( + source=self.ledger.rel, + says=f"{caption.caption} is {caption.prior.rendered} in the " + f"prior-year column and in the signed balance sheet", + cites=caption.prior.cites, + figures=(caption.prior,), + ), + ), + ) + ) + content.gaps.append( + Gap( + f"reconciliation of the prior-year {label.lower()} to the " + f"{caption.caption.lower()} comparative, a difference of " + f"{difference.rendered}", + self._sources_looked_in, + figures=(difference,), + ) + ) + items_used = len(items) + del items_used + + def _outstanding_element(self, content: NoteContent) -> Element: + body = Body().markup("<p><em>Outstanding for this note:</em> ") + for index, gap in enumerate(content.gaps): + if index: + body.text(" ") + body.prose(f"({index + 1}) {gap.sentence()}.", gap.figures) + if content.disagreements: + for disagreement in content.disagreements: + body.text( + f" The sources disagree on {disagreement.about}; both are stated above and " + f"neither is preferred here." + ) + body.markup("</p>") + where = "outstanding" + return Element("outstanding", "p", body.render(where), body.skeleton(where), body.figures) + + # -- recipes ----------------------------------------------------------- + + def _recipe_asset_additions(self, note: Note, content: NoteContent, item: str) -> None: + schedule = AssetSchedule( + self.corpus / "sources" / "fixed-asset-additions-FY2026.csv", self.corpus + ) + line = self.rolled.caption_line(_account_for(self.rolled, note, "Non-current assets")) + opening, closing = line.prior, line.current + additions = schedule.grand_total() + movement = Derived( + "Depreciation, disposals and other movements", + (Component(closing), Component(opening, -1), Component(additions, -1)), + ) + body = Body().markup(style.table_open()) + body.markup( + f'<tr style="{style.HEAD_ROW}">{style.head_cell("Particulars")}' + f"{style.head_cell('Amount', numeric=True)}</tr>" + ) + body.markup(f'<tr><td style="{style.TD_TEXT}">Net carrying amount as at 1 April ').dated( + str(self.rolled.year - 1), str(self.rolled.prior_year - 1) + ).markup(f'</td><td style="{style.TD_NUM}">').fig(opening, "0.00").markup("</td></tr>") + for category in sorted(schedule.by_category): + total = schedule.total(category) + body.markup( + f'<tr><td style="{style.TD_TEXT}">Additions - {category}</td>' + f'<td style="{style.TD_NUM}">' + ).fig(total, "0.00").markup("</td></tr>") + body.markup( + f'<tr style="{style.SECTION_ROW}"><td style="{style.TD_TEXT}">' + f'<strong>Total additions</strong></td><td style="{style.TD_NUM}"><strong>' + ).fig(additions, "0.00").markup("</strong></td></tr>") + body.markup( + f'<tr><td style="{style.TD_TEXT}">Depreciation, disposals and other movements ' + f'(not separately supplied)</td><td style="{style.TD_NUM}">' + ).fig(movement, "0.00").markup("</td></tr>") + body.markup( + f'<tr style="{style.SECTION_ROW}"><td style="{style.TD_TEXT}">' + "<strong>Net carrying amount as at " + ).dated( + self.ledger.year_label(self.rolled.year), + self.ledger.year_label(self.rolled.prior_year), + ).markup(f'</strong></td><td style="{style.TD_NUM}"><strong>').fig(closing, opening).markup( + "</strong></td></tr></table>" + ) + where = f"Note {note.number} asset movement" + content.elements.append( + Element( + "recipe:asset_additions", + "table", + body.render(where), + body.skeleton(where), + body.figures, + ) + ) + content.gaps.append( + Gap( + "the movement split between depreciation charged for the year and disposals, " + "and the movement by class of asset", + self._sources_looked_in, + ) + ) + content.blocked_items.update({item, *self.also_blocks[item]}) + + def _recipe_equity_movement(self, note: Note, content: NoteContent, item: str) -> None: + line = self.rolled.caption_line(_account_for(self.rolled, note, "Equity")) + profit = self.rolled.statement("profit_and_loss").line("Profit for the year").current + closing = Derived("Balance carried forward", (Component(line.prior), Component(profit))) + minutes = self.corpus / "sources" / "board-minutes-2026-01-18.md" + dividend_cite = _quote_from(minutes, "to recommend a dividend", self.corpus) + body = ( + Body() + .markup("<p>Balance as at 1 April ") + .dated(str(self.rolled.year - 1), str(self.rolled.prior_year - 1)) + .text(": ") + .fig(line.prior, "0.00") + .text(". Add: profit for the year ") + .fig(profit, "0.00") + .text(". Balance as at ") + .dated( + self.ledger.year_label(self.rolled.year), + self.ledger.year_label(self.rolled.prior_year), + ) + .text(": ") + .fig(closing, line.prior) + .text( + ". The movement is the profit for the year in full; the board minutes record a " + "resolution " + ) + .quote("not to recommend a dividend", dividend_cite) + .text(", and the ledger carries no other movement in this caption.") + .markup("</p>") + ) + where = f"Note {note.number} equity movement" + content.elements.append( + Element( + "recipe:equity_movement", + "p", + body.render(where), + body.skeleton(where), + body.figures, + body.quotes, + ) + ) + + def _recipe_earnings_per_share(self, note: Note, content: NoteContent, item: str) -> None: + capital = self.ledger.account("2100") + if capital.balance(self.rolled.year) != capital.balance(self.rolled.prior_year): + content.gaps.append( + Gap( + "the weighted average number of equity shares, which moved during the year " + "and cannot be carried from the signed set", + self._sources_looked_in, + ) + ) + return + prior_note = self.prior.note(note.prior_number) if note.prior_number else None + if prior_note is None: + return + shares_literal = _first_token(prior_note.text, lambda t: "," in t and "." not in t) + eps_literal = _last_token(prior_note.text) + shares = Figure( + "weighted average number of equity shares", + Decimal(shares_literal.replace(",", "")), + "document", + (self.prior.cite_in_note(prior_note.number, shares_literal),), + literal=shares_literal, + ) + prior_eps = Figure( + "earnings per share (comparative)", + Decimal(eps_literal), + "document", + (self.prior.cite_in_note(prior_note.number, eps_literal),), + ) + profit = self.rolled.statement("profit_and_loss").line("Profit for the year").current + eps = Quotient("basic and diluted earnings per share", profit, shares, Decimal(100000)) + body = ( + Body() + .markup("<p>Profit attributable to equity shareholders ") + .fig(profit, "0.00") + .text("; weighted average number of equity shares ") + .fig(shares, shares_literal) + .text("; basic and diluted earnings per share INR ") + .fig(eps, "0.00") + .text(" (year ended ") + .dated( + self.ledger.year_label(self.rolled.prior_year), + self.ledger.year_label(self.rolled.prior_year - 1), + ) + .text(": INR ") + .fig(prior_eps, "0.00") + .text( + "). The share count is carried from the signed set because the ledger shows no " + "movement in equity share capital between the two years." + ) + .markup("</p>") + ) + where = f"Note {note.number} earnings per share" + content.elements.append( + Element( + "recipe:earnings_per_share", + "p", + body.render(where), + body.skeleton(where), + body.figures, + body.quotes, + ) + ) + + def _recipe_lease_measurement(self, note: Note, content: NoteContent, item: str) -> None: + deed = self.corpus / "sources" / "lease-agreement-warehouse-2025.md" + area = _figure_from(deed, "42,000", self.corpus, "area of the identified asset") + rent = _figure_from(deed, "6,50,000", self.corpus, "monthly rent, months 1-12") + deposit = _figure_from(deed, "39,00,000", self.corpus, "refundable security deposit") + body = Body().markup("<p>") + body.quote( + "The Company leases a warehouse and dispatch facility. The lease commenced on " + "1 July 2025 for a term of 5 years, with a lock-in period of 36 months.", + _quote_from(deed, "Commencement of lease term", self.corpus), + ) + body.text(" The identified asset is a warehouse unit of ").fig(area, "0.00").text( + " sq ft; the lessor may not substitute an alternative property. Monthly rent for " + "the first twelve months is INR " + ).fig(rent, "0.00").text( + ", escalating at 5% per annum on each anniversary of the commencement date. A " + "refundable security deposit of INR " + ).fig(deposit, "0.00").text( + " is held by the lessor. The incremental borrowing rate applied on initial " + "recognition is 9.15% per annum. The renewal option has been excluded from the " + "lease term because its exercise is not reasonably certain." + ).markup("</p>") + where = f"Note {note.number} lease measurement" + content.elements.append( + Element( + "recipe:lease_measurement", + "p", + body.render(where), + body.skeleton(where), + body.figures, + body.quotes, + ) + ) + content.gaps.append( + Gap( + "the movement in the right-of-use asset for the year, including the " + "depreciation charge and the class-wise split", + self._sources_looked_in, + ) + ) + content.gaps.append( + Gap( + "depreciation on right-of-use assets and interest on lease liabilities " + "recognised in profit or loss, and the total cash outflow for leases", + self._sources_looked_in, + ) + ) + content.blocked_items.update({item, *self.also_blocks[item]}) + + def _recipe_lease_maturity(self, note: Note, content: NoteContent, item: str) -> None: + current = self.rolled.caption_line("2310") + non_current = self.rolled.caption_line("2210") + total = Derived( + "Total lease liabilities", + (Component(current.current), Component(non_current.current)), + ) + body = ( + Body() + .markup("<p>Lease liabilities as at ") + .dated( + self.ledger.year_label(self.rolled.year), + self.ledger.year_label(self.rolled.prior_year), + ) + .text(": falling due within one year ") + .fig(current.current, current.prior) + .text("; falling due after one year ") + .fig(non_current.current, non_current.prior) + .text("; total ") + .fig(total, "0.00") + .text( + ". The split is the current and non-current classification carried in the ledger." + ) + .markup("</p>") + ) + where = f"Note {note.number} lease maturity" + content.elements.append( + Element( + "recipe:lease_maturity", + "p", + body.render(where), + body.skeleton(where), + body.figures, + body.quotes, + ) + ) + content.gaps.append( + Gap( + "the contractual undiscounted maturity analysis of lease liabilities by time band", + self._sources_looked_in, + ) + ) + content.blocked_items.add(item) + + def _recipe_business_combination(self, note: Note, content: NoteContent, item: str) -> None: + minutes = self.corpus / "sources" / "board-minutes-2026-01-18.md" + consideration = _figure_from(minutes, "185.00", self.corpus, "consideration approved") + pattern, what = self.absence_probes[item] + keywords = [k.strip().lower() for k in pattern.split("|")] + searched = [a.name_cite for a in self.ledger.accounts] + matches = [a for a in self.ledger.accounts if any(k in a.name.lower() for k in keywords)] + body = Body().markup("<p>") + body.quote( + "The board minutes record the acquisition of 100% of the equity share capital of " + "Kompally Tooling Works Private Limited for a total cash consideration of ", + _quote_from(minutes, "total cash consideration", self.corpus), + ) + body.fig(consideration, "0.00") + body.quote( + ", with control passing to the Company on 29 January 2026.", + _quote_from(minutes, "control of KTW passed to the Company", self.corpus), + ) + body.text( + f" The trial balance for the year carries no balance matching " + f"{' or '.join(keywords)}: all {len(searched)} account names were searched and " + f"{len(matches)} matched. The two sources disagree; both are stated and neither is " + f"preferred in this note." + ) + body.markup("</p>") + where = f"Note {note.number} business combination" + content.elements.append( + Element( + "recipe:business_combination", + "p", + body.render(where), + body.skeleton(where), + body.figures, + body.quotes, + ) + ) + content.disagreements.append( + Disagreement( + about="whether a business combination is recognised in the FY" + f"{self.rolled.year} ledger", + sides=( + Side( + source=minutes.relative_to(self.corpus).as_posix(), + says="100% of KTW acquired for 185.00, control passing 29 January 2026", + cites=consideration.cites, + figures=(consideration,), + ), + Side( + source=self.ledger.rel, + says=f"no account name matches {what}; {len(searched)} account names " + f"searched, {len(matches)} matched", + cites=tuple(searched), + searched=len(searched), + ), + ), + ) + ) + for gap_text in ( + "the purchase price allocation: fair values of assets acquired and liabilities assumed", + "goodwill or bargain purchase gain arising, and the qualitative factors behind it", + "the basis of consolidation for the entity over which control was obtained", + ): + content.gaps.append(Gap(gap_text, self._sources_looked_in)) + content.blocked_items.update({item, *self.also_blocks[item]}) + + +# -- helpers --------------------------------------------------------------- + + +def _sentences(raw: str) -> list[tuple[str, int]]: + inner_start = raw.index(">") + 1 + inner = raw[inner_start : raw.rindex("<")] + out: list[tuple[str, int]] = [] + cursor = 0 + for piece in SENTENCE_SPLIT.split(inner): + position = inner.index(piece, cursor) + out.append((piece, inner_start + position)) + cursor = position + len(piece) + return out + + +def _qualifying_groups(prior_note, prior: PriorYearSet): + """(label, derived total, item figures) for every group that qualifies.""" + out = [] + seen: dict[str, int] = {} + for element in prior_note.body: + if element.tag != "p": + continue + text = strip_markup(element.raw) + for sentence in BREAKDOWN_SPLIT_SENTENCE.split(text): + match = GROUP.match(sentence.strip()) + if match is None or ";" not in match.group(2): + continue + label = match.group(1).strip() + items = list(match.group(2).split(";")) + figures: list[Figure] = [] + for piece in items: + tokens = money_tokens(piece) + if len(tokens) != 1: + figures = [] + break + literal = tokens[0] + occurrence = seen.get(literal, 0) + seen[literal] = occurrence + 1 + figures.append( + Figure( + piece.strip()[:60], + _decimal(literal), + "document", + (prior.cite_in_note(prior_note.number, literal, occurrence),), + ) + ) + if len(figures) < 2: + continue + out.append( + ( + label, + Derived(f"{label} (prior year)", tuple(Component(f) for f in figures)), + tuple(figures), + ) + ) + return out + + +def _decimal(literal: str) -> Decimal: + from .money import parse_money + + return parse_money(literal) + + +def _breakdown_labels(prior_note) -> list[str]: + """The labels last year's note used, for the components we cannot reproduce. + + A segment qualifies only if it holds exactly one figure - the same narrowing + that stops "for the year: 276.00 (intangible amortisation of 9.00 is + disclosed in Note 4...)" being read as a two-item breakdown. + """ + labels: list[str] = [] + for element in prior_note.body: + if element.tag != "p": + continue + for segment in BREAKDOWN_SPLIT.split(strip_markup(element.raw)): + tokens = money_tokens(segment) + if len(tokens) != 1: + continue + label = segment.replace(tokens[0], "").strip(" :,()").strip() + label = re.sub(r"\s+", " ", label) + # "Total" and "Total current" are the grand-total line; "Total + # outstanding dues of micro and small enterprises" is a caption and + # dropping it would silently mark the MSMED disclosure reproduced. + if not label or (DROP_LABEL.match(label) and len(label.split()) <= 2): + continue + labels.append(label[:70]) + return labels[:8] + + +def _account_for(rolled: RollForward, note: Note, group: str) -> str: + for statement in rolled.statements: + for line in statement.lines: + if line.note_ref == note.number and line.account and line.account.group == group: + return line.account.code + raise KeyError(f"Note {note.number} has no {group} caption") + + +def _first_token(text: str, predicate) -> str: + for token in money_tokens(text): + if predicate(token): + return token + raise KeyError(f"no token matching the predicate in {text[:60]!r}") + + +def _last_token(text: str) -> str: + return money_tokens(text)[-1] + + +def _quote_from(path: Path, phrase: str, corpus_root: Path) -> Cite: + raw = path.read_bytes() + position = raw.find(phrase.encode("utf-8")) + if position < 0: + raise KeyError( + f"{path.name}: the phrase {phrase!r} is not in the source. A quotation that " + f"cannot be located is not a quotation." + ) + return Cite( + path.relative_to(corpus_root).as_posix(), + position, + position + len(phrase.encode("utf-8")), + phrase, + f"{path.name}, quoted passage", + ) + + +def _figure_from(path: Path, literal: str, corpus_root: Path, where: str) -> Figure: + cite = _quote_from(path, literal, corpus_root) + return Figure( + where, + Decimal(literal.replace(",", "")), + "document", + (Cite(cite.path, cite.byte_start, cite.byte_end, literal, f"{path.name}, {where}"),), + literal=literal, + ) diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/checklist.py b/use-cases/preetham1930/statutory-statements-builder/statutory/checklist.py new file mode 100644 index 000000000..3b10e211b --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/checklist.py @@ -0,0 +1,189 @@ +"""The disclosure checklist: 32 rows of data, and a grammar that evaluates them. + +Nothing in this module contains an item id, a standard name or a trigger. The +checklist is found by its **columns**, never its filename, and every condition is +parsed from the CSV at run time. Adding a 33rd row is a data edit. + + condition := "always" | account_test | event_name + account_test := "account" code ("or" code)* "balance" [scope] operator number + scope := "current" | "prior" (default: current) + operator := ">" | ">=" | "<" | "<=" | "=" | "==" | "<>" | "!=" + +Three loud failures where the tempting default is a quiet one. An unparseable +condition, an unknown account code and an event with no detector each raise. A +silent `False` in any of them means "this disclosure is not required", which is +exactly the gap this app exists to close; a silent `True` invents work. Neither +is visible in the output, which is what makes stopping the right answer. + +Several account codes joined by `or` are a **disjunction**: the item is required +if any one satisfies the comparison. A conjunctive reading would suppress the +lease maturity item whenever one of the current/non-current pair happened to be +nil - which is precisely the year it matters. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from decimal import Decimal +from pathlib import Path + +from .csvspans import as_records, parse_csv +from .errors import UnknownAccountError, UnknownConditionError, UnknownEventError +from .figures import Cite + +REQUIRED_COLUMNS = ("item_id", "requirement", "trigger_condition", "satisfied_by_note") + +ACCOUNT_TEST = re.compile( + r"^account\s+(?P<codes>\d+(?:\s+or\s+\d+)*)\s+balance" + r"(?:\s+(?P<scope>current|prior))?\s*" + r"(?P<op>>=|<=|<>|!=|==|=|>|<)\s*" + r"(?P<number>-?\d+(?:\.\d+)?)\s*$", + re.I, +) + +OPERATORS = { + ">": lambda a, b: a > b, + ">=": lambda a, b: a >= b, + "<": lambda a, b: a < b, + "<=": lambda a, b: a <= b, + "=": lambda a, b: a == b, + "==": lambda a, b: a == b, + "<>": lambda a, b: a != b, + "!=": lambda a, b: a != b, +} + +GRAMMAR = ( + 'condition := "always" | account_test | event_name; ' + 'account_test := "account" code ("or" code)* "balance" [current|prior] ' + "(>|>=|<|<=|=|==|<>|!=) number" +) + + +@dataclass(frozen=True) +class ChecklistItem: + item_id: str + standard: str + requirement: str + trigger: str + prior_note: int | None # the note number in *last year's* numbering + severity: str + row_cite: Cite + + @property + def mapped_last_year(self) -> bool: + return self.prior_note is not None + + +@dataclass(frozen=True) +class Trigger: + """Why an item is required, with the citations that make it true.""" + + item: ChecklistItem + required: bool + because: str + cites: tuple[Cite, ...] + + +class Checklist: + def __init__(self, path: Path, corpus_root: Path) -> None: + self.path = path + self.rel = path.relative_to(corpus_root).as_posix() + header, records = as_records(parse_csv(path.read_bytes())) + missing = [c for c in REQUIRED_COLUMNS if c not in header] + if missing: + raise ValueError( + f"{self.rel}: a disclosure checklist is recognised by its columns, not its " + f"name. Missing: {', '.join(missing)}. Found: {', '.join(header)}" + ) + self.items: list[ChecklistItem] = [] + for record in records: + note_cell = record["satisfied_by_note"] + self.items.append( + ChecklistItem( + item_id=record["item_id"].text, + standard=record["standard"].text if "standard" in record else "", + requirement=record["requirement"].text, + trigger=record["trigger_condition"].text, + prior_note=int(note_cell.text) if note_cell.text else None, + severity=record["severity"].text if "severity" in record else "", + row_cite=Cite( + path=self.rel, + byte_start=record["requirement"].byte_start, + byte_end=record["requirement"].byte_end, + snippet=record["requirement"].value, + where=f"disclosure checklist, item {record['item_id'].text}", + ), + ) + ) + + def __len__(self) -> int: + return len(self.items) + + def evaluate(self, ledger, events) -> list[Trigger]: + return [self._evaluate_one(item, ledger, events) for item in self.items] + + def _evaluate_one(self, item: ChecklistItem, ledger, events) -> Trigger: + condition = item.trigger.strip() + if condition.lower() == "always": + return Trigger(item, True, "required in every set of statements", ()) + + match = ACCOUNT_TEST.match(condition) + if match: + return self._evaluate_accounts(item, match, ledger) + + if events.defines(condition): + hits = events.hits(condition) + if hits: + return Trigger( + item, + True, + f"event '{condition}' detected in {hits[0].source}", + tuple(c for h in hits for c in h.cites), + ) + return Trigger(item, False, f"event '{condition}' not detected in any source", ()) + + raise UnknownConditionError( + f"{self.rel}: item {item.item_id} has trigger {condition!r}, which is neither " + f"'always', nor an account test, nor an event this build defines. " + f"Grammar: {GRAMMAR}. Events defined: {', '.join(events.names()) or '(none)'}. " + f"An unevaluable trigger is not 'not required' - it stops the run." + ) + + def _evaluate_accounts(self, item: ChecklistItem, match, ledger) -> Trigger: + codes = re.split(r"\s+or\s+", match.group("codes"), flags=re.I) + scope = (match.group("scope") or "current").lower() + year = ledger.year if scope == "current" else ledger.prior_year + operator = OPERATORS[match.group("op")] + threshold = Decimal(match.group("number")) + + unknown = [c for c in codes if not ledger.has(c)] + if unknown: + raise UnknownAccountError( + f"{self.rel}: item {item.item_id} tests account(s) {', '.join(unknown)}, which " + f"the trial balance does not have. The ledger holds {len(ledger)} account(s): " + f"{', '.join(ledger.codes())}. A missing account is not a nil balance." + ) + + cites: list[Cite] = [] + satisfied: list[str] = [] + for code in codes: + account = ledger.account(code) + cites.append(account.cite(year)) + if operator(account.balance(year), threshold): + satisfied.append(code) + if satisfied: + detail = ", ".join( + f"{c} ({ledger.account(c).name}) = {ledger.account(c).balance(year)}" + for c in satisfied + ) + return Trigger(item, True, f"{detail} in FY{year}", tuple(cites)) + return Trigger(item, False, f"no account in {', '.join(codes)} satisfies the test", ()) + + +def check_unknown_event(name: str, events) -> None: + """Kept separate so the error can be raised from a caller that knows the row.""" + if not events.defines(name): + raise UnknownEventError( + f"event {name!r} has no detector. Defined: {', '.join(events.names()) or '(none)'}" + ) diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/chunks.py b/use-cases/preetham1930/statutory-statements-builder/statutory/chunks.py new file mode 100644 index 000000000..4d82cf3c6 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/chunks.py @@ -0,0 +1,183 @@ +"""Chunks: the unit SuperDocs actually addresses, and the unit we verify. + +Measured rather than assumed (PROGRESS.md 2026-08-09): an HTML upload chunks at +the top-level element - every paragraph, every heading, and **the entire table as +one chunk**. Rows and cells carry no id. So the skeleton is built to match that +granularity: whatever we intend to replace later is its own top-level element, +and a note schedule is one table and therefore one chunk (Decision 37). + +`canonical()` is what "the expected bytes" means in practice. A document editor +is entitled to re-serialise whitespace, attribute order and character entities, +and treating that as a failure would make the verifier cry wolf on every step. +Everything that carries meaning - the tag sequence, every attribute value, and +every character of text including every digit - is compared exactly. The chunk +id is dropped because it is the address, not the content. + +The entity and void-element rules were measured, not assumed: we uploaded +`·` and got back the character it stands for, and we uploaded `<br>` and +got back `<br/>`. The first live run +stopped on it, which is the verbatim check doing exactly its job; the honest +resolution is to define "verbatim" at the level of content rather than of bytes, +and to say so here. +""" + +from __future__ import annotations + +import html as _html +import re +from dataclasses import dataclass +from html.parser import HTMLParser + +CHUNK_ID = re.compile(r'\sdata-chunk-id="[^"]*"') +TAG = re.compile(r"<(/?)([a-zA-Z0-9]+)((?:\s+[a-zA-Z0-9:-]+\s*=\s*\"[^\"]*\")*)\s*(/?)>") +ATTR = re.compile(r"([a-zA-Z0-9:-]+)\s*=\s*\"([^\"]*)\"") +WS = re.compile(r"\s+") +TOP_LEVEL = ("h1", "h2", "h3", "h4", "p", "table", "ul", "ol", "div", "section", "blockquote") +VOID = {"br", "meta", "hr", "img", "link", "input", "col"} +# HTML parsers insert these whether or not the source had them. Measured: we +# uploaded a table with no <tbody> and it came back wrapped in one. +IMPLICIT = {"tbody", "thead", "tfoot"} + + +def canonical(fragment: str) -> str: + """Whitespace-collapsed, attribute-sorted, chunk-id-free form of an HTML fragment.""" + out: list[str] = [] + cursor = 0 + for match in TAG.finditer(fragment): + text = fragment[cursor : match.start()] + if text: + out.append(WS.sub(" ", _html.unescape(text))) + closing, tag, attrs, selfclose = match.groups() + pairs = sorted( + (k.lower(), WS.sub(" ", v).strip()) + for k, v in ATTR.findall(attrs) + if k.lower() != "data-chunk-id" + ) + if tag.lower() in IMPLICIT: + cursor = match.end() + continue + rendered = "".join(f' {k}="{v}"' for k, v in pairs) + # `<br>` and `<br/>` are the same element; the product re-serialises the + # one we send as the other, and that is punctuation, not content. + marker = "/" if selfclose and tag.lower() not in VOID else "" + out.append(f"<{closing}{tag.lower()}{rendered}{marker}>") + cursor = match.end() + tail = fragment[cursor:] + if tail: + out.append(WS.sub(" ", _html.unescape(tail))) + return WS.sub(" ", "".join(out)).replace("> <", "><").strip() + + +@dataclass(frozen=True) +class Chunk: + chunk_id: str + tag: str + html: str + + @property + def canonical(self) -> str: + return canonical(self.html) + + +class _Scanner(HTMLParser): + def __init__(self) -> None: + super().__init__(convert_charrefs=False) + self.spans: list[tuple[str, str, int, int]] = [] + self._depth = 0 + self._tag: str | None = None + self._start = 0 + self._chunk_id = "" + self._text = "" + + def _offset(self) -> int: + line, col = self.getpos() + return self._line_start[line - 1] + col + + def feed_text(self, text: str) -> None: + self._text = text + self._line_start = [0] + offset = 0 + for line in text.splitlines(keepends=True): + offset += len(line) + self._line_start.append(offset) + self.feed(text) + self.close() + + def handle_starttag(self, tag: str, attrs) -> None: + if tag in VOID: + return + if self._tag is None: + if tag in TOP_LEVEL: + self._tag = tag + self._depth = 1 + self._start = self._offset() + self._chunk_id = dict(attrs).get("data-chunk-id") or "" + return + if tag == self._tag: + self._depth += 1 + + def handle_endtag(self, tag: str) -> None: + if self._tag is None or tag != self._tag: + return + self._depth -= 1 + if self._depth == 0: + end = self._offset() + len(f"</{tag}>") + self.spans.append((self._chunk_id, self._tag, self._start, end)) + self._tag = None + + +class ChunkMap: + """An ordered map of chunk id -> chunk, as the document currently stands.""" + + def __init__(self, html: str) -> None: + scanner = _Scanner() + scanner.feed_text(html) + self.chunks: list[Chunk] = [ + Chunk(chunk_id, tag, html[start:end]) for chunk_id, tag, start, end in scanner.spans + ] + self.by_id = {c.chunk_id: c for c in self.chunks if c.chunk_id} + + def __len__(self) -> int: + return len(self.chunks) + + @property + def ids(self) -> list[str]: + return [c.chunk_id for c in self.chunks] + + def headings(self) -> dict[int, str]: + """Note number -> heading text, read back from the document itself.""" + from .money import strip_markup + from .priorset import NOTE_HEADING + + found: dict[int, str] = {} + for chunk in self.chunks: + if chunk.tag not in ("h3", "h4"): + continue + text = strip_markup(chunk.html).strip() + match = NOTE_HEADING.match(text) + if match: + found[int(match.group("number"))] = text + return found + + def text_of(self, chunk_id: str) -> str: + from .money import strip_markup + + return strip_markup(self.by_id[chunk_id].html).strip() + + +def strip_chunk_ids(html: str) -> str: + return CHUNK_ID.sub("", html) + + +def tidy(fragment: str) -> str: + """Drop the whitespace an HTML serialiser drops, before we ever send it. + + Measured: we sent `<br>\nPrepared in accordance...` and the edited chunk came + back as `<br>Prepared in accordance...`. Rather than keep widening + `canonical()` until it forgives everything, the document we compute is + written the way a serialiser would write it, so there is less for the + comparison to forgive. Whitespace *inside* a line is left alone - that is + where the meaningful spaces between words are. + """ + fragment = re.sub(r">[ \t]*\n[ \t]*", ">", fragment) + return re.sub(r"[ \t]*\n[ \t]*<", "<", fragment) diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/csvspans.py b/use-cases/preetham1930/statutory-statements-builder/statutory/csvspans.py new file mode 100644 index 000000000..e4e5e47f9 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/csvspans.py @@ -0,0 +1,101 @@ +"""An RFC 4180 scanner that keeps byte positions. + +`csv.reader` yields values and throws positions away, and the positions are the +whole point: a figure we write into the statements has to name the byte range it +came from. Rebuilt here rather than imported (Decision 31); the idea is copied +from `system/ingest`, the code is not. + +Assumption: the corpus CSVs are UTF-8 with `\\n` or `\\r\\n` line endings and no +embedded newlines inside quoted fields. Both hold across all three corpus CSVs +and `.gitattributes` pins the endings; a file that breaks it raises rather than +mis-parsing, because a silently wrong byte offset is worse than a loud stop. +""" + +from __future__ import annotations + +from dataclasses import dataclass + + +@dataclass(frozen=True) +class Cell: + value: str + byte_start: int + byte_end: int + + @property + def text(self) -> str: + return self.value.strip() + + +def parse_csv(raw: bytes) -> list[list[Cell]]: + """Rows of cells. `byte_start`/`byte_end` span the field's *value* bytes. + + For a quoted field the span is the bytes inside the quotes, so a citation + reads back as the value rather than as the value plus its punctuation. + """ + rows: list[list[Cell]] = [] + row: list[Cell] = [] + i = 0 + n = len(raw) + while i <= n: + if i == n: + if row or (rows and raw.endswith(b"\n")): + pass + break + byte = raw[i : i + 1] + if byte == b'"': + start = i + 1 + j = start + chunks: list[bytes] = [] + while j < n: + if raw[j : j + 1] == b'"': + if raw[j + 1 : j + 2] == b'"': + chunks.append(b'"') + j += 2 + continue + break + chunks.append(raw[j : j + 1]) + j += 1 + if j >= n: + raise ValueError(f"unterminated quoted field starting at byte {i}") + row.append(Cell(b"".join(chunks).decode("utf-8"), start, j)) + i = j + 1 + # A quoted field must be followed by a delimiter, a newline or EOF. + if i < n and raw[i : i + 1] not in (b",", b"\r", b"\n"): + raise ValueError(f"stray bytes after a quoted field at byte {i}") + else: + j = i + while j < n and raw[j : j + 1] not in (b",", b"\r", b"\n"): + j += 1 + row.append(Cell(raw[i:j].decode("utf-8"), i, j)) + i = j + if i < n and raw[i : i + 1] == b",": + i += 1 + continue + if i < n and raw[i : i + 1] == b"\r": + i += 1 + if i < n and raw[i : i + 1] == b"\n": + i += 1 + rows.append(row) + row = [] + if i >= n: + break + if row: + rows.append(row) + return [r for r in rows if r and not (len(r) == 1 and r[0].value == "")] + + +def as_records(rows: list[list[Cell]]) -> tuple[list[str], list[dict[str, Cell]]]: + """Header row plus one dict per data row, keyed by column name.""" + if not rows: + raise ValueError("empty CSV: no header row") + header = [c.text for c in rows[0]] + records = [] + for row in rows[1:]: + if len(row) != len(header): + raise ValueError( + f"row has {len(row)} field(s) but the header has {len(header)}: " + f"{[c.text for c in row]}" + ) + records.append(dict(zip(header, row, strict=True))) + return header, records diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/disagreement.py b/use-cases/preetham1930/statutory-statements-builder/statutory/disagreement.py new file mode 100644 index 000000000..b3544d65f --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/disagreement.py @@ -0,0 +1,51 @@ +"""Two sources that disagree, both quoted, neither preferred. + +There is no `resolution`, no `resolved`, no `correct_value`, no `preferred_side`, +no `status` and no `approved` on anything in this module - and there is a test +that proves it by reflection rather than by reading. Decision 15's argument, +rebuilt for a document builder: a rule in a docstring is broken by the next +contributor by accident; a type with nowhere to write an opinion is not. + +Sides are ordered by source path, a stable and meaningless order. Ordering by +credibility would be a resolution smuggled in through the ordering. +""" + +from __future__ import annotations + +from dataclasses import dataclass + +from .figures import AnyFigure, Cite + + +@dataclass(frozen=True) +class Side: + source: str # corpus-relative path + says: str # what this source says, in words + cites: tuple[Cite, ...] + figures: tuple[AnyFigure, ...] = () + searched: int = 0 # how many locations were examined, when the side is an absence + + def __post_init__(self) -> None: + if not self.cites: + raise ValueError( + f"{self.source}: a side of a disagreement that cannot cite its own bytes is " + f"not evidence, and a disagreement missing a side is not emitted at all" + ) + + +@dataclass(frozen=True) +class Disagreement: + about: str + sides: tuple[Side, ...] + + def __post_init__(self) -> None: + if len(self.sides) < 2: + raise ValueError( + f"{self.about}: a disagreement needs at least two sides, each with its own " + f"citations. One side is an observation, not a disagreement." + ) + ordered = tuple(sorted(self.sides, key=lambda s: s.source)) + object.__setattr__(self, "sides", ordered) + + def sentence(self) -> str: + return " ".join(f"{s.source} says: {s.says}" for s in self.sides) diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/editplan.py b/use-cases/preetham1930/statutory-statements-builder/statutory/editplan.py new file mode 100644 index 000000000..035161fd4 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/editplan.py @@ -0,0 +1,150 @@ +"""The edit plan: one single-target chunk replacement per step, and nothing else. + +Every step is a `replace`. There is no other verb (Decision 28), no step targets +a heading (Decision 29 put the headings at their final numbers before upload), +and no step carries more than one chunk (Decision 35). Those three are checked +here, before anything is sent, because a plan that breaks a wire rule should +never reach the wire. + +The expected post-state is the **whole chunk's HTML**, not the value inside it +(Decision 37). That is what we compare on read-back, and it is also what we send: +the instruction hands SuperDocs the finished chunk rather than describing an edit +to make. +""" + +from __future__ import annotations + +from dataclasses import dataclass + +from .chunks import ChunkMap, canonical +from .errors import PlanRefusedError +from .skeleton import Block, StatutoryDocument + +FORBIDDEN_VERBS = ("create", "insert", "add a", "append", "renumber", "move ", "delete") + + +@dataclass +class Step: + index: int + role: str + tag: str + note_number: int | None + expected_html: str # what the chunk must hold afterwards (no chunk id) + was_html: str # what it holds before + chunk_id: str = "" # filled in once the document is uploaded and read back + + @property + def verb(self) -> str: + return "replace" + + def instruction(self) -> str: + """One instruction, one chunk, fully resolved. Nothing is left to plan.""" + return ( + f"Replace the entire contents of the chunk whose data-chunk-id is " + f"{self.chunk_id} with exactly the following HTML, character for character. " + f"The chunk must remain a single <{self.tag}> element and keep the same " + f"data-chunk-id; do not wrap it in a <div> or in any other element. " + f"Do not add any section. Do not change any heading. Do not touch any other " + f"chunk anywhere in the document.\n\n{self.expected_html}" + ) + + def describe(self) -> str: + where = f"Note {self.note_number}" if self.note_number else "front matter" + return f"[{self.index:02d}] replace {self.tag:5s} {self.role:34s} ({where})" + + +@dataclass +class Plan: + steps: list[Step] + document: StatutoryDocument + + def __len__(self) -> int: + return len(self.steps) + + def sample(self, limit: int | None) -> Plan: + if limit is None or limit >= len(self.steps): + return self + return Plan(self.steps[:limit], self.document) + + +def build_plan(document: StatutoryDocument) -> Plan: + # Ordered so that a halt loses the least verified work. Paragraph chunks + # first, table chunks last: measured on this build's live runs, a whole-table + # replacement is the one unit the product does not apply reliably - clean + # once, silently not applied twice, and once wrapped in a <div> nobody asked + # for. The order changes nothing about what is sent or checked; it changes + # which failure a reviewer is left holding. + blocks = sorted(document.editable_changed(), key=lambda b: (b.tag == "table",)) + steps = [ + Step( + index=index, + role=block.role, + tag=block.tag, + note_number=block.note_number, + expected_html=block.target, + was_html=block.skeleton, + ) + for index, block in enumerate(blocks, start=1) + ] + plan = Plan(steps, document) + assert_plan_is_legal(plan, document) + return plan + + +def assert_plan_is_legal(plan: Plan, document: StatutoryDocument) -> None: + headings = {b.role for b in document.blocks if not b.editable} + for step in plan.steps: + if step.verb != "replace": + raise PlanRefusedError( + f"step {step.index} uses the verb {step.verb!r}. There is one verb and it is " + f"replace (Decision 28): the only operation that inserts is also the one " + f"measured to fabricate sections nobody asked for." + ) + if step.role in headings or step.tag in ("h1", "h2", "h3"): + raise PlanRefusedError( + f"step {step.index} targets {step.role!r}, which is a heading. Headings carry " + f"the note numbering and are uploaded at their final values (Decision 29); " + f"there is no renumbering step to get wrong." + ) + if step.expected_html == step.was_html: + raise PlanRefusedError( + f"step {step.index} ({step.role}) would send a chunk that is already correct. " + f"An edit that changes nothing cannot be verified by read-back and reads as " + f"'not applied'." + ) + + +def bind_chunk_ids(plan: Plan, uploaded: ChunkMap) -> None: + """Match each step to the chunk it will address, by the bytes we uploaded. + + Matched on canonical content rather than on position, so a document that came + back re-ordered fails here rather than quietly editing the wrong chunk. + """ + by_canonical: dict[str, list[str]] = {} + for chunk in uploaded.chunks: + by_canonical.setdefault(chunk.canonical, []).append(chunk.chunk_id) + for step in plan.steps: + key = canonical(step.was_html) + candidates = by_canonical.get(key, []) + if len(candidates) != 1: + raise PlanRefusedError( + f"step {step.index} ({step.role}) matches {len(candidates)} uploaded chunk(s). " + f"A single-target edit needs exactly one target; {len(uploaded)} chunks came " + f"back. Content we uploaded:\n {step.was_html[:200]}" + ) + step.chunk_id = candidates[0] + + +def expected_after(before: ChunkMap, step: Step) -> dict[str, str]: + """The canonical form of every chunk after this step, computed before sending.""" + expected = {chunk.chunk_id: chunk.canonical for chunk in before.chunks} + if step.chunk_id not in expected: + raise PlanRefusedError( + f"step {step.index} targets chunk {step.chunk_id}, which is not in the document" + ) + expected[step.chunk_id] = canonical(step.expected_html) + return expected + + +def unused(block: Block) -> Block: # pragma: no cover - keeps the import honest + return block diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/errors.py b/use-cases/preetham1930/statutory-statements-builder/statutory/errors.py new file mode 100644 index 000000000..7da6ecd63 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/errors.py @@ -0,0 +1,76 @@ +"""Every failure this app can have, and the message it must carry. + +Hard rule 13's argument, applied to a document: a bare failure tells the +reviewer something is wrong but not what to do about it. Every error below names +what could not be done and where we looked. + +Non-ASCII characters are kept out of these messages on purpose: they are read in +a terminal, and Windows consoles mangled exactly the message that carried Phase +1's thesis (PROGRESS.md 2026-08-08, Assumption 19). +""" + +from __future__ import annotations + + +class StatutoryError(Exception): + """Base for everything raised by this package.""" + + +class CitationDriftError(StatutoryError): + """A citation no longer holds the literal it was created for.""" + + +class UnaccountedFigureError(StatutoryError): + """A money token reached a rendered body without resolving to a source.""" + + +class ComparativeMismatchError(StatutoryError): + """The ledger's prior-year column and the signed prior-year set disagree.""" + + +class TieBreakError(StatutoryError): + """A statement did not add up, or a note did not tie to its caption.""" + + +class CrossReferenceError(StatutoryError): + """A cross-reference does not resolve to a heading that exists.""" + + +class ChecklistDriftError(StatutoryError): + """A checklist item names a note the final document does not have.""" + + +class UnknownConditionError(StatutoryError): + """A checklist trigger could not be parsed. Never read as 'not required'.""" + + +class UnknownAccountError(StatutoryError): + """A trigger names an account the ledger does not have.""" + + +class UnknownEventError(StatutoryError): + """A trigger names an event no detector defines.""" + + +class PlanRefusedError(StatutoryError): + """An edit plan step broke one of the wire rules before it was sent.""" + + +class EditNotAppliedError(StatutoryError): + """The target chunk came back unchanged.""" + + +class EditAppliedWrongError(StatutoryError): + """The target chunk changed, but not to the bytes we computed.""" + + +class CollateralDamageError(StatutoryError): + """A chunk we did not target moved, appeared or vanished.""" + + +class DecisionAlreadyRecordedError(StatutoryError): + """A second decision on an item that already has one, without supersede.""" + + +class NotConfiguredError(StatutoryError): + """A live call was attempted with no SUPERDOCS_API_KEY set.""" diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/events.py b/use-cases/preetham1930/statutory-statements-builder/statutory/events.py new file mode 100644 index 000000000..25bb39bb3 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/events.py @@ -0,0 +1,170 @@ +"""Events are detected from this year's sources, never remembered. + +An event fires on a **block** when every phrase group has a phrase present in +that same block, and every group contributes a citation. One block rather than +one document is what keeps it honest: the prior-year set says "control of the +goods transfers to the customer" in the revenue note and carries "Equity share +capital" on the balance sheet, and a document-wide match would read those two as +a business combination. + +The vocabulary holds no company name, no acquiree name, no date, no amount and +no standard number. It is a CSV, so a new event is a data edit. + +Quarantine runs in both directions. A source carrying an instruction aimed at an +automated reader is reported as quarantined and then not consulted at all - +neither to suppress an event nor to manufacture one. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from pathlib import Path + +from .csvspans import as_records, parse_csv +from .figures import Cite + +COMMENT = re.compile(r"<!--(.*?)-->", re.S) +PARAGRAPH_SPLIT = re.compile(rb"\n[ \t]*\n") + + +@dataclass(frozen=True) +class Block: + source: str # corpus-relative path + byte_start: int + text: str + + +@dataclass(frozen=True) +class EventHit: + event: str + source: str + block_text: str + cites: tuple[Cite, ...] + + +@dataclass(frozen=True) +class Quarantine: + source: str + marker: str + cite: Cite + + +class EventDetector: + def __init__(self, vocabulary: Path, markers: Path) -> None: + _, records = as_records(parse_csv(vocabulary.read_bytes())) + self._groups: dict[str, list[list[str]]] = {} + for record in records: + name = record["event_name"].text + groups = [ + [phrase.strip().lower() for phrase in group.split("|") if phrase.strip()] + for group in record["phrase_groups"].text.split(";;") + ] + if not groups or not all(groups): + raise ValueError(f"{vocabulary.name}: event {name!r} has an empty phrase group") + self._groups[name] = groups + self._markers = [ + line.strip().lower() + for line in markers.read_text(encoding="utf-8").splitlines() + if line.strip() and not line.startswith("#") + ] + self.hits_by_event: dict[str, list[EventHit]] = {name: [] for name in self._groups} + self.quarantined: list[Quarantine] = [] + self.scanned: list[str] = [] + + def names(self) -> list[str]: + return sorted(self._groups) + + def defines(self, name: str) -> bool: + return name in self._groups + + def hits(self, name: str) -> list[EventHit]: + return self.hits_by_event.get(name, []) + + def is_quarantined(self, rel: str) -> bool: + return any(q.source == rel for q in self.quarantined) + + # -- scanning ---------------------------------------------------------- + + def scan(self, paths: list[Path], corpus_root: Path) -> None: + for path in sorted(paths): + rel = path.relative_to(corpus_root).as_posix() + raw = path.read_bytes() + marker = self._quarantine_marker(raw, rel) + if marker is not None: + self.quarantined.append(marker) + continue + self.scanned.append(rel) + for block in _blocks(raw, rel): + self._match(block, rel) + + def _quarantine_marker(self, raw: bytes, rel: str) -> Quarantine | None: + text = raw.decode("utf-8", errors="replace") + for match in COMMENT.finditer(text): + lowered = match.group(1).lower() + for marker in self._markers: + if marker in lowered: + start = len(text[: match.start()].encode("utf-8")) + end = len(text[: match.end()].encode("utf-8")) + return Quarantine( + source=rel, + marker=marker, + cite=Cite( + path=rel, + byte_start=start, + byte_end=end, + snippet=match.group(0), + where=f"{rel}, embedded comment addressed to an automated reader", + ), + ) + return None + + def _match(self, block: Block, rel: str) -> None: + lowered = block.text.lower() + for name, groups in self._groups.items(): + cites: list[Cite] = [] + for group in groups: + found = next((p for p in group if p in lowered), None) + if found is None: + cites = [] + break + position = lowered.index(found) + start = block.byte_start + len(block.text[:position].encode("utf-8")) + literal = block.text[position : position + len(found)] + cites.append( + Cite( + path=rel, + byte_start=start, + byte_end=start + len(literal.encode("utf-8")), + snippet=literal, + where=f"{rel}, phrase evidencing '{name}'", + ) + ) + if cites: + self.hits_by_event[name].append( + EventHit(name, rel, block.text.strip(), tuple(cites)) + ) + + +def _blocks(raw: bytes, rel: str) -> list[Block]: + """Blank-line separated blocks, with the byte offset of each.""" + blocks: list[Block] = [] + offset = 0 + for piece in PARAGRAPH_SPLIT.split(raw): + text = piece.decode("utf-8", errors="replace") + if text.strip(): + blocks.append(Block(rel, offset, text)) + offset += len(piece) + offset += 2 # the separator we split on; exact offsets are recomputed below + # Recompute exactly: splitting loses the separator width, and a byte offset + # that is approximately right is a citation that does not read back. + exact: list[Block] = [] + cursor = 0 + for block in blocks: + encoded = block.text.encode("utf-8") + position = raw.find(encoded, cursor) + if position < 0: # pragma: no cover - the block came from this file + raise ValueError(f"{rel}: could not locate a block in its own source") + exact.append(Block(rel, position, block.text)) + cursor = position + len(encoded) + return exact diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/figures.py b/use-cases/preetham1930/statutory-statements-builder/statutory/figures.py new file mode 100644 index 000000000..327c96f48 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/figures.py @@ -0,0 +1,375 @@ +"""Figures, their provenance, and the guarantee that nothing else reaches paper. + +Four bases and there is no fifth (INVARIANTS.md item 6): + +- `ledger` the figure is a trial-balance cell for the reporting year +- `comparative` a prior-year figure, cited twice - to the ledger's prior-year + column *and* to the signed prior-year statements - which is what + makes "the comparatives tie exactly" a checked claim +- `document` a figure quoted from a source document (the lease deed, the + minutes, the fixed-asset schedule) +- `derived` computed from figures that are themselves on one of the three + bases above, every component cited + +`Derived` has no `value` field. The total is a property recomputed on every +read, so a printed total cannot drift from the citations that support it and +cannot be stated apart from them. That is Decision 16's trick, rebuilt here +against a different substrate (Decision 31) - twenty lines, and copying it is +cheaper than any coupling. + +`Gap` is the honest alternative to inventing an analysis: it names what is +missing and every place we looked for it. +""" + +from __future__ import annotations + +import html as _html +from collections import Counter +from dataclasses import dataclass, field +from decimal import Decimal +from pathlib import Path + +from .errors import CitationDriftError, UnaccountedFigureError +from .money import format_money, money_tokens_in_html + +BASES = ("ledger", "comparative", "document", "derived") +SOURCE_BASES = ("ledger", "comparative", "document") + + +@dataclass(frozen=True) +class Cite: + """A byte range in a corpus file that physically contains `snippet`.""" + + path: str # corpus-relative, forward slashes + byte_start: int + byte_end: int + snippet: str + where: str # human location, e.g. "trial balance row 1110, column fy2026_inr_lakhs" + + def __post_init__(self) -> None: + if self.byte_end < self.byte_start: + raise ValueError(f"{self.path}: byte range runs backwards") + if not self.snippet: + raise ValueError(f"{self.path}: a citation with an empty snippet cites nothing") + + def read_back(self, root: Path) -> str: + raw = (root / self.path).read_bytes()[self.byte_start : self.byte_end] + return raw.decode("utf-8") + + def verify(self, root: Path) -> None: + """Hard rule 2: re-read the bytes and confirm they still say what we cited.""" + actual = self.read_back(root) + if actual != self.snippet: + raise CitationDriftError( + f"citation drift in {self.path} at bytes {self.byte_start}-{self.byte_end} " + f"({self.where}): cited {self.snippet!r}, the file now holds {actual!r}" + ) + + +@dataclass(frozen=True) +class Figure: + """One figure with its provenance and the exact string it is written as.""" + + label: str + value: Decimal + basis: str + cites: tuple[Cite, ...] + literal: str | None = None # when the source spells it differently (6,50,000) + + def __post_init__(self) -> None: + if self.basis not in SOURCE_BASES: + raise ValueError( + f"{self.label}: basis {self.basis!r} is not one of {SOURCE_BASES}. " + f"A derived total is a Derived, which has no value field." + ) + if not self.cites: + raise ValueError( + f"{self.label}: a figure with no citation has no provenance and may not exist" + ) + if self.basis == "comparative" and len(self.cites) < 2: + raise ValueError( + f"{self.label}: a comparative is cited twice - to the ledger's prior-year " + f"column and to the signed prior-year statements - or it is not a comparative" + ) + + @property + def rendered(self) -> str: + return self.literal if self.literal is not None else format_money(self.value) + + +@dataclass(frozen=True) +class Component: + figure: Figure | Derived + sign: int = 1 + + def __post_init__(self) -> None: + if self.sign not in (1, -1): + raise ValueError("a component contributes +1 or -1, nothing else") + + @property + def contribution(self) -> Decimal: + return self.sign * self.figure.value + + +@dataclass(frozen=True) +class Derived: + """A total that appears in no source. Its value is computed, never stored.""" + + label: str + components: tuple[Component, ...] + basis: str = field(default="derived", init=False) + + def __post_init__(self) -> None: + if not self.components: + raise ValueError(f"{self.label}: a derived figure with no components is a bare number") + + @property + def value(self) -> Decimal: + return sum((c.contribution for c in self.components), Decimal("0")).quantize( + Decimal("0.01") + ) + + @property + def rendered(self) -> str: + return format_money(self.value) + + @property + def cites(self) -> tuple[Cite, ...]: + out: list[Cite] = [] + for component in self.components: + out.extend(component.figure.cites) + return tuple(out) + + def workings(self) -> str: + parts = [] + for index, component in enumerate(self.components): + sign = "-" if component.sign < 0 else ("+" if index else "") + parts.append(f"{sign} {format_money(abs(component.figure.value))}".strip()) + return f"{' '.join(parts)} = {self.rendered}" + + +@dataclass(frozen=True) +class Gap: + """What could not be produced, and every place we looked for it. + + Not a finding (INVARIANTS.md item 20): it is a property of the document - + this note cannot state that analysis from the sources supplied. + """ + + what: str + looked_in: tuple[str, ...] + figures: tuple[AnyFigure, ...] = () + + def __post_init__(self) -> None: + if not self.what.strip(): + raise ValueError("a gap must say what is missing") + if not self.looked_in: + raise ValueError( + f"{self.what}: a gap that does not say where we looked is a shrug, not a gap" + ) + + def sentence(self) -> str: + return f"{self.what} (looked in: {', '.join(self.looked_in)})" + + +# -------------------------------------------------------------------------- +# The body builder, and the guarantee +# -------------------------------------------------------------------------- + + +@dataclass(frozen=True) +class Quotient: + """A per-share figure. Division does not fit a signed sum, so it gets a type. + + `scale` is a declared unit conversion (lakhs to rupees), not a fudge factor, + and it is written at the call site where the units are visible. + """ + + label: str + numerator: AnyFigure + denominator: AnyFigure + scale: Decimal = Decimal(1) + basis: str = field(default="derived", init=False) + + @property + def value(self) -> Decimal: + if self.denominator.value == 0: + raise ZeroDivisionError(f"{self.label}: the denominator is nil") + return (self.numerator.value * self.scale / self.denominator.value).quantize( + Decimal("0.01") + ) + + @property + def rendered(self) -> str: + return format_money(self.value) + + @property + def cites(self) -> tuple[Cite, ...]: + return tuple(self.numerator.cites) + tuple(self.denominator.cites) + + +AnyFigure = Figure | Derived | Quotient + + +@dataclass(frozen=True) +class _Slot: + """One figure as it appears twice: in the target, and in the skeleton.""" + + target: str + skeleton: str + figure: AnyFigure | None + + +class Body: + """A rendered fragment plus the figures that account for every token in it. + + Text goes through `text()`, markup through `markup()`, and every figure goes + through `fig()`. Then `render()` re-scans its own output and refuses to + return a fragment holding a token no figure accounts for. + + A body renders twice. `render()` is the FY figures we computed; `skeleton()` + is the same structure carrying last year's, which is what gets uploaded + (Decision 29). Both are scanned, and the plan only contains the chunks where + the two differ - an identical chunk needs no edit and is never sent. + """ + + def __init__(self) -> None: + self._parts: list[str | _Slot] = [] + self._figures: list[AnyFigure] = [] + self._quotes: list[Cite] = [] + + def markup(self, raw: str) -> Body: + self._parts.append(raw) + return self + + def text(self, value: str) -> Body: + self._parts.append(_html.escape(value, quote=False)) + return self + + def dated(self, target: str, skeleton: str) -> Body: + """A label that differs between the two renderings, e.g. the year.""" + self._parts.append( + _Slot(_html.escape(target, quote=False), _html.escape(skeleton, quote=False), None) + ) + return self + + def fig(self, figure: AnyFigure, skeleton: AnyFigure | str | None = None) -> Body: + self._figures.append(figure) + if isinstance(skeleton, str): + shadow = skeleton + elif skeleton is None: + shadow = format_money(Decimal(0)) + else: + shadow = skeleton.rendered + self._parts.append( + _Slot( + _html.escape(figure.rendered, quote=False), + _html.escape(shadow, quote=False), + figure, + ) + ) + return self + + def prose(self, text: str, figures: tuple[AnyFigure, ...] = ()) -> Body: + """Sentence text whose figures are named, emitted as figures where they occur. + + There is deliberately no "declare this figure without printing it" door: + a figure only counts once its rendered string has actually been found in + the sentence and emitted as a slot. A sentence that mentions a figure the + caller did not declare still fails the scan. + """ + remaining = text + while remaining: + hit = None + for figure in figures: + position = remaining.find(figure.rendered) + if position >= 0 and (hit is None or position < hit[0]): + hit = (position, figure) + if hit is None: + self.text(remaining) + return self + position, figure = hit + self.text(remaining[:position]) + self.fig(figure) + remaining = remaining[position + len(figure.rendered) :] + return self + + def quote(self, text: str, cite: Cite) -> Body: + """Prose lifted from a source, with the bytes it came from recorded.""" + self._quotes.append(cite) + self._parts.append(_html.escape(text, quote=False)) + return self + + def carried(self, raw_html: str, figures: tuple[AnyFigure, ...] = ()) -> Body: + """Prose carried from the prior-year set verbatim, with its figures declared.""" + self._parts.append(raw_html) + self._figures.extend(figures) + return self + + @property + def figures(self) -> tuple[AnyFigure, ...]: + return tuple(self._figures) + + @property + def quotes(self) -> tuple[Cite, ...]: + return tuple(self._quotes) + + def _join(self, skeleton: bool) -> str: + return "".join( + (p.skeleton if skeleton else p.target) if isinstance(p, _Slot) else p + for p in self._parts + ) + + def render(self, where: str) -> str: + fragment = self._join(skeleton=False) + assert_accounted(fragment, self._figures, where) + return fragment + + def skeleton(self, where: str) -> str: + """The same structure with last year's figures. Uploaded, then replaced.""" + fragment = self._join(skeleton=True) + declared = [ + _Shadow(p.skeleton) + for p in self._parts + if isinstance(p, _Slot) and p.figure is not None + ] + assert_accounted(fragment, declared, f"{where} (skeleton)") + return fragment + + +@dataclass(frozen=True) +class _Shadow: + """A skeleton token. Carries no provenance because it never reaches an export.""" + + rendered: str + + +def assert_accounted(fragment: str, figures: list[AnyFigure] | tuple[AnyFigure, ...], where: str): + """Every money token in `fragment` is the rendered form of a declared figure. + + Counted as a multiset, so two nil balances need two figures. A token with no + figure behind it is a hard error naming the token and the place - never a + warning, never a caveat (hard rule 13). + """ + found = Counter(money_tokens_in_html(fragment)) + declared = Counter(f.rendered for f in figures) + unaccounted = found - declared + if unaccounted: + tokens = ", ".join(sorted(unaccounted.elements())) + raise UnaccountedFigureError( + f"{where}: {sum(unaccounted.values())} money token(s) reached the document with " + f"no source behind them: {tokens}. " + f"Declared figures were: {', '.join(sorted(declared.elements())) or '(none)'}. " + f"Every figure is ledger, comparative, document or derived; there is no fifth " + f"basis, and this is a hard error rather than a warning." + ) + + +def verify_all(figures: list[AnyFigure] | tuple[AnyFigure, ...], root: Path) -> int: + """Re-read every citation off disk. Returns how many were checked.""" + checked = 0 + for figure in figures: + for cite in figure.cites: + cite.verify(root) + checked += 1 + return checked diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/ledger.py b/use-cases/preetham1930/statutory-statements-builder/statutory/ledger.py new file mode 100644 index 000000000..a28803836 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/ledger.py @@ -0,0 +1,153 @@ +"""The trial balance: the ledger every primary-statement caption is drawn from. + +The reporting year is the highest `fy<year>` column present and the comparative +year is the next highest, so no year is written down in this module and a +FY2027 corpus needs a new column, not a code change (hard rule 6). + +Signs are the ledger's own. Liabilities, equity and income are carried negative, +and no absolute value is ever taken - `balance > 0` has to keep meaning "debit +balance" or the trigger grammar is lying about what it evaluates (Assumption 20, +carried over as a principle and re-checked here against all 32 conditions). + +Presentation sign is a property of the caption's *group*, not of the account: a +statutory balance sheet shows liabilities positive. The map below is the whole +of that rule and it is configuration, not logic. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from decimal import Decimal +from pathlib import Path + +from .csvspans import as_records, parse_csv +from .figures import Cite +from .money import parse_money + +FY_COLUMN = re.compile(r"^fy(?P<year>\d{4})_") + +# presented = balance * factor, keyed on the ledger's own `group` column. +PRESENTATION_SIGN: dict[str, int] = { + "Non-current assets": 1, + "Current assets": 1, + "Equity": -1, + "Non-current liabilities": -1, + "Current liabilities": -1, + "Income": -1, + "Expenses": 1, + "Tax expense": 1, +} + + +@dataclass(frozen=True) +class Account: + code: str + name: str + statement: str # BS | PL + group: str + balances: dict[int, Decimal] # year -> signed balance + cites: dict[int, Cite] # year -> citation of that cell + name_cite: Cite + order: int # position in the file; the statutory presentation order + + def balance(self, year: int) -> Decimal: + return self.balances[year] + + def presented(self, year: int) -> Decimal: + return self.balances[year] * PRESENTATION_SIGN[self.group] + + def cite(self, year: int) -> Cite: + return self.cites[year] + + +class TrialBalance: + def __init__(self, path: Path, corpus_root: Path) -> None: + self.path = path + self.rel = path.relative_to(corpus_root).as_posix() + raw = path.read_bytes() + header, records = as_records(parse_csv(raw)) + years: list[int] = [] + for column in header: + match = FY_COLUMN.match(column) + if match: + years.append(int(match.group("year"))) + if len(years) < 2: + raise ValueError( + f"{self.rel}: a roll-forward needs a reporting year and a comparative; " + f"found fy columns for {years or 'nothing'}" + ) + self.years = sorted(years, reverse=True) + self.year = self.years[0] + self.prior_year = self.years[1] + self._columns = {y: next(c for c in header if c.startswith(f"fy{y}_")) for y in self.years} + + self.accounts: list[Account] = [] + for order, record in enumerate(records): + balances: dict[int, Decimal] = {} + cites: dict[int, Cite] = {} + for year in self.years: + cell = record[self._columns[year]] + balances[year] = parse_money(cell.text) + cites[year] = Cite( + path=self.rel, + byte_start=cell.byte_start, + byte_end=cell.byte_end, + snippet=cell.value, + where=( + f"trial balance, account {record['account_code'].text}, " + f"column {self._columns[year]}" + ), + ) + name_cell = record["account_name"] + group = record["group"].text + if group not in PRESENTATION_SIGN: + raise ValueError( + f"{self.rel}: account {record['account_code'].text} is in group {group!r}, " + f"which has no presentation sign. Known groups: " + f"{', '.join(sorted(PRESENTATION_SIGN))}" + ) + self.accounts.append( + Account( + code=record["account_code"].text, + name=name_cell.text, + statement=record["statement"].text, + group=group, + balances=balances, + cites=cites, + name_cite=Cite( + path=self.rel, + byte_start=name_cell.byte_start, + byte_end=name_cell.byte_end, + snippet=name_cell.value, + where=f"trial balance, account {record['account_code'].text}, name", + ), + order=order, + ) + ) + self._by_code = {a.code: a for a in self.accounts} + + def __len__(self) -> int: + return len(self.accounts) + + def has(self, code: str) -> bool: + return code in self._by_code + + def account(self, code: str) -> Account: + return self._by_code[code] + + def codes(self) -> list[str]: + return [a.code for a in self.accounts] + + def in_group(self, statement: str, group: str) -> list[Account]: + return [a for a in self.accounts if a.statement == statement and a.group == group] + + def groups(self, statement: str) -> list[str]: + seen: list[str] = [] + for account in self.accounts: + if account.statement == statement and account.group not in seen: + seen.append(account.group) + return seen + + def year_label(self, year: int) -> str: + return f"31 March {year}" diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/money.py b/use-cases/preetham1930/statutory-statements-builder/statutory/money.py new file mode 100644 index 000000000..d183b2c58 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/money.py @@ -0,0 +1,91 @@ +"""Money, and the recogniser that decides what counts as one. + +The recogniser is deliberately narrow, and it is narrow for the same reason the +ingester's was (PROGRESS.md 2026-08-08, Assumption 13): a token is a figure only +if it is thousands-grouped (Indian grouping included, so `50,00,000` works) or +written to exactly two decimals. That excludes bare years, pin codes, survey +numbers, the CIN, plain counts and bare percentages. + +It has to be narrow in both directions here. Too wide and the year in "31 March +2026" becomes a figure that must be cited; too narrow and a figure slips into +the document without provenance, which is the one thing INVARIANTS.md item 6 +exists to prevent. Both directions are tested. +""" + +from __future__ import annotations + +import html as _html +import re +from decimal import Decimal, InvalidOperation + +# Grouped (2,180.00 / 50,00,000 / 42,000) or exactly two decimals (0.00 / 8.14). +# Brackets are the accounting negative. +# Brackets are matched as a *pair* or not at all. An optional bracket on each +# side independently reads "(703.00 + 87.00)" as the token "(703.00", which is +# a bracket belonging to the sentence - the exact silent-skip we caught in +# Phase 3 (PROGRESS.md 2026-08-09). Bracketed alternatives come first so the +# accounting negative wins where both could match. +_MONEY = re.compile( + r"\(\d{1,3}(?:,\d{2,3})+(?:\.\d{2})?\)" # bracketed, thousands-grouped + r"|\(\d+\.\d{2}\)" # bracketed, two decimals + r"|\d{1,3}(?:,\d{2,3})+(?:\.\d{2})?" # thousands-grouped + r"|\d+\.\d{2}" # exactly two decimals +) +_TAG = re.compile(r"<[^>]*>") + + +def format_money(value: Decimal) -> str: + """Accounting presentation: grouped to two decimals, negatives in brackets.""" + q = Decimal(value).quantize(Decimal("0.01")) + if q == 0: + # A nil balance carried through a presentation sign flip is Decimal("-0.00"), + # which formats as "-0.00" and reads as a figure nobody wrote. + q = abs(q) + if q < 0: + return f"({-q:,.2f})" + return f"{q:,.2f}" + + +def parse_money(token: str) -> Decimal: + """Inverse of `format_money` for a single recognised token.""" + text = token.strip() + negative = text.startswith("(") and text.endswith(")") + text = text.strip("()").replace(",", "") + try: + value = Decimal(text) + except InvalidOperation as exc: # pragma: no cover - guarded by the recogniser + raise ValueError(f"not a money token: {token!r}") from exc + return -value if negative else value + + +def strip_markup(fragment: str) -> str: + """Tag-free text of an HTML fragment, entities resolved. + + The scan runs over text, never over markup: a chunk id is a hex string and a + style rule is full of numbers, and neither is a figure anybody wrote. + """ + return _html.unescape(_TAG.sub(" ", fragment)) + + +def money_tokens(text: str) -> list[str]: + """Every money token in plain text, in order of appearance. + + A match is rejected when it is glued to more digits (so a longer number is + never read as a short one) or immediately followed by `%` (so `9.15%` is a + rate, not 9.15 lakhs). + """ + found: list[str] = [] + for match in _MONEY.finditer(text): + start, end = match.span() + before = text[start - 1] if start else "" + after = text[end] if end < len(text) else "" + if before.isdigit() or before in {",", "."}: + continue + if after.isdigit() or after == "%": + continue + found.append(match.group(0)) + return found + + +def money_tokens_in_html(fragment: str) -> list[str]: + return money_tokens(strip_markup(fragment)) diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/notetree.py b/use-cases/preetham1930/statutory-statements-builder/statutory/notetree.py new file mode 100644 index 000000000..4dab7fdb9 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/notetree.py @@ -0,0 +1,282 @@ +"""The note tree: where a new note goes, what everything downstream becomes. + +This is the module INVARIANTS.md items 3, 4 and 5 are about, and it is the +reason there is no mid-document insertion anywhere in this build. Every +consequence of adding a note is computed **here**, before a single byte is sent: +the new note's number, the new number of every note after it, every note -> +primary cross-reference, every `Note N` in prose, and every checklist mapping. +The document is then uploaded already in that shape (Decision 29). + +The insertion point is derived from the data, not from a list of note names: + + a new note triggered by an account sits immediately after the note that the + nearest *preceding* account in the ledger already points at + +For the warehouse lease that reads: account 1110 is required and unmapped; the +account before it in the ledger is 1100, which the signed balance sheet +cross-references to Note 3; therefore the lease note is Note 4 and notes 4..25 +become 5..26. Nothing here knows what a lease is. + +A new note triggered only by an event has no account to sit beside, so it is +appended. That is also a rule about the data, not about the standard. + +The English *title* of a new note is configuration (`config/new-note-titles.csv`, +keyed on the checklist's own `standard` column). The requirement for the note is +detected; the label is a label. Recorded as an assumption rather than smuggled. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from itertools import pairwise +from pathlib import Path + +from .checklist import Trigger +from .csvspans import as_records, parse_csv +from .errors import ChecklistDriftError, CrossReferenceError +from .figures import Cite +from .ledger import TrialBalance +from .priorset import NOTE_REFERENCE, PriorYearSet + + +@dataclass +class Note: + key: str # stable identity across the roll-forward + title: str + number: int = 0 + prior_number: int | None = None + is_new: bool = False + trigger_accounts: tuple[str, ...] = () + checklist_items: tuple[str, ...] = () + why: str = "" + why_cites: tuple[Cite, ...] = () + + @property + def heading(self) -> str: + return f"Note {self.number} — {self.title}" + + +@dataclass +class NoteTree: + notes: list[Note] + renumber: dict[int, int] = field(default_factory=dict) # prior number -> final number + account_note: dict[str, int] = field(default_factory=dict) # account code -> final number + inserted: list[Note] = field(default_factory=list) + + def by_number(self, number: int) -> Note: + return next(n for n in self.notes if n.number == number) + + def by_key(self, key: str) -> Note: + return next(n for n in self.notes if n.key == key) + + def has_number(self, number: int) -> bool: + return any(n.number == number for n in self.notes) + + def repoint_prose(self, text: str) -> str: + """Rewrite every `Note N` in prose through the same map that renumbered.""" + + def swap(match) -> str: + old = int(match.group(1)) + if old not in self.renumber: + raise CrossReferenceError( + f"prose points at Note {old}, which the prior-year set does not have. " + f"Known prior notes: {sorted(self.renumber)}" + ) + return f"Note {self.renumber[old]}" + + return NOTE_REFERENCE.sub(swap, text) + + +def load_new_note_titles(path: Path) -> dict[str, str]: + _, records = as_records(parse_csv(path.read_bytes())) + return {r["standard"].text: r["new_note_title"].text for r in records} + + +def build_note_tree( + rolled, + triggers: list[Trigger], + new_note_titles: dict[str, str], +) -> NoteTree: + prior = rolled.prior + ledger = rolled.ledger + notes: list[Note] = [ + Note(key=f"prior-{n.number}", title=n.title, prior_number=n.number) for n in prior.notes + ] + + clusters: dict[str, list[Trigger]] = {} + for trigger in triggers: + if not trigger.required or trigger.item.mapped_last_year: + continue + standard = trigger.item.standard + title = new_note_titles.get(standard) + if title is None: + raise ChecklistDriftError( + f"item {trigger.item.item_id} ({standard}) is required and maps to no note in " + f"the prior-year set, and no new-note title is configured for {standard!r}. " + f"Configured: {', '.join(sorted(new_note_titles)) or '(none)'}. " + f"A required disclosure with nowhere to live stops the run rather than " + f"quietly vanishing from the checklist." + ) + clusters.setdefault(title, []).append(trigger) + + # The account -> prior-note map comes from the roll-forward, which already + # matched each ledger account to its signed-set row (and disambiguated the + # two "Borrowings" and the two "Provisions" rows by their section). Matching + # a second time here would be a second chance to disagree with itself. + account_note_of_prior: dict[str, int] = { + line.account.code: line.prior_note_ref + for statement in rolled.statements + for line in statement.lines + if line.account is not None and line.prior_note_ref is not None + } + + inserted: list[Note] = [] + for title, cluster in clusters.items(): + accounts = _trigger_accounts(cluster, ledger) + note = Note( + key=f"new-{title.lower().replace(' ', '-')}", + title=title, + is_new=True, + trigger_accounts=accounts, + checklist_items=tuple(t.item.item_id for t in cluster), + why="; ".join(sorted({t.because for t in cluster})), + why_cites=tuple(c for t in cluster for c in t.cites), + ) + position = _insertion_index(note, notes, ledger, account_note_of_prior) + notes.insert(position, note) + inserted.append(note) + + for index, note in enumerate(notes, start=1): + note.number = index + + renumber = {n.prior_number: n.number for n in notes if n.prior_number is not None} + account_note = { + code: renumber[number] + for code, number in account_note_of_prior.items() + if number in renumber + } + for note in inserted: + for code in note.trigger_accounts: + account_note[code] = note.number + + tree = NoteTree(notes=notes, renumber=renumber, account_note=account_note, inserted=inserted) + _validate(tree, prior) + return tree + + +def _trigger_accounts(cluster: list[Trigger], ledger: TrialBalance) -> tuple[str, ...]: + codes: list[str] = [] + for trigger in cluster: + for cite in trigger.cites: + marker = "account " + if marker in cite.where: + code = cite.where.split(marker, 1)[1].split(",", 1)[0].strip() + if ledger.has(code) and code not in codes: + codes.append(code) + return tuple(sorted(codes, key=lambda c: ledger.account(c).order)) + + +def _insertion_index( + note: Note, + notes: list[Note], + ledger: TrialBalance, + account_note_of_prior: dict[str, int], +) -> int: + if not note.trigger_accounts: + return len(notes) # event-only: nothing to sit beside + anchor = ledger.account(note.trigger_accounts[0]) + preceding = [ + a for a in ledger.accounts if a.order < anchor.order and a.code in account_note_of_prior + ] + following = [ + a for a in ledger.accounts if a.order > anchor.order and a.code in account_note_of_prior + ] + if not preceding: + return 0 + before = account_note_of_prior[preceding[-1].code] + if following: + after = account_note_of_prior[following[0].code] + if after < before: + raise CrossReferenceError( + f"cannot place the {note.title!r} note from the data: account " + f"{anchor.code} sits between account {preceding[-1].code} (Note {before}) and " + f"account {following[0].code} (Note {after}), and the notes are out of ledger " + f"order. The insertion point is ambiguous and guessing it is exactly what " + f"this build refuses to do." + ) + index = next(i for i, n in enumerate(notes) if n.prior_number == before) + while index + 1 < len(notes) and notes[index + 1].prior_number == before: + index += 1 + return index + 1 + + +def _validate(tree: NoteTree, prior: PriorYearSet) -> None: + numbers = [n.number for n in tree.notes] + if numbers != list(range(1, len(numbers) + 1)): + raise CrossReferenceError(f"note numbers are {numbers}, which is not 1..n") + if len(tree.renumber) != len(prior.notes): + raise CrossReferenceError( + f"{len(prior.notes)} prior-year notes went in and {len(tree.renumber)} came out " + f"of the renumbering map; a note was lost." + ) + ordered = sorted(tree.renumber.items()) + for (old_a, new_a), (old_b, new_b) in pairwise(ordered): + if new_a >= new_b: + raise CrossReferenceError( + f"renumbering is not order-preserving: prior Note {old_a} -> {new_a} but prior " + f"Note {old_b} -> {new_b}. A roll-forward may insert, never reorder." + ) + for old, new in tree.renumber.items(): + if not tree.has_number(new): + raise CrossReferenceError(f"prior Note {old} maps to Note {new}, which does not exist") + + +def remap_checklist(triggers: list[Trigger], tree: NoteTree) -> dict[str, int | None]: + """checklist item id -> the FY2026 note that carries it, through the same map.""" + mapping: dict[str, int | None] = {} + for trigger in triggers: + item = trigger.item + if item.prior_note is not None: + if item.prior_note not in tree.renumber: + raise ChecklistDriftError( + f"checklist item {item.item_id} points at prior-year Note " + f"{item.prior_note}, which the signed set does not have. Prior notes run " + f"1..{max(tree.renumber)}." + ) + mapping[item.item_id] = tree.renumber[item.prior_note] + continue + carrier = next((n for n in tree.inserted if item.item_id in n.checklist_items), None) + mapping[item.item_id] = carrier.number if carrier else None + return mapping + + +def assert_checklist_consistent( + mapping: dict[str, int | None], tree: NoteTree, headings: dict[int, str] +) -> None: + """Every mapped item names a heading that exists, at the number we expect. + + `headings` is read back from the document, not from our own model - that is + the point. A drift here is a hard failure, never a warning (hard rule 13's + argument, applied to structure). + """ + for item_id, number in sorted(mapping.items()): + if number is None: + continue + if number not in headings: + raise ChecklistDriftError( + f"checklist item {item_id} maps to Note {number}, and the document has no " + f"heading at that number. Headings present: {sorted(headings)}" + ) + expected = tree.by_number(number).heading + if headings[number] != expected: + raise ChecklistDriftError( + f"checklist item {item_id} maps to Note {number}; we expect the heading " + f"{expected!r} and the document reads {headings[number]!r}" + ) + unaccounted = sorted(set(headings) - {n.number for n in tree.notes}) + if unaccounted: + raise ChecklistDriftError( + f"the document carries note heading(s) {unaccounted} that our tree does not have. " + f"An unaccounted heading is a section nobody planned - which is the shape of the " + f"fabrication we measured on 2026-08-09." + ) diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/orchestrator.py b/use-cases/preetham1930/statutory-statements-builder/statutory/orchestrator.py new file mode 100644 index 000000000..06d69a8b8 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/orchestrator.py @@ -0,0 +1,254 @@ +"""The run: upload once, then one verified chunk replacement at a time. + +The shape of a step, and every part of it is load-bearing: + + compute the expected post-state of the WHOLE document + -> one instruction, one chunk, on the async route + -> poll to the gate + -> refuse anything that is not exactly our one change + -> write OUR decision row, with an actor + -> approve (the actuator) + -> poll to completed + -> read the whole document back + -> classify: ok | not applied | applied wrong | collateral damage + +On any failure the queue **halts**. No downstream step runs, because a +downstream step is the consequence of work that may not have happened - that is +the 2026-08-07 corrupted-document mechanism and the halt is the whole defence. +Then: revert to the last verified-good state, read back again to confirm the +revert actually happened, surface the diff, and produce **no export**. + +Retry policy: at most one narrow retry of the same instruction, never a re-plan. +Re-planning is the degradation mode, measured twice; and a retry diffs the whole +document, because failed content has been seen arriving a turn later. +""" + +from __future__ import annotations + +import time +from dataclasses import dataclass, field +from pathlib import Path + +from .chunks import ChunkMap +from .editplan import Plan, bind_chunk_ids +from .superdocs.client import SuperDocsClient, select_our_change +from .superdocs.decisions import DecisionLedger +from .superdocs.verifier import Verification, verify, verify_upload_verbatim + + +@dataclass +class StepOutcome: + index: int + role: str + verification: Verification + retried: bool = False + job_id: str = "" + + @property + def ok(self) -> bool: + return self.verification.ok + + +@dataclass +class RunResult: + run_id: str + document_id: str = "" + session_id: str = "" + planned: int = 0 + outcomes: list[StepOutcome] = field(default_factory=list) + upload_receipt: str = "" + halted_at: int | None = None + exported: list[str] = field(default_factory=list) + reverted: str = "" + checklist_receipt: str = "" + + @property + def applied(self) -> int: + return sum(1 for o in self.outcomes if o.ok) + + @property + def ok(self) -> bool: + return self.halted_at is None and self.applied == self.planned + + def sentence(self, ledger: DecisionLedger) -> str: + """Computed from the outcomes and the rows, at the moment it is emitted.""" + if self.halted_at is not None: + return ( + f"HALTED at step {self.halted_at}. {self.applied} of {self.planned} planned " + f"change(s) verified applied; the rest were not attempted. No export was " + f"produced. {ledger.describe(self.run_id, self.planned)}." + ) + return ( + f"{self.applied} of {self.planned} planned change(s) verified applied by reading " + f"the whole document back after each one. " + f"{ledger.describe(self.run_id, self.planned)}." + ) + + +class Orchestrator: + def __init__( + self, + client: SuperDocsClient, + ledger: DecisionLedger, + actor: str, + *, + sleep=time.sleep, + export_formats: tuple[str, ...] = ("docx",), + export_dir: Path | None = None, + ) -> None: + self.client = client + self.ledger = ledger + self.actor = actor + self.sleep = sleep + self.export_formats = export_formats + self.export_dir = export_dir + self.log: list[str] = [] + + def _say(self, line: str) -> None: + self.log.append(line) + + def run(self, run_id: str, plan: Plan, filename: str) -> RunResult: + result = RunResult(run_id=run_id, planned=len(plan)) + skeleton_blocks = [b.skeleton for b in plan.document.blocks] + + document_id, session_id = self.client.upload_verbatim( + filename, plan.document.html("skeleton") + ) + result.document_id, result.session_id = document_id, session_id + uploaded, _ = self.client.read_back(document_id) + result.upload_receipt = verify_upload_verbatim(uploaded, skeleton_blocks) + self._say(result.upload_receipt) + bind_chunk_ids(plan, uploaded) + + before = uploaded + last_good = before + for step in plan.steps: + outcome = self._one_step(run_id, step, before, session_id, document_id) + result.outcomes.append(outcome) + if not outcome.ok: + result.halted_at = step.index + self._say(outcome.verification.report()) + self._say( + "halting: no downstream step runs on the consequences of an unconfirmed one" + ) + result.reverted = self._revert(session_id, document_id, last_good, result.applied) + return result + self._say(outcome.verification.report()) + before, _ = self.client.read_back(document_id) + last_good = before + + for fmt in self.export_formats: + written = self.client.export(session_id, fmt, self.export_dir) + result.exported.append(f"{fmt} -> {written}" if written else fmt) + return result + + def _one_step( + self, run_id: str, step, before: ChunkMap, session_id: str, document_id: str + ) -> StepOutcome: + verification = self._attempt(run_id, step, before, session_id, document_id) + if verification.ok: + return StepOutcome(step.index, step.role, verification) + # One narrow retry of the same instruction. Never a re-plan: re-planning + # is the measured degradation mode. Collateral damage is not retried at + # all - the document already holds content nobody asked for. + if any(p.kind == "collateral_damage" for p in verification.problems): + return StepOutcome(step.index, step.role, verification) + self._say(f"step {step.index}: one narrow retry, same instruction, no re-plan") + # The retry is compared against the **last verified-good** state, not + # against whatever the failed attempt left behind. Comparing against the + # wreckage turns "it applied the wrong thing twice" into "it did + # nothing", which is a different diagnosis and the wrong one. Found on + # the first live run of this build. + retried = self._attempt(run_id, step, before, session_id, document_id, supersede=True) + return StepOutcome(step.index, step.role, retried, retried=True) + + def _attempt( + self, + run_id: str, + step, + before: ChunkMap, + session_id: str, + document_id: str, + supersede: bool = False, + ) -> Verification: + job = self.client.chat_gated(session_id, document_id, step.instruction()) + if not job.awaiting: + return Verification( + step.index, + step.chunk_id, + problems=[ + _problem( + "not_applied", + f"the job reached status {job.status!r} without ever offering a change " + f"to approve. Nothing was decided and nothing is claimed.", + ) + ], + ) + change, unasked = select_our_change(job, step.chunk_id) + # Our row first, then the actuator (Decision 33). + self.ledger.record( + run_id, + step.index, + step.chunk_id, + "accept", + self.actor, + f"replacement for {step.role} matches the post-state computed from the ledger", + supersede=supersede, + ) + self.client.approve(session_id, job.job_id, change.change_id) + for extra in unasked: + # Denied bare: no feedback, ever (Decision 34). A rejected change + # still gets a row with an actor, because "who decided this" has to + # have an answer for every change the product offered, not only for + # the ones we wanted. + self.ledger.record( + run_id, + step.index, + extra.chunk_id, + "reject", + self.actor, + "not in the computed plan; the product proposed it unasked", + supersede=True, + ) + self.client.deny(session_id, job.job_id, extra.change_id) + self._say( + f"step {step.index}: denied an unasked change on chunk {extra.chunk_id} " + f"(bare, no feedback)" + ) + settled = self.client.poll(job.job_id, until_terminal=True, sleep=self.sleep) + del settled # the job's own account of itself is not evidence + after, _ = self.client.read_back(document_id) + return verify(step.index, step.chunk_id, before, after, step.expected_html) + + def _revert(self, session_id: str, document_id: str, last_good: ChunkMap, applied: int) -> str: + """Revert to the last verified-good turn, then read back and say what happened.""" + try: + self.client.revert(session_id, turn_index=applied) + except Exception as exc: + return f"revert call failed: {exc}. The document is left as it is and NOT exported." + after, _ = self.client.read_back(document_id) + same = [c.canonical for c in after.chunks] == [c.canonical for c in last_good.chunks] + if same: + return "reverted to the last verified-good state, confirmed by read-back" + return ( + "revert was requested and the read-back does NOT match the last verified-good " + f"state ({len(after)} chunk(s) now, {len(last_good)} then). The document is left " + f"as it is, no export was produced, and this is reported rather than smoothed over." + ) + + +def _problem(kind: str, detail: str): + from .superdocs.verifier import Problem + + return Problem(kind, detail) + + +def write_artifacts(out_dir: Path, plan: Plan) -> list[Path]: + """The computed document, both shapes, on disk - so a reviewer can read them.""" + out_dir.mkdir(parents=True, exist_ok=True) + written = [] + for which in ("skeleton", "target"): + path = out_dir / f"statements-{which}.html" + path.write_text(plan.document.html(which), encoding="utf-8") + written.append(path) + return written diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/pipeline.py b/use-cases/preetham1930/statutory-statements-builder/statutory/pipeline.py new file mode 100644 index 000000000..8eb347e09 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/pipeline.py @@ -0,0 +1,167 @@ +"""One function that turns `corpus/` into the document we intend to end up with. + +Everything before this point is ours and nothing here touches the network. The +whole roll-forward - parse, tie, detect, renumber, repoint, regenerate, render - +runs offline against the corpus, which is why the test suite can exercise real +behaviour with no API key (hard rule 3). +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from pathlib import Path + +from .bodies import NoteContent +from .checklist import Checklist, Trigger +from .events import EventDetector +from .figures import verify_all +from .ledger import TrialBalance +from .notetree import NoteTree, build_note_tree, load_new_note_titles, remap_checklist +from .priorset import PriorYearSet +from .skeleton import StatutoryDocument, assert_note_captions_tie, build_document +from .statements import RollForward, roll_forward + +DEFAULT_CONFIG = Path(__file__).resolve().parent.parent / "config" + + +@dataclass +class ChecklistLine: + item_id: str + standard: str + requirement: str + required: bool + because: str + prior_note: int | None + note: int | None + status: str # satisfied | outstanding | unsatisfied | not required + + @property + def flagged(self) -> str: + return "" if self.status == "satisfied" else self.status + + +@dataclass +class Rolled: + rolled: RollForward + tree: NoteTree + document: StatutoryDocument + triggers: list[Trigger] + mapping: dict[str, int | None] + checklist: list[ChecklistLine] = field(default_factory=list) + quarantined: list[str] = field(default_factory=list) + caption_ties: list[str] = field(default_factory=list) + cites_verified: int = 0 + + @property + def unsatisfied(self) -> list[ChecklistLine]: + return [line for line in self.checklist if line.status != "satisfied"] + + @property + def disagreements(self) -> list: + return [d for note in self.document.notes for d in note.disagreements] + + def note(self, number: int) -> NoteContent: + return next(n for n in self.document.notes if n.note.number == number) + + +def prepare(corpus_root: Path, config_root: Path = DEFAULT_CONFIG) -> Rolled: + prior = PriorYearSet( + next((corpus_root / "prior-year").glob("*.html")), + corpus_root, + ) + ledger = TrialBalance(_find_ledger(corpus_root), corpus_root) + rolled = roll_forward(ledger, prior, corpus_root) + + events = EventDetector( + config_root / "event-vocabulary.csv", config_root / "quarantine-markers.txt" + ) + events.scan( + [p for p in sorted((corpus_root / "sources").glob("*")) if p.is_file()], corpus_root + ) + + checklist = Checklist(_find_checklist(corpus_root), corpus_root) + triggers = checklist.evaluate(ledger, events) + + tree = build_note_tree( + rolled, triggers, load_new_note_titles(config_root / "new-note-titles.csv") + ) + mapping = remap_checklist(triggers, tree) + + # The primary statements' cross-references are repointed through the same + # map that renumbered the headings. This is the only place a note reference + # is written onto a statement line, so there is one map and no second chance + # to disagree with itself. + for statement in rolled.statements: + for line in statement.lines: + if line.account is not None: + line.note_ref = tree.account_note.get(line.account.code) + + document = build_document(rolled, tree, mapping, corpus_root, config_root) + + out = Rolled(rolled, tree, document, triggers, mapping) + out.quarantined = [q.source for q in events.quarantined] + out.caption_ties = assert_note_captions_tie(document, rolled) + out.checklist = _score_checklist(triggers, mapping, document) + out.cites_verified = verify_all(document.figures, corpus_root) + for cite in document.cites: + cite.verify(corpus_root) + return out + + +def _score_checklist( + triggers: list[Trigger], mapping: dict[str, int | None], document: StatutoryDocument +) -> list[ChecklistLine]: + blocked: dict[int, set[str]] = {} + gapped: set[int] = set() + for note in document.notes: + blocked.setdefault(note.note.number, set()).update(note.blocked_items) + if note.gaps or note.disagreements: + gapped.add(note.note.number) + lines: list[ChecklistLine] = [] + for trigger in triggers: + item = trigger.item + number = mapping.get(item.item_id) + if not trigger.required: + status = "not required" + elif number is None: + status = "unsatisfied" + elif item.item_id in blocked.get(number, set()) or number in gapped: + status = "outstanding" + else: + status = "satisfied" + lines.append( + ChecklistLine( + item_id=item.item_id, + standard=item.standard, + requirement=item.requirement, + required=trigger.required, + because=trigger.because, + prior_note=item.prior_note, + note=number, + status=status, + ) + ) + return lines + + +def _find_ledger(corpus_root: Path) -> Path: + """Found by its columns, not its filename.""" + for path in sorted((corpus_root / "sources").glob("*.csv")): + header = path.read_bytes().split(b"\n", 1)[0].decode("utf-8") + if "account_code" in header and "statement" in header: + return path + raise FileNotFoundError( + f"no trial balance in {corpus_root / 'sources'}: a trial balance is a CSV whose header " + f"carries account_code and statement columns" + ) + + +def _find_checklist(corpus_root: Path) -> Path: + for path in sorted((corpus_root / "rules").glob("*.csv")): + header = path.read_bytes().split(b"\n", 1)[0].decode("utf-8") + if "trigger_condition" in header and "satisfied_by_note" in header: + return path + raise FileNotFoundError( + f"no disclosure checklist in {corpus_root / 'rules'}: a checklist is a CSV whose header " + f"carries trigger_condition and satisfied_by_note columns" + ) diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/priorset.py b/use-cases/preetham1930/statutory-statements-builder/statutory/priorset.py new file mode 100644 index 000000000..ada38ab52 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/priorset.py @@ -0,0 +1,335 @@ +"""The prior-year signed statements, parsed so we can cite them and roll them on. + +Two things are needed from last year's set and nothing else: + +1. **Structure.** The order of the top-level elements, the note headings and + their numbers, and the `Note` column of the primary statements. That column + is real data - it is the note/caption cross-reference graph, supplied rather + than supplied by us - and every repoint rides on it. +2. **Citable figures.** A comparative is only a comparative if it can be cited + to the signed set as well as to the ledger, so every figure in the primary + tables and in the note prose is located by byte range. + +Byte offsets, not character offsets. The file is full of em dashes and +`·`, and a character offset would be silently wrong from the first note +onwards - which is the failure mode that is invisible until a citation is read +back on a different machine. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from html.parser import HTMLParser +from pathlib import Path + +from .figures import Cite +from .money import money_tokens, strip_markup + +TOP_LEVEL = ("h1", "h2", "h3", "p", "table") +VOID = {"br", "meta", "hr", "img", "link", "input"} + +# The dash class covers em, en and hyphen: the signed set uses an em dash and a +# different engagement will not. +NOTE_HEADING = re.compile(r"^\s*Note\s+(?P<number>\d+)\s*[—–-]\s*(?P<title>.+?)\s*$") # noqa: RUF001 +CELL = re.compile(r"<t([dh])\b[^>]*>(.*?)</t\1>", re.S) +ROW = re.compile(r"<tr\b[^>]*>(.*?)</tr>", re.S) +NOTE_REFERENCE = re.compile(r"\bNote\s+(\d+)\b") + + +class _ByteIndex: + """Character (line, col) from HTMLParser -> absolute byte offset.""" + + def __init__(self, text: str) -> None: + self._line_start: list[int] = [0] + self._lines: list[str] = text.splitlines(keepends=True) + offset = 0 + for line in self._lines: + offset += len(line.encode("utf-8")) + self._line_start.append(offset) + + def byte_at(self, line: int, col: int) -> int: + prefix = self._lines[line - 1][:col] if line - 1 < len(self._lines) else "" + return self._line_start[line - 1] + len(prefix.encode("utf-8")) + + +@dataclass(frozen=True) +class Element: + tag: str + raw: str # exact source slice, tags included + byte_start: int + byte_end: int + + @property + def text(self) -> str: + return strip_markup(self.raw).strip() + + @property + def inner(self) -> str: + return self.raw[self.raw.index(">") + 1 : self.raw.rindex("<")] + + +class _TopLevelScanner(HTMLParser): + def __init__(self, index: _ByteIndex) -> None: + super().__init__(convert_charrefs=False) + self.index = index + self.spans: list[tuple[str, int, int]] = [] + self._depth = 0 + self._open_tag: str | None = None + self._start: int = 0 + + def handle_starttag(self, tag: str, attrs) -> None: + if tag in VOID: + return + if self._open_tag is None: + if tag in TOP_LEVEL: + self._open_tag = tag + self._depth = 1 + line, col = self.getpos() + self._start = self.index.byte_at(line, col) + return + if tag == self._open_tag: + self._depth += 1 + + def handle_endtag(self, tag: str) -> None: + if self._open_tag is None or tag != self._open_tag: + return + self._depth -= 1 + if self._depth == 0: + line, col = self.getpos() + end = self.index.byte_at(line, col) + len(f"</{tag}>") + self.spans.append((self._open_tag, self._start, end)) + self._open_tag = None + + +@dataclass(frozen=True) +class StatementRow: + caption: str + note_ref: int | None + values: tuple[str, ...] # raw cell text, current year first + cites: tuple[Cite | None, ...] + is_total: bool + section: str = "" # the enclosing section header row, e.g. "Non-current liabilities" + is_section: bool = False + + +@dataclass(frozen=True) +class PrimaryStatement: + heading: Element + table: Element + rows: tuple[StatementRow, ...] + + def row(self, caption: str) -> StatementRow | None: + for candidate in self.rows: + if candidate.caption == caption: + return candidate + return None + + +@dataclass(frozen=True) +class PriorNote: + number: int + title: str + heading: Element + body: tuple[Element, ...] + + @property + def text(self) -> str: + return " ".join(e.text for e in self.body) + + +class PriorYearSet: + """Last year's signed statements: elements, primary statements, notes.""" + + def __init__(self, path: Path, corpus_root: Path) -> None: + self.path = path + self.rel = path.relative_to(corpus_root).as_posix() + raw = path.read_bytes() + self.raw = raw + text = raw.decode("utf-8") + scanner = _TopLevelScanner(_ByteIndex(text)) + scanner.feed(text) + scanner.close() + self.elements: list[Element] = [ + Element(tag, raw[s:e].decode("utf-8"), s, e) for tag, s, e in scanner.spans + ] + if not self.elements: + raise ValueError(f"{self.rel}: no top-level elements found; is this the signed set?") + + self.header: list[Element] = [] + self.statements: list[PrimaryStatement] = [] + self.notes_heading: Element | None = None + self.notes: list[PriorNote] = [] + self._split() + + # -- structure --------------------------------------------------------- + + def _split(self) -> None: + index = 0 + elements = self.elements + while index < len(elements) and elements[index].tag in ("h1", "p"): + self.header.append(elements[index]) + index += 1 + while index < len(elements): + element = elements[index] + if ( + element.tag == "h2" + and index + 1 < len(elements) + and elements[index + 1].tag == "table" + ): + table = elements[index + 1] + self.statements.append(PrimaryStatement(element, table, self._parse_table(table))) + index += 2 + continue + if element.tag == "h2": + self.notes_heading = element + index += 1 + break + index += 1 + current: PriorNote | None = None + body: list[Element] = [] + while index < len(elements): + element = elements[index] + if element.tag == "h3": + if current is not None: + self.notes.append( + PriorNote(current.number, current.title, current.heading, tuple(body)) + ) + match = NOTE_HEADING.match(element.text) + if not match: + raise ValueError( + f"{self.rel}: heading {element.text!r} is not 'Note N - Title'. " + f"Note numbering is read from the signed set, never guessed." + ) + current = PriorNote(int(match.group("number")), match.group("title"), element, ()) + body = [] + elif current is not None: + body.append(element) + index += 1 + if current is not None: + self.notes.append( + PriorNote(current.number, current.title, current.heading, tuple(body)) + ) + numbers = [n.number for n in self.notes] + if numbers != list(range(1, len(numbers) + 1)): + raise ValueError(f"{self.rel}: notes are numbered {numbers}, which is not 1..n") + + def _parse_table(self, table: Element) -> tuple[StatementRow, ...]: + rows: list[StatementRow] = [] + base = table.byte_start + prefix_bytes = len(table.raw[:0].encode("utf-8")) # 0; kept explicit for clarity + del prefix_bytes + section = "" + for row_match in ROW.finditer(table.raw): + row_html = row_match.group(1) + row_offset = row_match.start(1) + cells = list(CELL.finditer(row_html)) + texts = [strip_markup(c.group(2)).strip() for c in cells] + if cells and cells[0].group(1) == "h": + continue + if len(cells) < 2: + # A single spanning cell is a section header: "Non-current + # liabilities". It is what tells the two "Borrowings" rows apart. + if texts and texts[0]: + section = texts[0] + rows.append( + StatementRow( + caption=texts[0], + note_ref=None, + values=(), + cites=(), + is_total=False, + section=section, + is_section=True, + ) + ) + continue + caption = texts[0] + note_ref: int | None = None + if len(texts) > 1 and texts[1].isdigit(): + note_ref = int(texts[1]) + values: list[str] = [] + cites: list[Cite | None] = [] + for cell in cells[2:]: + inner = cell.group(2) + value = strip_markup(inner).strip() + values.append(value) + tokens = money_tokens(value) + if len(tokens) == 1: + literal = tokens[0] + position = inner.find(literal) + if position >= 0: + start_char = row_offset + cell.start(2) + position + byte_start = base + len(table.raw[:start_char].encode("utf-8")) + cites.append( + Cite( + path=self.rel, + byte_start=byte_start, + byte_end=byte_start + len(literal.encode("utf-8")), + snippet=literal, + where=f"prior-year signed statements, row {caption!r}", + ) + ) + continue + cites.append(None) + rows.append( + StatementRow( + caption=caption, + note_ref=note_ref, + values=tuple(values), + cites=tuple(cites), + is_total=caption.upper().startswith("TOTAL"), + section=section, + ) + ) + return tuple(rows) + + # -- lookups ----------------------------------------------------------- + + def note(self, number: int) -> PriorNote: + for candidate in self.notes: + if candidate.number == number: + return candidate + raise KeyError(f"{self.rel}: no note {number}") + + def statement_row(self, caption: str) -> tuple[PrimaryStatement, StatementRow] | None: + for statement in self.statements: + row = statement.row(caption) + if row is not None: + return statement, row + return None + + def cite_in_note(self, number: int, literal: str, occurrence: int = 0) -> Cite: + """Byte range of the `occurrence`-th `literal` inside note `number`'s body.""" + note = self.note(number) + seen = 0 + for element in note.body: + start = 0 + while True: + position = element.raw.find(literal, start) + if position < 0: + break + if seen == occurrence: + byte_start = element.byte_start + len(element.raw[:position].encode("utf-8")) + return Cite( + path=self.rel, + byte_start=byte_start, + byte_end=byte_start + len(literal.encode("utf-8")), + snippet=literal, + where=f"prior-year signed statements, Note {number} ({note.title})", + ) + seen += 1 + start = position + len(literal) + raise KeyError( + f"{self.rel}: {literal!r} occurrence {occurrence} is not in Note {number}. " + f"A comparative that is not in the signed set is not a comparative." + ) + + def note_references_in_prose(self) -> dict[int, list[int]]: + """note number -> the note numbers its prose points at.""" + out: dict[int, list[int]] = {} + for note in self.notes: + hits = [int(m.group(1)) for e in note.body for m in NOTE_REFERENCE.finditer(e.text)] + if hits: + out[note.number] = hits + return out diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/report.py b/use-cases/preetham1930/statutory-statements-builder/statutory/report.py new file mode 100644 index 000000000..2324d4935 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/report.py @@ -0,0 +1,149 @@ +"""What the run says for itself, in the terminal and in a file. + +Two rules shape it. Every number is computed from the objects at the moment it +is printed, never from a counter kept alongside them; and where the sources do +not support an analysis, the report says which analysis and where we looked +rather than reporting a smaller, tidier truth. +""" + +from __future__ import annotations + +from .money import format_money +from .pipeline import Rolled + +RULE = "-" * 78 + + +def render(rolled: Rolled, plan_size: int) -> str: + lines: list[str] = [] + ledger = rolled.rolled.ledger + year, prior_year = rolled.rolled.year, rolled.rolled.prior_year + + lines.append(RULE) + lines.append(f"STATUTORY ROLL-FORWARD FY{prior_year} -> FY{year}") + lines.append(RULE) + lines.append( + f"ledger: {len(ledger)} accounts, caption granularity, columns fy{year} and fy{prior_year}" + ) + lines.append( + f"prior-year set: {len(rolled.rolled.prior.notes)} notes, " + f"{len(rolled.rolled.prior.statements)} primary statements" + ) + if rolled.quarantined: + lines.append( + f"quarantined and not consulted (in either direction): {', '.join(rolled.quarantined)}" + ) + + lines.append("") + lines.append("TIES") + for label, left, right, ok in rolled.rolled.ties: + mark = "ok " if ok else "BAD" + lines.append(f" {mark} {label:58s} {format_money(left):>12s} {format_money(right):>12s}") + matched = [c for c in rolled.rolled.comparative_checks if c.signed_value is not None] + new_nil = [c for c in rolled.rolled.comparative_checks if c.signed_value is None] + lines.append( + f" comparatives: {len(matched)} caption(s) agree between the ledger's fy{prior_year} " + f"column and the signed set; {len(new_nil)} new caption(s) nil in both" + ) + for check in new_nil: + lines.append(f" new this year: {check.caption}") + lines.append(f" note <-> primary cross-references checked: {len(rolled.caption_ties)}") + coverage = rolled.document.coverage + lines.append( + f" prior-year breakdown groups that qualified for a tie-out: {coverage['groups']} " + f"({coverage['tied']} tied, {coverage['raised']} raised)" + ) + + lines.append("") + lines.append("NOTE TREE") + inserted = rolled.tree.inserted + lines.append( + f" {len(rolled.tree.notes)} notes ({len(rolled.tree.renumber)} rolled forward, " + f"{len(inserted)} required by this year's data)" + ) + for note in inserted: + lines.append(f" + Note {note.number} {note.title}") + lines.append(f" required because: {note.why}") + lines.append(f" carries checklist items: {', '.join(note.checklist_items)}") + shifted = [(old, new) for old, new in sorted(rolled.tree.renumber.items()) if old != new] + if shifted: + lines.append( + f" {len(shifted)} note(s) renumbered by the insertion: " + + ", ".join(f"{old}->{new}" for old, new in shifted[:6]) + + (" ..." if len(shifted) > 6 else "") + ) + + lines.append("") + lines.append("DISCLOSURE CHECKLIST") + unmapped_before = [ + line for line in rolled.checklist if line.required and line.prior_note is None + ] + lines.append( + f" {len(rolled.checklist)} items evaluated from the CSV, " + f"{sum(1 for line in rolled.checklist if line.required)} required" + ) + lines.append( + f" detected from THIS YEAR'S DATA and unsatisfied in the set we rolled forward: " + f"{len(unmapped_before)}" + ) + for line in unmapped_before: + lines.append( + f" {line.item_id} {line.standard:10s} no note in the prior-year set " + f"-> now Note {line.note}" + ) + lines.append(f" {line.requirement}") + lines.append(f" because: {line.because}") + by_status: dict[str, int] = {} + for line in rolled.checklist: + by_status[line.status] = by_status.get(line.status, 0) + 1 + lines.append( + " after the roll-forward: " + + ", ".join(f"{count} {status}" for status, count in sorted(by_status.items())) + ) + lines.append( + " 'outstanding' means the note exists and carries the caption, and the analysis the " + ) + lines.append( + " item requires is not derivable from the sources supplied (the trial balance is at " + ) + lines.append(" caption granularity). Each note names what is missing and where we looked.") + lines.append("") + lines.append(" item standard prior note FY note status") + for line in rolled.checklist: + prior = str(line.prior_note) if line.prior_note else "-" + current = str(line.note) if line.note else "-" + lines.append( + f" {line.item_id:7s} {line.standard:11s} {prior:>10s} {current:>9s} {line.status}" + ) + + if rolled.disagreements: + lines.append("") + lines.append("SOURCES THAT DISAGREE (stated, never resolved)") + for disagreement in rolled.disagreements: + lines.append(f" * {disagreement.about}") + for side in disagreement.sides: + extra = f" [{side.searched} location(s) searched]" if side.searched else "" + lines.append(f" {side.source}: {side.says}{extra}") + lines.append(f" cited: {len(side.cites)} byte range(s)") + + gaps = [(n.note.number, g) for n in rolled.document.notes for g in n.gaps] + lines.append("") + lines.append( + f"OUTSTANDING ANALYSIS ({len(gaps)} item(s) across {len(rolled.document.notes)} notes)" + ) + for number, gap in gaps: + lines.append(f" Note {number:2d}: {gap.what}") + + lines.append("") + lines.append("DOCUMENT") + lines.append( + f" {len(rolled.document.blocks)} chunk-aligned blocks; " + f"{sum(1 for b in rolled.document.blocks if not b.editable)} are headings and are " + f"uploaded at their final numbers (never edited)" + ) + lines.append( + f" {plan_size} chunk(s) differ between the skeleton and the target -> {plan_size} single-target replacements" + ) + lines.append(f" {rolled.cites_verified} citation(s) re-read from the corpus bytes and matched") + lines.append(RULE) + return "\n".join(lines) diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/skeleton.py b/use-cases/preetham1930/statutory-statements-builder/statutory/skeleton.py new file mode 100644 index 000000000..2e450e81e --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/skeleton.py @@ -0,0 +1,275 @@ +"""The whole document, computed here, in both its shapes. + +Decision 29, stated as code: we compute the **entire** final skeleton - every +heading at its final number, every cross-reference already repointed, the new +note's heading already in place - and upload that verbatim. There is no +insertion to perform, so the insertion bug cannot be reached; SuperDocs only +ever replaces the contents of a chunk that already exists. + +Two renderings come out of one structure: + +- `skeleton` - the document as uploaded. Same structure, last year's figures. +- `target` - the document we intend to end up with. + +The edit plan is exactly the blocks where the two differ, and **no heading block +is ever one of them** (INVARIANTS.md item 2). Blocks that are identical need no +edit and are never sent, which is what makes this a focused update rather than a +full rewrite. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from decimal import Decimal +from pathlib import Path + +from . import style +from .bodies import BodyBuilder, Element, NoteContent +from .chunks import tidy +from .errors import CrossReferenceError +from .figures import AnyFigure, Body, Cite +from .money import money_tokens_in_html +from .notetree import NoteTree +from .priorset import NOTE_REFERENCE +from .statements import Line, RollForward, Statement + + +@dataclass +class Block: + """One top-level element, which becomes one chunk on upload.""" + + role: str + tag: str + skeleton: str + target: str + note_number: int | None = None + figures: tuple[AnyFigure, ...] = () + quotes: tuple[Cite, ...] = () + editable: bool = True # headings are structure and are never targeted + + def __post_init__(self) -> None: + object.__setattr__(self, "skeleton", tidy(self.skeleton)) + object.__setattr__(self, "target", tidy(self.target)) + + @property + def changed(self) -> bool: + return self.skeleton != self.target + + +@dataclass +class StatutoryDocument: + blocks: list[Block] = field(default_factory=list) + notes: list[NoteContent] = field(default_factory=list) + tree: NoteTree | None = None + coverage: dict[str, int] = field(default_factory=dict) + + def html(self, which: str = "target") -> str: + return "\n".join(getattr(b, which) for b in self.blocks) + + @property + def figures(self) -> tuple[AnyFigure, ...]: + return tuple(f for b in self.blocks for f in b.figures) + + @property + def cites(self) -> tuple[Cite, ...]: + out = [c for f in self.figures for c in f.cites] + out.extend(c for b in self.blocks for c in b.quotes) + out.extend(c for n in self.notes for d in n.disagreements for s in d.sides for c in s.cites) + return tuple(out) + + def editable_changed(self) -> list[Block]: + return [b for b in self.blocks if b.editable and b.changed] + + +def build_document( + rolled: RollForward, + tree: NoteTree, + mapping: dict[str, int | None], + corpus_root: Path, + config_root: Path, +) -> StatutoryDocument: + prior = rolled.prior + document = StatutoryDocument(tree=tree) + year = rolled.year + prior_year = rolled.prior_year + + # -- header ------------------------------------------------------------ + document.blocks.append( + Block("entity", "h1", prior.header[0].raw, prior.header[0].raw, editable=False) + ) + document.blocks.append( + Block("registered-office", "p", prior.header[1].raw, prior.header[1].raw, editable=False) + ) + # The basis-of-preparation block is the signed set's own paragraph with the + # year moved on. Retyping it here would put the standard's name into our + # source, and a build that knows the name of the standard has started to + # remember the domain instead of reading it (there is a test). + basis_prior = prior.header[2].raw + basis_target = basis_prior.replace(f"31 March {prior_year}", f"31 March {year}") + if basis_target == basis_prior: + raise CrossReferenceError( + f"the signed set's basis-of-preparation paragraph does not name " + f"'31 March {prior_year}', so the reporting date cannot be rolled forward from it: " + f"{basis_prior[:160]}" + ) + document.blocks.append(Block("basis-of-preparation", "p", basis_prior, basis_target)) + document.blocks.append( + Block( + "status", + "p", + prior.header[3].raw, + "<p><em>Draft prepared by roll-forward for review. Not approved and not signed.</em></p>", + ) + ) + + # -- primary statements ------------------------------------------------ + for statement in rolled.statements: + heading = f"<h2>{statement.title} 31 March {year}</h2>" + document.blocks.append( + Block(f"{statement.key}-heading", "h2", heading, heading, editable=False) + ) + skeleton_html, target_html, figures = _render_statement(statement, rolled, prior) + document.blocks.append( + Block(statement.key, "table", skeleton_html, target_html, figures=figures) + ) + + notes_heading = "<h2>Notes to the Financial Statements</h2>" + document.blocks.append( + Block("notes-heading", "h2", notes_heading, notes_heading, editable=False) + ) + + # -- notes ------------------------------------------------------------- + builder = BodyBuilder(rolled, tree, mapping, corpus_root, config_root) + for note in tree.notes: + heading = f"<h3>Note {note.number} — {note.title}</h3>" + document.blocks.append( + Block("note-heading", "h3", heading, heading, note_number=note.number, editable=False) + ) + content = builder.build(note) + document.notes.append(content) + for element in content.elements: + document.blocks.append( + Block( + f"note-{note.number}-{element.role}", + element.tag, + element.skeleton, + element.target, + note_number=note.number, + figures=element.figures, + quotes=element.quotes, + ) + ) + document.coverage = dict(builder.coverage) + _assert_cross_references_resolve(document, tree) + return document + + +def _render_statement( + statement: Statement, rolled: RollForward, prior +) -> tuple[str, str, tuple[AnyFigure, ...]]: + year, prior_year = rolled.year, rolled.prior_year + body = Body().markup( + f'{style.table_open()}\n<tr style="{style.HEAD_ROW}">' + f"{style.head_cell('Particulars')}{style.head_cell('Note', numeric=True)}" + f'<th style="{style.TH_NUM}">' + ) + body.dated(f"31 March {year}", f"31 March {prior_year}").markup( + f'</th><th style="{style.TH_NUM}">' + ) + body.dated(f"31 March {prior_year}", f"31 March {prior_year - 1}").markup("</th></tr>\n") + for line in statement.lines: + if line.kind in ("section", "subheader"): + wrap = ("<strong>", "</strong>") if line.kind == "section" else ("<em>", "</em>") + body.markup( + f'<tr style="{style.SECTION_ROW}"><td colspan="4" style="{style.TD_TEXT}">{wrap[0]}' + ).text(line.caption).markup(f"{wrap[1]}</td></tr>\n") + continue + open_tag, close_tag = ("<strong>", "</strong>") if line.emphasis else ("", "") + body.markup(f'<tr><td style="{style.TD_TEXT}">').markup(open_tag).text(line.caption).markup( + close_tag + ) + body.markup(f'</td><td style="{style.TD_NUM}">') + if line.note_ref is not None: + body.text(str(line.note_ref)) + body.markup(f'</td><td style="{style.TD_NUM}">').markup(open_tag) + body.fig(line.current, line.prior) + body.markup(f'{close_tag}</td><td style="{style.TD_NUM}">{open_tag}') + body.fig(line.prior, _signed_prior_prior(prior, line)) + body.markup(f"{close_tag}</td></tr>\n") + body.markup("</table>") + where = f"{statement.key} table" + return body.skeleton(where), body.render(where), body.figures + + +def _signed_prior_prior(prior, line: Line) -> str: + """The signed set's own second column, used only as skeleton scaffolding.""" + found = prior.statement_row(line.caption) + if found is not None and len(found[1].values) > 1: + return found[1].values[1] + return "0.00" + + +def _assert_cross_references_resolve(document: StatutoryDocument, tree: NoteTree) -> None: + """Every `Note N` anywhere in the final document names a heading that exists.""" + numbers = {note.number for note in tree.notes} + for block in document.blocks: + if block.role == "note-heading": + continue + for match in NOTE_REFERENCE.finditer(block.target): + referenced = int(match.group(1)) + if referenced not in numbers: + raise CrossReferenceError( + f"block {block.role!r} points at Note {referenced}, which the final " + f"document does not have. Notes run 1..{max(numbers)}. Every reference is " + f"repointed through the same map that renumbered the headings, so this is " + f"a bug in the map, not a caveat." + ) + for statement in ("balance_sheet", "profit_and_loss"): + block = next(b for b in document.blocks if b.role == statement) + del block + + +def assert_note_captions_tie(document: StatutoryDocument, rolled: RollForward) -> list[str]: + """Every figure a note states for a caption is the figure the statement prints. + + Checked on the **rendered HTML** rather than on the objects, because the + objects being shared is the reason it holds and the rendering is what a + reader sees. Returns one line per checked caption for the run summary. + """ + checked: list[str] = [] + for statement in rolled.statements: + table = next(b for b in document.blocks if b.role == statement.key) + table_tokens = money_tokens_in_html(table.target) + for line in statement.lines: + if line.account is None or line.note_ref is None: + continue + note_blocks = [ + b + for b in document.blocks + if b.note_number == line.note_ref and b.role.endswith("captions") + ] + if not note_blocks: + continue + note_tokens = money_tokens_in_html(note_blocks[0].target) + current = line.current.rendered + if current not in table_tokens: + raise CrossReferenceError(f"{line.caption}: the statement does not print {current}") + if current not in note_tokens: + raise CrossReferenceError( + f"{line.caption}: Note {line.note_ref} does not print {current}, which the " + f"statement does. The note and the primary statement must tie exactly." + ) + checked.append(f"{line.caption} -> Note {line.note_ref}: {current}") + return checked + + +def zero() -> Decimal: # pragma: no cover - keeps Decimal imported for callers + return Decimal(0) + + +def note_body_blocks(document: StatutoryDocument, number: int) -> list[Block]: + return [b for b in document.blocks if b.note_number == number and b.role != "note-heading"] + + +def elements_of(content: NoteContent) -> list[Element]: # pragma: no cover - convenience + return content.elements diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/statements.py b/use-cases/preetham1930/statutory-statements-builder/statutory/statements.py new file mode 100644 index 000000000..21f3bf6a1 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/statements.py @@ -0,0 +1,432 @@ +"""The primary statements, rolled forward, with comparatives that are checked. + +The reporting-year column comes from the ledger. The comparative column comes +from the ledger *and* from the signed prior-year set, and the two must agree to +the paisa. That double citation is what turns "the comparatives tie" from a +claim into a check: if the trial balance's prior-year column had drifted from +what was signed, this raises rather than printing a number that reconciles to +nothing. + +Every subtotal and total is a `Derived` - computed on every read from the +account figures beneath it - so a statement cannot print a total that its own +rows do not support. The prior-year totals are then compared against the totals +the signed set actually prints, which is the second half of the same check. + +The layout (which groups sit under which section header, and what the subtotal +is called) is configuration keyed on the ledger's own `group` column. A new +group is a data change plus one line here, not a rewrite. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from decimal import Decimal +from pathlib import Path + +from .errors import ComparativeMismatchError, TieBreakError +from .figures import AnyFigure, Cite, Component, Derived, Figure +from .ledger import Account, TrialBalance +from .money import format_money, parse_money +from .priorset import PriorYearSet + +# section header -> the ledger groups shown under it, then the section total. +BALANCE_SHEET_LAYOUT: tuple[tuple[str, tuple[str, ...], str | None], ...] = ( + ("ASSETS", ("Non-current assets", "Current assets"), "TOTAL ASSETS"), + ( + "EQUITY AND LIABILITIES", + ("Equity", "Non-current liabilities", "Current liabilities"), + "TOTAL EQUITY AND LIABILITIES", + ), +) +SUBTOTAL_CAPTION = { + "Non-current assets": "Total non-current assets", + "Current assets": "Total current assets", + "Equity": "Total equity", + "Non-current liabilities": "Total non-current liabilities", + "Current liabilities": "Total current liabilities", + "Income": "Total income", + "Expenses": "Total expenses", +} +# Groups that get a printed sub-header row above them (Equity does not, in the +# signed set, and the roll-forward keeps the signed layout). +SUBHEADED = { + "Non-current assets", + "Current assets", + "Non-current liabilities", + "Current liabilities", +} + +_ALIASES = (("wip", "work-in-progress"),) + + +def normalise_caption(text: str) -> str: + lowered = " ".join(text.lower().split()) + for suffix in (" - non-current", " - current", " (net)"): + if lowered.endswith(suffix): + lowered = lowered[: -len(suffix)] + for short, long in _ALIASES: + lowered = lowered.replace(f" and {short}", f" and {long}") + return lowered.strip() + + +@dataclass +class Line: + """One printed row. `kind` decides how it is typeset, not what it means.""" + + caption: str + kind: str # section | subheader | account | subtotal | total | derivedline + current: AnyFigure | None = None + prior: AnyFigure | None = None + account: Account | None = None + prior_note_ref: int | None = None + note_ref: int | None = None # filled in once the note tree exists + emphasis: bool = False + + @property + def figures(self) -> tuple[AnyFigure, ...]: + return tuple(f for f in (self.current, self.prior) if f is not None) + + +@dataclass +class Statement: + key: str # "balance_sheet" | "profit_and_loss" + title: str # without the date, which the renderer appends + lines: list[Line] = field(default_factory=list) + column_labels: tuple[str, str] = ("", "") + + def line(self, caption: str) -> Line | None: + return next((line for line in self.lines if line.caption == caption), None) + + @property + def figures(self) -> tuple[AnyFigure, ...]: + return tuple(f for line in self.lines for f in line.figures) + + +@dataclass +class ComparativeCheck: + caption: str + ledger_value: Decimal + signed_value: Decimal | None + matched: bool + reason: str + + +@dataclass +class RollForward: + ledger: TrialBalance + prior: PriorYearSet + statements: list[Statement] + comparative_checks: list[ComparativeCheck] + ties: list[tuple[str, Decimal, Decimal, bool]] + + @property + def year(self) -> int: + return self.ledger.year + + @property + def prior_year(self) -> int: + return self.ledger.prior_year + + def statement(self, key: str) -> Statement: + return next(s for s in self.statements if s.key == key) + + def figure_for(self, code: str, year: int) -> AnyFigure: + line = next( + line for line in self._all_lines() if line.account and line.account.code == code + ) + return line.current if year == self.year else line.prior + + def caption_line(self, code: str) -> Line: + return next( + line for line in self._all_lines() if line.account and line.account.code == code + ) + + def total(self, caption: str) -> Derived: + for line in self._all_lines(): + if line.caption == caption and isinstance(line.current, Derived): + return line.current + raise KeyError(caption) + + def prior_total(self, caption: str) -> Derived: + for line in self._all_lines(): + if line.caption == caption and isinstance(line.prior, Derived): + return line.prior + raise KeyError(caption) + + def _all_lines(self) -> list[Line]: + return [line for statement in self.statements for line in statement.lines] + + +class _ComparativeSource: + """Locates a signed prior-year cell for a ledger account, once.""" + + def __init__(self, prior: PriorYearSet) -> None: + self._rows: dict[str, list] = {} + for statement in prior.statements: + for row in statement.rows: + if row.is_section or row.is_total or not row.values: + continue + self._rows.setdefault(normalise_caption(row.caption), []).append(row) + + def find(self, account: Account): + candidates = self._rows.get(normalise_caption(account.name), []) + if len(candidates) == 1: + return candidates[0] + for candidate in candidates: + if normalise_caption(candidate.section) == normalise_caption(account.group): + return candidate + return None + + +def roll_forward(ledger: TrialBalance, prior: PriorYearSet, corpus_root: Path) -> RollForward: + source = _ComparativeSource(prior) + checks: list[ComparativeCheck] = [] + + def figures_for(account: Account) -> tuple[Figure, Figure]: + current = Figure( + label=account.name, + value=account.presented(ledger.year), + basis="ledger", + cites=(account.cite(ledger.year),), + ) + prior_value = account.presented(ledger.prior_year) + row = source.find(account) + if row is None: + if prior_value != 0: + raise ComparativeMismatchError( + f"{account.code} ({account.name}) has a prior-year balance of " + f"{format_money(prior_value)} but no row in the signed prior-year set. " + f"A comparative that is not in the signed set cannot be presented as one; " + f"looked for caption {normalise_caption(account.name)!r} in " + f"{prior.rel}." + ) + checks.append( + ComparativeCheck( + account.name, + prior_value, + None, + True, + "new caption this year; nil in the ledger's prior-year column and absent " + "from the signed set - consistent", + ) + ) + return current, Figure( + label=f"{account.name} (comparative)", + value=prior_value, + basis="ledger", + cites=(account.cite(ledger.prior_year),), + ) + cell = row.values[0] + cell_cite = row.cites[0] + signed = parse_money(cell) + if signed != prior_value: + raise ComparativeMismatchError( + f"comparative mismatch on {account.name!r}: the trial balance's " + f"fy{ledger.prior_year} column presents {format_money(prior_value)} and the " + f"signed prior-year set prints {cell} at {prior.rel} bytes " + f"{cell_cite.byte_start}-{cell_cite.byte_end}. The comparatives must tie " + f"exactly; neither side is preferred and this is a hard error." + ) + checks.append( + ComparativeCheck(account.name, prior_value, signed, True, "ledger and signed set agree") + ) + return current, Figure( + label=f"{account.name} (comparative)", + value=prior_value, + basis="comparative", + cites=(account.cite(ledger.prior_year), cell_cite), + ) + + def note_ref_for(account: Account) -> int | None: + row = source.find(account) + return row.note_ref if row is not None else None + + balance_sheet = Statement("balance_sheet", "Balance Sheet as at") + for section, groups, section_total in BALANCE_SHEET_LAYOUT: + balance_sheet.lines.append(Line(section, "section", emphasis=True)) + section_components_current: list[Component] = [] + section_components_prior: list[Component] = [] + for group in groups: + accounts = ledger.in_group("BS", group) + if not accounts: + continue + if group in SUBHEADED: + balance_sheet.lines.append(Line(group, "subheader")) + group_current: list[Component] = [] + group_prior: list[Component] = [] + for account in accounts: + current, prior_figure = figures_for(account) + balance_sheet.lines.append( + Line( + caption=account.name, + kind="account", + current=current, + prior=prior_figure, + account=account, + prior_note_ref=note_ref_for(account), + ) + ) + group_current.append(Component(current)) + group_prior.append(Component(prior_figure)) + caption = SUBTOTAL_CAPTION[group] + balance_sheet.lines.append( + Line( + caption, + "subtotal", + Derived(caption, tuple(group_current)), + Derived(f"{caption} (comparative)", tuple(group_prior)), + emphasis=True, + ) + ) + section_components_current.extend(group_current) + section_components_prior.extend(group_prior) + if section_total: + balance_sheet.lines.append( + Line( + section_total, + "total", + Derived(section_total, tuple(section_components_current)), + Derived(f"{section_total} (comparative)", tuple(section_components_prior)), + emphasis=True, + ) + ) + + profit_and_loss = Statement( + "profit_and_loss", "Statement of Profit and Loss for the year ended" + ) + pl_totals: dict[str, tuple[Derived, Derived]] = {} + for group in ("Income", "Expenses", "Tax expense"): + accounts = ledger.in_group("PL", group) + if not accounts: + continue + current_components: list[Component] = [] + prior_components: list[Component] = [] + for account in accounts: + current, prior_figure = figures_for(account) + profit_and_loss.lines.append( + Line( + caption=account.name, + kind="account", + current=current, + prior=prior_figure, + account=account, + prior_note_ref=note_ref_for(account), + ) + ) + current_components.append(Component(current)) + prior_components.append(Component(prior_figure)) + if group in SUBTOTAL_CAPTION: + caption = SUBTOTAL_CAPTION[group] + current_total = Derived(caption, tuple(current_components)) + prior_total = Derived(f"{caption} (comparative)", tuple(prior_components)) + profit_and_loss.lines.append( + Line(caption, "subtotal", current_total, prior_total, emphasis=True) + ) + pl_totals[group] = (current_total, prior_total) + else: + pl_totals[group] = ( + Derived(f"Total {group.lower()}", tuple(current_components)), + Derived(f"Total {group.lower()} (comparative)", tuple(prior_components)), + ) + if group == "Expenses": + pbt_current = Derived( + "Profit before tax", + (Component(pl_totals["Income"][0]), Component(current_total, -1)), + ) + pbt_prior = Derived( + "Profit before tax (comparative)", + (Component(pl_totals["Income"][1]), Component(prior_total, -1)), + ) + profit_and_loss.lines.insert( + len(profit_and_loss.lines), + Line("Profit before tax", "derivedline", pbt_current, pbt_prior, emphasis=True), + ) + pl_totals["PBT"] = (pbt_current, pbt_prior) + tax_current, tax_prior = pl_totals["Tax expense"] + profit_and_loss.lines.append( + Line( + "Profit for the year", + "derivedline", + Derived( + "Profit for the year", + (Component(pl_totals["PBT"][0]), Component(tax_current, -1)), + ), + Derived( + "Profit for the year (comparative)", + (Component(pl_totals["PBT"][1]), Component(tax_prior, -1)), + ), + emphasis=True, + ) + ) + + rolled = RollForward(ledger, prior, [balance_sheet, profit_and_loss], checks, []) + _check_ties(rolled, prior) + return rolled + + +def _check_ties(rolled: RollForward, prior: PriorYearSet) -> None: + """The four ties in TASK.md section 2, plus every prior-year total the signed set prints.""" + ties = rolled.ties + for year_label, getter in ( + (rolled.year, lambda c: rolled.total(c)), + (rolled.prior_year, lambda c: rolled.prior_total(c)), + ): + assets = getter("TOTAL ASSETS").value + equity_and_liabilities = getter("TOTAL EQUITY AND LIABILITIES").value + ok = assets == equity_and_liabilities + ties.append( + ( + f"FY{year_label} total assets = total equity and liabilities", + assets, + equity_and_liabilities, + ok, + ) + ) + if not ok: + raise TieBreakError( + f"FY{year_label} balance sheet does not balance: total assets " + f"{format_money(assets)} against total equity and liabilities " + f"{format_money(equity_and_liabilities)}, a difference of " + f"{format_money(assets - equity_and_liabilities)}." + ) + + # Every total the signed set prints must equal the total our comparatives + # derive. This is the half of "the comparatives tie" that a caption-by-caption + # check cannot see: a set can agree row by row and still print a wrong total. + for statement in prior.statements: + for row in statement.rows: + if not row.values: + continue + try: + ours = rolled.prior_total(row.caption) + except KeyError: + continue + signed = parse_money(row.values[0]) + ok = ours.value == signed + ties.append( + (f"FY{rolled.prior_year} {row.caption} vs signed set", ours.value, signed, ok) + ) + if not ok: + raise ComparativeMismatchError( + f"comparative total mismatch on {row.caption!r}: our roll-forward derives " + f"{format_money(ours.value)} from {len(ours.components)} ledger row(s), the " + f"signed prior-year set prints {row.values[0]}. Working: {ours.workings()}" + ) + + other_equity = rolled.caption_line("2110") + profit = rolled.statement("profit_and_loss").line("Profit for the year").current + opening = other_equity.prior.value + closing = other_equity.current.value + ok = opening + profit.value == closing + ties.append((f"{other_equity.caption} rolls forward", opening + profit.value, closing, ok)) + if not ok: + raise TieBreakError( + f"{other_equity.caption} does not roll forward: {format_money(opening)} + " + f"{format_money(profit.value)} = {format_money(opening + profit.value)}, but the " + f"ledger closes at {format_money(closing)}. The difference " + f"{format_money(closing - opening - profit.value)} is undistributed and unexplained." + ) + + +def unused_cite(cite: Cite) -> Cite: # pragma: no cover - keeps the import honest + return cite diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/style.py b/use-cases/preetham1930/statutory-statements-builder/statutory/style.py new file mode 100644 index 000000000..f881df684 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/style.py @@ -0,0 +1,33 @@ +"""Table styling, as inline CSS, because that is what measurably survives. + +The signed prior-year set uses the legacy `border="1" cellpadding="4"` +attributes. Measured on the first live upload: the product drops them and the +table comes back unstyled. Inline `style` attributes are the ones that survive - +on the styled template fixture all 21 borders, both zebra rows, the header +shading and all 15 right-alignments came back identical, through an upload, a +targeted replace and a DOCX export. + +So the roll-forward re-styles the tables on the way through. That is a change to +the signed set's markup and it is deliberate: the deliverable is a typeset +document, and this is the styling the export preserves. +""" + +from __future__ import annotations + +RULE = "1px solid #1f3864" + +TABLE = f"border-collapse:collapse;width:100%;border:{RULE}" +HEAD_ROW = "background-color:#1f3864;color:#ffffff" +TH_TEXT = f"border:{RULE};padding:6px;text-align:left" +TH_NUM = f"border:{RULE};padding:6px;text-align:right" +TD_TEXT = f"border:{RULE};padding:6px" +TD_NUM = f"border:{RULE};padding:6px;text-align:right" +SECTION_ROW = "background-color:#eaeef7" + + +def table_open() -> str: + return f'<table style="{TABLE}">' + + +def head_cell(text: str, numeric: bool = False) -> str: + return f'<th style="{TH_NUM if numeric else TH_TEXT}">{text}</th>' diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/superdocs/__init__.py b/use-cases/preetham1930/statutory-statements-builder/statutory/superdocs/__init__.py new file mode 100644 index 000000000..5999721ea --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/superdocs/__init__.py @@ -0,0 +1,5 @@ +"""The SuperDocs wire: transport, client, decision ledger, verifier. + +Nothing above this package knows an HTTP path or a JSON key, and nothing in it +knows what a statutory note is. +""" diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/superdocs/client.py b/use-cases/preetham1930/statutory-statements-builder/statutory/superdocs/client.py new file mode 100644 index 000000000..ee0180b68 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/superdocs/client.py @@ -0,0 +1,290 @@ +"""The SuperDocs client, with the wire rules enforced in the client itself. + +Four measured decisions live here rather than in a caller's discipline: + +- **Decision 32** - gated changes go to `/v1/chat/async`. The gate takes a + `job_id` and jobs exist only on the async route; passing `approval_mode` to + the synchronous route leaves nothing to approve against. `chat_gated` is the + only method that can produce a pending change and it cannot reach `/v1/chat`. +- **Decision 35** - one change per job, on the sending side. The answering side + does not always agree: the first live run of this build asked for one chunk by + id and got a two-change batch back. So we approve exactly the change we + computed and deny every other one bare, and the read-back is what makes that + safe rather than the batch size. +- **Decision 34** - a denial carries no feedback. `deny()` takes no feedback + parameter, so there is no call shape in which one can be sent. Feedback + triggers a revision pass, and the revision pass proposed edits to two chunks + nobody targeted, aimed at the section headings. +- **Decision 33** - `approve` is an actuator, not a record. This client returns + what the API returned and claims nothing about what was decided; the decision + ledger is written by the caller, before the call. + +Nothing here believes a reply. `read_back()` is the only method whose result is +evidence, and it is the only one the verifier consults. +""" + +from __future__ import annotations + +import base64 +import time +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from ..chunks import ChunkMap +from ..errors import PlanRefusedError +from .transport import Transport + +TERMINAL = {"completed", "failed", "cancelled", "error"} + + +@dataclass +class PendingChange: + change_id: str + chunk_id: str + operation: str + old_html: str + new_html: str + + +@dataclass +class Job: + job_id: str + status: str + progress: int + pending: list[PendingChange] + raw: dict[str, Any] + + @property + def awaiting(self) -> bool: + return self.status == "awaiting_approval" + + @property + def finished(self) -> bool: + return self.status in TERMINAL + + +class SuperDocsClient: + def __init__(self, transport: Transport, *, poll_seconds: float = 3.0, max_polls: int = 80): + self.transport = transport + self.poll_seconds = poll_seconds + self.max_polls = max_polls + self.billable_calls = 0 + self.export_content_types: list[str] = [] + + # -- documents --------------------------------------------------------- + + def upload_verbatim(self, filename: str, html: str) -> tuple[str, str]: + """Upload our computed skeleton. Returns (document_id, session_id). + + Verbatim is the load-bearing property (Decision 29) and it is checked by + the caller against a read-back, never assumed from this response. + """ + # Three calls, and the third is the one that makes the other two + # checkable. An upload with no session is not persisted, and the upload + # response carries no document id at all - so the id comes from the + # session's own roster, which is a read rather than a claim. + session = self.transport.request("POST", "/v1/sessions/init", json_body={}) + session_id = session.get("session_id") + if not session_id: + raise RuntimeError(f"sessions/init returned no session_id; keys {sorted(session)}") + payload = { + "filename": filename, + "file_base64": base64.b64encode(html.encode("utf-8")).decode("ascii"), + "session_id": session_id, + } + # One retry, because a 503 with an empty body was observed on a 23 KB + # upload and succeeded immediately on the next attempt. A retried POST + # could leave two documents in the session, so the roster below is + # required to hold exactly one - the retry is checked, not trusted. + try: + uploaded = self.transport.request( + "POST", "/v1/documents/upload-base64", json_body=payload + ) + except RuntimeError as exc: + if "HTTP 5" not in str(exc): + raise + uploaded = self.transport.request( + "POST", "/v1/documents/upload-base64", json_body=payload + ) + if not uploaded.get("persisted"): + raise RuntimeError( + f"the upload reports persisted={uploaded.get('persisted')!r}. An unpersisted " + f"document cannot be read back, and a read-back is the only evidence there is." + ) + roster = self.transport.request("GET", f"/v1/sessions/{session_id}/documents") + documents = roster.get("documents") or [] + if len(documents) > 1: + raise RuntimeError( + f"the session holds {len(documents)} documents and this build works on one. " + f"If an upload was retried after a 5xx, both may have landed; nothing is edited " + f"until a human has looked." + ) + durable = next( + (d.get("durable_document_id") for d in documents if d.get("durable_document_id")), None + ) + if not durable: + raise RuntimeError( + f"the session roster carries no durable document id; {len(documents)} " + f"document(s), keys {sorted(documents[0]) if documents else '(none)'}. " + f"Without one there is nothing to verify against." + ) + return durable, session_id + + def read_back( + self, document_id: str, *, attempts: int = 4, sleep=time.sleep + ) -> tuple[ChunkMap, dict[str, Any]]: + """`GET /v1/documents/{id}?include_html=true` - the only evidence there is. + + Retried on a 404 because the durable record is not always queryable the + instant the upload returns: one run 404'd immediately and the same call + returned 200 on the next attempt. A read is safe to repeat; the retry is + bounded and a persistent 404 stops the run rather than being smoothed + over into "nothing to verify". + """ + body: dict[str, Any] = {} + for attempt in range(attempts): + try: + body = self.transport.request( + "GET", f"/v1/documents/{document_id}", params={"include_html": "true"} + ) + break + except RuntimeError as exc: + if "HTTP 404" not in str(exc) or attempt == attempts - 1: + raise + sleep(2.0 * (attempt + 1)) + html = body.get("html") + if not html: + raise RuntimeError( + f"read-back for {document_id} carried no html. A verification step that " + f"cannot see the document verifies nothing." + ) + return ChunkMap(html), body + + def export(self, session_id: str, fmt: str, out_dir: Path | None = None) -> Path | None: + """The export answers with a file, not with JSON, so it gets its own path. + + Written to disk when `out_dir` is given, because an export nobody can + open is not evidence of anything. + """ + content_type, blob = self.transport.download( + "POST", "/v1/documents/export", json_body={"session_id": session_id, "format": fmt} + ) + self.export_content_types.append(content_type) + if out_dir is None or not blob: + return None + out_dir.mkdir(parents=True, exist_ok=True) + path = out_dir / f"statements.{fmt}" + path.write_bytes(blob) + return path + + def revert(self, session_id: str, turn_index: int) -> dict[str, Any]: + return self.transport.request( + "POST", f"/v1/sessions/{session_id}/revert", json_body={"turn_index": turn_index} + ) + + # -- the gated edit loop ---------------------------------------------- + + def chat_gated(self, session_id: str, document_id: str, instruction: str) -> Job: + """One instruction, on the async route, held at the gate (Decisions 32 and 35). + + `document_id` is not sent: the session holds exactly one document and it + is focused, and the durable id we verify against is not the id the chat + surface uses for it. Naming the wrong one is worse than naming none. + """ + del document_id + body = self.transport.request( + "POST", + "/v1/chat/async", + json_body={ + "session_id": session_id, + "message": instruction, + "approval_mode": "ask_every_time", + "response_mode": "compact", + }, + ) + self.billable_calls += 1 + job_id = body.get("job_id") + if not job_id: + raise RuntimeError(f"chat/async returned no job_id; keys {sorted(body)}") + return self.poll(job_id) + + def poll(self, job_id: str, *, until_terminal: bool = False, sleep=time.sleep) -> Job: + for _ in range(self.max_polls): + body = self.transport.request("GET", f"/v1/jobs/{job_id}") + job = _job_from(body) + if job.finished or (job.awaiting and not until_terminal): + return job + sleep(self.poll_seconds) + raise TimeoutError( + f"job {job_id} did not settle in {self.max_polls} polls. Nothing is reported as " + f"applied on a timeout; the document is read back and the run halts." + ) + + def approve(self, session_id: str, job_id: str, change_id: str) -> dict[str, Any]: + return self.transport.request( + "POST", + f"/v1/chat/{session_id}/approve", + json_body={"job_id": job_id, "change_id": change_id, "approved": True}, + ) + + def deny(self, session_id: str, job_id: str, change_id: str) -> dict[str, Any]: + """Bare. There is no feedback parameter, because feedback re-plans (Decision 34).""" + return self.transport.request( + "POST", + f"/v1/chat/{session_id}/approve", + json_body={"job_id": job_id, "change_id": change_id, "approved": False}, + ) + + +def select_our_change( + job: Job, expected_chunk_id: str +) -> tuple[PendingChange, list[PendingChange]]: + """Split a batch into the one change we asked for and everything else. + + We send one chunk per job (Decision 35) and the product does not always + answer with one. Measured on the first live run of this build: an + instruction naming a single chunk id came back as a **two**-change batch, + the second on a chunk nobody targeted - the same shape as the unprompted + heading edits recorded on 2026-08-09. + + So the rule is not "the batch must be one"; it is **we approve exactly the + change we computed and deny every other one, bare** (Decision 34). The + read-back afterwards is what makes that safe: if a denial did not hold, the + non-target chunk moved and the verifier calls it collateral damage. + """ + ours = next((c for c in job.pending if c.chunk_id == expected_chunk_id), None) + if ours is None: + raise PlanRefusedError( + f"job {job.job_id} proposes {len(job.pending)} change(s), none of them on chunk " + f"{expected_chunk_id}, which is the one the step targets. Chunks proposed: " + f"{[c.chunk_id for c in job.pending]}. Every one is denied; nothing is approved on " + f"the strength of a proposal we did not ask for." + ) + if ours.operation != "edit": + raise PlanRefusedError( + f"job {job.job_id} proposes operation {ours.operation!r} on our target. There is " + f"one verb and it is replace (Decision 28)." + ) + return ours, [c for c in job.pending if c is not ours] + + +def _job_from(body: dict[str, Any]) -> Job: + metadata = body.get("metadata") or {} + pending = [ + PendingChange( + change_id=c.get("change_id", ""), + chunk_id=c.get("chunk_id", ""), + operation=c.get("operation", ""), + old_html=c.get("old_html", "") or "", + new_html=c.get("new_html", "") or "", + ) + for c in (metadata.get("pending_changes") or []) + ] + return Job( + job_id=body.get("job_id", ""), + status=body.get("status", ""), + progress=int(body.get("progress") or 0), + pending=pending, + raw=body, + ) diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/superdocs/decisions.py b/use-cases/preetham1930/statutory-statements-builder/statutory/superdocs/decisions.py new file mode 100644 index 000000000..2ac98122d --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/superdocs/decisions.py @@ -0,0 +1,123 @@ +"""Our decision ledger. SuperDocs' `approve` is only an actuator (Decision 33). + +Measured on 2026-08-09: the product's gate is **write-only**. Before any +decision `pending_changes` listed three changes with `status: None`; after +approving one and denying another it listed the same three, still with +`status: None`. The approve call returns `{"status":"ok","batch_complete":...}` +and nothing else. There is no API answer to "what has been decided, and by whom". + +So the row is ours, it is written **before** the actuator is called, and it +carries a named actor. An item is undecided exactly when it has no row - there +is no default, no pending row and no status column. A second decision on the +same item is refused unless it explicitly supersedes, and the superseded row is +kept with both actors. + +The completion sentence is computed from the rows at the moment it is emitted, +never from counters, because we watched a product report "Successfully updated +all 0 sections. 2 change(s) were denied" when two had been applied and three +denied. +""" + +from __future__ import annotations + +import json +from dataclasses import asdict, dataclass +from datetime import UTC, datetime +from pathlib import Path + +from ..errors import DecisionAlreadyRecordedError + +VERBS = ("accept", "reject") + + +@dataclass(frozen=True) +class Decision: + run_id: str + step_index: int + chunk_id: str + verb: str + actor: str + reason: str + at: str + supersedes: str | None = None + + def __post_init__(self) -> None: + if self.verb not in VERBS: + raise ValueError( + f"the vocabulary is {VERBS} and nothing else. There is no 'resolve' and no " + f"'choose_side': a reviewer decides whether a change enters the document, not " + f"which of two disagreeing sources is right." + ) + if not self.actor.strip(): + raise ValueError("a decision with no actor answers nobody's question about who") + + +class DecisionLedger: + def __init__(self, path: Path | None = None) -> None: + self.path = path + self.rows: list[Decision] = [] + + def record( + self, + run_id: str, + step_index: int, + chunk_id: str, + verb: str, + actor: str, + reason: str, + supersede: bool = False, + ) -> Decision: + existing = self.find(run_id, step_index) + if existing is not None and not supersede: + raise DecisionAlreadyRecordedError( + f"step {step_index} of run {run_id} was already decided '{existing.verb}' by " + f"{existing.actor} at {existing.at}. Quiet replacement is the mechanism by " + f"which a decision gets quietly dropped; pass supersede=True and both rows are " + f"kept." + ) + row = Decision( + run_id=run_id, + step_index=step_index, + chunk_id=chunk_id, + verb=verb, + actor=actor, + reason=reason, + at=datetime.now(UTC).isoformat(timespec="seconds"), + supersedes=existing.at if existing is not None else None, + ) + self.rows.append(row) + if self.path is not None: + self.path.parent.mkdir(parents=True, exist_ok=True) + with self.path.open("a", encoding="utf-8") as handle: + handle.write(json.dumps(asdict(row)) + "\n") + return row + + def find(self, run_id: str, step_index: int) -> Decision | None: + matches = [r for r in self.rows if r.run_id == run_id and r.step_index == step_index] + return matches[-1] if matches else None + + def decided(self, run_id: str) -> list[Decision]: + latest: dict[int, Decision] = {} + for row in self.rows: + if row.run_id == run_id: + latest[row.step_index] = row + return [latest[k] for k in sorted(latest)] + + # No count fields anywhere: these are properties of the rows being held, so + # the sentence cannot assert a decision that was never made (Decision 22's + # trick, rebuilt). + def accepted(self, run_id: str) -> int: + return sum(1 for r in self.decided(run_id) if r.verb == "accept") + + def rejected(self, run_id: str) -> int: + return sum(1 for r in self.decided(run_id) if r.verb == "reject") + + def describe(self, run_id: str, planned: int) -> str: + accepted = self.accepted(run_id) + rejected = self.rejected(run_id) + undecided = planned - accepted - rejected + actors = sorted({r.actor for r in self.decided(run_id)}) + return ( + f"{accepted} accepted, {rejected} rejected, {undecided} undecided of {planned} " + f"planned change(s); decided by {', '.join(actors) or 'nobody'}" + ) diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/superdocs/transport.py b/use-cases/preetham1930/statutory-statements-builder/statutory/superdocs/transport.py new file mode 100644 index 000000000..6fe4d5d9d --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/superdocs/transport.py @@ -0,0 +1,172 @@ +"""How a request reaches SuperDocs - or, in the test suite, how it does not. + +Two implementations behind one protocol: + +- `HttpTransport` talks to `api.superdocs.app`. It is used by the demo and by + nothing else. The key is read from the environment, never printed, never + written to a file, and never included in a message we log. +- `ReplayTransport` answers from recorded responses under `docs/evidence/`. It + is what the whole test suite runs on, so `make test` passes on a clean + checkout with no `.env` and with the network blocked (hard rule 3). + +A replay transport that is asked for a request it has no recording of **raises**, +naming the request. A transport that invents a plausible response would be the +2026-08-07 failure mode with a different noun: a success message for work that +did not happen. +""" + +from __future__ import annotations + +import json +import os +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Protocol + +from ..errors import NotConfiguredError + +DEFAULT_BASE_URL = "https://api.superdocs.app" + + +class Transport(Protocol): + def request( + self, + method: str, + path: str, + *, + json_body: dict[str, Any] | None = None, + params: dict[str, Any] | None = None, + ) -> dict[str, Any]: ... + + def download( + self, method: str, path: str, *, json_body: dict[str, Any] | None = None + ) -> tuple[str, bytes]: + """For the one endpoint that answers with a file rather than with JSON.""" + ... + + +def redacted_key() -> str: + key = os.environ.get("SUPERDOCS_API_KEY", "") + if not key: + return "(not set)" + return f"...{key[-4:]}" + + +class HttpTransport: + def __init__(self, base_url: str | None = None, timeout: float = 180.0) -> None: + self.base_url = ( + base_url or os.environ.get("SUPERDOCS_BASE_URL") or DEFAULT_BASE_URL + ).rstrip("/") + self.timeout = timeout + self.calls = 0 + key = os.environ.get("SUPERDOCS_API_KEY", "") + if not key: + raise NotConfiguredError( + "SUPERDOCS_API_KEY is not set. This build reads it from .env only, never from " + "a command line and never from a file it writes. The test suite does not need " + "it: tests run on recorded responses under docs/evidence/." + ) + self._key = key + + def request( + self, + method: str, + path: str, + *, + json_body: dict[str, Any] | None = None, + params: dict[str, Any] | None = None, + ) -> dict[str, Any]: + import httpx # imported here so the package imports with no network stack + + self.calls += 1 + response = httpx.request( + method, + f"{self.base_url}{path}", + json=json_body, + params=params, + headers={"Authorization": f"Bearer {self._key}", "Content-Type": "application/json"}, + timeout=self.timeout, + ) + if response.status_code >= 400: + # The body can echo the request; never let it carry the key onward. + raise RuntimeError( + f"{method} {path} -> HTTP {response.status_code}. " + f"{response.text[:400].replace(self._key, '<redacted>')}" + ) + return response.json() + + def download( + self, method: str, path: str, *, json_body: dict[str, Any] | None = None + ) -> tuple[str, bytes]: + """The export endpoint answers with a file, so it does not go through `request`.""" + import httpx + + self.calls += 1 + response = httpx.request( + method, + f"{self.base_url}{path}", + json=json_body, + headers={"Authorization": f"Bearer {self._key}", "Content-Type": "application/json"}, + timeout=self.timeout, + ) + if response.status_code >= 400: + raise RuntimeError( + f"{method} {path} -> HTTP {response.status_code}. " + f"{response.text[:400].replace(self._key, '<redacted>')}" + ) + return response.headers.get("content-type", ""), response.content + + +@dataclass +class RecordedCall: + method: str + path: str + response: dict[str, Any] + match: dict[str, Any] = field(default_factory=dict) + + +class ReplayTransport: + """Answers from recordings, in order, and refuses anything it was not given.""" + + def __init__(self, calls: list[RecordedCall]) -> None: + self._calls = list(calls) + self.seen: list[tuple[str, str, dict[str, Any] | None]] = [] + + def request( + self, + method: str, + path: str, + *, + json_body: dict[str, Any] | None = None, + params: dict[str, Any] | None = None, + ) -> dict[str, Any]: + self.seen.append((method, path, json_body)) + for index, call in enumerate(self._calls): + if call.method != method or call.path != path: + continue + if call.match and not all((json_body or {}).get(k) == v for k, v in call.match.items()): + continue + self._calls.pop(index) + return call.response + raise AssertionError( + f"no recorded response for {method} {path} with body keys " + f"{sorted(json_body or {})}. A replay transport never invents one: an invented " + f"response is a success message for work that did not happen. " + f"{len(self._calls)} recording(s) left: " + f"{[(c.method, c.path) for c in self._calls]}" + ) + + def download( + self, method: str, path: str, *, json_body: dict[str, Any] | None = None + ) -> tuple[str, bytes]: + """A replayed export records that we asked. There is no file to replay.""" + self.request(method, path, json_body=json_body) + return "application/octet-stream", b"" + + @property + def exhausted(self) -> bool: + return not self._calls + + +def load_recording(path: Path) -> dict[str, Any]: + return json.loads(path.read_text(encoding="utf-8")) diff --git a/use-cases/preetham1930/statutory-statements-builder/statutory/superdocs/verifier.py b/use-cases/preetham1930/statutory-statements-builder/statutory/superdocs/verifier.py new file mode 100644 index 000000000..6edce2bde --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/statutory/superdocs/verifier.py @@ -0,0 +1,180 @@ +"""Read the whole document back and decide what actually happened. + +Three classes, and the third is the dangerous one: + +- **NOT APPLIED** - the target chunk came back unchanged. This is the + 2026-08-07 corrupted-document mechanism: the change silently does not land and + the caller executes its consequences anyway. +- **APPLIED WRONG** - the target changed, but not to the post-state we computed + before sending. +- **COLLATERAL DAMAGE** - a chunk we did not target moved, appeared or vanished. + Measured on 2026-08-09: one requested change *succeeded* while four fabricated + sections appeared alongside it, and again when a template instantiation kept + the table and silently deleted the letterhead. A verifier that checked only its + own target would have reported success both times. + +So the comparison is over the **whole document**, every time, including on a +retry - failed content has been observed arriving a turn later. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field + +from ..chunks import ChunkMap, canonical + + +@dataclass +class Problem: + kind: str # not_applied | applied_wrong | collateral_damage + detail: str + diff: str = "" + + +@dataclass +class Verification: + step_index: int + chunk_id: str + problems: list[Problem] = field(default_factory=list) + receipt: str = "" + exact_bytes: bool = False + + @property + def ok(self) -> bool: + return not self.problems + + def report(self) -> str: + if self.ok: + return f"step {self.step_index}: ok - {self.receipt}" + lines = [f"step {self.step_index}: FAILED on chunk {self.chunk_id}"] + for problem in self.problems: + lines.append(f" {problem.kind.upper().replace('_', ' ')}: {problem.detail}") + if problem.diff: + lines.append(problem.diff) + return "\n".join(lines) + + +def verify( + step_index: int, + chunk_id: str, + before: ChunkMap, + after: ChunkMap, + expected_html: str, +) -> Verification: + result = Verification(step_index=step_index, chunk_id=chunk_id) + expected = canonical(expected_html) + + before_ids = before.ids + after_ids = after.ids + added = [i for i in after_ids if i not in set(before_ids)] + removed = [i for i in before_ids if i not in set(after_ids)] + if added or removed: + result.problems.append( + Problem( + "collateral_damage", + f"the chunk set changed: {len(before_ids)} chunk(s) before, {len(after_ids)} " + f"after; {len(added)} added, {len(removed)} removed. A replacement never " + f"changes the chunk set.", + diff="\n".join( + f" + {after.by_id[i].tag}: {after.by_id[i].html[:120]}" for i in added[:6] + ) + + "\n".join(f" - {i}" for i in removed[:6]), + ) + ) + moved = [ + i + for i in before_ids + if i + and i != chunk_id + and i in after.by_id + and after.by_id[i].canonical != before.by_id[i].canonical + ] + if moved: + result.problems.append( + Problem( + "collateral_damage", + f"{len(moved)} chunk(s) we did not target changed: {moved[:6]}", + diff="\n".join( + f" was: {before.by_id[i].html[:150]}\n now: {after.by_id[i].html[:150]}" + for i in moved[:3] + ), + ) + ) + + if chunk_id not in after.by_id: + result.problems.append( + Problem("not_applied", f"the target chunk {chunk_id} is not in the read-back at all") + ) + return result + + was = before.by_id[chunk_id].canonical + now = after.by_id[chunk_id].canonical + if now == was: + result.problems.append( + Problem( + "not_applied", + "the target chunk came back byte-identical to what it held before. The chat " + "reply is not evidence and is not consulted here.", + diff=f" unchanged: {before.by_id[chunk_id].html[:200]}", + ) + ) + elif now != expected: + result.problems.append( + Problem( + "applied_wrong", + "the target chunk changed, but not to the post-state computed before sending", + diff=_diff(expected, now), + ) + ) + else: + result.exact_bytes = _strip_id(after.by_id[chunk_id].html) == _strip_id(expected_html) + result.receipt = ( + f"read back {len(after_ids)} chunk(s) from GET /v1/documents/" + f"{{id}}?include_html=true; target chunk {chunk_id} matches the computed " + f"post-state{' byte for byte' if result.exact_bytes else ' (canonical form)'}; " + f"{len(after_ids) - 1} non-target chunk(s) unchanged" + ) + return result + + +def _strip_id(html: str) -> str: + from ..chunks import strip_chunk_ids + + return strip_chunk_ids(html) + + +def _diff(expected: str, actual: str) -> str: + import difflib + + lines = list( + difflib.unified_diff( + expected.split(">"), actual.split(">"), "expected", "actual", lineterm="", n=1 + ) + ) + return "\n".join(f" {line}" for line in lines[:24]) + + +def verify_upload_verbatim(uploaded: ChunkMap, sent_blocks: list[str]) -> str: + """The upload is only useful if it is verbatim, so that is checked, not assumed. + + Decision 29 rests entirely on `upload-base64` being a verbatim load. It was + measured once; it is re-checked on every run, because a design resting on a + measured property should re-measure it rather than remember it. + """ + ours = [canonical(block) for block in sent_blocks] + theirs = [chunk.canonical for chunk in uploaded.chunks] + if ours != theirs: + mismatch = next( + (i for i, (a, b) in enumerate(zip(ours, theirs, strict=False)) if a != b), + min(len(ours), len(theirs)), + ) + raise AssertionError( + f"the upload was not verbatim: we sent {len(ours)} block(s) and {len(theirs)} came " + f"back, first difference at block {mismatch}.\n" + f" sent: {ours[mismatch] if mismatch < len(ours) else '(nothing)'}\n" + f" got : {theirs[mismatch] if mismatch < len(theirs) else '(nothing)'}\n" + f"Decision 29 rests on the upload being verbatim; if it is not, the whole " + f"skeleton approach is unsound and the run stops here rather than editing a " + f"document that is not the one we computed." + ) + return f"upload verified verbatim: {len(theirs)} chunk(s), every block canonically identical" diff --git a/use-cases/preetham1930/statutory-statements-builder/tests/recorded.py b/use-cases/preetham1930/statutory-statements-builder/tests/recorded.py new file mode 100644 index 000000000..f7d83b997 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/tests/recorded.py @@ -0,0 +1,38 @@ +"""Recorded SuperDocs responses, replayed. No key, no network, no invention. + +Everything here comes out of `docs/evidence/`, which was captured on 2026-08-09 +against the live API and saved with `redemption_id` stripped. The failure-class +tests are driven by those recordings rather than by hand-written HTML, because +the point is that these three failures are things the product actually did. +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +REPO_ROOT = Path(__file__).resolve().parent.parent.parent.parent +REVERIFY = REPO_ROOT / "docs" / "evidence" / "2026-08-09-superdocs-reverify" +GATE = REPO_ROOT / "docs" / "evidence" / "2026-08-09-gate-and-templates" + + +def load(path: Path) -> dict[str, Any]: + return json.loads(path.read_text(encoding="utf-8")) + + +def read_back_html(name: str) -> str: + """`html` from a recorded GET /v1/documents/{id}?include_html=true.""" + return load(REVERIFY / f"{name}.json")["html"] + + +def gate_job(name: str) -> dict[str, Any]: + return load(GATE / f"gate-job-{name}.json") + + +def targeted_edit_response() -> dict[str, Any]: + return load(REVERIFY / "A-targeted-edit.response.json") + + +def append_response() -> dict[str, Any]: + return load(REVERIFY / "C-append-structural.response.json") diff --git a/use-cases/preetham1930/statutory-statements-builder/tests/simulator.py b/use-cases/preetham1930/statutory-statements-builder/tests/simulator.py new file mode 100644 index 000000000..9cb3c440f --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/tests/simulator.py @@ -0,0 +1,213 @@ +"""A SuperDocs stand-in whose behaviours are the ones we measured. + +This is not a mock that returns what it was told to return. It holds a real +document, parses it with our own chunk parser, and reproduces four behaviours +recorded on 2026-08-09: + +- an upload is verbatim, and comes back with a `data-chunk-id` per top-level + element (measured: five chunks for a styled HTML upload, table atomic); +- `chat/async` with `approval_mode: ask_every_time` holds the change at + `awaiting_approval` and applies **nothing** until the batch settles; +- an approved change lands only when the batch completes; +- the failure modes are optional and each is the shape of a recorded one: + `not_applied` (the mid-document insertion that vanished), `applied_wrong`, + `collateral` (the create that appended sections nobody asked for). + +The assertions in the tests are about **our** behaviour - does the queue halt, +is the ledger row written before the actuator, is an export produced - not about +the simulator's. The simulator only has to be a faithful enough environment for +those questions to be meaningful. +""" + +from __future__ import annotations + +import re +import uuid +from dataclasses import dataclass, field +from typing import Any + +from statutory.chunks import ChunkMap + +CHUNK_TAG = re.compile(r"<([a-zA-Z0-9]+)(\s|>)") + + +@dataclass +class Behaviour: + """What the simulator does on a given **chat call** (1-based, not step). + + A retry is a second chat call for the same step, so `not_applied={2, 3}` + means "step 2 fails and so does its one narrow retry", while + `not_applied={2}` means "step 2 fails and the retry lands". + """ + + not_applied: set[int] = field(default_factory=set) + applied_wrong: set[int] = field(default_factory=set) + collateral: set[int] = field(default_factory=set) + fail_upload_verbatim: bool = False + revert_works: bool = True + + +class SimulatedSuperDocs: + def __init__(self, behaviour: Behaviour | None = None) -> None: + self.behaviour = behaviour or Behaviour() + self.html = "" + self.pristine = "" + self.document_id = "doc-" + uuid.uuid4().hex[:8] + self.session_id = "session_init_" + uuid.uuid4().hex[:12] + self.jobs: dict[str, dict[str, Any]] = {} + self.step_index = 0 + self.exports: list[str] = [] + self.reverts = 0 + self.approve_calls: list[str] = [] + self.event_log: list[str] = [] + + # -- transport --------------------------------------------------------- + + def request( + self, + method: str, + path: str, + *, + json_body: dict[str, Any] | None = None, + params: dict[str, Any] | None = None, + ) -> dict[str, Any]: + body = json_body or {} + if method == "POST" and path == "/v1/sessions/init": + return {"session_id": self.session_id, "opened": [], "focused_document_id": None} + if method == "POST" and path == "/v1/documents/upload-base64": + import base64 + + raw = base64.b64decode(body["file_base64"]).decode("utf-8") + if self.behaviour.fail_upload_verbatim: + raw = raw.replace("</p>", " (edited on ingest)</p>", 1) + self.html = _assign_chunk_ids(raw) + self.pristine = self.html + return { + "session_id": self.session_id, + "filename": body["filename"], + "chunks_count": len(ChunkMap(self.html)), + "persisted": True, + } + if method == "GET" and path == f"/v1/sessions/{self.session_id}/documents": + return { + "session_id": self.session_id, + "focused_document_id": "doc_primary", + "documents": [ + { + "document_id": "doc_primary", + "durable_document_id": self.document_id, + "chunks_count": len(ChunkMap(self.html)), + "focused": True, + } + ], + } + if method == "GET" and path == f"/v1/documents/{self.document_id}": + return {"document_id": self.document_id, "html": self.html, "version": 1} + if method == "POST" and path == "/v1/chat/async": + return {"job_id": self._plan_job(body["message"])} + if method == "GET" and path.startswith("/v1/jobs/"): + return self.jobs[path.rsplit("/", 1)[1]] + if path.endswith("/approve"): + self.approve_calls.append(body["change_id"]) + self.event_log.append(f"approve:{body['change_id']}") + self._settle(body["job_id"], body["approved"]) + return {"status": "ok", "batch_complete": True} + if path == "/v1/documents/export": + self.exports.append(body.get("format", "docx")) + return {"download_url": "https://example.invalid/x"} + if path.endswith("/revert"): + self.reverts += 1 + if self.behaviour.revert_works: + self.html = self.last_good + else: + # The call returns ok and the document is not restored. This is + # the shape the whole build is built against: a 200 means the + # call was accepted, not that the state is what it claims. + self.html += '\n<p data-chunk-id="rv-1">not actually reverted</p>' + return {"status": "ok"} + raise AssertionError(f"the simulator was asked for {method} {path}") + + def download( + self, method: str, path: str, *, json_body: dict[str, Any] | None = None + ) -> tuple[str, bytes]: + self.request(method, path, json_body=json_body) + return ( + "application/vnd.openxmlformats-officedocument.wordprocessingml.document", + b"PKsimulated-docx", + ) + + # -- behaviour --------------------------------------------------------- + + def _plan_job(self, message: str) -> str: + self.step_index += 1 + chunk_id = re.search(r"data-chunk-id is ([0-9a-f-]+)", message).group(1) + new_html = message.split("\n\n", 1)[1] + job_id = "job-" + uuid.uuid4().hex[:8] + self.jobs[job_id] = { + "job_id": job_id, + "status": "awaiting_approval", + "progress": 88, + "metadata": { + "pending_changes": [ + { + "change_id": "chg-" + uuid.uuid4().hex[:8], + "chunk_id": chunk_id, + "operation": "edit", + "old_html": ChunkMap(self.html).by_id[chunk_id].html, + "new_html": new_html, + } + ] + }, + "_chunk_id": chunk_id, + "_new_html": new_html, + "_step": self.step_index, + } + return job_id + + def _settle(self, job_id: str, approved: bool) -> None: + job = self.jobs[job_id] + job["status"] = "completed" + job["progress"] = 100 + job["metadata"]["pending_changes"] = [] + if not approved: + return + step = job["_step"] + chunk_id, new_html = job["_chunk_id"], job["_new_html"] + if step in self.behaviour.not_applied: + self.event_log.append(f"step{step}:silently-did-nothing") + return + if step in self.behaviour.applied_wrong: + new_html = new_html.replace("</p>", " (paraphrased)</p>") + self._replace(chunk_id, new_html) + if step in self.behaviour.collateral: + self.event_log.append(f"step{step}:appended-sections-nobody-asked-for") + self.html += ( + '\n<h3 data-chunk-id="fab-1">Note 99 — Quality Assurance</h3>' + '\n<p data-chunk-id="fab-2">The Company maintains a quality system.</p>' + ) + + def _replace(self, chunk_id: str, new_html: str) -> None: + current = ChunkMap(self.html) + old = current.by_id[chunk_id].html + replacement = _with_chunk_id(new_html, chunk_id) + self.html = self.html.replace(old, replacement, 1) + + @property + def last_good(self) -> str: + return self.pristine + + +def _assign_chunk_ids(html: str) -> str: + out: list[str] = [] + for line in html.split("\n"): + match = CHUNK_TAG.match(line) + if match: + out.append(_with_chunk_id(line, "c-" + uuid.uuid4().hex[:12])) + else: + out.append(line) + return "\n".join(out) + + +def _with_chunk_id(fragment: str, chunk_id: str) -> str: + fragment = re.sub(r'\sdata-chunk-id="[^"]*"', "", fragment, count=1) + return re.sub(r"<([a-zA-Z0-9]+)", rf'<\1 data-chunk-id="{chunk_id}"', fragment, count=1) diff --git a/use-cases/preetham1930/statutory-statements-builder/tests/test_bodies.py b/use-cases/preetham1930/statutory-statements-builder/tests/test_bodies.py new file mode 100644 index 000000000..3ed2f406a --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/tests/test_bodies.py @@ -0,0 +1,144 @@ +"""What a regenerated note may say, and what it must refuse to say.""" + +from __future__ import annotations + +import re + +from statutory.money import money_tokens, money_tokens_in_html + +PRIOR_ONLY_FIGURES = { + "312.00", # inventories: raw materials + "196.00", # inventories: finished goods + "141.00", # cash: balances with banks + "542.00", # employee benefits: salaries and wages + "4,196.00", # revenue: sale of manufactured goods + "68.00", # related party: KMP remuneration + "34.00", # contingent: disputed GST demand + "2,503.00", # cost of materials: purchases +} + + +def test_no_regenerated_body_reuses_a_prior_year_breakdown(rolled) -> None: + """Last year's analysis never appears under this year's heading.""" + html = rolled.document.html("target") + tokens = set(money_tokens_in_html(html)) + reused = sorted(PRIOR_ONLY_FIGURES & tokens) + assert not reused, f"prior-year breakdown figures reached the FY set: {reused}" + + +def test_the_analysis_we_cannot_reproduce_is_named_rather_than_dropped(rolled) -> None: + gaps = [g for note in rolled.document.notes for g in note.gaps] + assert len(gaps) >= 25 + text = " ".join(g.what for g in gaps) + for phrase in ("Raw materials", "Salaries and wages", "Remuneration to key management"): + assert phrase in text, f"the outstanding list should name {phrase!r}" + for gap in gaps: + assert any("caption granularity" in place for place in gap.looked_in) + + +def test_a_carried_policy_sentence_has_no_figure_and_no_period_reference(rolled) -> None: + policies = [block for block in rolled.document.blocks if block.role.endswith("policy")] + assert policies, "some policy prose should survive the roll-forward" + for block in policies: + assert not money_tokens_in_html(block.target) + lowered = block.target.lower() + for word in ("during the year", "for the year", "as at"): + assert word not in lowered, f"{block.role} carries a period claim: {block.target}" + + +def test_the_share_capital_claim_from_last_year_is_not_carried(rolled) -> None: + """ "There was no movement in share capital during the year" is a claim about 2025.""" + assert "no movement in share capital during the year" not in rolled.document.html("target") + + +def test_every_note_says_what_the_statement_says(rolled) -> None: + assert len(rolled.caption_ties) == 29 + for tie in rolled.caption_ties: + assert re.match(r".+ -> Note \d+: [\d,()\.]+$", tie) + + +def test_the_prior_year_tie_out_is_narrow_and_reports_its_own_selectivity(rolled) -> None: + """Two groups qualified out of 25 notes; one tied, one raised.""" + assert rolled.document.coverage == {"groups": 2, "tied": 1, "raised": 1} + tieouts = [b for b in rolled.document.blocks if b.role.endswith("comparative_tieout")] + assert len(tieouts) == 1 + body = tieouts[0].target + assert "821.00" in body and "812.00" in body and "9.00" in body + assert "is not resolved" in body + + +def test_the_lease_note_states_the_deed_terms_and_the_ledger_balances(rolled) -> None: + note = rolled.note(4) + text = " ".join(e.target for e in note.elements) + for token in ("268.00", "212.00", "62.00", "274.00", "6,50,000", "39,00,000"): + assert token in text, token + sources = {cite.path for figure in note.figures for cite in figure.cites} + assert "sources/lease-agreement-warehouse-2025.md" in sources + assert "sources/trial-balance-FY2026.csv" in sources + + +def test_the_business_combination_note_quotes_both_sides_and_resolves_neither( + rolled, +) -> None: + note = rolled.note(27) + text = " ".join(e.target for e in note.elements) + assert "185.00" in text + assert "29 January 2026" in text + assert "all 29 account names were searched and 0 matched" in text + assert "neither is preferred" in text + assert len(note.disagreements) == 1 + sides = note.disagreements[0].sides + assert len(sides) == 2 + assert {s.source for s in sides} == { + "sources/board-minutes-2026-01-18.md", + "sources/trial-balance-FY2026.csv", + } + assert max(s.searched for s in sides) == 29 + + +def test_earnings_per_share_is_computed_not_carried(rolled) -> None: + note = rolled.note(26) + text = " ".join(e.target for e in note.elements) + assert "8.14" in text, "407.00 lakhs over 50,00,000 shares" + assert "6.18" in text, "the comparative, cited to the signed set" + assert "50,00,000" in text + + +def test_the_injected_instruction_has_no_channel_to_act_through(rolled) -> None: + """Defect 4: the quarantined source is reported and then not consulted. + + The stronger half of the defence is structural: nothing in this build reads + a satisfied-flag out of a document. A checklist item is satisfied because a + note exists and carries the analysis, and there is no code path from source + text to that decision. + """ + assert rolled.quarantined == ["sources/vendor-correspondence-2026-02.md"] + html = rolled.document.html("target") + assert "SATISFIED" not in html + assert "APPROVED" not in html + assert "vendor-correspondence" not in html + cited = {cite.path for cite in rolled.document.cites} + assert "sources/vendor-correspondence-2026-02.md" not in cited + # and the checklist did not become all-satisfied, which is what it asked for + assert sum(1 for line in rolled.checklist if line.status == "satisfied") < 5 + + +def test_a_figure_only_reaches_the_document_through_one_of_four_bases(rolled) -> None: + bases = {f.basis for f in rolled.document.figures} + assert bases <= {"ledger", "comparative", "document", "derived"} + assert bases == {"ledger", "comparative", "document", "derived"}, "all four are in use" + + +def test_every_money_token_in_the_final_document_is_accounted_for(rolled) -> None: + """The whole document, re-scanned, not just each body as it was built.""" + from collections import Counter + + declared = Counter(f.rendered for f in rolled.document.figures) + found = Counter(money_tokens(_visible_text(rolled.document))) + assert not (found - declared), f"unaccounted: {sorted((found - declared).elements())}" + + +def _visible_text(document) -> str: + from statutory.money import strip_markup + + return " ".join(strip_markup(block.target) for block in document.blocks) diff --git a/use-cases/preetham1930/statutory-statements-builder/tests/test_checklist.py b/use-cases/preetham1930/statutory-statements-builder/tests/test_checklist.py new file mode 100644 index 000000000..f461d4b12 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/tests/test_checklist.py @@ -0,0 +1,159 @@ +"""The rule set is data, the triggers are evaluated, and nothing fails quietly.""" + +from __future__ import annotations + +import re +import shutil +from pathlib import Path + +import pytest +from statutory.checklist import Checklist +from statutory.errors import UnknownAccountError, UnknownConditionError +from statutory.events import EventDetector +from statutory.ledger import TrialBalance +from statutory.pipeline import prepare + +PACKAGE = Path(__file__).resolve().parent.parent / "statutory" + + +def _bits(root: Path, config: Path): + ledger = TrialBalance(root / "sources" / "trial-balance-FY2026.csv", root) + events = EventDetector(config / "event-vocabulary.csv", config / "quarantine-markers.txt") + events.scan([p for p in sorted((root / "sources").glob("*")) if p.is_file()], root) + return ledger, events + + +def test_all_thirty_two_triggers_are_evaluated_from_the_csv(rolled) -> None: + assert len(rolled.checklist) == 32 + assert all(line.required for line in rolled.checklist) + assert all(line.because for line in rolled.checklist) + + +def test_the_eight_unmapped_items_are_detected_from_the_data(rolled) -> None: + unmapped = [line for line in rolled.checklist if line.prior_note is None] + assert [line.item_id for line in unmapped] == [f"DC-0{n}" for n in range(25, 33)] + lease = [line for line in unmapped if line.item_id in {f"DC-0{n}" for n in range(25, 29)}] + assert all("1110" in line.because or "2210" in line.because for line in lease) + combination = [line for line in unmapped if line.item_id in ("DC-029", "DC-031")] + assert all("board-minutes" in line.because for line in combination) + + +def test_the_package_never_names_an_item_id_or_a_standard() -> None: + """The checklist is found by its columns; no *code* knows what is in it. + + `config/` may name an item id and a standard - that is what configuration + is, and it is how a 33rd row is wired to a computation without a code edit. + The line this test draws is between data and code, not between the repo and + the world. + """ + for path in PACKAGE.rglob("*.py"): + text = path.read_text(encoding="utf-8-sig") + assert not re.search(r"DC-\d{3}", text), f"{path.name} names a checklist item" + assert "Ind AS" not in text, f"{path.name} names a standard" + configured = (PACKAGE.parent / "config" / "note-recipes.csv").read_text(encoding="utf-8") + assert re.search(r"DC-\d{3}", configured), "the wiring lives in config, and it is there" + + +def test_a_thirty_third_row_changes_behaviour_with_zero_code_edits( + corpus: Path, config: Path, tmp_path: Path +) -> None: + root = tmp_path / "corpus" + shutil.copytree(corpus, root) + checklist_path = root / "rules" / "disclosure-checklist-indas.csv" + with checklist_path.open("a", encoding="utf-8") as handle: + handle.write("DC-033,Ind AS 7,Cash flow statement,account 1220 balance > 0,7,high\n") + result = prepare(root, config) + line = next(line for line in result.checklist if line.item_id == "DC-033") + assert line.required + assert line.note == 8, "prior note 7 -> 8, through the same map as everything else" + assert "1220" in line.because and "189.00" in line.because + + +def test_an_unparseable_trigger_stops_the_run_and_prints_the_grammar( + corpus: Path, config: Path, tmp_path: Path +) -> None: + root = tmp_path / "corpus" + shutil.copytree(corpus, root) + path = root / "rules" / "disclosure-checklist-indas.csv" + with path.open("a", encoding="utf-8") as handle: + handle.write("DC-099,Ind AS 1,Something,when it feels right,2,high\n") + ledger, events = _bits(root, config) + with pytest.raises(UnknownConditionError) as caught: + Checklist(path, root).evaluate(ledger, events) + message = str(caught.value) + assert "DC-099" in message and "when it feels right" in message + assert "Grammar:" in message + assert "not 'not required'" in message + + +def test_a_trigger_naming_an_account_the_ledger_lacks_stops_the_run( + corpus: Path, config: Path, tmp_path: Path +) -> None: + root = tmp_path / "corpus" + shutil.copytree(corpus, root) + path = root / "rules" / "disclosure-checklist-indas.csv" + with path.open("a", encoding="utf-8") as handle: + handle.write("DC-098,Ind AS 1,Something,account 9999 balance > 0,2,high\n") + ledger, events = _bits(root, config) + with pytest.raises(UnknownAccountError) as caught: + Checklist(path, root).evaluate(ledger, events) + message = str(caught.value) + assert "9999" in message + assert "29 account(s)" in message + assert "not a nil balance" in message + + +def test_an_event_with_no_detector_stops_the_run( + corpus: Path, config: Path, tmp_path: Path +) -> None: + root = tmp_path / "corpus" + shutil.copytree(corpus, root) + path = root / "rules" / "disclosure-checklist-indas.csv" + with path.open("a", encoding="utf-8") as handle: + handle.write("DC-097,Ind AS 1,Something,a merger happened,2,high\n") + ledger, events = _bits(root, config) + with pytest.raises(UnknownConditionError) as caught: + Checklist(path, root).evaluate(ledger, events) + assert "Events defined:" in str(caught.value) + + +def test_an_or_condition_is_a_disjunction(corpus: Path, config: Path) -> None: + """The lease maturity item fires because either leg is non-nil.""" + ledger, events = _bits(corpus, config) + checklist = Checklist(corpus / "rules" / "disclosure-checklist-indas.csv", corpus) + trigger = next(t for t in checklist.evaluate(ledger, events) if t.item.item_id == "DC-026") + assert trigger.required + assert "2210" in trigger.because and "2310" in trigger.because + assert len(trigger.cites) == 2 + + +def test_the_signed_balance_is_what_the_grammar_evaluates(corpus: Path, config: Path) -> None: + """`balance > 0` keeps meaning 'debit balance'; no absolute value is taken.""" + ledger, _ = _bits(corpus, config) + assert ledger.account("1110").balance(2026) > 0 + assert ledger.account("2210").balance(2026) < 0 + assert ledger.account("2210").presented(2026) > 0 + + +def test_events_fire_on_a_block_not_on_a_document(corpus: Path, config: Path) -> None: + """The revenue note says 'control ... transfers to the customer'; that is not a + business combination, and requiring one block is what keeps them apart.""" + _, events = _bits(corpus, config) + for name in events.names(): + hits = events.hits(name) + assert len(hits) == 1 + assert hits[0].source.endswith("board-minutes-2026-01-18.md") + assert all(cite.path == hits[0].source for cite in hits[0].cites) + + +def test_a_quarantined_source_can_neither_suppress_nor_manufacture_an_event( + corpus: Path, config: Path +) -> None: + _, events = _bits(corpus, config) + assert [q.source for q in events.quarantined] == ["sources/vendor-correspondence-2026-02.md"] + assert "sources/vendor-correspondence-2026-02.md" not in events.scanned + quarantine = events.quarantined[0] + assert quarantine.cite.read_back(corpus) == quarantine.cite.snippet + # the forged source mentions "business combinations"; it contributed nothing + for name in events.names(): + assert all(h.source != quarantine.source for h in events.hits(name)) diff --git a/use-cases/preetham1930/statutory-statements-builder/tests/test_client.py b/use-cases/preetham1930/statutory-statements-builder/tests/test_client.py new file mode 100644 index 000000000..38a88824e --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/tests/test_client.py @@ -0,0 +1,156 @@ +"""The four wire rules, enforced in the client rather than in a caller's care.""" + +from __future__ import annotations + +import inspect + +import pytest +from recorded import gate_job +from statutory.errors import NotConfiguredError, PlanRefusedError +from statutory.superdocs.client import SuperDocsClient, select_our_change +from statutory.superdocs.transport import HttpTransport, RecordedCall, ReplayTransport + + +def _client(calls: list[RecordedCall]) -> tuple[SuperDocsClient, ReplayTransport]: + transport = ReplayTransport(calls) + return SuperDocsClient(transport, poll_seconds=0), transport + + +def test_gated_changes_go_to_the_async_route(monkeypatch) -> None: + """Decision 32: approve takes a job_id and jobs exist only on /v1/chat/async.""" + job = gate_job("awaiting-approval") + client, transport = _client( + [ + RecordedCall("POST", "/v1/chat/async", {"job_id": job["job_id"]}), + RecordedCall("GET", f"/v1/jobs/{job['job_id']}", job), + ] + ) + result = client.chat_gated("session_x", "doc_x", "replace chunk abc with ...") + assert result.awaiting + paths = [path for _, path, _ in transport.seen] + assert paths[0] == "/v1/chat/async" + assert "/v1/chat" not in paths, "the synchronous route has no gate to reach" + body = transport.seen[0][2] + assert body["approval_mode"] == "ask_every_time" + assert body["response_mode"] == "compact" + + +def test_a_batch_is_split_into_ours_and_everything_else() -> None: + """We send one chunk; the product does not always answer with one change.""" + from statutory.superdocs.client import _job_from + + job = _job_from(gate_job("awaiting-approval")) + assert len(job.pending) == 3 + ours, unasked = select_our_change(job, job.pending[1].chunk_id) + assert ours is job.pending[1] + assert len(unasked) == 2 + assert ours not in unasked + + +def test_a_batch_that_does_not_contain_our_target_is_refused_in_full() -> None: + from statutory.superdocs.client import Job, PendingChange + + job = Job("j", "awaiting_approval", 88, [PendingChange("c", "other-chunk", "edit", "", "")], {}) + with pytest.raises(PlanRefusedError) as caught: + select_our_change(job, "our-chunk") + assert "none of them on chunk our-chunk" in str(caught.value) + assert "Every one is denied" in str(caught.value) + + +def test_a_create_operation_on_our_target_is_refused() -> None: + from statutory.superdocs.client import Job, PendingChange + + job = Job("j", "awaiting_approval", 88, [PendingChange("c", "ours", "create", "", "")], {}) + with pytest.raises(PlanRefusedError, match="one verb"): + select_our_change(job, "ours") + + +def test_denials_carry_no_feedback() -> None: + """Decision 34: there is no call shape in which feedback can be sent.""" + signature = inspect.signature(SuperDocsClient.deny) + assert "feedback" not in signature.parameters + + client, transport = _client( + [RecordedCall("POST", "/v1/chat/s/approve", {"status": "ok", "batch_complete": True})] + ) + client.deny("s", "j", "c") + body = transport.seen[0][2] + assert body == {"job_id": "j", "change_id": "c", "approved": False} + assert "feedback" not in body + + +def test_a_read_back_with_no_html_verifies_nothing_and_says_so() -> None: + client, _ = _client([RecordedCall("GET", "/v1/documents/d", {"version": 2})]) + with pytest.raises(RuntimeError, match="carried no html"): + client.read_back("d") + + +def test_an_upload_that_is_not_persisted_stops_the_run() -> None: + """Measured: an upload with no session is not persisted and carries no id.""" + client, _ = _client( + [ + RecordedCall("POST", "/v1/sessions/init", {"session_id": "s"}), + RecordedCall( + "POST", + "/v1/documents/upload-base64", + {"filename": "f.html", "chunks_count": 3, "persisted": False}, + ), + ] + ) + with pytest.raises(RuntimeError, match="only evidence there is"): + client.upload_verbatim("f.html", "<p>x</p>") + + +def test_the_document_id_comes_from_the_session_roster_not_from_the_upload() -> None: + """The upload response has no document id at all; the roster is a read, not a claim.""" + client, transport = _client( + [ + RecordedCall("POST", "/v1/sessions/init", {"session_id": "s"}), + RecordedCall("POST", "/v1/documents/upload-base64", {"persisted": True}), + RecordedCall( + "GET", + "/v1/sessions/s/documents", + {"documents": [{"document_id": "doc_primary", "durable_document_id": "dur-1"}]}, + ), + ] + ) + document_id, session_id = client.upload_verbatim("f.html", "<p>x</p>") + assert (document_id, session_id) == ("dur-1", "s") + assert [path for _, path, _ in transport.seen][-1] == "/v1/sessions/s/documents" + + +def test_a_roster_with_no_durable_id_stops_the_run() -> None: + client, _ = _client( + [ + RecordedCall("POST", "/v1/sessions/init", {"session_id": "s"}), + RecordedCall("POST", "/v1/documents/upload-base64", {"persisted": True}), + RecordedCall("GET", "/v1/sessions/s/documents", {"documents": []}), + ] + ) + with pytest.raises(RuntimeError, match="nothing to verify against"): + client.upload_verbatim("f.html", "<p>x</p>") + + +def test_the_replay_transport_never_invents_a_response() -> None: + client, _ = _client([]) + with pytest.raises(AssertionError) as caught: + client.read_back("d") + assert "never invents one" in str(caught.value) + + +def test_a_live_transport_without_a_key_refuses_to_exist(monkeypatch) -> None: + monkeypatch.delenv("SUPERDOCS_API_KEY", raising=False) + with pytest.raises(NotConfiguredError) as caught: + HttpTransport() + assert "recorded responses" in str(caught.value) + assert "SUPERDOCS_API_KEY" in str(caught.value) + + +def test_the_key_is_never_in_a_message_this_module_can_produce(monkeypatch) -> None: + fake = "sk-" + "live-" + "0123456789abcdef" + monkeypatch.setenv("SUPERDOCS_API_KEY", fake) + transport = HttpTransport() + from statutory.superdocs.transport import redacted_key + + assert redacted_key() == "...cdef" + assert fake not in repr(transport.__dict__.get("base_url", "")) diff --git a/use-cases/preetham1930/statutory-statements-builder/tests/test_edit_plan.py b/use-cases/preetham1930/statutory-statements-builder/tests/test_edit_plan.py new file mode 100644 index 000000000..b795c91f6 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/tests/test_edit_plan.py @@ -0,0 +1,118 @@ +"""One verb, no heading, one chunk. Checked before anything reaches the wire.""" + +from __future__ import annotations + +import ast +import re +from pathlib import Path + +import pytest +from statutory.chunks import ChunkMap +from statutory.editplan import ( + Step, + assert_plan_is_legal, + bind_chunk_ids, + build_plan, + expected_after, +) +from statutory.errors import PlanRefusedError + +PACKAGE = Path(__file__).resolve().parent.parent / "statutory" + + +def test_every_step_is_a_replacement_of_an_existing_chunk(rolled) -> None: + plan = build_plan(rolled.document) + assert len(plan) == 33 + assert {step.verb for step in plan.steps} == {"replace"} + for step in plan.steps: + assert step.expected_html != step.was_html + + +def test_no_step_targets_a_heading_chunk(rolled) -> None: + plan = build_plan(rolled.document) + assert not [s for s in plan.steps if s.tag in ("h1", "h2", "h3")] + headings = [b for b in rolled.document.blocks if not b.editable] + assert len(headings) == 32, "27 note headings, 3 section headings, entity and address" + assert not [b for b in headings if b.changed], ( + "a heading that differs between skeleton and target would need an edit, and this " + "design has no step that can make one" + ) + + +def test_the_instruction_never_asks_for_a_structural_operation(rolled) -> None: + plan = build_plan(rolled.document) + uploaded = ChunkMap(rolled.document.html("skeleton")) + for index, chunk in enumerate(uploaded.chunks): + object.__setattr__(chunk, "chunk_id", f"chunk-{index}") + uploaded.by_id = {c.chunk_id: c for c in uploaded.chunks} + bind_chunk_ids(plan, uploaded) + for step in plan.steps: + instruction = step.instruction().lower() + assert "replace the entire contents" in instruction + assert "do not add any section" in instruction + assert "do not change any heading" in instruction + assert not re.search(r"\b(insert|renumber|create a|append a)\b", instruction) + + +def test_no_create_verb_anywhere_in_the_build(rolled) -> None: + """Decision 28: one verb. Checked against our own string literals.""" + banned = ("create_document", "insert_section", "add_section", '"create"', "'create'") + for path in PACKAGE.rglob("*.py"): + text = path.read_text(encoding="utf-8-sig") + tree = ast.parse(text) + literals = [ + node.value + for node in ast.walk(tree) + if isinstance(node, ast.Constant) and isinstance(node.value, str) + ] + joined = "\n".join(literals) + for word in banned: + assert word not in joined or "refused" in joined, f"{path.name} mentions {word}" + + +def test_a_no_op_step_is_refused_because_it_cannot_be_verified(rolled) -> None: + plan = build_plan(rolled.document) + plan.steps.append(Step(99, "probe", "p", None, "<p>same</p>", "<p>same</p>")) + with pytest.raises(PlanRefusedError, match="reads as 'not applied'"): + assert_plan_is_legal(plan, rolled.document) + + +def test_a_step_that_targets_a_heading_is_refused(rolled) -> None: + plan = build_plan(rolled.document) + plan.steps.append(Step(99, "note-heading", "h3", 4, "<h3>a</h3>", "<h3>b</h3>")) + with pytest.raises(PlanRefusedError, match="no renumbering step to get wrong"): + assert_plan_is_legal(plan, rolled.document) + + +def test_chunk_ids_are_bound_by_content_not_by_position(rolled) -> None: + plan = build_plan(rolled.document).sample(3) + uploaded = ChunkMap(rolled.document.html("skeleton")) + for index, chunk in enumerate(uploaded.chunks): + object.__setattr__(chunk, "chunk_id", f"chunk-{index}") + uploaded.by_id = {c.chunk_id: c for c in uploaded.chunks} + bind_chunk_ids(plan, uploaded) + assert all(step.chunk_id for step in plan.steps) + + plan.steps[0].was_html = "<p>never uploaded</p>" + with pytest.raises(PlanRefusedError, match="matches 0 uploaded chunk"): + bind_chunk_ids(plan, uploaded) + + +def test_the_expected_post_state_is_the_whole_document(rolled) -> None: + plan = build_plan(rolled.document).sample(1) + uploaded = ChunkMap(rolled.document.html("skeleton")) + for index, chunk in enumerate(uploaded.chunks): + object.__setattr__(chunk, "chunk_id", f"chunk-{index}") + uploaded.by_id = {c.chunk_id: c for c in uploaded.chunks} + bind_chunk_ids(plan, uploaded) + expected = expected_after(uploaded, plan.steps[0]) + assert len(expected) == len(uploaded) + changed = [k for k in expected if expected[k] != uploaded.by_id[k].canonical] + assert changed == [plan.steps[0].chunk_id], "exactly one chunk is expected to move" + + +def test_sample_mode_takes_a_prefix_and_nothing_else(rolled) -> None: + plan = build_plan(rolled.document) + sampled = plan.sample(4) + assert [s.index for s in sampled.steps] == [1, 2, 3, 4] + assert plan.sample(None) is plan diff --git a/use-cases/preetham1930/statutory-statements-builder/tests/test_figures.py b/use-cases/preetham1930/statutory-statements-builder/tests/test_figures.py new file mode 100644 index 000000000..227379965 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/tests/test_figures.py @@ -0,0 +1,163 @@ +"""No figure reaches paper without a source, and no type has room for an opinion.""" + +from __future__ import annotations + +import dataclasses +import re +from decimal import Decimal +from pathlib import Path + +import pytest +from statutory import bodies, disagreement, figures +from statutory.errors import CitationDriftError, UnaccountedFigureError +from statutory.figures import ( + SOURCE_BASES, + Body, + Cite, + Component, + Derived, + Figure, + Gap, + Quotient, + assert_accounted, +) + +CITE = Cite("sources/trial-balance-FY2026.csv", 200, 204, "0.00", "probe") +FORBIDDEN = ( + "resolution", + "resolved", + "resolve", + "conclusion", + "correct_value", + "preferred_side", + "preferred", + "status", + "approved", + "verdict", + "answer", + "winner", + "truth", +) + + +def _figure(value: str, basis: str = "ledger") -> Figure: + return Figure("probe", Decimal(value), basis, (CITE,)) + + +def test_an_unaccounted_token_is_a_hard_error_naming_the_token() -> None: + with pytest.raises(UnaccountedFigureError) as caught: + Body().markup("<p>a total of 1,234.00</p>").render("probe note") + message = str(caught.value) + assert "1,234.00" in message + assert "probe note" in message + assert "hard error rather than a warning" in message + + +def test_a_declared_figure_accounts_for_its_own_token() -> None: + body = Body().markup("<p>total ").fig(_figure("1234.00")).markup("</p>") + assert "1,234.00" in body.render("probe") + + +def test_two_identical_tokens_need_two_figures() -> None: + one = _figure("5.00") + with pytest.raises(UnaccountedFigureError): + assert_accounted("<p>5.00 and 5.00</p>", [one], "probe") + assert_accounted("<p>5.00 and 5.00</p>", [one, one], "probe") + + +def test_a_derived_total_has_no_value_field() -> None: + names = {f.name for f in dataclasses.fields(Derived)} + assert "value" not in names, "a stored total can drift from its citations" + total = Derived("t", (Component(_figure("2.00")), Component(_figure("3.00")))) + assert total.value == Decimal("5.00") + with pytest.raises(dataclasses.FrozenInstanceError): + total.value = Decimal("9.00") # type: ignore[misc] + + +def test_a_derived_total_recomputes_from_its_components_on_every_read() -> None: + part = _figure("2.00") + total = Derived("t", (Component(part), Component(_figure("3.00"), -1))) + assert total.value == Decimal("-1.00") + assert total.cites == (CITE, CITE) + assert total.workings() == "2.00 - 3.00 = (1.00)" + + +def test_a_derived_total_with_no_components_is_a_bare_number() -> None: + with pytest.raises(ValueError, match="bare number"): + Derived("t", ()) + + +def test_a_figure_needs_a_basis_that_exists_and_a_citation() -> None: + with pytest.raises(ValueError, match="not one of"): + Figure("x", Decimal("1"), "vibes", (CITE,)) + with pytest.raises(ValueError, match="no provenance"): + Figure("x", Decimal("1"), "ledger", ()) + assert set(SOURCE_BASES) == {"ledger", "comparative", "document"} + + +def test_a_quotient_divides_and_carries_both_sides_citations() -> None: + profit = _figure("407.00") + shares = Figure("shares", Decimal("5000000"), "document", (CITE,), literal="50,00,000") + eps = Quotient("eps", profit, shares, Decimal(100000)) + assert eps.rendered == "8.14" + assert len(eps.cites) == 2 + + +def test_a_gap_that_does_not_say_where_we_looked_is_a_shrug() -> None: + with pytest.raises(ValueError, match="shrug"): + Gap("something is missing", ()) + assert "looked in" in Gap("x", ("the ledger",)).sentence() + + +def test_prose_cannot_smuggle_an_undeclared_figure() -> None: + body = Body().markup("<p>") + with pytest.raises(UnaccountedFigureError): + body.prose("a difference of 9.00", ()).render("probe") + + +def test_a_citation_that_no_longer_reads_back_is_a_hard_error(corpus: Path) -> None: + CITE.verify(corpus) + with pytest.raises(CitationDriftError, match="citation drift"): + Cite(CITE.path, CITE.byte_start, CITE.byte_end, "9.99", "probe").verify(corpus) + + +def test_no_outcome_type_has_a_field_an_opinion_could_be_written_into() -> None: + """Decision 15's defence, rebuilt: the absence of a field, proved by reflection.""" + types = [ + figures.Figure, + figures.Derived, + figures.Component, + figures.Gap, + figures.Quotient, + figures.Cite, + disagreement.Side, + disagreement.Disagreement, + bodies.NoteContent, + bodies.Element, + ] + for candidate in types: + names = {f.name.lower() for f in dataclasses.fields(candidate)} + offenders = sorted(names & set(FORBIDDEN)) + assert not offenders, f"{candidate.__name__} has {offenders}" + + +def test_no_assignment_to_a_resolution_name_in_the_modules_that_carry_outcomes() -> None: + """Scoped to the outcome modules on purpose. + + A checklist item legitimately has a disclosure `status`, and an HTTP job has + a transport `status`; neither is an opinion about which of two disagreeing + sources is right. The rule is about the types that carry an outcome, and + those are the three scanned here. + """ + package = Path(figures.__file__).resolve().parent + pattern = re.compile(rf"\b(?:{'|'.join(FORBIDDEN)})\s*=", re.I) + hits = [] + for name in ("figures.py", "disagreement.py", "bodies.py"): + path = package / name + for number, line in enumerate(path.read_text(encoding="utf-8-sig").splitlines(), 1): + stripped = line.strip() + if stripped.startswith("#"): + continue + if pattern.search(stripped): + hits.append(f"{path.name}:{number}: {stripped}") + assert not hits, f"a resolution is being written somewhere: {hits}" diff --git a/use-cases/preetham1930/statutory-statements-builder/tests/test_money_and_chunks.py b/use-cases/preetham1930/statutory-statements-builder/tests/test_money_and_chunks.py new file mode 100644 index 000000000..059870997 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/tests/test_money_and_chunks.py @@ -0,0 +1,103 @@ +"""The recogniser, in both directions, and the chunk model it feeds.""" + +from __future__ import annotations + +from decimal import Decimal + +import pytest +from recorded import read_back_html +from statutory.chunks import ChunkMap, canonical, strip_chunk_ids +from statutory.money import format_money, money_tokens, money_tokens_in_html, parse_money + + +@pytest.mark.parametrize( + ("text", "expected"), + [ + ("total 2,180.00 here", ["2,180.00"]), + ("a loss of (45.00) this year", ["(45.00)"]), + ("50,00,000 equity shares", ["50,00,000"]), + ("INR 6,50,000 per month", ["6,50,000"]), + ("earnings per share 8.14", ["8.14"]), + ("42,000 sq ft", ["42,000"]), + ("sums to 821.00 (703.00 + 87.00)", ["821.00", "703.00", "87.00"]), + ], +) +def test_the_recogniser_finds_what_it_must(text: str, expected: list[str]) -> None: + assert money_tokens(text) == expected + + +@pytest.mark.parametrize( + "text", + [ + "for the year ended 31 March 2026", + "interest at 9.15% per annum", + "CIN: U29253TG2016PLC112447", + "Plot 42, IDA Nacharam, Hyderabad 500076", + "a term of 5 years with a lock-in of 36 months", + "100% of the equity share capital", + "Companies (Indian Accounting Standards) Rules, 2015", + ], +) +def test_the_recogniser_leaves_what_is_not_a_figure(text: str) -> None: + assert money_tokens(text) == [] + + +def test_brackets_are_matched_as_a_pair_or_not_at_all() -> None: + """The Phase 3 silent-skip: an independent optional bracket matched "(703.00".""" + assert "(703.00" not in money_tokens("sums to 821.00 (703.00 + 87.00)") + assert money_tokens("(1,462.00)") == ["(1,462.00)"] + + +def test_format_and_parse_round_trip() -> None: + for value in ("0.00", "-45.00", "4857.00", "-1691.00", "12.50"): + assert parse_money(format_money(Decimal(value))) == Decimal(value) + assert format_money(Decimal("-0.00")) == "0.00", "a nil balance is never '-0.00'" + + +def test_markup_never_contributes_a_figure() -> None: + fragment = ( + '<td data-chunk-id="1f2adf88-8741-40b3-a5c7-f443314369ab" ' + 'style="border:1px solid #1f3864;padding:6px">40.00</td>' + ) + assert money_tokens_in_html(fragment) == ["40.00"] + + +def test_canonical_ignores_whitespace_attribute_order_and_the_chunk_id() -> None: + left = '<p data-chunk-id="abc" style="color:red" class="x">a b</p>' + right = '<p class="x" style="color:red">a b</p>' + assert canonical(left) == canonical(right) + + +def test_canonical_treats_an_entity_and_its_character_as_the_same_content() -> None: + """Measured on the first live upload: we sent `·` and got back `·`.""" + assert canonical("<p>a · b</p>") == canonical("<p>a · b</p>") + assert canonical("<h3>Note 4 — Leases</h3>") == canonical("<h3>Note 4 — Leases</h3>") + + +def test_canonical_treats_a_void_element_the_same_either_way() -> None: + """Measured on the first live upload: we sent `<br>` and got back `<br/>`.""" + assert canonical("<p>a<br>b</p>") == canonical("<p>a<br/>b</p>") + + +def test_canonical_never_ignores_a_digit() -> None: + assert canonical("<p>724</p>") != canonical("<p>750</p>") + assert canonical('<td style="a">1.00</td>') != canonical('<td style="a">1.01</td>') + + +def test_a_recorded_document_parses_into_the_chunks_the_product_addressed() -> None: + chunks = ChunkMap(read_back_html("read0-baseline")) + assert len(chunks) == 18 + assert all(chunk.chunk_id for chunk in chunks.chunks) + assert chunks.chunks[0].tag == "h1" + assert "724" in chunks.by_id["8ec79daa-5de5-4973-84a5-1d1139da29be"].html + + +def test_headings_are_read_back_out_of_the_document(rolled) -> None: + headings = ChunkMap(rolled.document.html("target")).headings() + assert len(headings) == 27 + assert headings[4].startswith("Note 4") + assert "Leases" in headings[4] + + +def test_strip_chunk_ids_is_exact() -> None: + assert strip_chunk_ids('<p data-chunk-id="x">a</p>') == "<p>a</p>" diff --git a/use-cases/preetham1930/statutory-statements-builder/tests/test_note_tree.py b/use-cases/preetham1930/statutory-statements-builder/tests/test_note_tree.py new file mode 100644 index 000000000..0c59e4525 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/tests/test_note_tree.py @@ -0,0 +1,180 @@ +"""The insertion, and everything it drags with it. + +The second half of the grading line: a newly required note is detected from the +data, and note numbering plus every cross-reference stays consistent after an +insertion in the middle of the document. +""" + +from __future__ import annotations + +import re +import shutil +from itertools import pairwise +from pathlib import Path + +import pytest +from statutory.checklist import Checklist +from statutory.errors import ChecklistDriftError, CrossReferenceError +from statutory.events import EventDetector +from statutory.ledger import TrialBalance +from statutory.notetree import ( + assert_checklist_consistent, + build_note_tree, + load_new_note_titles, + remap_checklist, +) +from statutory.priorset import PriorYearSet +from statutory.statements import roll_forward + +PACKAGE = Path(__file__).resolve().parent.parent / "statutory" + + +def test_the_lease_note_is_required_by_the_data_not_by_a_list(rolled) -> None: + lease = next(n for n in rolled.tree.inserted if n.checklist_items[0] == "DC-025") + assert lease.number == 4 + assert lease.trigger_accounts[0] == "1110" + assert "1110" in lease.why and "268.00" in lease.why + assert lease.why_cites, "the trigger must carry the ledger bytes that made it true" + + +def test_insertion_point_comes_from_account_order( + corpus: Path, config: Path, tmp_path: Path +) -> None: + """Move the right-of-use account down the ledger; the note moves with it. + + If the position were remembered rather than computed, this would still land + the lease note at 4. + """ + root = tmp_path / "corpus" + shutil.copytree(corpus, root) + ledger_path = root / "sources" / "trial-balance-FY2026.csv" + lines = ledger_path.read_text(encoding="utf-8").splitlines(keepends=True) + rou = next(line for line in lines if line.startswith("1110,")) + lines.remove(rou) + at = next(i for i, line in enumerate(lines) if line.startswith("1230,")) + lines.insert(at + 1, rou) + ledger_path.write_text("".join(lines), encoding="utf-8") + + tree = _tree_for(root, config) + lease = next(n for n in tree.inserted if "DC-025" in n.checklist_items) + assert lease.number == 9, "after the note the preceding account points at, not after PPE" + assert tree.renumber[8] == 8 and tree.renumber[9] == 10 + + +def test_every_prior_note_survives_and_the_order_is_preserved(rolled) -> None: + tree = rolled.tree + assert len(tree.renumber) == 25 + assert tree.renumber[3] == 3 + assert tree.renumber[4] == 5 + assert tree.renumber[25] == 26 + shifted = [(old, new) for old, new in tree.renumber.items() if old != new] + assert len(shifted) == 22 + ordered = sorted(tree.renumber.items()) + assert all(a[1] < b[1] for a, b in pairwise(ordered)) + + +def test_every_cross_reference_resolves_after_insertion(rolled) -> None: + numbers = {note.number for note in rolled.tree.notes} + html = rolled.document.html("target") + referenced = {int(m.group(1)) for m in re.finditer(r"\bNote (\d+)\b", html)} + assert referenced <= numbers + # the one prose cross-reference in the signed set was repointed + assert "disclosed in Note 5" in html + assert "disclosed in Note 4" not in html + + +def test_the_primary_statements_point_at_the_new_numbers(rolled) -> None: + lines = { + line.account.code: line.note_ref + for statement in rolled.rolled.statements + for line in statement.lines + if line.account is not None + } + assert lines["1130"] == 5, "intangibles moved from note 4 to note 5" + assert lines["1210"] == 7 + assert lines["1110"] == 4 and lines["2210"] == 4 and lines["2310"] == 4 + + +def test_no_note_title_is_hardcoded_in_the_package(rolled) -> None: + """Every rolled-forward title comes from the signed set, not from our source.""" + titles = [note.title for note in rolled.tree.notes if not note.is_new] + literals = _string_literals(PACKAGE) + offenders = sorted({t for t in titles for lit in literals if t in lit}) + assert not offenders, f"these note titles are written into our code: {offenders}" + + +def test_the_checklist_is_remapped_through_the_same_map(rolled) -> None: + mapping = rolled.mapping + assert mapping["DC-005"] == 5, "prior note 4 -> 5" + assert mapping["DC-007"] == mapping["DC-008"] == 7 + assert mapping["DC-024"] == 26 + assert {mapping[f"DC-0{n}"] for n in (25, 26, 27, 28)} == {4} + assert {mapping[f"DC-0{n}"] for n in (29, 30, 31, 32)} == {27} + + +def test_checklist_consistency_is_asserted_against_the_document_not_our_model( + rolled, +) -> None: + from statutory.chunks import ChunkMap + + headings = ChunkMap(rolled.document.html("target")).headings() + assert len(headings) == 27 + assert_checklist_consistent(rolled.mapping, rolled.tree, headings) + + drifted = dict(headings) + drifted[5] = "Note 5 — Something else" + with pytest.raises(ChecklistDriftError, match="DC-005"): + assert_checklist_consistent(rolled.mapping, rolled.tree, drifted) + + extra = dict(headings) + extra[99] = "Note 99 - Quality Assurance" + with pytest.raises(ChecklistDriftError, match="fabrication"): + assert_checklist_consistent(rolled.mapping, rolled.tree, extra) + + +def test_a_prose_reference_to_a_note_that_never_existed_raises(rolled) -> None: + with pytest.raises(CrossReferenceError, match="Note 99"): + rolled.tree.repoint_prose("see Note 99 for details") + + +def _string_literals(package: Path) -> list[str]: + """Every string constant in our source that is not a docstring. + + Comments are excluded on purpose - a comment naming "Borrowings" to explain + why two rows need disambiguating is documentation, not a hardcoded title. + """ + import ast + + out: list[str] = [] + for path in package.rglob("*.py"): + tree = ast.parse(path.read_text(encoding="utf-8-sig")) + docstrings = { + id(node.body[0].value) + for node in ast.walk(tree) + if isinstance(node, (ast.Module, ast.ClassDef, ast.FunctionDef, ast.AsyncFunctionDef)) + and node.body + and isinstance(node.body[0], ast.Expr) + and isinstance(node.body[0].value, ast.Constant) + and isinstance(node.body[0].value.value, str) + } + for node in ast.walk(tree): + if ( + isinstance(node, ast.Constant) + and isinstance(node.value, str) + and id(node) not in docstrings + ): + out.append(node.value) + return out + + +def _tree_for(root: Path, config: Path): + prior = PriorYearSet(next((root / "prior-year").glob("*.html")), root) + ledger = TrialBalance(root / "sources" / "trial-balance-FY2026.csv", root) + forward = roll_forward(ledger, prior, root) + events = EventDetector(config / "event-vocabulary.csv", config / "quarantine-markers.txt") + events.scan([p for p in sorted((root / "sources").glob("*")) if p.is_file()], root) + checklist = Checklist(root / "rules" / "disclosure-checklist-indas.csv", root) + triggers = checklist.evaluate(ledger, events) + tree = build_note_tree(forward, triggers, load_new_note_titles(config / "new-note-titles.csv")) + remap_checklist(triggers, tree) + return tree diff --git a/use-cases/preetham1930/statutory-statements-builder/tests/test_orchestrator.py b/use-cases/preetham1930/statutory-statements-builder/tests/test_orchestrator.py new file mode 100644 index 000000000..5c89914eb --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/tests/test_orchestrator.py @@ -0,0 +1,152 @@ +"""The failure path is the build, not an afterthought. + +Four things are asserted here and each of them is a thing we watched a product +get wrong on 2026-08-07 and again on 2026-08-09: the queue halts, the downstream +steps do not run, no export is produced, and the sentence at the end is computed +from what actually happened. +""" + +from __future__ import annotations + +import pytest +from simulator import Behaviour, SimulatedSuperDocs +from statutory.editplan import build_plan +from statutory.errors import DecisionAlreadyRecordedError +from statutory.orchestrator import Orchestrator +from statutory.superdocs.client import SuperDocsClient +from statutory.superdocs.decisions import DecisionLedger + + +def _run(rolled, behaviour: Behaviour, steps: int = 4, ledger: DecisionLedger | None = None): + plan = build_plan(rolled.document).sample(steps) + product = SimulatedSuperDocs(behaviour) + client = SuperDocsClient(product, poll_seconds=0) + ledger = ledger or DecisionLedger() + orchestrator = Orchestrator(client, ledger, "K. Latha (CFO)", sleep=lambda _: None) + result = orchestrator.run("run-1", plan, "statements.html") + return result, product, ledger, orchestrator + + +def test_a_clean_run_applies_every_step_and_exports(rolled) -> None: + result, product, ledger, _ = _run(rolled, Behaviour()) + assert result.ok + assert result.applied == 4 + assert product.exports == ["docx"] + assert ledger.accepted("run-1") == 4 + assert "4 of 4 planned change(s) verified applied" in result.sentence(ledger) + assert "decided by K. Latha (CFO)" in result.sentence(ledger) + + +def test_a_failure_halts_the_queue_and_downstream_steps_never_run(rolled) -> None: + result, product, _ledger, orchestrator = _run(rolled, Behaviour(not_applied={2, 3})) + assert not result.ok + assert result.halted_at == 2 + assert result.applied == 1, "step 1 landed; 3 and 4 were never attempted" + assert product.step_index == 3, "step 2 plus its one narrow retry, and nothing after" + assert any("halting" in line for line in orchestrator.log) + + +def test_no_export_after_a_failure(rolled) -> None: + result, product, ledger, _ = _run(rolled, Behaviour(not_applied={2, 3})) + assert product.exports == [] + assert result.exported == [] + assert "No export was produced" in result.sentence(ledger) + assert "HALTED at step 2" in result.sentence(ledger) + + +def test_at_most_one_narrow_retry_and_never_a_replan(rolled) -> None: + """The retry is the same instruction. A second failure ends it.""" + result, product, _, orchestrator = _run(rolled, Behaviour(not_applied={2, 3})) + assert product.step_index == 3, "the step, its one retry, and nothing more" + assert result.outcomes[-1].retried + assert sum("narrow retry" in line for line in orchestrator.log) == 1 + + messages = [line for line in orchestrator.log if "retry" in line] + assert all("no re-plan" in line for line in messages) + + +def test_a_retry_that_succeeds_lets_the_run_continue(rolled) -> None: + """One failure, one retry of the same instruction, and the run carries on.""" + result, product, ledger, _orchestrator = _run(rolled, Behaviour(not_applied={2}), steps=3) + assert result.ok + assert product.step_index == 4, "three steps plus one retry" + assert [o.retried for o in result.outcomes] == [False, True, False] + rows = ledger.decided("run-1") + assert len(rows) == 3 + assert any(r.supersedes for r in ledger.rows), "the retry supersedes; both rows are kept" + assert len(ledger.rows) == 4 + + +def test_collateral_damage_is_never_retried(rolled) -> None: + """The document already holds content nobody asked for; another turn is worse.""" + result, product, _, orchestrator = _run(rolled, Behaviour(collateral={1})) + assert result.halted_at == 1 + assert product.step_index == 1, "no retry at all" + assert any("collateral" in line.lower() for line in orchestrator.log) + + +def test_a_failed_run_reverts_and_says_whether_the_revert_worked(rolled) -> None: + result, product, _, _ = _run(rolled, Behaviour(not_applied={1, 2})) + assert product.reverts == 1 + assert "reverted to the last verified-good state, confirmed by read-back" in result.reverted + + result2, product2, _, _ = _run(rolled, Behaviour(not_applied={1, 2}, revert_works=False)) + assert "does NOT match the last verified-good state" in result2.reverted + assert product2.exports == [] + + +def test_an_upload_that_is_not_verbatim_stops_before_any_edit(rolled) -> None: + with pytest.raises(AssertionError, match="was not verbatim"): + _run(rolled, Behaviour(fail_upload_verbatim=True)) + + +def test_the_decision_row_is_written_before_the_actuator_is_called(rolled) -> None: + """Decision 33: the product's gate is write-only, so the record has to be ours. + + Asserted on one shared timeline rather than on two counts, because two counts + that happen to be equal say nothing about the order. + """ + plan = build_plan(rolled.document).sample(2) + product = SimulatedSuperDocs(Behaviour()) + client = SuperDocsClient(product, poll_seconds=0) + ledger = DecisionLedger() + original = ledger.record + + def spy(*args, **kwargs): + product.event_log.append("ledger-row") + return original(*args, **kwargs) + + ledger.record = spy # type: ignore[method-assign] + result = Orchestrator(client, ledger, "K. Latha (CFO)", sleep=lambda _: None).run( + "run-1", plan, "statements.html" + ) + assert result.ok + timeline = [ + "ledger-row" if e == "ledger-row" else "approve" + for e in product.event_log + if e == "ledger-row" or e.startswith("approve:") + ] + assert timeline == ["ledger-row", "approve", "ledger-row", "approve"] + assert ledger.decided("run-1")[0].at, "the row carries a timestamp and an actor" + assert ledger.decided("run-1")[0].actor == "K. Latha (CFO)" + + +def test_a_second_decision_on_the_same_step_is_refused_unless_it_supersedes() -> None: + ledger = DecisionLedger() + ledger.record("r", 1, "c", "accept", "A", "because") + with pytest.raises(DecisionAlreadyRecordedError, match="quietly dropped"): + ledger.record("r", 1, "c", "reject", "B", "because") + superseding = ledger.record("r", 1, "c", "reject", "B", "because", supersede=True) + assert superseding.supersedes + assert len(ledger.rows) == 2, "the superseded row is kept" + assert ledger.accepted("r") == 0 and ledger.rejected("r") == 1 + + +def test_the_completion_sentence_is_computed_from_the_rows_it_holds() -> None: + ledger = DecisionLedger() + ledger.record("r", 1, "c1", "accept", "A", "x") + ledger.record("r", 2, "c2", "reject", "B", "y") + sentence = ledger.describe("r", planned=5) + assert "1 accepted, 1 rejected, 3 undecided of 5 planned" in sentence + ledger.rows.clear() + assert "0 accepted, 0 rejected, 5 undecided" in ledger.describe("r", planned=5) diff --git a/use-cases/preetham1930/statutory-statements-builder/tests/test_rollforward.py b/use-cases/preetham1930/statutory-statements-builder/tests/test_rollforward.py new file mode 100644 index 000000000..f447d0d1a --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/tests/test_rollforward.py @@ -0,0 +1,130 @@ +"""The comparatives tie exactly, and a disagreement stops the run. + +This is the first half of the grading line. It is checked twice: caption by +caption against the signed set, and total by total, because a set can agree row +by row and still print a wrong total. +""" + +from __future__ import annotations + +import shutil +from decimal import Decimal +from pathlib import Path + +import pytest +from statutory.errors import ComparativeMismatchError, TieBreakError +from statutory.figures import Figure +from statutory.ledger import TrialBalance +from statutory.pipeline import prepare +from statutory.priorset import PriorYearSet +from statutory.statements import roll_forward + + +def _copy_corpus(corpus: Path, tmp_path: Path) -> Path: + destination = tmp_path / "corpus" + shutil.copytree(corpus, destination) + return destination + + +def test_the_four_ties_in_the_task_hold(rolled) -> None: + document = rolled.rolled + assert document.total("TOTAL ASSETS").value == Decimal("4857.00") + assert document.total("TOTAL EQUITY AND LIABILITIES").value == Decimal("4857.00") + assert document.prior_total("TOTAL ASSETS").value == Decimal("4068.00") + assert document.prior_total("TOTAL EQUITY AND LIABILITIES").value == Decimal("4068.00") + + profit_and_loss = document.statement("profit_and_loss") + assert profit_and_loss.line("Profit before tax").current.value == Decimal("577.00") + assert profit_and_loss.line("Profit before tax").prior.value == Decimal("414.50") + assert profit_and_loss.line("Profit for the year").current.value == Decimal("407.00") + assert profit_and_loss.line("Profit for the year").prior.value == Decimal("309.00") + + other_equity = document.caption_line("2110") + assert other_equity.prior.value + Decimal("407.00") == other_equity.current.value + assert other_equity.current.value == Decimal("1691.00") + assert all(ok for _, _, _, ok in document.ties) + + +def test_every_total_the_signed_set_prints_is_reproduced(rolled) -> None: + labels = [label for label, _, _, _ in rolled.rolled.ties] + assert sum(1 for label in labels if "vs signed set" in label) >= 11 + + +def test_a_comparative_is_cited_twice_or_it_is_not_a_comparative(rolled) -> None: + comparatives = [ + line.prior + for statement in rolled.rolled.statements + for line in statement.lines + if line.account is not None and isinstance(line.prior, Figure) + ] + assert comparatives + for figure in comparatives: + if figure.basis == "comparative": + assert len(figure.cites) == 2 + paths = {cite.path for cite in figure.cites} + assert len(paths) == 2, "one cite to the ledger, one to the signed set" + + with pytest.raises(ValueError, match="cited twice"): + Figure("x", Decimal("1.00"), "comparative", (comparatives[0].cites[0],)) + + +def test_a_comparative_mismatch_is_a_hard_error(corpus: Path, tmp_path: Path) -> None: + """Doctor the ledger's prior-year column; the signed set now disagrees.""" + root = _copy_corpus(corpus, tmp_path) + ledger_path = root / "sources" / "trial-balance-FY2026.csv" + raw = ledger_path.read_text(encoding="utf-8") + ledger_path.write_text( + raw.replace( + "1210,Trade receivables,BS,Current assets,812.00", + "1210,Trade receivables,BS,Current assets,800.00", + ), + encoding="utf-8", + ) + prior = PriorYearSet(next((root / "prior-year").glob("*.html")), root) + ledger = TrialBalance(ledger_path, root) + with pytest.raises(ComparativeMismatchError) as caught: + roll_forward(ledger, prior, root) + message = str(caught.value) + assert "Trade receivables" in message + assert "800.00" in message and "812.00" in message + assert "bytes" in message, "the error must say where to look, not only that it failed" + + +def test_a_balance_sheet_that_does_not_balance_stops_the_run(corpus: Path, tmp_path: Path) -> None: + root = _copy_corpus(corpus, tmp_path) + ledger_path = root / "sources" / "trial-balance-FY2026.csv" + raw = ledger_path.read_text(encoding="utf-8") + ledger_path.write_text( + raw.replace( + ",1200,Inventories,BS,Current assets,685.00,747.00", + ",1200,Inventories,BS,Current assets,685.00,777.00", + ).replace( + "1200,Inventories,BS,Current assets,685.00,747.00", + "1200,Inventories,BS,Current assets,685.00,777.00", + ), + encoding="utf-8", + ) + prior = PriorYearSet(next((root / "prior-year").glob("*.html")), root) + with pytest.raises(TieBreakError, match="does not balance"): + roll_forward(TrialBalance(ledger_path, root), prior, root) + + +def test_new_captions_are_nil_in_the_prior_year_and_absent_from_the_signed_set( + rolled, +) -> None: + new = [c for c in rolled.rolled.comparative_checks if c.signed_value is None] + assert {c.caption for c in new} == { + "Right-of-use assets", + "Lease liabilities - non-current", + "Lease liabilities - current", + } + assert all(c.ledger_value == 0 for c in new) + + +def test_the_whole_roll_forward_runs_with_no_key_and_no_network( + corpus: Path, config: Path, monkeypatch +) -> None: + monkeypatch.delenv("SUPERDOCS_API_KEY", raising=False) + result = prepare(corpus, config) + assert result.cites_verified > 400 + assert len(result.document.blocks) > 90 diff --git a/use-cases/preetham1930/statutory-statements-builder/tests/test_verifier.py b/use-cases/preetham1930/statutory-statements-builder/tests/test_verifier.py new file mode 100644 index 000000000..f0e38f923 --- /dev/null +++ b/use-cases/preetham1930/statutory-statements-builder/tests/test_verifier.py @@ -0,0 +1,101 @@ +"""The three failure classes, each driven by a response the product actually gave. + +Nothing here is hand-written HTML. `read0`..`read3` are four recorded +`GET /v1/documents/{id}?include_html=true` bodies from 2026-08-09: + +- read0 -> read1 a single-value targeted edit that worked +- read1 -> read2 a mid-document insertion that silently did not happen, while + five heading chunks nobody targeted were rewritten around it +- read2 -> read3 one `create` that appended six sections, four of them invented + +So the verifier is tested against the real shapes of the real failures. +""" + +from __future__ import annotations + +from recorded import read_back_html +from statutory.chunks import ChunkMap +from statutory.superdocs.verifier import verify, verify_upload_verbatim + +TARGET = "8ec79daa-5de5-4973-84a5-1d1139da29be" + + +def _maps(*names: str) -> list[ChunkMap]: + return [ChunkMap(read_back_html(name)) for name in names] + + +def test_a_clean_single_target_replacement_verifies() -> None: + before, after = _maps("read0-baseline", "read1-after-A") + expected = after.by_id[TARGET].html + result = verify(1, TARGET, before, after, expected) + assert result.ok + assert "matches the computed post-state" in result.receipt + assert "17 non-target chunk(s) unchanged" in result.receipt + assert result.exact_bytes + + +def test_a_reply_claiming_success_does_not_make_a_step_ok() -> None: + """The chat reply for the failing turn read 'nothing actually changed'. + + Five changes had been auto-approved and were in the document. The verifier + never sees a reply, so it cannot be wrong in either direction. + """ + before, after = _maps("read1-after-A", "read2-after-B") + result = verify(2, TARGET, before, after, before.by_id[TARGET].html + "<!--x-->") + assert not result.ok + kinds = {p.kind for p in result.problems} + assert "not_applied" in kinds, "the target chunk came back untouched" + assert "collateral_damage" in kinds, "five heading chunks moved" + detail = next(p for p in result.problems if p.kind == "collateral_damage").detail + assert "5 chunk(s) we did not target changed" in detail + + +def test_collateral_damage_is_caught_when_the_requested_change_also_succeeded() -> None: + """The dangerous one: the append worked *and* four fabricated sections arrived. + + A verifier checking only its own target would report success here. + """ + before, after = _maps("read2-after-B", "read3-after-C") + # Pretend our step targeted a chunk in the document; the requested append + # also succeeded, which is exactly what makes this class dangerous. + changed = before.ids[0] + result = verify(3, changed, before, after, after.by_id[changed].html) + assert not result.ok + problem = next(p for p in result.problems if p.kind == "collateral_damage") + assert "the chunk set changed" in problem.detail + assert "18 chunk(s) before, 19 after" in problem.detail + assert "Quality Assurance" in problem.diff or "Humidity" in problem.diff + + +def test_applied_wrong_is_its_own_class() -> None: + before, after = _maps("read0-baseline", "read1-after-A") + expected = after.by_id[TARGET].html.replace("750", "751") + result = verify(4, TARGET, before, after, expected) + assert not result.ok + assert [p.kind for p in result.problems] == ["applied_wrong"] + assert "751" in result.problems[0].diff + + +def test_a_missing_target_chunk_is_not_applied_rather_than_a_crash() -> None: + before, after = _maps("read0-baseline", "read1-after-A") + result = verify(5, "no-such-chunk", before, after, "<p>x</p>") + assert not result.ok + assert any(p.kind == "not_applied" for p in result.problems) + + +def test_the_upload_is_checked_verbatim_rather_than_assumed() -> None: + """Decision 29 rests on a measured property, so it is re-measured every run.""" + uploaded = ChunkMap(read_back_html("read0-baseline")) + blocks = [chunk.html for chunk in uploaded.chunks] + receipt = verify_upload_verbatim(uploaded, blocks) + assert "18 chunk(s)" in receipt + + tampered = list(blocks) + tampered[3] = tampered[3].replace("</p>", " and one more sentence.</p>") + try: + verify_upload_verbatim(uploaded, tampered) + except AssertionError as exc: + assert "was not verbatim" in str(exc) + assert "first difference at block 3" in str(exc) + else: # pragma: no cover + raise AssertionError("a non-verbatim upload must stop the run")