diff --git a/use-cases/shivansh193/multi-agent-editorial-review/.env.example b/use-cases/shivansh193/multi-agent-editorial-review/.env.example new file mode 100644 index 00000000..f611a7f0 --- /dev/null +++ b/use-cases/shivansh193/multi-agent-editorial-review/.env.example @@ -0,0 +1 @@ +SUPERDOCS_API_KEY=your-key-here diff --git a/use-cases/shivansh193/multi-agent-editorial-review/.gitignore b/use-cases/shivansh193/multi-agent-editorial-review/.gitignore new file mode 100644 index 00000000..0703dfd8 --- /dev/null +++ b/use-cases/shivansh193/multi-agent-editorial-review/.gitignore @@ -0,0 +1,5 @@ +.env +output/ +__pycache__/ +*.pyc +.venv/ diff --git a/use-cases/shivansh193/multi-agent-editorial-review/PROGRESS.md b/use-cases/shivansh193/multi-agent-editorial-review/PROGRESS.md new file mode 100644 index 00000000..4cb48f92 --- /dev/null +++ b/use-cases/shivansh193/multi-agent-editorial-review/PROGRESS.md @@ -0,0 +1,116 @@ +# Progress log -- Multi-Agent Editorial Review Loop + +## Before any API calls: validated `verify_issues()` against known ground truth + +Same discipline as the other two builds. Hand-built a bloated, +pre-review draft (94-word Executive Summary, stale TAM/market-share +figures, one passive-voice sentence) and a fully-fixed version, ran +`verify_issues()` against both with zero API cost: the broken draft +correctly failed all four real checks (and correctly passed the +already-fine "risk mitigations untouched" control), the fixed version +correctly passed all five. + +## Run 1: the concurrency test found something more interesting than planned + +The original design fired the fact-check and style-review turns as +genuinely concurrent HTTP requests (`ThreadPoolExecutor`, not a +sequential loop) against two disjoint sets of sections, to empirically +test whether two simultaneous edits to one session could collide. + +They didn't get the chance to collide: **`POST /v1/chat/async` returned +`409 Conflict` outright** the instant the second submission landed while +the first chat job was still active on that session. This is a stronger +answer than the empirical test was designed to produce -- collisions +aren't just absent, they're structurally prevented, because the platform +won't accept two simultaneous chat jobs against the same session in the +first place. + +Fixed by adding retry-with-backoff on 409 to `run_turn` (`MAX_409_RETRIES += 40`, 5s interval) -- treating the rejection as "wait your turn" instead +of a hard failure, which is what a real caller has to do against this API +if it wants two logically-independent agents both editing one document. + +## Run 2: a script crash, and a false failure underneath it + +Re-ran with the 409 retry in place. Round 1 completed (style-review hit +7 real 409s waiting out fact-check's job, then ran and completed +normally). Verification reported `risk_mitigations_untouched: count 1` +(expected 2) -- looked like a real content problem, so the loop correctly +continued to round 2 per its own termination logic. Round 2's style-review +turn then exceeded the client's 400-second wait and the script crashed +with an unhandled `TimeoutError`, before writing any output file. + +Diagnosed by querying the session directly (`GET /v1/sessions/{id}/documents`, +authoritative, unaffected by the local crash) rather than re-running +blind. The real document state showed round 1 had actually fully +succeeded: both risks' mitigations were genuinely present in the text -- +one said `"Mitigation: ..."` (noun), the other said `"To mitigate the +risks..."` (verb). `verify_issues()`'s regex, `[Mm]itigation`, only +matched the noun form. Fixed to `[Mm]itigat` (the stem, catching +"mitigate," "mitigation," "mitigating," all forms). Re-checked against +the actual round-1 document with the fixed regex, no API cost: all five +checks passed. The crash-triggering round 2 had been chasing a problem +that didn't exist. + +(`GET /v1/sessions/{id}/jobs` came back with zero jobs for this session +despite real chat activity, which is odd and possibly worth a closer look +some other time -- not investigated further here since the document-state +endpoint gave a complete, authoritative answer on its own.) + +## Run 3: a clean run, and a real, different failure + +Re-ran once more, cleanly, with the regex fix in place -- no crash this +time. Round 1: `risk_mitigations_untouched: count 1` again, same as +before the crash's "fix." This time it's real, not a verification bug: + +The second risk's mitigation was reworded from the original +`"Mitigation: qualify a second supplier"` to `"To address this +vulnerability, we intend to qualify a second supplier"` -- no form of +"mitigate" anywhere in it. The mitigating *action* is still there (the +sentence still says what will be done about the risk); the specific word +the style guardrail's language leans on is not. This happened even +though `FACT_CHECK_INSTRUCTION` -- the only turn scoped to touch Risk +Assessment at all -- explicitly says "do not change anything else in the +document... no other sentence in Market Analysis or Risk Assessment +beyond the one figure in each." The fact-check turn rewrote both risk +items into fuller prose while correcting the one figure, not just the +figure in place. + +Deliberately did not patch the regex a second time to also match +"address." The first fix (noun form to stem, same underlying word) was a +correction of a real bug in the check. Chasing this one the same way +would mean adding every synonym a model might reach for until the check +stops meaning anything -- at that point it's not verifying the guardrail +anymore, it's verifying "did the reviewer output some sentence." This is +a real, different finding, not the same bug twice: **`FACT_CHECK_INSTRUCTION`'s +explicit "don't change anything else" scope wasn't fully honored -- the +substance of the edit stayed correct, but the turn reworded surrounding +content it was told to leave alone, and that reword happened to drift +away from the literal word a downstream guardrail cares about.** Round 2 +re-ran identically (fact-check and style-review both had nothing new to +find, since the only remaining issue isn't something either instruction +is scoped to fix) and the loop correctly hit its hard cap and stopped -- +the termination guarantee held exactly as designed, independent of +whether the document ever converged. + +## Final result + +**Overall: FAIL, 4 of 5 checks** (`tam_corrected`, +`competitor_share_corrected`, `exec_summary_length`, +`passive_voice_fixed`: PASS; `risk_mitigations_untouched`: FAIL, for the +real reason above). 2 of 2 rounds run, hard cap reached, loop terminated +as designed. `output/reviewed_launch_brief.docx`, +`output/final_document.html`, `output/verification_result.json`, and +`output/round_log.json` all reflect this run. + +Both things this build set out to prove came back with real answers, not +the ones originally hypothesized: + +- **No section collisions**: true, but not because two concurrent edits + landed side by side without conflict -- true because the platform + rejects the second concurrent submission outright with 409 Conflict. + Structurally prevented, not just empirically absent. +- **Provable loop termination**: true, and cleanly demonstrated -- the + loop ran its full `MAX_ROUNDS = 2` and stopped on the hard cap when + convergence didn't happen, exactly as the bounded `for` loop guarantees + it would. diff --git a/use-cases/shivansh193/multi-agent-editorial-review/README.md b/use-cases/shivansh193/multi-agent-editorial-review/README.md new file mode 100644 index 00000000..db0c771f --- /dev/null +++ b/use-cases/shivansh193/multi-agent-editorial-review/README.md @@ -0,0 +1,129 @@ +# Multi-Agent Editorial Review Loop + +Built by Shivansh Kalra for the SuperDocs task. + +A writer agent expands a bullet-point outline into a full first draft; a +fact-checker agent and a style-reviewer agent then work the same +document, on two genuinely disjoint sets of sections, fired as real +concurrent API calls rather than sequential turns, in a loop that's +bounded by construction rather than hoped to stop. + +All content is synthetic: a fictional smart-home hub (Aurora Home Hub) +and a fictional internal fact sheet and style checklist. + +Two things this build is specifically built to prove, not just assert -- +see [Verified result](#verified-result) for what actually held up: + +1. **No section collisions.** The fact-checker's target sections (Market + Analysis, Risk Assessment) and the style-reviewer's target sections + (Executive Summary, Technical Specifications) are disjoint by + construction, and both turns' submissions are fired at the same + instant against the same session. +2. **Provable loop termination.** The review loop is a plain `for` loop + over `range(1, MAX_ROUNDS + 1)` with an early `break` on convergence -- + it terminates in at most `MAX_ROUNDS` iterations by construction, + whether or not the agents ever agree the document is clean. + +## What it does + +1. Uploads the bullet-point outline (focused) plus two reference + documents as background: `verified_facts` (the two real figures) and + `style_guardrails` (four editorial rules, two of which the draft + already satisfies). +2. **Writer turn**: expands each section's bullets into full prose, + keeping every fact and number exactly as stated. +3. **Review rounds** (up to `MAX_ROUNDS = 2`): each round fires the + fact-checker and style-reviewer's *submissions* at the same instant + via a `ThreadPoolExecutor`, checks all four planted issues against the + resulting document, and stops early if everything's resolved. +4. Exports the result and verifies it programmatically by inspecting the + real returned HTML, not by asserting success. + +`verify_issues()` was validated in both directions before any real API +call: a hand-built bloated pre-review draft (all four checks correctly +fail) and a hand-built fully-fixed version (all five correctly pass). + +## How to run it + +```bash +python -m venv .venv +.venv/Scripts/activate # or source .venv/bin/activate on macOS/Linux +pip install -r requirements.txt +cp .env.example .env # then set SUPERDOCS_API_KEY +python build.py --dry-run # prints the full plan, zero API calls +python build.py # runs it for real +``` + +## SuperDocs features used + +- **Multi-document sessions** (`open_mode: "replace"` / `"background"`) -- + the draft plus two reference documents open together +- **Chat / async edit** (`POST /v1/chat/async`) with + `approval_mode: "ask_every_time"`, fired concurrently against one + session via a `ThreadPoolExecutor` +- **Export** (`POST /v1/documents/export`, `.docx`) + +## Verified result + +Two real runs against the live API (a third was a script crash from +chasing a false failure -- full trace in +[`PROGRESS.md`](PROGRESS.md)). Final, clean run: + +| Check | Result | +|---|---| +| TAM figure corrected ($2.8B, not $4.2B) | PASS | +| Competitor market share corrected (38%, not 61%) | PASS | +| Executive Summary <= 80 words | PASS | +| No passive voice in Technical Specifications | PASS | +| Risk Assessment mitigations untouched | **FAIL** | + +Overall: **FAIL, 4 of 5.** Both fact corrections landed, both style +fixes landed, and both guardrails that were already satisfied stayed +untouched everywhere except one place: the fact-checker, while correcting +the one figure it was scoped to touch, reworded the *other* risk's +mitigation sentence too -- from `"Mitigation: qualify a second supplier"` +to `"To address this vulnerability, we intend to qualify a second +supplier"` -- despite an explicit instruction not to change anything else +in that section. The mitigating action is still there in substance; the +literal word a downstream guardrail's language depends on isn't. That's +a real scope-discipline finding about the fact-checker turn, not a +verification-script bug -- see PROGRESS.md for how that was told apart +from two verification bugs that *did* turn up along the way and got +fixed instead of reported as findings. + +**Both things this build set out to prove came back true, for different +reasons than expected:** + +- **No section collisions**: true, but because `POST /v1/chat/async` + returns `409 Conflict` outright when a second chat request lands on a + session that already has one active -- collisions are structurally + prevented by the platform, not just empirically absent. `run_turn` + retries on 409 with backoff, which is what two agents genuinely racing + to edit one document have to do against this API. +- **Provable loop termination**: true, and directly demonstrated -- the + loop ran its full 2 rounds and stopped on the hard cap without + converging, exactly as the bounded `for` loop guarantees regardless of + outcome. + +## Honest limitations + +- The fact-checker turn's scope discipline isn't perfect: it corrected + the right figure but also touched wording elsewhere in its assigned + section that it was told to leave alone. Not investigated further as + its own bug report -- noted here as what it is. +- `output/` is gitignored; run `python build.py` to regenerate + `reviewed_launch_brief.docx`, `final_document.html`, + `verification_result.json`, and `round_log.json`. + +## Files + +- `build.py` -- upload -> writer -> review rounds (concurrent + fact-check + style-review, verify, retry up to `MAX_ROUNDS`) -> verify + -> export flow, plus `--dry-run` +- `content/brief_outline.html` -- the launch brief outline, authored with + two planted factual errors and two planted style violations +- `content/verified_facts.html`, `content/style_guardrails.html` -- the + fact-checker's and style-reviewer's reference documents +- `PROGRESS.md` -- full diagnostic trace across three runs, including how + two real verification-script bugs were told apart from the one real + platform/scope finding diff --git a/use-cases/shivansh193/multi-agent-editorial-review/build.py b/use-cases/shivansh193/multi-agent-editorial-review/build.py new file mode 100644 index 00000000..d51a0716 --- /dev/null +++ b/use-cases/shivansh193/multi-agent-editorial-review/build.py @@ -0,0 +1,391 @@ +"""Multi-Agent Editorial Review Loop -- built against the real, hosted +SuperDocs product. A writer agent expands a bullet-point outline into a +full first draft; a fact-checker agent and a style-reviewer agent then +work the same document, on two genuinely disjoint sets of sections, and +are fired as real concurrent API calls rather than sequential turns. + +Two things this build is specifically built to prove, not just assert: + +1. No section collisions. The fact-checker's target sections (Market + Analysis, Risk Assessment) and the style-reviewer's target sections + (Executive Summary, Technical Specifications) are disjoint by + construction. Both turns' *submissions* are fired at the same instant + (a ThreadPoolExecutor, not a sequential loop), genuinely racing to + start on the same session. The first real run found that SuperDocs + itself prevents the race: `POST /v1/chat/async` returns 409 Conflict + outright when a chat request lands on a session that already has + another chat job active, rather than accepting both and risking a + collision. That's a stronger answer than what this build originally + set out to test empirically -- collisions aren't just absent, they're + structurally prevented by the API rejecting concurrent submissions. + `run_turn` retries on 409 with backoff, which is what "two agents + racing to edit one document" has to do against this API in practice. + Verification still checks all four planted issues after every round, + so a real collision (if the platform's serialization ever had a gap) + would still show up as exactly one side's fixes missing. + +2. Provable loop termination. The review loop is a plain `for` loop over + `range(1, MAX_ROUNDS + 1)` with an early `break` on convergence -- it + terminates in at most MAX_ROUNDS iterations by construction, whether + or not the agents ever agree the document is clean. Not a sophisticated + proof, but a real and correct one: a bounded for-loop cannot run + forever. + +Ground truth: two deliberately wrong figures (a market-size number and a +competitor market-share number) that only the fact-checker, reading the +verified_facts document, can correct; and two deliberate style violations +(an over-length Executive Summary, one passive-voice sentence) that only +the style-reviewer, reading the style_guardrails document, can fix. Two +further guardrails in that same document are already satisfied by the +draft and must NOT get spuriously "fixed" -- getting that split right is +part of what's verified. + +Run `python build.py --dry-run` first: prints the full plan with zero API +calls. Only run for real (`python build.py`) after reading that output. +""" + +import argparse +import json +import os +import re +import sys +import time +import uuid +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path + +import httpx +from dotenv import load_dotenv + +load_dotenv() + +BASE_URL = "https://api.superdocs.app" +HERE = Path(__file__).parent +CONTENT_DIR = HERE / "content" +OUTPUT_DIR = HERE / "output" +OUTPUT_DIR.mkdir(exist_ok=True) + +MAX_ROUNDS = 2 + + +def log(msg: str) -> None: + print(f"[{time.strftime('%H:%M:%S')}] {msg}", flush=True) + + +# ---------- API helpers (same shape as the other two builds) ---------- + + +class Client: + def __init__(self, api_key: str): + self.http = httpx.Client(base_url=BASE_URL, headers={"Authorization": f"Bearer {api_key}"}, timeout=240.0) + + def upload_document(self, path: Path, session_id: str, open_mode: str = "replace") -> dict: + with open(path, "rb") as f: + resp = self.http.post( + "/v1/documents/upload", + files={"file": (path.name, f, "text/html")}, + data={"session_id": session_id, "open_mode": open_mode}, + ) + resp.raise_for_status() + return resp.json() + + def start_chat(self, message: str, session_id: str, approval_mode: str = "ask_every_time") -> dict: + resp = self.http.post( + "/v1/chat/async", + json={"message": message, "session_id": session_id, "approval_mode": approval_mode}, + ) + resp.raise_for_status() + return resp.json() + + def get_job(self, job_id: str) -> dict: + resp = self.http.get(f"/v1/jobs/{job_id}") + resp.raise_for_status() + return resp.json() + + def approve_all(self, session_id: str, job_id: str, pending_changes: list[dict]) -> None: + changes = [{"change_id": c["change_id"], "approved": True} for c in pending_changes] + resp = self.http.post(f"/v1/chat/{session_id}/approve", json={"job_id": job_id, "approved": True, "changes": changes}) + resp.raise_for_status() + + def continue_job(self, session_id: str, job_id: str) -> None: + resp = self.http.post(f"/v1/chat/{session_id}/continue", json={"job_id": job_id, "continue": True}) + resp.raise_for_status() + + def wait_for_job(self, session_id: str, job_id: str, label: str, max_wait_s: int = 400) -> dict: + start = time.time() + while time.time() - start < max_wait_s: + job = self.get_job(job_id) + status = job["status"] + if status == "completed": + log(f" [{label}] completed") + return job + if status in ("failed", "cancelled"): + raise RuntimeError(f"{label} job {status}: {job.get('error')}") + if status == "awaiting_approval": + metadata = job.get("metadata") or {} + if metadata.get("awaiting_kind") == "continue_prompt": + log(f" [{label}] paused mid-edit, continuing") + self.continue_job(session_id, job_id) + else: + pending = metadata.get("pending_changes") or [] + log(f" [{label}] awaiting approval on {len(pending)} change(s) -- approving") + self.approve_all(session_id, job_id, pending) + else: + log(f" [{label}] {status}...") + time.sleep(4) + raise TimeoutError(f"{label} job did not complete in time") + + def session_documents(self, session_id: str, include_html: bool = True) -> dict: + resp = self.http.get(f"/v1/sessions/{session_id}/documents", params={"include_html": str(include_html).lower()}) + resp.raise_for_status() + return resp.json() + + def export_html(self, html: str, filename: str, fmt: str = "docx") -> Path: + resp = self.http.post("/v1/documents/export", json={"html": html, "format": fmt, "options": {"filename": filename}}) + resp.raise_for_status() + ext = {"docx": "docx", "pdf": "pdf", "html": "html"}.get(fmt, fmt) + out_path = OUTPUT_DIR / f"{filename}.{ext}" + if "application/json" in resp.headers.get("content-type", ""): + data = resp.json() + url = data.get("download_url") or data.get("url") + out_path.write_bytes(self.http.get(url).content) + else: + out_path.write_bytes(resp.content) + return out_path + + +def _norm(s: str) -> str: + return re.sub(r"[_\-\s]+", " ", (s or "")).strip().lower() + + +def find_document_html(doc_list: dict, title_substring: str) -> str: + needle = _norm(title_substring) + for d in doc_list.get("documents", []): + if needle in _norm(d.get("title")): + html = d.get("html") + if not html: + raise ValueError(f"document matching '{title_substring}' found but has no html: {d}") + return html + raise ValueError(f"no open document matching '{title_substring}' -- got {doc_list}") + + +MAX_409_RETRIES = 40 +RETRY_409_INTERVAL_S = 5 + + +def run_turn(client: Client, session_id: str, label: str, instruction: str) -> int: + """Returns how many 409s this turn hit before its start_chat call was + accepted -- the platform rejects a chat request outright with 409 + Conflict while another chat job is already active on the same + session, rather than risking a race between them. Discovered on the + first real run of this build: retry-with-backoff on 409 turns that + rejection into "wait your turn," which is the only way two turns + submitted at the same instant against one session can both land.""" + log(f"[{label}] starting") + conflicts = 0 + while True: + try: + job = client.start_chat(instruction, session_id, approval_mode="ask_every_time") + break + except httpx.HTTPStatusError as e: + if e.response.status_code == 409 and conflicts < MAX_409_RETRIES: + conflicts += 1 + log(f" [{label}] 409 Conflict (session busy with another chat job) -- " + f"retrying in {RETRY_409_INTERVAL_S}s (attempt {conflicts}/{MAX_409_RETRIES})") + time.sleep(RETRY_409_INTERVAL_S) + continue + raise + client.wait_for_job(session_id, job["job_id"], label) + return conflicts + + +def run_concurrent_turns(client: Client, session_id: str, turns: list[tuple[str, str]]) -> dict[str, int]: + """Fires every (label, instruction) turn's *submission* at the same + instant (a ThreadPoolExecutor, not a sequential loop) so they're + genuinely racing to start on the same session. Whichever one the + platform rejects with 409 retries until the other's job frees the + session. Returns each label's 409 count -- the real evidence of + whether/how much contention actually happened.""" + with ThreadPoolExecutor(max_workers=len(turns)) as ex: + futures = {label: ex.submit(run_turn, client, session_id, label, instruction) for label, instruction in turns} + return {label: f.result() for label, f in futures.items()} + + +# ---------- the plan ---------- + +WRITER_INSTRUCTION = ( + "Expand each section's bullet list into 2-3 full prose sentences per section. Keep every fact, name, " + "and number exactly as stated in the bullets -- do not invent, add, or change any fact, and do not " + "soften or qualify any number. Keep each section's heading exactly as it is now. Do not add new " + "sections and do not remove any existing bullet's content, just turn it into prose." +) + +FACT_CHECK_INSTRUCTION = ( + "There is another document open in this session called verified_facts. Read it specifically. Check " + "exactly two claims in THIS document against it: the total addressable market figure in the Market " + "Analysis section, and HearthLink's market share figure in the Risk Assessment section. For each claim " + "that doesn't match verified_facts, correct the number in THIS document to match verified_facts " + "exactly, and add a short parenthetical note right after the corrected number: '(fact-checked against " + "verified_facts)'. If a claim already matches verified_facts, leave it untouched. Do not change " + "anything else in the document -- not the Executive Summary section, not the Technical Specifications " + "section, and no other sentence in Market Analysis or Risk Assessment beyond the one figure in each." +) + +REVIEW_INSTRUCTION = ( + "There is another document open in this session called style_guardrails. Read it specifically. Check " + "THIS document's Executive Summary section against guardrail 1 (must be 80 words or fewer) and THIS " + "document's Technical Specifications section against guardrail 2 (no passive voice -- every sentence " + "must have an explicit, active-voice subject performing the action). If the Executive Summary is " + "longer than 80 words, rewrite it to 80 words or fewer while keeping all three of its factual points, " + "and add a short parenthetical note at the end of the section: '(style-reviewed: trimmed for length)'. " + "If any sentence in Technical Specifications uses passive voice, rewrite that sentence in active voice, " + "and add a short parenthetical note right after it: '(style-reviewed: active voice)'. Do not change " + "anything else in the document -- not the Market Analysis section, not the Risk Assessment section, " + "and no other sentence in Executive Summary or Technical Specifications beyond what these two " + "guardrails require." +) + + +def print_dry_run() -> None: + print("=== DRY RUN -- no API calls will be made ===\n") + print(f"Document that would be uploaded to a single session: {CONTENT_DIR / 'brief_outline.html'}") + print(f"Background documents: {CONTENT_DIR / 'verified_facts.html'}, {CONTENT_DIR / 'style_guardrails.html'}") + print() + print("Planted issues (ground truth):") + print(" FACT : Market Analysis states TAM = $4.2B -- verified_facts says $2.8B") + print(" FACT : Risk Assessment states HearthLink share = 61% -- verified_facts says 38%") + print(" STYLE : Executive Summary will draft to >80 words -- guardrail caps it at 80") + print(" STYLE : Technical Specifications has one passive-voice sentence -- guardrail bans it") + print(" ALREADY OK, must NOT be touched: Risk Assessment mitigations (guardrail 3), Market") + print(" Analysis figures are already specific numbers, just wrong ones (guardrail 4)") + print() + print("Chat instruction 1 (writer, expands bullets to prose, no document_id set):") + print(f" {WRITER_INSTRUCTION[:200]}...") + print() + print(f"Then up to {MAX_ROUNDS} review round(s). Each round fires both instructions' submissions at the") + print("same instant (ThreadPoolExecutor, not sequential) against disjoint sections. SuperDocs itself") + print("rejects the second submission with 409 Conflict while the first is still active on that") + print("session -- run_turn retries on 409 with backoff. Checks convergence after each round, stops") + print("early if all four planted issues are resolved:") + print(f" fact-check (targets Market Analysis, Risk Assessment): {FACT_CHECK_INSTRUCTION[:120]}...") + print(f" style-review (targets Exec Summary, Tech Specs): {REVIEW_INSTRUCTION[:120]}...") + print() + print("Loop termination: a plain `for round in range(1, MAX_ROUNDS + 1)` with an early `break` on") + print("convergence -- bounded by construction, terminates in at most MAX_ROUNDS rounds regardless.") + print() + print(f"API calls this would make for real: 1 upload + 1 writer turn, then 2 turns per round") + print(f"(up to {MAX_ROUNDS} rounds), plus 1 export. No cross_session_search used.") + print("Re-run without --dry-run once this plan looks right.") + + +# ---------- verification ---------- + + +def verify_issues(html: str) -> dict: + results = {} + + tam_fixed = "$2.8B" in html or "2.8B" in html + tam_stale = "$4.2B" in html or "4.2B" in html + results["tam_corrected"] = {"correct": tam_fixed and not tam_stale} + + share_fixed = bool(re.search(r"38%", html)) + share_stale = bool(re.search(r"61%", html)) + results["competitor_share_corrected"] = {"correct": share_fixed and not share_stale} + + exec_summary_match = re.search(r"Executive Summary(.*?)(?=]+>", " ", exec_summary_match.group(1)) if exec_summary_match else "" + exec_word_count = len(exec_text.split()) + results["exec_summary_length"] = {"word_count": exec_word_count, "correct": exec_word_count <= 80} + + tech_specs_match = re.search(r"Technical Specifications(.*?)(?== 2} + + all_resolved = all(r["correct"] for r in results.values()) + return {"all_resolved": all_resolved, "details": results} + + +# ---------- main ---------- + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--dry-run", action="store_true") + args = parser.parse_args() + + if args.dry_run: + print_dry_run() + return + + api_key = os.environ.get("SUPERDOCS_API_KEY") + if not api_key: + print("SUPERDOCS_API_KEY not set", file=sys.stderr) + sys.exit(1) + client = Client(api_key) + + session_id = f"editorial-{uuid.uuid4()}" + log(f"session: {session_id}") + for name, mode in [ + ("brief_outline.html", "replace"), + ("verified_facts.html", "background"), + ("style_guardrails.html", "background"), + ]: + client.upload_document(CONTENT_DIR / name, session_id, open_mode=mode) + log(f" opened {name} ({mode})") + + log("writer: expanding outline to prose") + run_turn(client, session_id, "writer", WRITER_INSTRUCTION) + + round_log = [] + for round_num in range(1, MAX_ROUNDS + 1): + log(f"round {round_num}/{MAX_ROUNDS}: firing fact-check and style-review concurrently") + conflicts = run_concurrent_turns( + client, + session_id, + [("fact-check", FACT_CHECK_INSTRUCTION), ("style-review", REVIEW_INSTRUCTION)], + ) + log(f" round {round_num} 409 conflicts encountered: {conflicts}") + docs = client.session_documents(session_id, include_html=True) + html = find_document_html(docs, "brief_outline") + check = verify_issues(html) + round_log.append({"round": round_num, "conflicts_409": conflicts, **check}) + log(f" round {round_num} result: {json.dumps(check['details'])}") + if check["all_resolved"]: + log(f" converged after round {round_num}/{MAX_ROUNDS}, stopping") + break + else: + log(f" did not converge within {MAX_ROUNDS} rounds -- stopping anyway (hard cap reached)") + + (OUTPUT_DIR / "round_log.json").write_text(json.dumps(round_log, indent=2), encoding="utf-8") + + docs = client.session_documents(session_id, include_html=True) + html = find_document_html(docs, "brief_outline") + result = verify_issues(html) + result["rounds_run"] = len(round_log) + log("final verification:") + for name, detail in result["details"].items(): + log(f" {name}: {json.dumps(detail)}") + log(f"OVERALL: {'PASS' if result['all_resolved'] else 'FAIL'} ({result['rounds_run']} round(s) run)") + + export_path = client.export_html(html, "reviewed_launch_brief", fmt="docx") + log(f"exported -> {export_path}") + + (OUTPUT_DIR / "final_document.html").write_text(html, encoding="utf-8") + (OUTPUT_DIR / "verification_result.json").write_text(json.dumps(result, indent=2), encoding="utf-8") + + if not result["all_resolved"]: + sys.exit(1) + + +if __name__ == "__main__": + main() diff --git a/use-cases/shivansh193/multi-agent-editorial-review/content/brief_outline.html b/use-cases/shivansh193/multi-agent-editorial-review/content/brief_outline.html new file mode 100644 index 00000000..97031879 --- /dev/null +++ b/use-cases/shivansh193/multi-agent-editorial-review/content/brief_outline.html @@ -0,0 +1,29 @@ +

Aurora Home Hub — Product Launch Brief

+

Synthetic document for demonstration purposes only. No real product, company, or figures.

+ +

Executive Summary

+ + +

Market Analysis

+ + +

Technical Specifications

+ + +

Risk Assessment

+ diff --git a/use-cases/shivansh193/multi-agent-editorial-review/content/style_guardrails.html b/use-cases/shivansh193/multi-agent-editorial-review/content/style_guardrails.html new file mode 100644 index 00000000..b22c6ff5 --- /dev/null +++ b/use-cases/shivansh193/multi-agent-editorial-review/content/style_guardrails.html @@ -0,0 +1,8 @@ +

Aurora Home Hub — Editorial Style Guardrails

+

Internal style checklist for reviewing the launch brief. Synthetic, for demonstration purposes only.

+
    +
  1. Executive Summary length — must be 80 words or fewer. Flag and trim if longer.
    2. +
  2. No passive voice in Technical Specifications — every sentence in that section must have an explicit, active-voice subject performing the action. Flag and rewrite any passive construction.
    3. +
  3. Risk Assessment mitigations — every listed risk must end with an explicit mitigation sentence. Flag any risk that doesn't have one.
    4. +
  4. Market Analysis sourcing — any market-size or share figure must be presented as a specific number, not a vague range. Flag any figure that isn't specific.
    5. +
diff --git a/use-cases/shivansh193/multi-agent-editorial-review/content/verified_facts.html b/use-cases/shivansh193/multi-agent-editorial-review/content/verified_facts.html new file mode 100644 index 00000000..391c44ac --- /dev/null +++ b/use-cases/shivansh193/multi-agent-editorial-review/content/verified_facts.html @@ -0,0 +1,8 @@ +

Aurora Home Hub — Verified Facts (Internal)

+

Internal source-of-truth reference for fact-checking the launch brief. Synthetic, for demonstration purposes only.

+ diff --git a/use-cases/shivansh193/multi-agent-editorial-review/requirements.txt b/use-cases/shivansh193/multi-agent-editorial-review/requirements.txt new file mode 100644 index 00000000..7507eb03 --- /dev/null +++ b/use-cases/shivansh193/multi-agent-editorial-review/requirements.txt @@ -0,0 +1,2 @@ +httpx>=0.27 +python-dotenv>=1.0 diff --git a/use-cases/shivansh193/owner-contractor-redline-workspace/.env.example b/use-cases/shivansh193/owner-contractor-redline-workspace/.env.example new file mode 100644 index 00000000..f611a7f0 --- /dev/null +++ b/use-cases/shivansh193/owner-contractor-redline-workspace/.env.example @@ -0,0 +1 @@ +SUPERDOCS_API_KEY=your-key-here diff --git a/use-cases/shivansh193/owner-contractor-redline-workspace/.gitignore b/use-cases/shivansh193/owner-contractor-redline-workspace/.gitignore new file mode 100644 index 00000000..0703dfd8 --- /dev/null +++ b/use-cases/shivansh193/owner-contractor-redline-workspace/.gitignore @@ -0,0 +1,5 @@ +.env +output/ +__pycache__/ +*.pyc +.venv/ diff --git a/use-cases/shivansh193/owner-contractor-redline-workspace/PROGRESS.md b/use-cases/shivansh193/owner-contractor-redline-workspace/PROGRESS.md new file mode 100644 index 00000000..b10ae55a --- /dev/null +++ b/use-cases/shivansh193/owner-contractor-redline-workspace/PROGRESS.md @@ -0,0 +1,180 @@ +# Progress log -- Owner-Contractor Agreement Redline Workspace + +## 2026-08-20 -- reconciliation verified, redline surfaces a real platform bug + +### Run 1 (first real run, no `--dry-run`) + +Both chat jobs (`reconcile`, `redline`) reported `status: "completed"`, but +`GET /v1/sessions/{id}/documents?include_html=true` afterward showed the +focused document completely unmodified from the original upload -- no +reconciliation, no redline. Cross-checked against `GET /v1/sessions/{id}/jobs`: +both jobs' `result.response` text admitted failure in prose +(`"I couldn't access the requested sections of the document."` / +`"I wasn't able to complete the per-document work."`) while the job status +itself never said `failed`. + +Diagnosed via two cheap, targeted live-API calls instead of blind retries: + +1. A minimal 2-document test with a unique marker string -- succeeded, + proving background-document reading works in principle. +2. A single targeted question on the still-live real session + ("what does SC-1 say?") -- succeeded with the fully correct answer. + +Conclusion: the original `RECONCILE_INSTRUCTION` and `REDLINE_INSTRUCTION` +were too open-ended for one turn (read N documents, synthesize a full +rewrite, invent formatting, all at once) -- not a categorical platform +limit. Rewrote both to be narrower and procedural: one document to read, +one thing to extract from it, one edit rule, applied one category at a +time. Also fixed a real bug in `main()`: it picked the first session +document with non-empty `html` rather than matching by title, which would +have silently graded the wrong document if the roster ever came back in a +different order. + +### Run 2 (after the instruction rewrite and the document-selection fix) + +**Reconciliation: verified working correctly, end to end.** Not just +inferred from the final export -- the reconcile job's own before/after +diff (`GET /v1/sessions/{id}/jobs`) shows Article 5.1 changed from +`"forty-five (45) days"` to `"twenty-one (21) days ... (as amended by +SC-1)"`, with the job's `ai_explanation` correctly noting that SC-2 and +SC-3 were reviewed and correctly judged not to be amendments to existing +Article text. This is the hardest check in the build (proving the base +agreement's 45-day term, which independently violates the playbook's +23-day threshold, was actually reconciled to the amended 21-day term +*before* redlining happened) and it held up under direct inspection. + +**Redline: a real, reproducible SuperDocs bug, not a build mistake.** +The redline chat call uses `cross_session_search: true` so the playbook's +five thresholds are retrieved from a separate prior session rather than +re-pasted into the instruction -- proof the agent genuinely searched +memory rather than pattern-matching generic contract norms. Its own +`intermediate_responses` log two `open_document` operations: one for +`risk_playbook` (expected -- that's the intended cross-session retrieval) +and a second, unrequested one for `base_agreement` -- the document already +open and freshly reconciled in *this* session. + +That second open pulled in a stale snapshot: its HTML shows Article 5.1 +back at **45 days** (the pre-reconciliation figure), with a completely +different set of `data-chunk-id` UUIDs than the session's actual live +document (e.g. `h1 data-chunk-id="33047d90-..."` in the redline job's +snapshot vs. `h1 data-chunk-id="1085cde4-..."` in the reconcile job's -- +same document, same content, different identity). Working from that stale +copy, the agent correctly judged 45 days > 23-day threshold and flagged +it -- a *locally correct* judgment made against the *wrong* document +state. The job reported the edit as `"approved"` and the job itself +`"completed"`. But because the chunk ID it edited doesn't exist in the +session's real current document, the edit never actually applied there: +`GET /v1/sessions/{id}/documents` immediately after both jobs still shows +the correctly-reconciled, unflagged 21-day text -- and neither of the two +genuinely-required flags (Article 6, notice period; Article 7, +indemnification) was ever computed at all, because the job's one +"parallel edit" pass was spent on the phantom Article 5 violation instead. + +Net effect: **a chat job can report `completed`, with a specific, +plausible-looking approved diff, while that diff has zero effect on the +document the session actually holds -- and nothing in the API response +signals the divergence.** The only way to catch it was comparing chunk-id +UUIDs across two different jobs' snapshots of "the same" document, which +isn't something a caller would normally think to do. This is a sharper +finding than Run 1's silent-failure-in-prose bug: that one at least made +the mismatch visible in the `response` text if you read it; this one +reports success at every layer that matters (`status`, `changes[].status`, +`ai_explanation`) and is only detectable via document-identity metadata a +caller has no obvious reason to cross-check. + +Working theory for the trigger, not confirmed: `cross_session_search` +resolves by document *title* across all of the account's sessions, not +scoped to "only search for things not already open here." Run 1's failed +session had also uploaded a document titled `base_agreement`, never +edited (since Run 1's reconcile job silently did nothing) -- a very +plausible candidate for the stale copy that got re-opened, though a +same-session naming collision without any Run 1 leftover would produce +the same symptom. + +### Decision: stop here, don't blind-retry against operations budget + +Also found and fixed, while diagnosing: `verify()` compared literal +`"Article 7"` against document text that actually reads `"ARTICLE 7"` +(all-caps headings), so every `article_found` check was silently `false` +regardless of real content -- fixed to case-insensitive search. Its +reconciliation regex (`21\s*day`) also didn't match the real text +`"twenty-one (21) days"` because of the parenthesis -- fixed to +`21\)?\s*day`. Both were bugs in this repo's own verification script, not +platform behavior; re-running the fixed `verify()` against the existing +Run 2 export (no API cost) gives the accurate final picture below. + +Given the root cause of the redline failure isn't fully pinned down +(cross-session title collision vs. a more general re-open-on-touch +behavior), a third run risks reproducing the same failure for the same +reason and spending ops without new information. Reported the honest, +well-diagnosed result instead of retrying blind. + +## Final verification result (Run 2, corrected `verify()`, no further API calls) + +| Check | Expected | Actual | Result | +|---|---|---|---| +| Article 7 (indemnification) flagged | yes | no | **FAIL** | +| Article 6 (notice period) flagged | yes | no | **FAIL** | +| Article 8 (damages waiver) flagged | no | no | PASS | +| Article 5 (payment terms) flagged | no | no | PASS | +| Article 9 (termination) flagged | no | no | PASS | +| Reconciliation applied (shows 21 days, not 45) | yes | yes | **PASS** | + +Overall: **FAIL** (4 of 6 checks correct). The hardest check -- +reconciliation actually landing before redlining ran -- passed cleanly. +Both failures trace to the single stale-reopen bug above, not to two +independent problems. + +## 2026-08-21 -- the known mitigation, run for real: a clean pass + +Everything above is left exactly as it was written. This is a second, +later result on top of it, not a replacement -- the platform bug it +found is still real and still worth reporting on its own. + +Implemented the fix the redline-step bug pointed at: dropped +`cross_session_search` entirely and uploaded `risk_playbook.html` into +the *main* session as a fifth background document, the same pattern +already used for the two Exhibits. `REDLINE_INSTRUCTION` now says "there +is another document open in this session called risk_playbook" instead +of "search your memory of previous sessions" -- otherwise unchanged. +This trades away one of the build's two original evidentiary properties +(proof the playbook was retrieved via genuine cross-session search, +not re-pasted) for a redline step that actually works; the other +property (the payment-terms reconciliation check, and the playbook's +arbitrary, non-guessable thresholds) is untouched. + +Ran for real. Reconciliation this time correctly picked up all three +Supplementary Conditions amendments (SC-1's payment term, SC-2's +submittal-schedule addition to Article 3, SC-3's site-access addition to +Article 10) rather than just SC-1 -- a more thorough read than either +prior run, not a regression; every added or changed sentence carries its +own "(As amended by SC-N)" note, and no Article's original text was lost. +The redline step then flagged both Article 6 (notice period, 5 business +days violates the 11-day threshold) and Article 7 (indemnification, +one-directional) correctly, left Article 8 (damages waiver) and Article 9 +(termination) correctly unflagged, and treated Article 5's reconciled +21-day payment term as compliant, exactly as the reconciliation-order +check is designed to catch. + +**A second, unrelated bug -- this time in this repo's own `verify()`, not +the platform.** The initial verification run reported `payment_terms_flagged: +correct: false`, i.e. Article 5 appeared to have been wrongly flagged. +Direct inspection of `output/final_document.html` showed no RISK FLAG +anywhere near Article 5 at all. The cause: `verify()`'s per-Article check +used a fixed 1200-character window after each Article's heading to look +for a flag. Article 5's own content (heading + two short paragraphs) is +short enough that the fixed window ran straight through Article 6's +heading and *into Article 6's own, genuinely-correct RISK FLAG paragraph*, +misattributing it to Article 5. Every earlier run happened not to trigger +this, because Article 5 was either itself flagged (wrongly, by the +platform) or the run failed before getting this far -- this is the first +run clean enough on the platform side to expose a bug that was sitting in +the verification script the whole time. Fixed by bounding each Article's +window to the position of the *next* "Article N" heading instead of a +fixed character count (confirmed via `grep -o "ARTICLE [0-9]*"` that this +document only ever uses the heading form, no inline mid-sentence +references, so the bound is unambiguous here). Re-ran `verify()` against +the same, unchanged export -- no new API call needed. + +**Final result: PASS, 6 of 6.** `output/reconciled_and_redlined_agreement.docx` +and `output/verification_result.json` reflect this run. diff --git a/use-cases/shivansh193/owner-contractor-redline-workspace/README.md b/use-cases/shivansh193/owner-contractor-redline-workspace/README.md new file mode 100644 index 00000000..4ff4535e --- /dev/null +++ b/use-cases/shivansh193/owner-contractor-redline-workspace/README.md @@ -0,0 +1,187 @@ +# Owner-Contractor Agreement Redline Workspace + +Built by Shivansh Kalra for the SuperDocs task. + +Reconciles a base Owner-Contractor Agreement with its Supplementary +Conditions into one effective document, then redlines that effective +document against a risk playbook -- indemnity mutuality, damages-waiver +mutuality, notice periods, payment terms, and termination-for-convenience +notice, each against a deliberately non-"standard" numeric threshold (11 +business days, 23 days, 17 days, not the round 10/14/30 a model would +guess from generic contract knowledge). + +All content is synthetic: a fictional owner (Riverside Medical Partners +LLC), a fictional contractor (Meridian Builders LLC), and a fictional +internal risk playbook. + +Two things were deliberately engineered to be independently verifiable, +not just plausible-looking -- see [Verified result](#verified-result) for +which one actually held up, and for what changed between the two real +runs documented there: + +1. **Originally**: the risk playbook was established in a separate prior + session and referenced only via `cross_session_search: true` -- never + re-pasted into the redline instruction, so a correct flag against one + of its specific, arbitrary thresholds would be evidence the search + genuinely retrieved the playbook, not that the model pattern-matched + typical contract norms. This surfaced a real SuperDocs bug (see below) + and was replaced with the playbook loaded into the main session as a + background document instead, the same pattern used for the two + Exhibits -- trading that specific evidentiary property for a redline + step that actually works. +2. The base agreement's payment term (45 days) genuinely violates the + playbook's threshold (>23 days) on its own -- but the Supplementary + Conditions amend it to 21 days, which is compliant. If the final + document treats payment terms as compliant, that's evidence real + reconciliation happened *before* redlining, not that the base document + was redlined in isolation while ignoring the amendment. This property + is untouched by the fix above and still holds. + +## What it does + +1. Uploads the risk playbook to a throwaway "setup" session and has + SuperDocs summarize it, so it exists in cross-session memory. +2. Opens four documents together in a second session: the base agreement + (focused), Supplementary Conditions, and two Exhibits (background). +3. **Reconcile step**: instructs SuperDocs to read the Supplementary + Conditions, find its numbered amendments, and edit the corresponding + Articles in the base agreement in place -- every other Article must + stay present and unchanged. +4. **Redline step**: instructs SuperDocs to read the risk playbook (open + in the same session as a background document), check the + now-reconciled document's actual terms against each of the playbook's + five thresholds in turn, and insert a red `RISK FLAG:` paragraph after + any Article that violates its threshold. +5. Exports the result as `.docx` and verifies it programmatically against + six checks (five per-Article flag/no-flag expectations plus the + reconciliation check itself) by inspecting the real returned HTML, not + by asserting success. + +## How to run it + +```bash +python -m venv .venv +.venv/Scripts/activate # or source .venv/bin/activate on macOS/Linux +pip install -r requirements.txt +cp .env.example .env # then set SUPERDOCS_API_KEY +python build.py --dry-run # prints the full plan, zero API calls +python build.py # runs it for real: ~5 uploads, 2 chat turns, 1 export +``` + +## SuperDocs features used + +- **Multi-document sessions** (`open_mode: "replace"` / `"background"`) -- + five related documents open together (base agreement, Supplementary + Conditions, two Exhibits, risk playbook), one focused +- **Chat / async edit** (`POST /v1/chat/async`) with + `approval_mode: "ask_every_time"` across two sequential instructions on + the same focused document +- **Export** (`POST /v1/documents/export`, `.docx`) +- **Job introspection** (`GET /v1/sessions/{id}/jobs`) -- used here not + just to poll status but to directly diff each job's approved + before/after HTML, which is how both real findings below were caught + +## Verified result + +**Reconciliation: verified working correctly, end to end.** Not inferred +from the final export -- the reconcile job's own before/after diff shows +Article 5.1 changed from `"forty-five (45) days"` to `"twenty-one (21) +days ... (as amended by SC-1)"`, and its `ai_explanation` correctly notes +that the Supplementary Conditions' other two clauses were reviewed and +correctly judged to be new obligations, not amendments to existing +Article text. This was the hardest of the six checks (proving the +independently-violating 45-day term was actually reconciled *before* +redlining ran) and it held up under direct inspection. + +**Redline: exposed a real, reproducible SuperDocs platform bug.** The +`cross_session_search`-enabled redline call, per its own +`intermediate_responses`, opened *two* documents by name: the intended +`risk_playbook`, and a second, unrequested `base_agreement` -- the +document already open and freshly reconciled in the same session. That +second open pulled in a stale snapshot (Article 5.1 back at 45 days, with +an entirely different set of `data-chunk-id` UUIDs than the session's real +current document). Working from that stale copy, the agent correctly +judged 45 > 23 days and flagged it -- a locally correct judgment against +the wrong document state. The job reported the resulting edit as +`"approved"` and the job itself `"completed"`, but because its chunk ID +doesn't exist in the session's real document, the edit never actually +applied there. The two genuinely-required flags (Article 6 notice period, +Article 7 indemnification) were never computed at all, because the job's +one edit pass went to the phantom Article 5 violation instead. + +Net effect: **a chat job can report `completed`, with a specific, +plausible-looking approved diff, while that diff has zero effect on the +document the session actually holds -- and nothing in the response +signals the divergence.** Full technical trace, including the exact job +diffs and chunk IDs involved, is in [`PROGRESS.md`](PROGRESS.md). + +| Check | Expected | Actual | Result | +|---|---|---|---| +| Article 7 (indemnification) flagged | yes | no | **FAIL** | +| Article 6 (notice period) flagged | yes | no | **FAIL** | +| Article 8 (damages waiver) flagged | no | no | PASS | +| Article 5 (payment terms) flagged | no | no | PASS | +| Article 9 (termination) flagged | no | no | PASS | +| Reconciliation applied (21 days, not 45) | yes | yes | **PASS** | + +Overall: **FAIL** (4 of 6). Both failures trace to the single stale-reopen +bug above, not to two independent problems -- and the check that was +actually the point of the exercise (real reconciliation before redlining) +passed cleanly. + +### Later result: the mitigation, run for real + +Dropped `cross_session_search` and loaded the playbook into the main +session as a background document instead (the same pattern already used +for the Exhibits). Ran again for real: + +| Check | Expected | Actual | Result | +|---|---|---|---| +| Article 7 (indemnification) flagged | yes | yes | PASS | +| Article 6 (notice period) flagged | yes | yes | PASS | +| Article 8 (damages waiver) flagged | no | no | PASS | +| Article 5 (payment terms) flagged | no | no | PASS | +| Article 9 (termination) flagged | no | no | PASS | +| Reconciliation applied (21 days, not 45) | yes | yes | PASS | + +Overall: **PASS, 6 of 6.** Reconciliation this run also picked up two +further Supplementary Conditions amendments (a submittal-schedule +addition and a site-access clause) that earlier runs had judged as new +obligations rather than amendments -- a more thorough read, not a +regression; every Article is still present and every change carries its +own "(As amended by SC-N)" note. + +One verification-script bug turned up along the way, in this repo's own +`verify()`, not the platform: it checked each Article for a nearby +`RISK FLAG` using a fixed 1200-character window, which was short enough +that Article 5's window ran into Article 6's own (correct) flag and +misattributed it, briefly reporting a false failure. Fixed by bounding +each Article's window to the next Article heading instead of a fixed +length. Full trace in [`PROGRESS.md`](PROGRESS.md). + +This result doesn't replace the one above -- the bug that first run found +is real and still worth reporting on its own; this is what fixing it +looks like once you actually apply the known mitigation. + +## Honest limitations + +- The original `cross_session_search` design (proof the playbook was + retrieved via genuine cross-session search, not re-pasted) was traded + away to get a working redline step -- see the two results above for + why. The reconciliation-order evidentiary property is untouched. +- `output/` is gitignored; run `python build.py` to regenerate + `reconciled_and_redlined_agreement.docx`, `final_document.html`, and + `verification_result.json` -- reflects the mitigated version's PASS + result as of the current `build.py`. + +## Files + +- `build.py` -- upload -> reconcile -> redline -> verify -> export flow, + plus `--dry-run` +- `content/base_agreement.html`, `supplementary_conditions.html`, + `exhibit_a_scope.html`, `exhibit_b_insurance.html` -- the contract + documents +- `content/risk_playbook.html` -- the internal risk checklist, retrieved + via cross-session search rather than re-pasted +- `PROGRESS.md` -- full diagnostic trace of both runs, including the + exact job diffs and chunk-ID evidence for the platform bug diff --git a/use-cases/shivansh193/owner-contractor-redline-workspace/build.py b/use-cases/shivansh193/owner-contractor-redline-workspace/build.py new file mode 100644 index 00000000..792c0dfd --- /dev/null +++ b/use-cases/shivansh193/owner-contractor-redline-workspace/build.py @@ -0,0 +1,328 @@ +"""Owner-Contractor Agreement Redline Workspace -- built against the real, +hosted SuperDocs product. Reconciles a base Owner-Contractor Agreement, +Supplementary Conditions, and two Exhibits into one effective document +(the task doc calls this "the genuinely hard part"), then redlines that +effective document against a risk playbook -- indemnity, damages waiver, +notice periods, payment terms, termination for convenience. + +The playbook was originally established in a *separate* prior session and +referenced only via `cross_session_search: true` in the redline step, to +prove the search genuinely retrieved it rather than the model pattern- +matching generic contract norms. That version is preserved as history in +PROGRESS.md: it surfaced a real SuperDocs bug (`cross_session_search` can +silently re-open a stale, pre-reconciliation snapshot of a document +already open in the same session, so an "approved" edit never actually +lands). This version implements the known mitigation instead -- the +playbook is uploaded into the *main* session as a background document, +the same pattern already used for the two Exhibits, so nothing needs +cross-session retrieval at all. This trades away one evidentiary property +(proof the retrieval was genuinely cross-session) for a working redline +step. The other evidentiary property below still holds either way. + +The base agreement's payment term (45 days) genuinely violates the +playbook's threshold (>23 days) on its own -- but the Supplementary +Conditions amend it to 21 days, which is compliant. If the final redline +treats payment terms as compliant, that's evidence real reconciliation +happened *before* redlining, not that the base document was redlined in +isolation while ignoring the amendment. Its thresholds are also arbitrary, +non-"standard" numbers (11 business days, 23 days, 17 days -- not the +round 10/14/30 a model would guess from generic contract knowledge), so a +correct flag still isn't just pattern-matching typical contract norms. + +Run `python build.py --dry-run` first: prints the full plan (uploads, +exact chat instructions, what verification will check) with zero API +calls. Only run for real (`python build.py`) after reading that output. +""" + +import argparse +import json +import os +import re +import sys +import time +import uuid +from pathlib import Path + +import httpx +from dotenv import load_dotenv + +load_dotenv() + +BASE_URL = "https://api.superdocs.app" +HERE = Path(__file__).parent +CONTENT_DIR = HERE / "content" +OUTPUT_DIR = HERE / "output" +OUTPUT_DIR.mkdir(exist_ok=True) + + +def log(msg: str) -> None: + print(f"[{time.strftime('%H:%M:%S')}] {msg}", flush=True) + + +# ---------- API helpers ---------- + + +class Client: + def __init__(self, api_key: str): + self.http = httpx.Client(base_url=BASE_URL, headers={"Authorization": f"Bearer {api_key}"}, timeout=240.0) + + def upload_document(self, path: Path, session_id: str, open_mode: str = "replace") -> dict: + with open(path, "rb") as f: + resp = self.http.post( + "/v1/documents/upload", + files={"file": (path.name, f, "text/html")}, + data={"session_id": session_id, "open_mode": open_mode}, + ) + resp.raise_for_status() + return resp.json() + + def start_chat(self, message: str, session_id: str, approval_mode: str = "ask_every_time", cross_session_search: bool = False) -> dict: + body = {"message": message, "session_id": session_id, "approval_mode": approval_mode} + if cross_session_search: + body["cross_session_search"] = True + resp = self.http.post("/v1/chat/async", json=body) + resp.raise_for_status() + return resp.json() + + def get_job(self, job_id: str) -> dict: + resp = self.http.get(f"/v1/jobs/{job_id}") + resp.raise_for_status() + return resp.json() + + def approve_all(self, session_id: str, job_id: str, pending_changes: list[dict]) -> None: + changes = [{"change_id": c["change_id"], "approved": True} for c in pending_changes] + resp = self.http.post(f"/v1/chat/{session_id}/approve", json={"job_id": job_id, "approved": True, "changes": changes}) + resp.raise_for_status() + + def continue_job(self, session_id: str, job_id: str) -> None: + resp = self.http.post(f"/v1/chat/{session_id}/continue", json={"job_id": job_id, "continue": True}) + resp.raise_for_status() + + def wait_for_job(self, session_id: str, job_id: str, label: str, max_wait_s: int = 400) -> dict: + start = time.time() + while time.time() - start < max_wait_s: + job = self.get_job(job_id) + status = job["status"] + if status == "completed": + log(f" {label}: completed") + return job + if status in ("failed", "cancelled"): + raise RuntimeError(f"{label} job {status}: {job.get('error')}") + if status == "awaiting_approval": + metadata = job.get("metadata") or {} + if metadata.get("awaiting_kind") == "continue_prompt": + log(f" {label}: paused mid-edit, continuing") + self.continue_job(session_id, job_id) + else: + pending = metadata.get("pending_changes") or [] + log(f" {label}: awaiting approval on {len(pending)} change(s) -- approving") + self.approve_all(session_id, job_id, pending) + else: + log(f" {label}: {status}...") + time.sleep(4) + raise TimeoutError(f"{label} job did not complete in time") + + def session_documents(self, session_id: str, include_html: bool = True) -> dict: + resp = self.http.get(f"/v1/sessions/{session_id}/documents", params={"include_html": str(include_html).lower()}) + resp.raise_for_status() + return resp.json() + + def export_html(self, html: str, filename: str, fmt: str = "docx") -> Path: + resp = self.http.post("/v1/documents/export", json={"html": html, "format": fmt, "options": {"filename": filename}}) + resp.raise_for_status() + ext = {"docx": "docx", "pdf": "pdf", "html": "html"}.get(fmt, fmt) + out_path = OUTPUT_DIR / f"{filename}.{ext}" + if "application/json" in resp.headers.get("content-type", ""): + data = resp.json() + url = data.get("download_url") or data.get("url") + out_path.write_bytes(self.http.get(url).content) + else: + out_path.write_bytes(resp.content) + return out_path + + +# ---------- the plan (shared by --dry-run and the real run) ---------- + +RECONCILE_INSTRUCTION = ( + "There is another document open in this session called supplementary_conditions. Read it specifically. " + "It contains one or more numbered amendments (labeled like 'SC-1'); each one names which Article of " + "THIS document it amends and states the new term. For each amendment you find in supplementary_conditions: " + "edit the corresponding Article in THIS document so it states the new term instead of the old one, and " + "add a short parenthetical note right after the changed sentence naming which amendment made the change, " + "for example '(as amended by SC-1)'. Do not change any Article that supplementary_conditions doesn't " + "amend, and do not summarize, shorten, or remove any other part of this document -- every Article must " + "still be present with its number and full text, unchanged except where an amendment applies." +) + +REDLINE_INSTRUCTION = ( + "There is another document open in this session called risk_playbook. Read it specifically. It lists " + "five numbered risk categories, each with one specific numeric threshold. Work through the five " + "categories one at a time, in order. For each one: find the Article in THIS document (not " + "risk_playbook) that covers that category, compare this document's actual current term for it against " + "that category's threshold, and only if it violates the threshold, insert one new paragraph directly " + "after that Article's text: start it with the literal text 'RISK FLAG:', explain which threshold is " + "violated and by how much, and make the whole paragraph red using style=\"color:#b00\". If a category's " + "term already meets the threshold, insert nothing for it and move to the next category. Some of the " + "five will need a flag and some won't. Do not edit risk_playbook itself." +) + + +def print_dry_run() -> None: + print("=== DRY RUN -- no API calls will be made ===\n") + print("Mitigated version: no setup session, no cross_session_search. The risk playbook goes into the") + print("main session as a fifth background document, same pattern as the two Exhibits.") + print() + print("Documents that would be uploaded to the main session, in order:") + for name, mode in [ + ("base_agreement.html", "replace (becomes focused)"), + ("supplementary_conditions.html", "background"), + ("exhibit_a_scope.html", "background"), + ("exhibit_b_insurance.html", "background"), + ("risk_playbook.html", "background"), + ]: + print(f" - {CONTENT_DIR / name} [{mode}]") + print() + print("Chat instruction 1 (reconcile, targets the focused base_agreement doc, no document_id set):") + print(f" {RECONCILE_INSTRUCTION[:200]}...") + print() + print("Chat instruction 2 (redline, same focused doc, playbook read from the same session, no") + print("cross_session_search):") + print(f" {REDLINE_INSTRUCTION[:200]}...") + print() + print("Expected verification result (against the source documents as authored):") + print(" FLAG expected : Article 7 (indemnification) -- one-directional in the base, never amended") + print(" FLAG expected : Article 6 (notice period) -- 5 business days < 11-day playbook threshold") + print(" NO FLAG expected: Article 8 (damages waiver) -- already mutual in the base") + print(" NO FLAG expected: Article 5 (payment terms) -- 45 days in the base (would violate on its") + print(" own) but Supplementary Conditions SC-1 amends it to 21 days, which is") + print(" compliant with the 23-day threshold -- this is the reconciliation check") + print(" NO FLAG expected: Article 9 (termination) -- 30 days >= 17-day threshold") + print() + print("API calls this would make for real: 5 uploads, 2 chat turns (+ approvals), 1 export.") + print("Re-run without --dry-run once this plan looks right.") + + +def _norm(s: str) -> str: + return re.sub(r"[_\-\s]+", " ", (s or "")).strip().lower() + + +def find_document_html(doc_list: dict, title_substring: str) -> str: + needle = _norm(title_substring) + for d in doc_list.get("documents", []): + if needle in _norm(d.get("title")): + html = d.get("html") + if not html: + raise ValueError(f"document matching '{title_substring}' found but has no html: {d}") + return html + raise ValueError(f"no open document matching '{title_substring}' -- got {doc_list}") + + +# ---------- verification ---------- + + +def verify(html: str) -> dict: + import re + + checks = { + "indemnification_flagged": ("Article 7", True), + "notice_period_flagged": ("Article 6", True), + "damages_waiver_flagged": ("Article 8", False), + "payment_terms_flagged": ("Article 5", False), + "termination_flagged": ("Article 9", False), + } + # Each Article's window is bounded by the *next* "Article N" heading (or end of + # document), not a fixed character count -- a fixed window can overrun into the + # next Article's own RISK FLAG and misattribute it, which a short, unflagged + # Article immediately followed by a flagged one will actually trigger. + all_article_starts = sorted(m.start() for m in re.finditer(r"Article\s+\d+", html, re.IGNORECASE)) + + results = {} + for check_name, (article, should_be_flagged) in checks.items(): + m = re.search(re.escape(article), html, re.IGNORECASE) + idx = m.start() if m else -1 + if idx == -1: + window = "" + else: + next_starts = [s for s in all_article_starts if s > idx] + end = next_starts[0] if next_starts else len(html) + window = html[idx:end] + has_flag = "RISK FLAG" in window + results[check_name] = { + "article_found": idx != -1, + "flagged": has_flag, + "expected_flagged": should_be_flagged, + "correct": has_flag == should_be_flagged, + } + + payment_shows_21 = bool(re.search(r"21\)?\s*day", html, re.IGNORECASE)) + payment_shows_stale_45 = bool(re.search(r"45\)?\s*day", html, re.IGNORECASE)) + results["reconciliation_applied"] = { + "shows_amended_21_days": payment_shows_21, + "still_shows_stale_45_days": payment_shows_stale_45, + "correct": payment_shows_21 and not payment_shows_stale_45, + } + + all_correct = all(r["correct"] for r in results.values()) + return {"pass": all_correct, "details": results} + + +# ---------- main ---------- + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--dry-run", action="store_true") + args = parser.parse_args() + + if args.dry_run: + print_dry_run() + return + + api_key = os.environ.get("SUPERDOCS_API_KEY") + if not api_key: + print("SUPERDOCS_API_KEY not set", file=sys.stderr) + sys.exit(1) + client = Client(api_key) + + # --- main session: open all five documents together, playbook included --- + main_session = f"redline-{uuid.uuid4()}" + log(f"main session: {main_session}") + for name, mode in [ + ("base_agreement.html", "replace"), + ("supplementary_conditions.html", "background"), + ("exhibit_a_scope.html", "background"), + ("exhibit_b_insurance.html", "background"), + ("risk_playbook.html", "background"), + ]: + client.upload_document(CONTENT_DIR / name, main_session, open_mode=mode) + log(f" opened {name} ({mode})") + + log("reconciling into one effective document") + job = client.start_chat(RECONCILE_INSTRUCTION, main_session, approval_mode="ask_every_time") + client.wait_for_job(main_session, job["job_id"], "reconciliation") + + log("redlining against the risk playbook (same-session document, no cross_session_search)") + job = client.start_chat(REDLINE_INSTRUCTION, main_session, approval_mode="ask_every_time") + client.wait_for_job(main_session, job["job_id"], "redline") + + docs = client.session_documents(main_session, include_html=True) + html = find_document_html(docs, "base_agreement") + + result = verify(html) + log("verification:") + for name, detail in result["details"].items(): + log(f" {name}: {json.dumps(detail)}") + log(f"OVERALL: {'PASS' if result['pass'] else 'FAIL'}") + + export_path = client.export_html(html, "reconciled_and_redlined_agreement", fmt="docx") + log(f"exported -> {export_path}") + + (OUTPUT_DIR / "final_document.html").write_text(html, encoding="utf-8") + (OUTPUT_DIR / "verification_result.json").write_text(json.dumps(result, indent=2), encoding="utf-8") + + if not result["pass"]: + sys.exit(1) + + +if __name__ == "__main__": + main() diff --git a/use-cases/shivansh193/owner-contractor-redline-workspace/content/base_agreement.html b/use-cases/shivansh193/owner-contractor-redline-workspace/content/base_agreement.html new file mode 100644 index 00000000..2daba2f2 --- /dev/null +++ b/use-cases/shivansh193/owner-contractor-redline-workspace/content/base_agreement.html @@ -0,0 +1,35 @@ +

Owner-Contractor Agreement

+

Synthetic document for demonstration purposes only. No real parties, project, or figures.

+

This Agreement is entered into as of February 10, 2026, by and between Riverside Medical Partners LLC ("Owner") and Meridian Builders LLC ("Contractor") for the project known as the Riverside Medical Office Renovation, located at 4180 Riverside Parkway, Suite 100, as further described in Exhibit A (Scope of Work).

+ +

ARTICLE 1 — THE CONTRACT DOCUMENTS

+

1.1 The Contract Documents consist of this Agreement, the Supplementary Conditions, Exhibit A (Scope of Work), Exhibit B (Insurance Requirements), the Drawings, and the Specifications.

+

1.2 Where the Supplementary Conditions modify a provision of this Agreement, the Supplementary Conditions govern as to that provision.

+ +

ARTICLE 2 — THE WORK

+

2.1 Contractor shall furnish all labor, materials, equipment, and services necessary to complete the Work described in Exhibit A.

+ +

ARTICLE 3 — DATE OF COMMENCEMENT AND SUBSTANTIAL COMPLETION

+

3.1 Contractor shall commence the Work on March 2, 2026, and shall achieve Substantial Completion no later than November 30, 2026.

+ +

ARTICLE 4 — CONTRACT SUM

+

4.1 Owner shall pay Contractor, for full and satisfactory performance of the Work, the sum of $2,140,000.00 (the "Contract Sum"), subject to Change Orders.

+ +

ARTICLE 5 — PAYMENTS

+

5.1 Contractor shall submit an Application for Payment monthly. Owner shall pay each undisputed Application for Payment within forty-five (45) days of receipt.

+

5.2 Retainage of ten percent (10%) shall be withheld from each payment and released upon Substantial Completion.

+ +

ARTICLE 6 — CLAIMS AND NOTICE

+

6.1 Any claim by either party arising out of or relating to this Agreement must be made by written notice to the other party within five (5) business days of the event giving rise to the claim, or the claim is waived.

+ +

ARTICLE 7 — INDEMNIFICATION

+

7.1 Contractor shall indemnify, defend, and hold harmless Owner from and against all claims, damages, losses, and expenses arising out of or resulting from performance of the Work, to the extent caused by the negligent acts or omissions of Contractor.

+ +

ARTICLE 8 — WAIVER OF CONSEQUENTIAL DAMAGES

+

8.1 Owner and Contractor mutually waive claims against each other for consequential damages arising out of or relating to this Agreement.

+ +

ARTICLE 9 — TERMINATION

+

9.1 Owner may terminate this Agreement for convenience upon thirty (30) days' written notice to Contractor, in which case Contractor shall be paid for Work properly performed through the date of termination plus reasonable demobilization costs.

+ +

ARTICLE 10 — MISCELLANEOUS PROVISIONS

+

10.1 This Agreement shall be governed by the laws of the State of Colorado.

diff --git a/use-cases/shivansh193/owner-contractor-redline-workspace/content/exhibit_a_scope.html b/use-cases/shivansh193/owner-contractor-redline-workspace/content/exhibit_a_scope.html new file mode 100644 index 00000000..6a7b8a07 --- /dev/null +++ b/use-cases/shivansh193/owner-contractor-redline-workspace/content/exhibit_a_scope.html @@ -0,0 +1,9 @@ +

Exhibit A — Scope of Work

+

Synthetic document for demonstration purposes only.

+

Referenced by Article 2.1 of the Owner-Contractor Agreement between Riverside Medical Partners LLC and Meridian Builders LLC.

+ diff --git a/use-cases/shivansh193/owner-contractor-redline-workspace/content/exhibit_b_insurance.html b/use-cases/shivansh193/owner-contractor-redline-workspace/content/exhibit_b_insurance.html new file mode 100644 index 00000000..37853b59 --- /dev/null +++ b/use-cases/shivansh193/owner-contractor-redline-workspace/content/exhibit_b_insurance.html @@ -0,0 +1,8 @@ +

Exhibit B — Insurance Requirements

+

Synthetic document for demonstration purposes only.

+

Referenced by Article 1.1 of the Owner-Contractor Agreement between Riverside Medical Partners LLC and Meridian Builders LLC.

+ diff --git a/use-cases/shivansh193/owner-contractor-redline-workspace/content/risk_playbook.html b/use-cases/shivansh193/owner-contractor-redline-workspace/content/risk_playbook.html new file mode 100644 index 00000000..54b3bc22 --- /dev/null +++ b/use-cases/shivansh193/owner-contractor-redline-workspace/content/risk_playbook.html @@ -0,0 +1,9 @@ +

Meridian Builders — Owner Contract Risk Playbook

+

Internal risk review checklist. Deliberately specific, non-"standard" thresholds below (not the usual round-number industry defaults) so that correctly applying them proves the reviewer actually consulted this playbook, not generic contract knowledge.

+
    +
  1. Indemnification — Flag if the indemnification obligation is one-directional (only the Contractor indemnifies the Owner). Require mutual indemnification, each party for its own negligent acts.
    2. +
  2. Waiver of consequential damages — Flag if there is no mutual waiver of consequential damages between Owner and Contractor.
    3. +
  3. Claims and notice periods — Flag if either party has fewer than eleven (11) business days to give written notice of a claim.
    4. +
  4. Payment terms — Flag if the Owner's payment period for an undisputed Application for Payment exceeds twenty-three (23) days.
    5. +
  5. Termination for convenience — Flag if the Owner's termination-for-convenience notice period is less than seventeen (17) days.
    6. +
diff --git a/use-cases/shivansh193/owner-contractor-redline-workspace/content/supplementary_conditions.html b/use-cases/shivansh193/owner-contractor-redline-workspace/content/supplementary_conditions.html new file mode 100644 index 00000000..8ad2abe8 --- /dev/null +++ b/use-cases/shivansh193/owner-contractor-redline-workspace/content/supplementary_conditions.html @@ -0,0 +1,12 @@ +

Supplementary Conditions

+

Synthetic document for demonstration purposes only.

+

These Supplementary Conditions modify the Owner-Contractor Agreement between Riverside Medical Partners LLC ("Owner") and Meridian Builders LLC ("Contractor") for the Riverside Medical Office Renovation. Where a provision below conflicts with the Agreement, this document governs as to that provision, per Article 1.2 of the Agreement.

+ +

SC-1 — AMENDMENT TO ARTICLE 5 (PAYMENTS)

+

SC-1.1 Article 5.1 of the Agreement is hereby amended: the forty-five (45) day payment period is deleted and replaced with twenty-one (21) days of receipt of an undisputed Application for Payment. All other provisions of Article 5 remain unchanged.

+ +

SC-2 — SUBMITTAL SCHEDULE

+

SC-2.1 Contractor shall submit a submittal schedule to Owner within fifteen (15) days of the commencement date stated in Article 3.1 of the Agreement.

+ +

SC-3 — SITE ACCESS

+

SC-3.1 The Riverside Medical Office building remains partially occupied during the Work. Contractor shall coordinate all noise-generating activity with the Owner's facility manager at least 48 hours in advance.

diff --git a/use-cases/shivansh193/owner-contractor-redline-workspace/requirements.txt b/use-cases/shivansh193/owner-contractor-redline-workspace/requirements.txt new file mode 100644 index 00000000..7507eb03 --- /dev/null +++ b/use-cases/shivansh193/owner-contractor-redline-workspace/requirements.txt @@ -0,0 +1,2 @@ +httpx>=0.27 +python-dotenv>=1.0 diff --git a/use-cases/shivansh193/self-healing-structure-agent/.env.example b/use-cases/shivansh193/self-healing-structure-agent/.env.example new file mode 100644 index 00000000..f611a7f0 --- /dev/null +++ b/use-cases/shivansh193/self-healing-structure-agent/.env.example @@ -0,0 +1 @@ +SUPERDOCS_API_KEY=your-key-here diff --git a/use-cases/shivansh193/self-healing-structure-agent/.gitignore b/use-cases/shivansh193/self-healing-structure-agent/.gitignore new file mode 100644 index 00000000..0703dfd8 --- /dev/null +++ b/use-cases/shivansh193/self-healing-structure-agent/.gitignore @@ -0,0 +1,5 @@ +.env +output/ +__pycache__/ +*.pyc +.venv/ diff --git a/use-cases/shivansh193/self-healing-structure-agent/PROGRESS.md b/use-cases/shivansh193/self-healing-structure-agent/PROGRESS.md new file mode 100644 index 00000000..df9f2df6 --- /dev/null +++ b/use-cases/shivansh193/self-healing-structure-agent/PROGRESS.md @@ -0,0 +1,210 @@ +# Progress log -- Self-Healing Document Agent for Structure and Numbering + +## Before any API calls: validated `verify()` against known ground truth + +Given the two self-inflicted bugs found in `verify()` on the redline-workspace +build (case-sensitivity, a regex that didn't tolerate a parenthesis), this +build's `verify()` was checked both directions before spending anything: + +- Run against the known-broken `content/manual.html` as authored -> all 8 + checks correctly reported `false`. +- Run against a hand-repaired copy (all 10 headings renumbered, both + cross-refs fixed, TOC repaired -- built with `sed`, not through the API) + -> all 8 checks correctly reported `true`. + +One real bug caught this way: the Table-of-Contents region regex used +`re.IGNORECASE`, so its own boundary pattern (`SECTION 1 --`) matched the +TOC's *own* first entry ("Section 1 -- Introduction and Scope") instead of +the real ALL-CAPS heading below it, collapsing the captured TOC region to +nothing. Fixed by making that one boundary match case-sensitive (body +headings are ALL CAPS, TOC entries are title case -- that distinction is +exactly what makes the boundary work once it's not case-blind). Caught for +free, before the first real API call. + +## Design note: no `cross_session_search` used anywhere in this build + +The redline-workspace build (sibling folder, same session) found that +`cross_session_search: true` can cause SuperDocs to silently re-open a +stale snapshot of a document already open and edited in the current +session. This build's task -- renumber, fix cross-refs, fix a TOC -- never +needs data from another session, so it structurally can't hit that bug: +one document, one session, three sequential same-document chat turns. + +## Run 1: renumbering and cross-refs correct, TOC silently skipped + +Real run against the live API. Result: 6 of 8 checks passed. + +- `headings_sequential_1_to_10`: **PASS** -- all ten headings renumbered + correctly, in order, titles untouched. +- Both cross-reference checks: **PASS** -- confidentiality reference + correctly updated to Section 8, termination reference to Section 9. +- All three TOC checks and the TOC entry-count check: **FAIL** -- the TOC + region was byte-for-byte identical to the original broken input. + +Diagnosed via the job's own response text (already-paid-for data, no +extra API cost): the second chat turn -- which had asked for two things +in one instruction, "fix the two cross-refs" *and* "fix the TOC" -- came +back with `"Successfully updated all 2 sections"` and exactly 2 changes +in its diff, both of them the cross-ref edits. The agent didn't attempt +the TOC part and fail on it; it silently redefined the task down to only +the part it planned for, then reported full, unqualified success on that +narrowed scope. + +This is the same failure class already diagnosed on the redline-workspace +build's Run 1: an instruction bundling two distinct sub-tasks into one +turn gets silently truncated to one of them, while the job still reports +`completed` with no error and a response that sounds like full success if +you don't check exactly what it claims to have updated ("all 2 sections" +undersells that 2 was never the whole ask). + +## Fix: split into two single-purpose turns instead of one bundled turn + +This is a verified fix pattern, not a guess -- the same narrowing (one +instruction, one job to do) already proved reliable for the reconcile +step on the redline-workspace build, across two separate real runs. +Replaced `CROSSREF_TOC_INSTRUCTION` with two instructions, +`CROSSREF_INSTRUCTION` and `TOC_INSTRUCTION`, run as two sequential chat +turns instead of one. Each instruction now also explicitly names what +*not* to touch, and `TOC_INSTRUCTION` states the expected end-state count +("there must be exactly ten Table of Contents entries") so a silently +narrowed interpretation has a concrete number to fall short of, not just +a qualitative goal. + +Proceeded straight to a second real run without checking in: the root +cause was specific and already independently confirmed by a working +comparison case in the sibling build, the fix directly targets that root +cause, and the incremental cost is one additional chat turn (~1 op) +against a 10,000-op promo grant with roughly 9,985 remaining at this +point -- a verified next step, not a speculative retry. + +## Run 2: the identical renumber instruction that worked cleanly in Run 1 +## produced a different, three-layered failure this time + +Result: 0 of 8 checks passed -- worse than Run 1, and wrong in ways Run 1 +never was. Pulled `GET /v1/sessions/{id}/jobs` (free, already-paid data) +and diffed all three jobs chronologically against their own reported +changes. What actually happened, in order: + +**Turn 1 (renumber) -- claimed full success, made almost no real +progress, and edited things it was told not to.** Response: `"✅ +Successfully updated all 10 sections."` Its own 10-change diff tells a +different story: 9 of the 10 changes were unrequested edits to the Table +of Contents -- capitalizing "Section" to "SECTION" in every TOC line +(RENUMBER_INSTRUCTION explicitly says "Do not touch the Table of Contents +... in this step"), incidentally fixing one TOC number as a side effect. +The 10th change touched exactly one real Section heading -- and instead +of changing its *number* (the entire ask), it left the number at 10 and +silently rewrote the heading's *title* from `TERMINATION` to +`Miscellaneous Provisions`, while that heading's own body paragraph +(`10.1 Employment may be terminated...`) stayed the original Termination +text -- creating a heading/body mismatch that didn't exist in the source +document at all. None of the other 9 Section headings were touched. So: +a confident, specific, false claim of complete success, covering an +instruction that was executed almost 0% correctly on its actual target +and violated its own explicit "don't touch this" constraint. + +**Turn 2 (crossref) -- trusted turn 1's false claim instead of checking +ground truth, made zero edits.** Response: `"The section numbers 'Section +9' and 'Section 10' are already correct following the renumbering of the +manual to 1 through 10. No edits were required."` This is wrong: nothing +had been renumbered (see above), and the cross-refs still said "9" and +"10" only because they'd never been touched -- coincidentally the same +literal digits as the stale, unfixed heading labels. The turn reasoned +from turn 1's claimed outcome rather than the document's actual state and +concluded, confidently and explicitly, that no work was needed. + +**Turn 3 (TOC) -- deleted the entire literal Table of Contents and +replaced it with an empty auto-generated widget.** Instead of editing the +nine `

` paragraphs as literal text (which is exactly what +they are -- plain HTML I authored), this turn's diff shows a `delete` of +the whole `

TABLE OF CONTENTS

` block plus all TOC paragraphs, and +a `create` of `
` -- an +empty placeholder for what looks like a SuperDocs-native live-TOC +feature. Whatever renders that widget doesn't populate it in the raw HTML +this build reads back via the API, so the exported document's TOC region +is now completely empty: zero entries, not nine, not ten. + +**Why this is a different finding from Run 1's, not a repeat of it.** The +RENUMBER_INSTRUCTION text was byte-identical between Run 1 and Run 2, and +Run 1 executed it cleanly -- all 10 headings correctly renumbered, no +scope violations, no false claims. Run 2, same instruction, same +document, produced three independent kinds of wrong: a false-success +claim covering near-total non-execution, a downstream turn trusting that +false claim instead of the real document, and an unrequested content +substitution (literal text -> a live-TOC widget) that this build has no +way to verify through the HTML API regardless of instruction wording. +That spread, from one identical input, points to real run-to-run +non-determinism in how these structural edit requests get executed, not +a wording problem this build's instructions can reliably fix. + +**Decision: stop, don't spend a third run's ops on a re-roll.** The two +earlier fixes (this build's split-instruction fix, and the +redline-workspace build's cross_session_search fix) each targeted a +specific, identified mechanism and were reasonable to expect to work. +A third attempt here would not be that -- Run 1 already proves the exact +same instruction *can* work, so a third run offers no new lever to pull, +only a chance the non-determinism lands favorably again. That is a guess +against operations budget, not a verified next step, which is exactly +where the standing instruction says to stop rather than continue. +`output/verification_result.json` and `output/final_document.html` are +left as Run 2 produced them -- an accurate record of the failure, not +patched over. + +## 2026-08-21 -- a verify-then-retry wrapper around the one non-deterministic step + +Everything above stays as it happened. This is a third run, with a +different mitigation targeted specifically at the non-determinism, not a +replacement for the earlier two. + +The renumber step is the one that showed non-determinism. Wrapped just +that step in a loop: each attempt uploads a *fresh* copy of the source +document into a *fresh* session (not a follow-up turn on a half-broken +document -- a genuinely independent attempt, matching how the original +two runs were also independent), sends the same `RENUMBER_INSTRUCTION`, +and checks the result against ground truth with a new `verify_headings()` +function (numbers 1-10 in order, titles unchanged) before deciding +whether to keep it or discard and retry, up to `MAX_RENUMBER_ATTEMPTS = 3`. +The cross-reference and Table of Contents turns are unchanged and stay +single-shot -- the non-determinism only ever showed up in renumbering. + +`verify_headings()` was checked against known ground truth before spending +any API calls on it, same discipline as the original `verify()`: run +against a hand-repaired copy (all correct) and the original broken source +(all incorrect), both came back as expected. + +**Result: converged immediately, attempt 1 of 3.** Real run against the +live API: `attempt 1 numbers: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] correct: True`. +No retry was needed this time -- which is itself real information, not a +non-event: it means the instruction *can* succeed reliably when nothing +else changes, consistent with Run 1's original clean pass. The retry +wrapper's actual value would show up on a run where the first attempt +fails and a later one succeeds, or where all three fail and that gets +reported honestly instead of silently retried into oblivion; this run +didn't need to exercise that path, but the mechanism is now in place and +its per-attempt log (`output/renumber_attempts.json`) makes every +attempt's real outcome inspectable regardless of which path a given run +takes. + +**The Table of Contents step failed again, the same way as before, now +confirmed twice.** Cross-refs and headings this run: clean. The TOC step, +asked to fix the same three planted defects, made no edit at all to any +of the nine existing `

` lines -- same stale title, same +duplicated/wrong number, same missing tenth entry as the unmodified +source. Instead it inserted an empty `

` widget in the middle of the list (this time between +the first and second entries; last time, replacing the whole block). This +happened on the only two real runs that got far enough to reach this +step, with different instruction wording each time (bundled, then +single-shot) -- reasonable evidence this is a specific, reproducible +platform behavior (asking to edit a literal, hand-authored Table of +Contents triggers a native "live TOC" substitution instead) rather than a +fluke. Left as-is per the current priority order, which scoped the retry +wrapper to renumbering only -- fixing the TOC step is separate, +not-yet-scoped work. + +**Final result: FAIL, 4 of 8** (`headings_sequential_1_to_10`, +`titles_unchanged_and_in_order`, both crossref checks: PASS; all four TOC +checks: FAIL). `output/repaired_manual.docx`, +`output/verification_result.json` (now also carries `renumber_attempts` +and `renumber_converged`), and `output/renumber_attempts.json` all +reflect this run. diff --git a/use-cases/shivansh193/self-healing-structure-agent/README.md b/use-cases/shivansh193/self-healing-structure-agent/README.md new file mode 100644 index 00000000..09a0ce23 --- /dev/null +++ b/use-cases/shivansh193/self-healing-structure-agent/README.md @@ -0,0 +1,179 @@ +# Self-Healing Document Agent for Structure and Numbering + +Built by Shivansh Kalra for the SuperDocs task. + +Takes a document whose internal structure has drifted -- Section numbers +with a gap, a duplicate, and a run past the real count; two body +cross-references that point at the wrong Section by number; a Table of +Contents that's stale in three independent ways (a wrong number, a wrong +title, a missing entry) -- and asks the real, hosted SuperDocs product to +repair all three problem classes against a fully known, exact ground +truth (all ten Sections are already in the correct reading order, so the +correct final number for each one is just its position). + +All content is synthetic: a fictional company (NorthPeak Logistics) and a +fictional driver safety manual. + +Deliberately a single document, single session, with no +`cross_session_search` anywhere -- the sibling +[owner-contractor-redline-workspace](../owner-contractor-redline-workspace/) +build found that `cross_session_search` can cause SuperDocs to silently +re-open a stale snapshot of a document already open in the current +session. This build's task never needs data from another session, so it +can't hit that specific bug -- and, as it turned out, still surfaced a +different, real problem on its own. + +## What it does + +1. Uploads the broken manual to a session. +2. **Renumber step**: asks SuperDocs to renumber the ten Section headings + sequentially 1-10, in the order they already appear, without touching + titles, body text, or the Table of Contents. Wrapped in a + verify-then-retry loop (up to 3 attempts, each from a fresh session and + a fresh copy of the source) since this exact instruction showed real + run-to-run non-determinism -- see [Verified result](#verified-result-fail-and-a-genuinely-interesting-one). +3. **Cross-reference step**: asks SuperDocs to find two body sentences + that reference another Section by number and correct each number to + match its target Section's new, corrected number. +4. **Table of Contents step**: asks SuperDocs to bring every Table of + Contents entry's number and title in line with its Section, and add + an entry for the one Section that has none. +5. Exports the result and verifies it programmatically against 8 checks + by inspecting the real returned HTML -- not asserted, checked. + +`verify()` was validated in both directions before any real API call: +run against the known-broken source (all 8 checks correctly `false`) and +against a hand-repaired copy built with `sed`, not the API (all 8 +correctly `true`). One real bug in the verification script itself was +caught this way, for free: a case-insensitive regex boundary was matching +the Table of Contents' own first entry instead of the real heading below +it, collapsing the captured TOC region to nothing. Fixed before spending +anything. + +## How to run it + +```bash +python -m venv .venv +.venv/Scripts/activate # or source .venv/bin/activate on macOS/Linux +pip install -r requirements.txt +cp .env.example .env # then set SUPERDOCS_API_KEY +python build.py --dry-run # prints the full plan, zero API calls +python build.py # runs it for real: 1 upload, 3 chat turns, 1 export +``` + +## SuperDocs features used + +- **Chat / async edit** (`POST /v1/chat/async`) with + `approval_mode: "ask_every_time"` across three sequential instructions + on the same document, in the same session +- **Export** (`POST /v1/documents/export`, `.docx`) +- **Job introspection** (`GET /v1/sessions/{id}/jobs`) -- used to + chronologically diff all three turns' own reported changes against + what they actually claimed, which is how the real finding below was + caught + +## Verified result: FAIL, and a genuinely interesting one + +Two real runs against the live API, same instructions both times. + +**Run 1**: 6 of 8 checks passed. Renumbering and both cross-reference +fixes landed correctly. The Table of Contents fix was silently dropped -- +the step's own response said `"Successfully updated all 2 sections"`, +meaning it had quietly narrowed a two-part instruction (fix cross-refs +*and* fix the TOC) down to just the first part while still reporting full +success. Diagnosed via the job's own response text, no extra API cost. +Fixed by splitting into two single-purpose turns instead of one bundled +one -- the same narrowing pattern already proven reliable on the +redline-workspace build. + +**Run 2**, same split instructions, same document: 0 of 8 checks passed +-- and wrong in ways Run 1 never was. The renumber turn claimed +`"Successfully updated all 10 sections"` while making almost no real +progress: 9 of its 10 changes were unrequested Table-of-Contents edits +(explicitly out of scope for that step), and the one change that touched +an actual Section heading left its number unchanged and instead silently +rewrote its *title* (`TERMINATION` -> `Miscellaneous Provisions`), leaving +that heading's body paragraph as the original Termination text -- a +mismatch that didn't exist in the source. The cross-reference turn then +trusted that false claim rather than checking the real document, decided +"no edits were required," and made none. The Table of Contents turn +deleted the entire hand-authored TOC and replaced it with an empty +`
` placeholder -- apparently a native live-TOC feature -- +leaving zero literal entries where nine had been. + +The renumber instruction's text was byte-identical between the two runs. +One execution was clean; the other was wrong in three independent, +compounding ways. That's evidence of real run-to-run non-determinism in +how SuperDocs executes structural edit requests, not something this +build's instruction wording controls -- so a third run wasn't attempted: +Run 1 already proves the same instruction *can* succeed, meaning a third +attempt would be spending operations on a re-roll with no new diagnostic +basis, not a verified fix. Full turn-by-turn diagnosis, including the +exact job diffs, is in [`PROGRESS.md`](PROGRESS.md). + +| Check | Run 1 | Run 2 | Run 3 (retry wrapper) | +|---|---|---|---| +| Headings renumbered 1-10 sequentially | PASS | FAIL | PASS | +| Titles unchanged and in order | PASS | FAIL | PASS | +| Confidentiality cross-ref -> Section 8 | PASS | FAIL | PASS | +| Termination cross-ref -> Section 9 | PASS | FAIL | PASS | +| TOC: stale title fixed | FAIL | FAIL | FAIL | +| TOC: stale number fixed | FAIL | FAIL | FAIL | +| TOC: missing entry added | FAIL | FAIL | FAIL | +| TOC: exactly 10 entries | FAIL | FAIL | FAIL | +| **Overall** | **FAIL (6/8)** | **FAIL (0/8)** | **FAIL (4/8)** | + +### Run 3: a verify-then-retry wrapper around the one non-deterministic step + +Run 1 and Run 2 used the byte-identical renumber instruction and got very +different results -- real non-determinism, not a wording problem. Run 3 +wraps just that step (the only one that showed non-determinism) in a +verify-then-retry loop: each attempt starts from a fresh session and a +fresh copy of the source document, and the result gets checked against +ground truth before deciding whether to keep it or discard and try again, +up to 3 attempts. Cross-refs and TOC are unchanged, single-shot turns. + +It converged on the first attempt: `[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]`, +correct, no retry needed. That's a real result, not a non-event -- Run 1 +already showed the instruction can succeed cleanly; Run 3 confirms it +again under the same fresh-session conditions. Every attempt's real +outcome is logged to `output/renumber_attempts.json` regardless of +whether a given run needs 1 attempt or all 3. + +The Table of Contents step failed again, the same way as Run 2: no edits +to any of the nine existing TOC lines, and an empty auto-generated +`
` widget inserted instead. Two real runs +that reached this step, two different instruction phrasings, the same +outcome both times -- this now reads as a specific, reproducible platform +behavior around editing a literal, hand-authored Table of Contents, +not a fluke. Out of scope for this pass (the retry wrapper was scoped to +renumbering only); fixing the TOC step is separate, not-yet-scoped work. + +## Honest limitations + +- Single-shot structural repair (renumbering specifically) was not + reliable run-to-run against the live API, based on two identical + attempts producing very different outcomes. The retry wrapper (Run 3) + mitigates this for renumbering by discarding a failed attempt and + trying again fresh, up to 3 times -- but that's a mitigation, not proof + the underlying non-determinism is gone; a run where all 3 attempts fail + is still possible and would be reported honestly if it happened. +- The Table of Contents step still fails, the same way, on both real runs + that reached it: given a literal, hand-authored TOC to edit, it replaces + the whole thing with an empty auto-generated widget rather than editing + the existing text -- something this build's HTML-based verification has + no way to see through, and not yet mitigated (out of scope for the + retry-wrapper pass). +- `output/` is gitignored; run `python build.py` to regenerate + `repaired_manual.docx`, `final_document.html`, + `verification_result.json` (now also carries the renumber retry log), + and `renumber_attempts.json`. + +## Files + +- `build.py` -- upload -> renumber (verify-then-retry, up to 3 attempts) + -> fix cross-refs -> fix TOC -> verify -> export flow, plus `--dry-run` +- `content/manual.html` -- the driver safety manual, authored with the + three planted structural defects described above +- `PROGRESS.md` -- full diagnostic trace of both runs, including the + exact job diffs behind both findings diff --git a/use-cases/shivansh193/self-healing-structure-agent/build.py b/use-cases/shivansh193/self-healing-structure-agent/build.py new file mode 100644 index 00000000..e83d2bad --- /dev/null +++ b/use-cases/shivansh193/self-healing-structure-agent/build.py @@ -0,0 +1,430 @@ +"""Self-Healing Document Agent for Structure and Numbering -- built against +the real, hosted SuperDocs product. Takes a document whose Section +numbering has drifted (a skipped number, a duplicated number, numbers that +run past the actual Section count), whose body text contains two +cross-references pointing at the wrong Section numbers, and whose Table of +Contents is stale in three independent ways (a wrong number, a wrong +title, a missing entry) -- and repairs all three problem classes. + +Deliberately a single document, single session, no cross_session_search: +the redline-workspace build (own-folder sibling to this one) found that +cross_session_search can cause SuperDocs to silently re-open a stale +snapshot of a document already open in the current session. This build's +task doesn't need cross-session data at all, so it structurally can't hit +that bug -- two narrow, sequential same-document instructions instead, +matching the instruction style that was proven reliable there (targeted +and procedural, not open-ended). + +Ground truth: the manual has exactly 10 Sections in document order. +Renumbered correctly, each Section's number must equal its position +(1st Section heading -> "SECTION 1", ..., 10th -> "SECTION 10"), because +they're already in the right order -- only the numbers are wrong. That +makes verification exact rather than approximate: the correct final state +is fully known in advance, not just "plausible." + +Two real runs of the byte-identical RENUMBER_INSTRUCTION produced very +different outcomes (see PROGRESS.md): one clean pass, one run with a false +"updated all 10 sections" claim covering near-zero real progress. That's +real run-to-run non-determinism, not a wording problem -- so this version +wraps the renumber turn specifically in a verify-then-retry loop: run it +against a fresh session and a fresh copy of the source document, check the +actual resulting headings against ground truth, and if it doesn't match, +throw the attempt away and try again from scratch, up to +MAX_RENUMBER_ATTEMPTS times. Every attempt's real outcome is logged, +whether or not retrying converges to a pass -- both are real information. +The cross-reference and Table of Contents turns are left single-shot; the +non-determinism only showed up in renumbering. + +Run `python build.py --dry-run` first: prints the full plan with zero API +calls. Only run for real (`python build.py`) after reading that output. +""" + +import argparse +import json +import os +import re +import sys +import time +import uuid +from pathlib import Path + +import httpx +from dotenv import load_dotenv + +load_dotenv() + +BASE_URL = "https://api.superdocs.app" +HERE = Path(__file__).parent +CONTENT_DIR = HERE / "content" +OUTPUT_DIR = HERE / "output" +OUTPUT_DIR.mkdir(exist_ok=True) + +MAX_RENUMBER_ATTEMPTS = 3 + + +def log(msg: str) -> None: + print(f"[{time.strftime('%H:%M:%S')}] {msg}", flush=True) + + +# ---------- API helpers (same shape as the redline-workspace build) ---------- + + +class Client: + def __init__(self, api_key: str): + self.http = httpx.Client(base_url=BASE_URL, headers={"Authorization": f"Bearer {api_key}"}, timeout=240.0) + + def upload_document(self, path: Path, session_id: str, open_mode: str = "replace") -> dict: + with open(path, "rb") as f: + resp = self.http.post( + "/v1/documents/upload", + files={"file": (path.name, f, "text/html")}, + data={"session_id": session_id, "open_mode": open_mode}, + ) + resp.raise_for_status() + return resp.json() + + def start_chat(self, message: str, session_id: str, approval_mode: str = "ask_every_time") -> dict: + resp = self.http.post( + "/v1/chat/async", + json={"message": message, "session_id": session_id, "approval_mode": approval_mode}, + ) + resp.raise_for_status() + return resp.json() + + def get_job(self, job_id: str) -> dict: + resp = self.http.get(f"/v1/jobs/{job_id}") + resp.raise_for_status() + return resp.json() + + def approve_all(self, session_id: str, job_id: str, pending_changes: list[dict]) -> None: + changes = [{"change_id": c["change_id"], "approved": True} for c in pending_changes] + resp = self.http.post(f"/v1/chat/{session_id}/approve", json={"job_id": job_id, "approved": True, "changes": changes}) + resp.raise_for_status() + + def continue_job(self, session_id: str, job_id: str) -> None: + resp = self.http.post(f"/v1/chat/{session_id}/continue", json={"job_id": job_id, "continue": True}) + resp.raise_for_status() + + def wait_for_job(self, session_id: str, job_id: str, label: str, max_wait_s: int = 400) -> dict: + start = time.time() + while time.time() - start < max_wait_s: + job = self.get_job(job_id) + status = job["status"] + if status == "completed": + log(f" {label}: completed") + return job + if status in ("failed", "cancelled"): + raise RuntimeError(f"{label} job {status}: {job.get('error')}") + if status == "awaiting_approval": + metadata = job.get("metadata") or {} + if metadata.get("awaiting_kind") == "continue_prompt": + log(f" {label}: paused mid-edit, continuing") + self.continue_job(session_id, job_id) + else: + pending = metadata.get("pending_changes") or [] + log(f" {label}: awaiting approval on {len(pending)} change(s) -- approving") + self.approve_all(session_id, job_id, pending) + else: + log(f" {label}: {status}...") + time.sleep(4) + raise TimeoutError(f"{label} job did not complete in time") + + def session_documents(self, session_id: str, include_html: bool = True) -> dict: + resp = self.http.get(f"/v1/sessions/{session_id}/documents", params={"include_html": str(include_html).lower()}) + resp.raise_for_status() + return resp.json() + + def export_html(self, html: str, filename: str, fmt: str = "docx") -> Path: + resp = self.http.post("/v1/documents/export", json={"html": html, "format": fmt, "options": {"filename": filename}}) + resp.raise_for_status() + ext = {"docx": "docx", "pdf": "pdf", "html": "html"}.get(fmt, fmt) + out_path = OUTPUT_DIR / f"{filename}.{ext}" + if "application/json" in resp.headers.get("content-type", ""): + data = resp.json() + url = data.get("download_url") or data.get("url") + out_path.write_bytes(self.http.get(url).content) + else: + out_path.write_bytes(resp.content) + return out_path + + +def _norm(s: str) -> str: + return re.sub(r"[_\-\s]+", " ", (s or "")).strip().lower() + + +def find_document_html(doc_list: dict, title_substring: str) -> str: + needle = _norm(title_substring) + for d in doc_list.get("documents", []): + if needle in _norm(d.get("title")): + html = d.get("html") + if not html: + raise ValueError(f"document matching '{title_substring}' found but has no html: {d}") + return html + raise ValueError(f"no open document matching '{title_substring}' -- got {doc_list}") + + +# ---------- the plan ---------- + +RENUMBER_INSTRUCTION = ( + "This document has ten Section headings (each one looks like 'SECTION — '), " + "already in the correct reading order from top to bottom, but their numbers are wrong -- some " + "are skipped, one number is used twice, and the numbers run past ten even though there are only " + "ten Sections. Go through the Section headings in top-to-bottom order and renumber them " + "sequentially: the first heading becomes 'SECTION 1', the second becomes 'SECTION 2', and so on " + "through 'SECTION 10' for the tenth and last one. Keep each heading's title text exactly as it " + "is now -- only change the number. Do not touch the Table of Contents or any body paragraph " + "text in this step." +) + +CROSSREF_INSTRUCTION = ( + "This document's Section numbers were just corrected so they now run 1 through 10 in order. " + "In the body text there are two sentences that reference another Section by number, written " + "like 'Section <number> of this Manual'. For each one, work out which Section it is actually " + "describing -- one refers to where confidentiality obligations for incident records are set " + "out, the other refers to where termination decisions are processed -- and update its number " + "to match that target Section's new, corrected number. Do not change anything else in the " + "document -- not the Table of Contents, not any heading, nothing else in the body text." +) + +TOC_INSTRUCTION = ( + "This document's Section numbers were just corrected so they now run 1 through 10 in order. " + "This document has a Table of Contents near the top, listing Sections by number and title. " + "Compare every Table of Contents entry against the Section it refers to: fix any entry whose " + "listed number no longer matches that Section's corrected number, fix any entry whose listed " + "title text no longer matches that Section's actual current title, and add a Table of Contents " + "entry for any Section that doesn't have one yet, in its correct position in the list. When you " + "are done there must be exactly ten Table of Contents entries, one per Section, in order. Do " + "not change anything else in the document -- not any heading, not any body paragraph text." +) + + +def print_dry_run() -> None: + print("=== DRY RUN -- no API calls will be made ===\n") + print(f"Document that would be uploaded to a single session: {CONTENT_DIR / 'manual.html'}") + print() + print("Known-broken structure as authored:") + print(" Section heading numbers in document order: 1, 2, 4, 5, 6, 6, 7, 9, 10, 11") + print(" (gap at 3, duplicate 6, gap at 8, runs to 11 instead of stopping at 10)") + print(" Body cross-ref in Section 'Incident Reporting': cites Section 9 for Confidentiality") + print(" -> Confidentiality is the 8th heading in order, so correct final number is 8") + print(" Body cross-ref in Section 'Disciplinary Actions': cites Section 10 for Termination") + print(" -> Termination is the 9th heading in order, so correct final number is 9") + print(" TOC entry for the 3rd Section: number correct (3), title stale ('Cargo Inspection") + print(" Requirements' instead of 'Vehicle Inspection Requirements')") + print(" TOC entry for 'Drug and Alcohol Policy': listed as Section 5 (duplicate of Incident") + print(" Reporting's entry), correct final number is 6") + print(" TOC: no entry at all for the 10th Section ('Miscellaneous Provisions')") + print() + print(f"Chat instruction 1 (renumber headings only, no document_id set), retried up to") + print(f"{MAX_RENUMBER_ATTEMPTS} times against a fresh session + fresh document each attempt,") + print("verified against ground truth after every attempt, since this exact instruction produced") + print("two very different real outcomes on two identical prior runs (see PROGRESS.md):") + print(f" {RENUMBER_INSTRUCTION[:200]}...") + print() + print("Chat instruction 2 (fix the two body cross-refs against the corrected numbers, only),") + print("single-shot, run once against whichever session's renumber attempt succeeded (or the") + print("last attempt, if none did):") + print(f" {CROSSREF_INSTRUCTION[:200]}...") + print() + print("Chat instruction 3 (fix the Table of Contents against the corrected numbers, only),") + print("single-shot:") + print(f" {TOC_INSTRUCTION[:200]}...") + print() + print("Split into three narrow, single-purpose turns rather than two: an earlier run bundled") + print("the cross-ref fix and the TOC fix into one instruction, and the agent silently completed") + print("only the cross-ref half while reporting full success -- see PROGRESS.md.") + print() + print("Expected final state: headings numbered 1-10 sequentially in order; both cross-refs") + print("updated (8 and 9 respectively); TOC has 10 correct entries, no stale title, no stale") + print("number, no missing entry.") + print() + print(f"API calls this would make for real: 1-{MAX_RENUMBER_ATTEMPTS} uploads + renumber turns") + print("(1 per attempt, until one verifies correct or the cap is hit), plus 2 more chat turns") + print("(crossref, TOC) and 1 export. No cross_session_search used anywhere.") + print("Re-run without --dry-run once this plan looks right.") + + +# ---------- verification ---------- + +SECTION_TITLES_IN_ORDER = [ + "INTRODUCTION AND SCOPE", + "DEFINITIONS", + "VEHICLE INSPECTION REQUIREMENTS", + "HOURS OF SERVICE", + "INCIDENT REPORTING", + "DRUG AND ALCOHOL POLICY", + "DISCIPLINARY ACTIONS", + "CONFIDENTIALITY", + "TERMINATION", + "MISCELLANEOUS PROVISIONS", +] + + +def verify_headings(html: str) -> dict: + """Narrow check used by the renumber retry loop: just the heading numbers + and titles, not cross-refs or TOC (those haven't run yet at this point).""" + headings = re.findall(r"SECTION\s+(\d+)\s*[—\-]\s*([A-Z ,&]+?)(?:</h\d>|\n)", html) + heading_numbers = [int(n) for n, _ in headings] + found_titles = [t.strip().rstrip(".") for _, t in headings] + expected_numbers = list(range(1, 11)) + numbers_correct = heading_numbers == expected_numbers + titles_correct = len(found_titles) == 10 and all( + SECTION_TITLES_IN_ORDER[i] in found_titles[i] for i in range(min(10, len(found_titles))) + ) + return { + "found_numbers": heading_numbers, + "found_titles": found_titles, + "numbers_correct": numbers_correct, + "titles_correct": titles_correct, + "correct": numbers_correct and titles_correct, + } + + +def verify(html: str) -> dict: + results = {} + + # 1. Heading sequence: every "SECTION <n> — <TITLE>" heading, in document order. + headings = re.findall(r"SECTION\s+(\d+)\s*[—\-]\s*([A-Z ,&]+?)(?:</h\d>|\n)", html) + heading_numbers = [int(n) for n, _ in headings] + expected_numbers = list(range(1, 11)) + results["headings_sequential_1_to_10"] = { + "found": heading_numbers, + "expected": expected_numbers, + "correct": heading_numbers == expected_numbers, + } + + # 2. Titles still in the same order and intact (renumbering shouldn't have touched titles). + found_titles = [t.strip().rstrip(".") for _, t in headings] + results["titles_unchanged_and_in_order"] = { + "found": found_titles, + "correct": len(found_titles) == 10 + and all(SECTION_TITLES_IN_ORDER[i] in found_titles[i] for i in range(min(10, len(found_titles)))), + } + + # 3. Cross-ref: confidentiality reference should now cite Section 8. + m = re.search(r"confidentiality obligations[^.]*?Section\s+(\d+)", html, re.IGNORECASE | re.DOTALL) + results["crossref_confidentiality_points_to_8"] = { + "found": m.group(1) if m else None, + "correct": m is not None and m.group(1) == "8", + } + + # 4. Cross-ref: termination reference should now cite Section 9. + m = re.search(r"processed pursuant to Section\s+(\d+)", html, re.IGNORECASE) + results["crossref_termination_points_to_9"] = { + "found": m.group(1) if m else None, + "correct": m is not None and m.group(1) == "9", + } + + # 5. TOC: entry for Section 3 has the current title, not the stale one. + # Body headings are ALL CAPS ("SECTION 1 -- INTRODUCTION..."); TOC entries are + # title case ("Section 1 -- Introduction..."). The boundary must be case-sensitive + # or it matches the TOC's own first entry instead of the real heading below it. + toc_region_match = re.search(r"TABLE OF CONTENTS(.*?)(?=SECTION 1\s*[—\-])", html, re.DOTALL) + toc_region = toc_region_match.group(1) if toc_region_match else "" + results["toc_section3_title_fixed"] = { + "correct": "Vehicle Inspection Requirements" in toc_region and "Cargo Inspection" not in toc_region, + } + + # 6. TOC: Drug and Alcohol Policy entry now says Section 6, not a duplicated Section 5. + dup5_count = len(re.findall(r"Section\s+5\s*[—\-]", toc_region, re.IGNORECASE)) + results["toc_drug_alcohol_number_fixed"] = { + "correct": bool(re.search(r"Section\s+6\s*[—\-]\s*Drug and Alcohol Policy", toc_region, re.IGNORECASE)) + and dup5_count == 1, + } + + # 7. TOC: Miscellaneous Provisions entry now present. + results["toc_missing_entry_added"] = { + "correct": bool(re.search(r"Section\s+10\s*[—\-]\s*Miscellaneous Provisions", toc_region, re.IGNORECASE)), + } + + # 8. TOC: exactly 10 entries total (sanity check against partial/duplicate fixes). + toc_entry_count = len(re.findall(r"Section\s+\d+\s*[—\-]", toc_region, re.IGNORECASE)) + results["toc_has_exactly_10_entries"] = { + "found": toc_entry_count, + "correct": toc_entry_count == 10, + } + + all_correct = all(r["correct"] for r in results.values()) + return {"pass": all_correct, "details": results} + + +# ---------- main ---------- + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--dry-run", action="store_true") + args = parser.parse_args() + + if args.dry_run: + print_dry_run() + return + + api_key = os.environ.get("SUPERDOCS_API_KEY") + if not api_key: + print("SUPERDOCS_API_KEY not set", file=sys.stderr) + sys.exit(1) + client = Client(api_key) + + # --- renumber: verify-then-retry, fresh session + fresh document each attempt --- + attempts_log = [] + session_id = None + for attempt in range(1, MAX_RENUMBER_ATTEMPTS + 1): + attempt_session = f"self-heal-{uuid.uuid4()}" + log(f"renumber attempt {attempt}/{MAX_RENUMBER_ATTEMPTS}, session: {attempt_session}") + client.upload_document(CONTENT_DIR / "manual.html", attempt_session, open_mode="replace") + job = client.start_chat(RENUMBER_INSTRUCTION, attempt_session, approval_mode="ask_every_time") + client.wait_for_job(attempt_session, job["job_id"], f"renumber (attempt {attempt})") + + docs = client.session_documents(attempt_session, include_html=True) + html = find_document_html(docs, "manual") + check = verify_headings(html) + attempts_log.append({"attempt": attempt, "session_id": attempt_session, **check}) + log(f" attempt {attempt} numbers: {check['found_numbers']} correct: {check['correct']}") + + if check["correct"]: + session_id = attempt_session + log(f" attempt {attempt} verified correct, proceeding with this session") + break + elif attempt < MAX_RENUMBER_ATTEMPTS: + log(f" attempt {attempt} failed verification, discarding and retrying fresh") + else: + log(f" attempt {attempt} failed verification, cap reached -- proceeding anyway with") + log(" this session's (incorrect) result, to see how the rest of the pipeline handles it") + session_id = attempt_session + + (OUTPUT_DIR / "renumber_attempts.json").write_text(json.dumps(attempts_log, indent=2), encoding="utf-8") + converged = any(a["correct"] for a in attempts_log) + log(f"renumber retry summary: {len(attempts_log)} attempt(s), converged to a correct result: {converged}") + + log("fixing cross-references") + job = client.start_chat(CROSSREF_INSTRUCTION, session_id, approval_mode="ask_every_time") + client.wait_for_job(session_id, job["job_id"], "crossref") + + log("fixing Table of Contents") + job = client.start_chat(TOC_INSTRUCTION, session_id, approval_mode="ask_every_time") + client.wait_for_job(session_id, job["job_id"], "toc") + + docs = client.session_documents(session_id, include_html=True) + html = find_document_html(docs, "manual") + + result = verify(html) + result["renumber_attempts"] = attempts_log + result["renumber_converged"] = converged + log("verification:") + for name, detail in result["details"].items(): + log(f" {name}: {json.dumps(detail)}") + log(f"OVERALL: {'PASS' if result['pass'] else 'FAIL'}") + + export_path = client.export_html(html, "repaired_manual", fmt="docx") + log(f"exported -> {export_path}") + + (OUTPUT_DIR / "final_document.html").write_text(html, encoding="utf-8") + (OUTPUT_DIR / "verification_result.json").write_text(json.dumps(result, indent=2), encoding="utf-8") + + if not result["pass"]: + sys.exit(1) + + +if __name__ == "__main__": + main() diff --git a/use-cases/shivansh193/self-healing-structure-agent/content/manual.html b/use-cases/shivansh193/self-healing-structure-agent/content/manual.html new file mode 100644 index 00000000..64a16937 --- /dev/null +++ b/use-cases/shivansh193/self-healing-structure-agent/content/manual.html @@ -0,0 +1,43 @@ +<h1>NorthPeak Logistics — Driver Safety & Compliance Manual</h1> +<p><em>Synthetic document for demonstration purposes only. No real company, drivers, or incidents.</em></p> + +<h2>TABLE OF CONTENTS</h2> +<p id="toc-1">Section 1 — Introduction and Scope</p> +<p id="toc-2">Section 2 — Definitions</p> +<p id="toc-3">Section 3 — Cargo Inspection Requirements</p> +<p id="toc-4">Section 4 — Hours of Service</p> +<p id="toc-5">Section 5 — Incident Reporting</p> +<p id="toc-6">Section 5 — Drug and Alcohol Policy</p> +<p id="toc-7">Section 7 — Disciplinary Actions</p> +<p id="toc-8">Section 8 — Confidentiality</p> +<p id="toc-9">Section 9 — Termination</p> + +<h2>SECTION 1 — INTRODUCTION AND SCOPE</h2> +<p>1.1 This Manual applies to all NorthPeak Logistics drivers operating company or leased vehicles on company business.</p> + +<h2>SECTION 2 — DEFINITIONS</h2> +<p>2.1 "Covered Driver" means any individual operating a vehicle under a NorthPeak Logistics dispatch. "Incident" means any collision, cargo loss, or safety violation required to be reported under this Manual.</p> + +<h2>SECTION 4 — VEHICLE INSPECTION REQUIREMENTS</h2> +<p>4.1 Each Covered Driver shall complete a pre-trip inspection before every dispatch and a post-trip inspection at the end of every shift, using the standard NorthPeak inspection checklist.</p> + +<h2>SECTION 5 — HOURS OF SERVICE</h2> +<p>5.1 No Covered Driver shall operate a vehicle for more than eleven (11) hours following ten (10) consecutive hours off duty.</p> + +<h2>SECTION 6 — INCIDENT REPORTING</h2> +<p>6.1 Any Incident must be reported to dispatch within one (1) hour of occurrence. Any records collected during an incident investigation shall be handled in accordance with the confidentiality obligations set out in Section 9 of this Manual.</p> + +<h2>SECTION 6 — DRUG AND ALCOHOL POLICY</h2> +<p>6.1 Covered Drivers are subject to random drug and alcohol testing consistent with applicable federal regulations. A confirmed positive result is grounds for immediate suspension pending investigation.</p> + +<h2>SECTION 7 — DISCIPLINARY ACTIONS</h2> +<p>7.1 Violations of this Manual are subject to progressive discipline, up to and including termination. Termination decisions arising from repeated violations under this Section shall be processed pursuant to Section 10 of this Manual.</p> + +<h2>SECTION 9 — CONFIDENTIALITY</h2> +<p>9.1 All incident records, personnel files, and investigation materials described in this Manual are confidential and shall not be disclosed outside NorthPeak Logistics except as required by law.</p> + +<h2>SECTION 10 — TERMINATION</h2> +<p>10.1 Employment may be terminated by either party at any time, consistent with NorthPeak Logistics' standard employment policies.</p> + +<h2>SECTION 11 — MISCELLANEOUS PROVISIONS</h2> +<p>11.1 This Manual may be amended by NorthPeak Logistics at any time by written notice to Covered Drivers. Headings are for convenience only.</p> diff --git a/use-cases/shivansh193/self-healing-structure-agent/requirements.txt b/use-cases/shivansh193/self-healing-structure-agent/requirements.txt new file mode 100644 index 00000000..7507eb03 --- /dev/null +++ b/use-cases/shivansh193/self-healing-structure-agent/requirements.txt @@ -0,0 +1,2 @@ +httpx>=0.27 +python-dotenv>=1.0 diff --git a/use-cases/shivansh193/two-agent-negotiation-referee/.env.example b/use-cases/shivansh193/two-agent-negotiation-referee/.env.example new file mode 100644 index 00000000..f611a7f0 --- /dev/null +++ b/use-cases/shivansh193/two-agent-negotiation-referee/.env.example @@ -0,0 +1 @@ +SUPERDOCS_API_KEY=your-key-here diff --git a/use-cases/shivansh193/two-agent-negotiation-referee/.gitignore b/use-cases/shivansh193/two-agent-negotiation-referee/.gitignore new file mode 100644 index 00000000..0703dfd8 --- /dev/null +++ b/use-cases/shivansh193/two-agent-negotiation-referee/.gitignore @@ -0,0 +1,5 @@ +.env +output/ +__pycache__/ +*.pyc +.venv/ diff --git a/use-cases/shivansh193/two-agent-negotiation-referee/PROGRESS.md b/use-cases/shivansh193/two-agent-negotiation-referee/PROGRESS.md new file mode 100644 index 00000000..50bb3fb7 --- /dev/null +++ b/use-cases/shivansh193/two-agent-negotiation-referee/PROGRESS.md @@ -0,0 +1,160 @@ +# Progress log -- Two-Agent Document Negotiation with a Human Referee + +## Before designing the build: tracked changes needed a real answer, not an assumption + +The pass bar requires the export to carry real, Word-readable tracked +changes. Checked the `ExportOptions` schema first -- no `track_changes` or +`redline` flag exists. Two cheap, targeted experiments before writing any +negotiation logic: + +1. Uploaded a tiny document, proposed one edit (`ask_every_time`, left + pending, never approved), and exported the *session* while that change + was still pending. Result: the export silently ignored the pending + diff and returned the pre-change document -- no tracked changes, no + sign the edit was ever proposed. +2. Constructed `<del>Net 45 days</del><ins>Net 30 days</ins>` HTML by + hand and exported it via the `html` field directly (not `session_id`). + Result: genuine Word tracked changes -- `<w:del w:id="1" + w:author="Unknown" w:date="...">` wrapping a proper `<w:delText>` + element, and a matching `<w:ins>`, both text values present. Verified + by unzipping the docx and reading `word/document.xml` directly. + +This settled the design: track each negotiated term's original anchor +value and final value, build a redline HTML with real `<ins>`/`<del>` +tags for just those two terms, and export *that* HTML directly rather +than exporting the session. + +## Playbook parsing validated before any API call + +`parse_playbook()` reads six numbers out of each playbook's own HTML +text at runtime -- nothing about either party's position is hardcoded in +`build.py`. Checked against all three playbook files with zero API cost: +correct extraction from `playbook_vendor.html`, `playbook_customer.html`, +and `playbook_customer_escalation.html`. Also caught a real design bug +here, for free: the escalation playbook was first written by *lowering* +the customer's liability floor from 2x to 1x, which actually *widens* +the customer's acceptable range and leaves the zone of agreement intact +at 2x -- the opposite of the intended effect. Fixed by *raising* it to +3x instead (above the vendor's ceiling of 2x), which creates a real, +unbridgeable gap. An exhaustive check across liability values 0-9 +confirmed no integer satisfies both playbooks under the fixed version. + +## Run 1 (convergent playbook): six rounds, zero real progress + +The negotiation never moved off the anchor values (`Net 45 days`, `four +(4) times`) across all 6 rounds, despite several rounds reporting +genuinely approved edits. Diagnosed by reading `audit_trail.json`'s +captured `old_html`/`new_html` per round rather than trusting the +round-by-round snapshots alone: every edit had landed on +**`playbook_vendor.html`'s own content**, not `msa_terms` -- the model +was editing its own reference playbook, not the document being +negotiated, in every single round. + +Root cause was in this build's own prompt, not the platform: both +instruction templates named `playbook_vendor` (or the customer playbook) +first, told the model to "read it," and then said "**This document's** +Section 4..." -- an ambiguous pronoun reference that most naturally +resolves to the document just named, not the actually-focused +`msa_terms`. Fixed by naming `msa_terms` explicitly, every time, in both +`opening_instruction()` and `counter_instruction()`, and by adding an +explicit "do not edit `{playbook_doc}` itself" clause. This is the same +family of lesson as everything else found by name-based instructions +tonight: an LLM will resolve an ambiguous "this document" to whatever +was mentioned most recently, not to what a human reader would obviously +intend from context. + +## Run 2 (convergent, fixed instructions): converged for real + +Rewired instructions, re-ran. Vendor's opening landed correctly on +`msa_terms` (`Net 15 days`, `one (1) times` -- its own preferred +position). Customer's round 2 counter jumped straight to its own +preferred liability figure (`five (5) times`) rather than moving by the +playbook's stated one-step increment, and left the clearly-violating +payment term (`15 days`, well outside its own ceiling) untouched. Not a +bug worth chasing -- the pass bar doesn't require every round to be +letter-perfect, and the imperfection didn't derail anything (the *audit +trail* is the design goal, and it caught this precisely). This run's +`reject-test` round also turned out to be a degenerate case: the job +completed with zero pending changes, so the "no residue" check passed +trivially rather than proving anything about an actual denied proposal. + +## Run 3 (convergent, reject-test hardened): real evidence, real convergence + +Added a hard check: if the reject-test proposes zero real edits, the +script now raises rather than silently accepting a degenerate pass. Ran +again for real: + +- **Reject-test**: 4 real proposed edits, all explicitly denied + (`approved: false` with feedback), document confirmed byte-equivalent + before and after (`{'payment_days': 45, 'liability_mult': 4}` both + times). This is now a genuine test of a real rejected proposal, not a + no-op. +- **Negotiation**: 5 real rounds. Vendor opened at its preferred position + (15 days, 1x). Customer moved liability in one uneven jump (1x -> 5x, + its full preferred ask, not a single step) and left payment untouched + despite violating its own ceiling. Vendor's every move was disciplined + -- exactly one step, only when its own floor was violated. By round 4, + customer corrected payment to exactly 30 days (one correct 15-day + step) but also nudged liability down from 4x to 3x -- a term that + already satisfied its own ceiling (>=2x) and didn't need to move at + all, an unprompted, unrequired concession. Vendor's round 5 (3x -> 2x, + its final permitted step) landed exactly on the true zone of + agreement, and this build's own independent check -- using both + playbooks' actual numbers, not anything the model claimed -- confirmed + it: **AGREED after round 5, final state 30 days / 2x, exactly matching + vendor's floor and customer's ceiling on both terms.** + +Customer's step-discipline was looser than vendor's throughout both real +runs (overshoots, one unprompted move on an already-satisfied term), but +never in a way that broke correctness -- the negotiation still converged +on the objectively correct point both times it had a real zone of +agreement to find. Worth naming as an honest observation, not smoothed +over: the two agents did not follow their playbooks with equal +discipline, even though both started from the same instruction template +with only the playbook name and direction word substituted. + +## Escalation run (`--customer-playbook playbook_customer_escalation.html`) + +No code changes from Run 3 -- only the `--customer-playbook` flag +differs, pointing at a file that's identical to the default customer +playbook except for one edited number. Payment terms resolved cleanly to +30 days by round 2 and correctly stayed there for the rest of the run +(both sides recognizing it as already-settled). Liability cap never +converged, for the deliberately designed reason: vendor's floor is 2x, +customer-escalation's ceiling is 3x, no integer satisfies both. The loop +ran its full `MAX_ROUNDS = 6` and stopped -- a `for` loop over +`range(1, 7)` with an early `break` on convergence cannot run past 6 +iterations by construction, agreement or not. `output/escalation_memo.json` +was written with the full round history, both playbooks' actual limits, +the final state, and an explicit per-term resolved/unresolved flag +(`payment_terms_resolved: true`, `liability_cap_resolved: false`) -- a +human referee gets everything needed to make the actual call. + +## Direct verification against the four pass-bar items + +1. **Every round genuinely auditable and reversible.** `audit_trail.json` + captures every round's actual proposed `old_html`/`new_html`, not + just a summary. The reject-test proposes and denies 4 real edits and + proves byte-for-byte no residue -- checked directly, not assumed. +2. **Export carries real tracked changes, readable in Word.** Verified + by unzipping both the AGREED and the ESCALATED `.docx` and reading + `word/document.xml` directly: real `<w:ins>`/`<w:del>` elements with + proper `<w:delText>` children, both the original and final text + present (`Net 45 days` -> `Net 30 days`, `four (4) times` -> `two (2) + times`). Not inferred from the automated check alone -- read by hand + a second time, same result. +3. **A deliberately non-converging case escalates instead of looping.** + Constructed on purpose (one number changed in one playbook file), + run for real, ran its full 6-round hard cap, escalated with a + complete memo. Confirmed via an exhaustive check that no integer + liability value could have satisfied both playbooks under that file. +4. **Both playbooks are genuinely swappable data files.** `parse_playbook()` + reads every number from the HTML at runtime. Two full real runs used + the same unmodified `build.py`, differing only in which + `--customer-playbook` file was passed, and produced the correct, + different outcome each time (AGREED vs. ESCALATED) -- not asserted, + run. + +All four hold up under direct evidence, not just the rendered output. +Committed and pushed to PR #115 per the standing instruction that a +build clearing all four bars doesn't need a check-in first. diff --git a/use-cases/shivansh193/two-agent-negotiation-referee/README.md b/use-cases/shivansh193/two-agent-negotiation-referee/README.md new file mode 100644 index 00000000..57674060 --- /dev/null +++ b/use-cases/shivansh193/two-agent-negotiation-referee/README.md @@ -0,0 +1,139 @@ +# Two-Agent Document Negotiation with a Human Referee + +Built by Shivansh Kalra for the SuperDocs task. + +Two agents, TechFlow Solutions (Vendor) and Meridian Retail Group +(Customer), negotiate two terms of a Master Services Agreement -- Payment +Terms and Limitation of Liability -- by alternately proposing changes to +one shared document. Every round is a real, reviewable proposed change +(`approval_mode: "ask_every_time"`, never `auto-apply`); nothing is ever +silently rewritten. The loop is bounded by construction, not by hope, and +escalates to a human referee with a complete memo when the two sides +genuinely can't agree. + +All content is synthetic: two fictional companies negotiating a +fictional contract. + +## What makes each agent's position real, not scripted + +Each agent's opening ask, walk-away limit, and step size per round lives +entirely in its own playbook -- an HTML data file +(`content/playbook_vendor.html`, `content/playbook_customer.html`), never +in `build.py`. `parse_playbook()` reads those six numbers back out of the +files at runtime and uses them for this build's own independent +convergence check too, so the check is data-driven, not hardcoded to one +scenario. `content/playbook_customer_escalation.html` is the same file +with exactly one number changed -- proof the playbooks are genuinely +swappable, not decorative, is in [Verified result](#verified-result). + +## What it does + +1. Uploads the MSA terms (focused) plus both playbooks (background). +2. **Pre-round test**: generates a real proposed change, then deliberately + *rejects* it, and confirms the document is byte-for-byte unchanged + afterward -- proof a denied round leaves no residue. +3. **Negotiation rounds** (up to `MAX_ROUNDS = 6`): vendor opens, then the + two sides alternate counters. Each round either leaves an + already-acceptable term untouched or moves it by exactly the + playbook's step size, never past that side's own walk-away limit. + Stops early the moment this build's own check -- using both playbooks' + real numbers, not the model's say-so -- confirms both terms are within + both parties' limits. +4. If `MAX_ROUNDS` passes without agreement: writes + `output/escalation_memo.json` with the full round history and both + playbooks' positions, and stops. No further rounds are attempted. +5. **Export**: builds a redline HTML with real `<ins>`/`<del>` tags from + the original anchor values to the final negotiated values, and exports + *that* directly via the API's `html` field. + +## How to run it + +```bash +python -m venv .venv +.venv/Scripts/activate # or source .venv/bin/activate on macOS/Linux +pip install -r requirements.txt +cp .env.example .env # then set SUPERDOCS_API_KEY +python build.py --dry-run # convergent case, zero API calls +python build.py # convergent case, for real +python build.py --customer-playbook playbook_customer_escalation.html # escalation case, for real +``` + +## SuperDocs features used + +- **Multi-document sessions** (`open_mode: "replace"` / `"background"`) -- + the negotiated document plus both playbooks open together +- **Chat / async edit** (`POST /v1/chat/async`) with + `approval_mode: "ask_every_time"`, including explicit denial + (`approved: false` with feedback) as a first-class, tested path +- **Export** (`POST /v1/documents/export`) with hand-built `<ins>`/`<del>` + HTML passed via the `html` field -- see Honest limitations for why + +## Verified result + +Two real negotiation runs, plus one earlier run that surfaced a real bug +in this build's own prompts before either of these (full trace in +[`PROGRESS.md`](PROGRESS.md)). + +**Convergent case** (`playbook_customer.html`, default): AGREED after 5 +rounds. Final state: Payment Terms 30 days, Liability Cap 2x -- exactly +vendor's floor and customer's ceiling on both terms, confirmed by this +build's independent check against both playbooks' real numbers. + +**Escalation case** (`playbook_customer_escalation.html`, one number +changed from the default): Payment Terms resolved cleanly to 30 days by +round 2. Liability Cap never converged -- a real, deliberately designed +gap between vendor's 2x floor and customer's 3x ceiling. Ran its full +6-round cap and escalated, with a complete memo (`payment_terms_resolved: +true`, `liability_cap_resolved: false`, full round history, both +playbooks' actual limits) for a human referee to act on. + +**The four pass-bar items, checked against direct evidence:** + +| Requirement | Evidence | +|---|---| +| Every round auditable and reversible | `audit_trail.json` captures every round's real proposed `old_html`/`new_html`. The reject-test proposes and denies 4 real edits and proves byte-for-byte no residue in the document afterward. | +| Export carries real tracked changes | Unzipped both the AGREED and ESCALATED `.docx` and read `word/document.xml` by hand: real `<w:ins>`/`<w:del>` with proper `<w:delText>` children, both old and new text present (`Net 45 days` -> `Net 30 days`, `four (4) times` -> `two (2) times`). | +| Non-converging case escalates, not loops | Constructed on purpose, run for real, hit its 6-round hard cap (a bounded `for` loop, not a hope), escalated with a complete memo. | +| Playbooks are genuinely swappable | `parse_playbook()` reads every number from the HTML at runtime. Two real runs of the same unmodified `build.py`, differing only in which `--customer-playbook` file was passed, produced the correct different outcome each time. | + +**A real, honest observation, not smoothed over**: the customer agent's +step-discipline was looser than vendor's in both real runs -- one round +jumped straight to its full preferred value instead of one step, another +moved a term that already satisfied its own ceiling and didn't need to +move at all. Neither broke correctness (both runs still converged +exactly where they should have, or correctly failed to), but it's a real +difference in how faithfully the two sides followed the same instruction +template with only the playbook name and direction word substituted. + +## Honest limitations + +- SuperDocs' export does not carry tracked changes for a session's + pending, unapproved edits -- tested directly before this build was + designed (see PROGRESS.md): exporting a session with a pending change + silently returns the pre-change document. Genuine `w:ins`/`w:del` only + comes from constructing `<ins>`/`<del>` HTML directly and exporting via + the `html` field, which is what this build does for its final export -- + it does not reflect the platform's own approval history natively. +- The reject-test and negotiation share one session; the negotiation's + own instructions are written defensively (explicit document naming, + explicit "do not edit the playbook" clauses) after an earlier run + showed how easily an ambiguous "this document" reference goes wrong -- + see PROGRESS.md for the full failure and fix. +- `output/` is gitignored; run `python build.py` (and the escalation + variant) to regenerate everything, including both `.docx` exports and + both JSON audit trails. + +## Files + +- `build.py` -- upload -> reject-test -> negotiation rounds (playbook- + driven, independently verified convergence) -> escalation-or-export -> + tracked-changes verification flow, plus `--dry-run` and + `--customer-playbook` +- `content/msa_terms.html` -- the two-term MSA excerpt being negotiated +- `content/playbook_vendor.html`, `content/playbook_customer.html` -- + each side's real negotiating position, as data +- `content/playbook_customer_escalation.html` -- the customer playbook + with exactly one number changed, proving both swappability and the + escalation path +- `PROGRESS.md` -- full diagnostic trace, including the wrong-document + bug this build's own instructions caused and how it was found and fixed diff --git a/use-cases/shivansh193/two-agent-negotiation-referee/build.py b/use-cases/shivansh193/two-agent-negotiation-referee/build.py new file mode 100644 index 00000000..438f1e8e --- /dev/null +++ b/use-cases/shivansh193/two-agent-negotiation-referee/build.py @@ -0,0 +1,490 @@ +"""Two-Agent Document Negotiation with a Human Referee -- built against +the real, hosted SuperDocs product. Two agents, TechFlow Solutions +(Vendor) and Meridian Retail Group (Customer), negotiate two terms of an +MSA -- Payment Terms and Limitation of Liability -- by alternately +proposing changes to one shared document. Every round is a real, +reviewable proposed change (`approval_mode: "ask_every_time"`, never +`auto-apply`); nothing is ever silently rewritten. + +Each agent's position -- opening ask, walk-away limit, step size per +round -- lives entirely in its own playbook, an HTML data file +(`content/playbook_vendor.html`, `content/playbook_customer.html`), never +in this script. `parse_playbook()` reads those numbers back out of the +files at runtime and uses them for this build's own independent +convergence check, so the check itself is also data-driven, not hardcoded +against one specific scenario. `--customer-playbook` selects which +customer playbook file to use; `content/playbook_customer_escalation.html` +is the same file with exactly one number changed (the liability ceiling, +pushed below Vendor's floor so no agreement is possible), used to prove +the escalation path. + +A finding from testing before this build was designed: exporting a +session with a pending, unapproved change does NOT produce tracked +changes -- it silently exports the pre-change state, ignoring the pending +diff entirely. Genuine Word-native tracked changes (`w:ins`/`w:del`) DO +come out of `POST /v1/documents/export` when you construct `<ins>`/`<del>` +HTML yourself and export it via the `html` field (not `session_id`). +`build_redline_html()` does exactly that for the two negotiated terms, +using each term's original anchor value and its final negotiated value. + +Run `python build.py --dry-run` first: prints the full plan with zero API +calls. Only run for real (`python build.py [--customer-playbook ...]`) +after reading that output. +""" + +import argparse +import json +import os +import re +import sys +import time +import uuid +import zipfile +from pathlib import Path + +import httpx +from dotenv import load_dotenv + +load_dotenv() + +BASE_URL = "https://api.superdocs.app" +HERE = Path(__file__).parent +CONTENT_DIR = HERE / "content" +OUTPUT_DIR = HERE / "output" +OUTPUT_DIR.mkdir(exist_ok=True) + +MAX_ROUNDS = 6 +NUM_WORDS = {1: "one", 2: "two", 3: "three", 4: "four", 5: "five", 6: "six", 7: "seven"} + + +def log(msg: str) -> None: + print(f"[{time.strftime('%H:%M:%S')}] {msg}", flush=True) + + +# ---------- API helpers (same shape as the other builds) ---------- + + +class Client: + def __init__(self, api_key: str): + self.http = httpx.Client(base_url=BASE_URL, headers={"Authorization": f"Bearer {api_key}"}, timeout=240.0) + + def upload_document(self, path: Path, session_id: str, open_mode: str = "replace") -> dict: + with open(path, "rb") as f: + resp = self.http.post( + "/v1/documents/upload", + files={"file": (path.name, f, "text/html")}, + data={"session_id": session_id, "open_mode": open_mode}, + ) + resp.raise_for_status() + return resp.json() + + def start_chat(self, message: str, session_id: str, approval_mode: str = "ask_every_time") -> dict: + resp = self.http.post( + "/v1/chat/async", + json={"message": message, "session_id": session_id, "approval_mode": approval_mode}, + ) + resp.raise_for_status() + return resp.json() + + def get_job(self, job_id: str) -> dict: + resp = self.http.get(f"/v1/jobs/{job_id}") + resp.raise_for_status() + return resp.json() + + def approve_all(self, session_id: str, job_id: str, pending_changes: list[dict]) -> None: + changes = [{"change_id": c["change_id"], "approved": True} for c in pending_changes] + resp = self.http.post(f"/v1/chat/{session_id}/approve", json={"job_id": job_id, "approved": True, "changes": changes}) + resp.raise_for_status() + + def reject_all(self, session_id: str, job_id: str, pending_changes: list[dict], feedback: str | None = None) -> None: + changes = [ + {"change_id": c["change_id"], "approved": False, **({"feedback": feedback} if feedback else {})} + for c in pending_changes + ] + resp = self.http.post(f"/v1/chat/{session_id}/approve", json={"job_id": job_id, "approved": False, "changes": changes}) + resp.raise_for_status() + + def continue_job(self, session_id: str, job_id: str) -> None: + resp = self.http.post(f"/v1/chat/{session_id}/continue", json={"job_id": job_id, "continue": True}) + resp.raise_for_status() + + def wait_for_job( + self, session_id: str, job_id: str, label: str, decision: str = "approve", feedback: str | None = None, max_wait_s: int = 400 + ) -> dict: + start = time.time() + while time.time() - start < max_wait_s: + job = self.get_job(job_id) + status = job["status"] + if status == "completed": + log(f" [{label}] completed") + return job + if status in ("failed", "cancelled"): + raise RuntimeError(f"{label} job {status}: {job.get('error')}") + if status == "awaiting_approval": + metadata = job.get("metadata") or {} + if metadata.get("awaiting_kind") == "continue_prompt": + log(f" [{label}] paused mid-edit, continuing") + self.continue_job(session_id, job_id) + else: + pending = metadata.get("pending_changes") or [] + if decision == "approve": + log(f" [{label}] awaiting approval on {len(pending)} change(s) -- approving") + self.approve_all(session_id, job_id, pending) + else: + log(f" [{label}] awaiting approval on {len(pending)} change(s) -- REJECTING (test)") + self.reject_all(session_id, job_id, pending, feedback=feedback) + else: + log(f" [{label}] {status}...") + time.sleep(4) + raise TimeoutError(f"{label} job did not complete in time") + + def session_documents(self, session_id: str, include_html: bool = True) -> dict: + resp = self.http.get(f"/v1/sessions/{session_id}/documents", params={"include_html": str(include_html).lower()}) + resp.raise_for_status() + return resp.json() + + def export_html(self, html: str, filename: str, fmt: str = "docx") -> Path: + resp = self.http.post("/v1/documents/export", json={"html": html, "format": fmt, "options": {"filename": filename}}) + resp.raise_for_status() + ext = {"docx": "docx", "pdf": "pdf", "html": "html"}.get(fmt, fmt) + out_path = OUTPUT_DIR / f"{filename}.{ext}" + if "application/json" in resp.headers.get("content-type", ""): + data = resp.json() + url = data.get("download_url") or data.get("url") + out_path.write_bytes(self.http.get(url).content) + else: + out_path.write_bytes(resp.content) + return out_path + + +def _norm(s: str) -> str: + return re.sub(r"[_\-\s]+", " ", (s or "")).strip().lower() + + +def find_document_html(doc_list: dict, title_substring: str) -> str: + needle = _norm(title_substring) + for d in doc_list.get("documents", []): + if needle in _norm(d.get("title")): + html = d.get("html") + if not html: + raise ValueError(f"document matching '{title_substring}' found but has no html: {d}") + return html + raise ValueError(f"no open document matching '{title_substring}' -- got {doc_list}") + + +# ---------- playbook parsing (data-driven, not hardcoded) ---------- + + +def parse_playbook(path: Path) -> dict: + text = re.sub(r"<[^>]+>", " ", path.read_text(encoding="utf-8")) + def find(pattern): + m = re.search(pattern, text) + return int(m.group(1)) if m else None + return { + "payment_opening": find(r"Opening ask:\s*Net\s+(\d+)\s+days"), + "payment_limit": find(r"Walk-away (?:floor|ceiling):\s*Net\s+(\d+)\s+days"), + "payment_step": find(r"Step size per round:\s*(\d+)\s+days"), + "liability_opening": find(r"Opening ask:\s*a cap equal to \w+\s*\((\d+)\)\s*times fees"), + "liability_limit": find(r"Walk-away (?:floor|ceiling):\s*\w+\s*\((\d+)\)\s*times fees"), + "liability_step": find(r"Step size per round:\s*\w+\s*\((\d+)\)\s*times fees"), + } + + +# ---------- document value extraction ---------- + + +def extract_payment_days(html: str) -> int | None: + m = re.search(r"Net\s+(\d+)\s+days", html) + return int(m.group(1)) if m else None + + +def extract_liability_multiplier(html: str) -> int | None: + m = re.search(r"\((\d+)\)\s*times the total fees", html) + return int(m.group(1)) if m else None + + +def snapshot(html: str) -> dict: + return {"payment_days": extract_payment_days(html), "liability_mult": extract_liability_multiplier(html)} + + +def check_agreement(snap: dict, vendor_pb: dict, customer_pb: dict) -> bool: + if snap["payment_days"] is None or snap["liability_mult"] is None: + return False + payment_ok = snap["payment_days"] <= vendor_pb["payment_limit"] and snap["payment_days"] >= customer_pb["payment_limit"] + liability_ok = snap["liability_mult"] <= vendor_pb["liability_limit"] and snap["liability_mult"] >= customer_pb["liability_limit"] + return payment_ok and liability_ok + + +# ---------- the plan ---------- + + +def opening_instruction() -> str: + return ( + "Read the document open in this session called playbook_vendor to get your negotiating position " + "from it -- but do not edit playbook_vendor itself, it is a reference only. Make your edit to the " + "document called msa_terms (the currently focused document). In msa_terms, Section 4 (Payment " + "Terms) and Section 7 (Limitation of Liability) currently state placeholder values. In msa_terms, " + "set Section 4's stated payment period to your playbook's opening ask for Payment Terms, phrased " + "exactly as 'Net N days' with a numeral N. In msa_terms, set Section 7's stated cap to your " + "playbook's opening ask for Limitation of Liability, phrased exactly as 'WORD (N) times the total " + "fees paid by Customer in the preceding twelve (12) months', spelling out both the word and the " + "numeral N (for example 'two (2) times'). Add a short note right after each sentence you changed " + "in msa_terms: '(TechFlow Solutions, round 1)'. Do not change anything else in msa_terms, and do " + "not make any edit to playbook_vendor." + ) + + +def counter_instruction(playbook_doc: str, party_label: str, limit_word: str, round_num: int) -> str: + return ( + f"Read the document open in this session called {playbook_doc} to get your negotiating position " + f"from it -- but do not edit {playbook_doc} itself, it is a reference only. Look at the document " + f"called msa_terms (the currently focused document), specifically its Section 4 (Payment Terms) " + f"and Section 7 (Limitation of Liability) -- these currently reflect the other party's latest " + f"position. For EACH of these two terms in msa_terms, compare the current stated value against " + f"your playbook's walk-away {limit_word} for that term. If the current value already satisfies " + f"your playbook (does not violate your walk-away {limit_word}), leave that term's sentence in " + f"msa_terms completely unchanged -- do not restate it. If the current value violates your " + f"walk-away {limit_word}, edit msa_terms to move it by exactly your playbook's step size for that " + f"term, in your favorable direction, and never move it past your own walk-away {limit_word} -- " + f"keep the exact same phrasing already used ('Net N days' for Payment Terms; 'WORD (N) times the " + f"total fees paid by Customer in the preceding twelve (12) months' for Limitation of Liability, " + f"spelling out both the word and the numeral). Add a short note right after any sentence you " + f"changed in msa_terms: '({party_label}, round {round_num})'. Do not change a term you did not " + f"need to move, do not change anything else in msa_terms, and do not make any edit to {playbook_doc}." + ) + + +def print_dry_run(customer_playbook_path: Path) -> None: + vendor_pb = parse_playbook(CONTENT_DIR / "playbook_vendor.html") + customer_pb = parse_playbook(customer_playbook_path) + print("=== DRY RUN -- no API calls will be made ===\n") + print(f"Document being negotiated: {CONTENT_DIR / 'msa_terms.html'}") + print(f"Vendor playbook: {CONTENT_DIR / 'playbook_vendor.html'} -> {vendor_pb}") + print(f"Customer playbook: {customer_playbook_path} -> {customer_pb}") + print() + zopa_payment = customer_pb["payment_limit"] <= vendor_pb["payment_limit"] + zopa_liability = customer_pb["liability_limit"] <= vendor_pb["liability_limit"] + print(f"Zone of possible agreement -- payment terms: {'EXISTS' if zopa_payment else 'NONE (will escalate)'}") + print(f"Zone of possible agreement -- liability cap: {'EXISTS' if zopa_liability else 'NONE (will escalate)'}") + print() + print("Plan:") + print(" 0. Pre-round test: vendor's opening proposal is generated, then deliberately REJECTED,") + print(" to prove a rejected round leaves no residue (document re-checked byte-for-byte).") + print(" 1. Vendor's real opening proposal (approved).") + print(f" 2. Up to {MAX_ROUNDS - 1} further rounds, alternating customer/vendor counters, each a real") + print(" reviewable proposed change (ask_every_time, never auto-apply). Stops early the moment") + print(" this build's own independent check (using both playbooks' actual numbers, not the model's") + print(" say-so) confirms both terms are within both parties' limits.") + print(f" 3. If {MAX_ROUNDS} rounds pass without agreement: ESCALATE -- write output/escalation_memo.json") + print(" with the full round history and both playbooks' positions for a human referee. No further") + print(" rounds are attempted after that -- this is a hard cap, not a hope.") + print(" 4. Export: build a redline HTML from the original anchor values vs the final negotiated") + print(" values (or the state at escalation), with real <ins>/<del> tags, and export THAT via the") + print(" html field directly -- exporting session_id alone does not carry tracked changes; this was") + print(" tested directly before this build was designed (see PROGRESS.md).") + print() + print(f"API calls this would make for real: 1 upload x3 (doc + 2 playbooks), 1 opening + 1 rejected-test") + print(f"turn, then up to {MAX_ROUNDS} more turns, plus 1 export. No cross_session_search used.") + print("Re-run without --dry-run once this plan looks right.") + + +# ---------- redline construction + verification ---------- + + +def build_redline_html(final_html: str, orig_payment: int, final_payment: int, orig_liability: int, final_liability: int) -> str: + html = final_html + if final_payment != orig_payment: + old_phrase = f"Net {orig_payment} days" + new_phrase = f"Net {final_payment} days" + html = html.replace(new_phrase, f"<del>{old_phrase}</del><ins>{new_phrase}</ins>", 1) + if final_liability != orig_liability: + old_word = NUM_WORDS.get(orig_liability, str(orig_liability)) + new_word = NUM_WORDS.get(final_liability, str(final_liability)) + old_phrase = f"{old_word} ({orig_liability}) times" + new_phrase = f"{new_word} ({final_liability}) times" + html = html.replace(new_phrase, f"<del>{old_phrase}</del><ins>{new_phrase}</ins>", 1) + return html + + +def verify_tracked_changes(docx_path: Path) -> dict: + with zipfile.ZipFile(docx_path) as z: + xml = z.read("word/document.xml").decode("utf-8") + ins_count = len(re.findall(r"<w:ins\b", xml)) + del_count = len(re.findall(r"<w:del\b", xml)) + has_deltext = "<w:delText" in xml + return { + "ins_count": ins_count, + "del_count": del_count, + "has_delText_element": has_deltext, + "correct": ins_count > 0 and del_count > 0 and has_deltext, + } + + +# ---------- main ---------- + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--dry-run", action="store_true") + parser.add_argument("--customer-playbook", default="playbook_customer.html") + args = parser.parse_args() + + customer_playbook_path = CONTENT_DIR / args.customer_playbook + customer_playbook_title = customer_playbook_path.stem + + if args.dry_run: + print_dry_run(customer_playbook_path) + return + + api_key = os.environ.get("SUPERDOCS_API_KEY") + if not api_key: + print("SUPERDOCS_API_KEY not set", file=sys.stderr) + sys.exit(1) + client = Client(api_key) + + vendor_pb = parse_playbook(CONTENT_DIR / "playbook_vendor.html") + customer_pb = parse_playbook(customer_playbook_path) + log(f"vendor playbook: {vendor_pb}") + log(f"customer playbook ({args.customer_playbook}): {customer_pb}") + + session_id = f"negotiation-{uuid.uuid4()}" + log(f"session: {session_id}") + client.upload_document(CONTENT_DIR / "msa_terms.html", session_id, open_mode="replace") + client.upload_document(CONTENT_DIR / "playbook_vendor.html", session_id, open_mode="background") + client.upload_document(customer_playbook_path, session_id, open_mode="background") + log(f" opened msa_terms.html, playbook_vendor.html, {args.customer_playbook}") + + docs = client.session_documents(session_id, include_html=True) + original_html = find_document_html(docs, "msa_terms") + original_snap = snapshot(original_html) + log(f"anchor values: {original_snap}") + + audit_trail = [] + + # --- pre-round test: propose, then reject, prove no residue --- + log("pre-round test: vendor opening proposal, deliberately REJECTED") + before_reject_html = find_document_html(client.session_documents(session_id, include_html=True), "msa_terms") + job = client.start_chat(opening_instruction(), session_id, approval_mode="ask_every_time") + reject_job_result = client.wait_for_job( + session_id, job["job_id"], "reject-test", decision="reject", + feedback="Rejected deliberately: proving a denied round leaves no residue in the document.", + ) + reject_changes = ((reject_job_result.get("result") or {}).get("document_changes") or {}).get("changes") or [] + reject_proposed = [ + {"chunk_id": c.get("chunk_id"), "status": c.get("status"), "old_html": c.get("old_html"), "new_html": c.get("new_html")} + for c in reject_changes if c.get("operation") == "edit" + ] + if not reject_proposed: + raise RuntimeError( + "reject-test proposed ZERO edits -- this test is supposed to prove a REAL rejected proposal " + "leaves no residue, not a no-op. Re-run: a genuine edit must be proposed and rejected here." + ) + log(f" reject-test proposed {len(reject_proposed)} real change(s), all denied: " + f"{[c['status'] for c in reject_proposed]}") + after_reject_html = find_document_html(client.session_documents(session_id, include_html=True), "msa_terms") + no_residue = snapshot(after_reject_html) == snapshot(before_reject_html) + log(f" residue check: before={snapshot(before_reject_html)} after={snapshot(after_reject_html)} no_residue={no_residue}") + audit_trail.append({ + "round": "reject-test", "actor": "vendor", "decision": "rejected", "proposed_changes": reject_proposed, + "before": snapshot(before_reject_html), "after": snapshot(after_reject_html), "no_residue": no_residue, + }) + + # --- round 1: vendor's real opening, approved --- + round_log = [] + outcome = None + for round_num in range(1, MAX_ROUNDS + 1): + actor = "vendor" if round_num % 2 == 1 else "customer" + if round_num == 1: + instruction = opening_instruction() + label = f"round{round_num}-vendor-opening" + elif actor == "vendor": + instruction = counter_instruction("playbook_vendor", "TechFlow Solutions", "floor", round_num) + label = f"round{round_num}-vendor-counter" + else: + instruction = counter_instruction(customer_playbook_title, "Meridian Retail Group", "ceiling", round_num) + label = f"round{round_num}-customer-counter" + + log(f"round {round_num}/{MAX_ROUNDS} ({actor}): {label}") + job = client.start_chat(instruction, session_id, approval_mode="ask_every_time") + job_result = client.wait_for_job(session_id, job["job_id"], label, decision="approve") + changes = ((job_result.get("result") or {}).get("document_changes") or {}).get("changes") or [] + proposed = [ + {"chunk_id": c.get("chunk_id"), "old_html": c.get("old_html"), "new_html": c.get("new_html")} + for c in changes if c.get("operation") == "edit" + ] + + docs = client.session_documents(session_id, include_html=True) + html = find_document_html(docs, "msa_terms") + snap = snapshot(html) + agreed = check_agreement(snap, vendor_pb, customer_pb) + round_log.append({"round": round_num, "actor": actor, "label": label, "proposed_changes": proposed, "state_after": snap, "agreed": agreed}) + audit_trail.append(round_log[-1]) + log(f" state after round {round_num}: {snap} agreement={agreed}") + + if agreed: + outcome = "AGREED" + log(f"AGREEMENT REACHED after round {round_num}/{MAX_ROUNDS}") + break + else: + outcome = "ESCALATED" + log(f"MAX_ROUNDS ({MAX_ROUNDS}) reached without agreement -- escalating to human referee") + + final_html = find_document_html(client.session_documents(session_id, include_html=True), "msa_terms") + final_snap = snapshot(final_html) + + if outcome == "ESCALATED": + memo = { + "outcome": "ESCALATED", + "rounds_run": MAX_ROUNDS, + "final_state": final_snap, + "vendor_playbook": vendor_pb, + "customer_playbook": customer_pb, + "payment_terms_resolved": final_snap["payment_days"] is not None + and final_snap["payment_days"] <= vendor_pb["payment_limit"] + and final_snap["payment_days"] >= customer_pb["payment_limit"], + "liability_cap_resolved": final_snap["liability_mult"] is not None + and final_snap["liability_mult"] <= vendor_pb["liability_limit"] + and final_snap["liability_mult"] >= customer_pb["liability_limit"], + "round_history": round_log, + "referral_note": ( + f"Automated negotiation did not converge within {MAX_ROUNDS} rounds. " + "Escalating to a human referee for a final decision. See round_history for the full " + "audit trail and the two playbook sections above for both parties' walk-away limits." + ), + } + (OUTPUT_DIR / "escalation_memo.json").write_text(json.dumps(memo, indent=2), encoding="utf-8") + log(f"escalation memo written -> {OUTPUT_DIR / 'escalation_memo.json'}") + + redline_html = build_redline_html( + final_html, original_snap["payment_days"], final_snap["payment_days"], + original_snap["liability_mult"], final_snap["liability_mult"], + ) + export_filename = "negotiated_msa_AGREED" if outcome == "AGREED" else "negotiated_msa_ESCALATED" + export_path = client.export_html(redline_html, export_filename, fmt="docx") + log(f"exported -> {export_path}") + + tc_result = verify_tracked_changes(export_path) + log(f"tracked-changes verification: {json.dumps(tc_result)}") + + result = { + "outcome": outcome, + "rounds_run": len(round_log), + "original_state": original_snap, + "final_state": final_snap, + "reject_test": audit_trail[0], + "tracked_changes_verification": tc_result, + "customer_playbook_used": args.customer_playbook, + } + log(f"OVERALL OUTCOME: {outcome}") + + (OUTPUT_DIR / "final_document.html").write_text(final_html, encoding="utf-8") + (OUTPUT_DIR / "redline_document.html").write_text(redline_html, encoding="utf-8") + (OUTPUT_DIR / "audit_trail.json").write_text(json.dumps(audit_trail, indent=2), encoding="utf-8") + (OUTPUT_DIR / "verification_result.json").write_text(json.dumps(result, indent=2), encoding="utf-8") + + if outcome != "AGREED" and args.customer_playbook == "playbook_customer.html": + sys.exit(1) + + +if __name__ == "__main__": + main() diff --git a/use-cases/shivansh193/two-agent-negotiation-referee/content/msa_terms.html b/use-cases/shivansh193/two-agent-negotiation-referee/content/msa_terms.html new file mode 100644 index 00000000..e0119cf2 --- /dev/null +++ b/use-cases/shivansh193/two-agent-negotiation-referee/content/msa_terms.html @@ -0,0 +1,8 @@ +<h1>Master Services Agreement — Negotiable Terms</h1> +<p><em>Synthetic document for demonstration purposes only. No real companies, terms, or figures. Between TechFlow Solutions ("Vendor") and Meridian Retail Group ("Customer").</em></p> + +<h2>Section 4 — Payment Terms</h2> +<p data-term="payment_terms">Customer shall pay each undisputed invoice within Net 45 days of receipt.</p> + +<h2>Section 7 — Limitation of Liability</h2> +<p data-term="liability_cap">Vendor's total liability under this Agreement shall not exceed an amount equal to four (4) times the total fees paid by Customer in the preceding twelve (12) months.</p> diff --git a/use-cases/shivansh193/two-agent-negotiation-referee/content/playbook_customer.html b/use-cases/shivansh193/two-agent-negotiation-referee/content/playbook_customer.html new file mode 100644 index 00000000..8e8aa877 --- /dev/null +++ b/use-cases/shivansh193/two-agent-negotiation-referee/content/playbook_customer.html @@ -0,0 +1,16 @@ +<h1>Meridian Retail Group — Negotiation Playbook (Customer)</h1> +<p><em>Internal negotiation positions. Synthetic, for demonstration purposes only. Edit the numbers below and re-run to change how Meridian negotiates — nothing about these positions is hardcoded in the build script.</em></p> + +<h2>Payment Terms</h2> +<ul> + <li>Opening ask: Net 60 days.</li> + <li>Walk-away ceiling: Net 30 days. Never agree to anything shorter than this.</li> + <li>Step size per round: 15 days.</li> +</ul> + +<h2>Limitation of Liability</h2> +<ul> + <li>Opening ask: a cap equal to five (5) times fees paid in the preceding twelve months.</li> + <li>Walk-away ceiling: two (2) times fees paid in the preceding twelve months. Never agree to a cap lower than this.</li> + <li>Step size per round: one (1) times fees.</li> +</ul> diff --git a/use-cases/shivansh193/two-agent-negotiation-referee/content/playbook_customer_escalation.html b/use-cases/shivansh193/two-agent-negotiation-referee/content/playbook_customer_escalation.html new file mode 100644 index 00000000..783dbf62 --- /dev/null +++ b/use-cases/shivansh193/two-agent-negotiation-referee/content/playbook_customer_escalation.html @@ -0,0 +1,16 @@ +<h1>Meridian Retail Group — Negotiation Playbook (Customer, Escalation Test Variant)</h1> +<p><em>Internal negotiation positions. Synthetic, for demonstration purposes only. Identical to playbook_customer.html except for ONE number: the Limitation of Liability walk-away ceiling, raised from two (2) times fees to three (3) times fees — above Vendor's own walk-away floor of two (2) times, so the two parties' acceptable ranges no longer overlap at all and no agreement on that term is possible. Used to prove the escalation path fires instead of looping forever. This file was produced by copying playbook_customer.html and editing exactly that one number; build.py was not touched.</em></p> + +<h2>Payment Terms</h2> +<ul> + <li>Opening ask: Net 60 days.</li> + <li>Walk-away ceiling: Net 30 days. Never agree to anything shorter than this.</li> + <li>Step size per round: 15 days.</li> +</ul> + +<h2>Limitation of Liability</h2> +<ul> + <li>Opening ask: a cap equal to five (5) times fees paid in the preceding twelve months.</li> + <li>Walk-away ceiling: three (3) times fees paid in the preceding twelve months. Never agree to a cap lower than this.</li> + <li>Step size per round: one (1) times fees.</li> +</ul> diff --git a/use-cases/shivansh193/two-agent-negotiation-referee/content/playbook_vendor.html b/use-cases/shivansh193/two-agent-negotiation-referee/content/playbook_vendor.html new file mode 100644 index 00000000..018e4699 --- /dev/null +++ b/use-cases/shivansh193/two-agent-negotiation-referee/content/playbook_vendor.html @@ -0,0 +1,16 @@ +<h1>TechFlow Solutions — Negotiation Playbook (Vendor)</h1> +<p><em>Internal negotiation positions. Synthetic, for demonstration purposes only. Edit the numbers below and re-run to change how TechFlow negotiates — nothing about these positions is hardcoded in the build script.</em></p> + +<h2>Payment Terms</h2> +<ul> + <li>Opening ask: Net 15 days.</li> + <li>Walk-away floor: Net 30 days. Never agree to anything longer than this.</li> + <li>Step size per round: 15 days.</li> +</ul> + +<h2>Limitation of Liability</h2> +<ul> + <li>Opening ask: a cap equal to one (1) times fees paid in the preceding twelve months.</li> + <li>Walk-away floor: two (2) times fees paid in the preceding twelve months. Never agree to a cap higher than this.</li> + <li>Step size per round: one (1) times fees.</li> +</ul> diff --git a/use-cases/shivansh193/two-agent-negotiation-referee/requirements.txt b/use-cases/shivansh193/two-agent-negotiation-referee/requirements.txt new file mode 100644 index 00000000..7507eb03 --- /dev/null +++ b/use-cases/shivansh193/two-agent-negotiation-referee/requirements.txt @@ -0,0 +1,2 @@ +httpx>=0.27 +python-dotenv>=1.0