Nojhi3 round 2: edit-quality rubric study and others

extensions/Nojhi3/edit-quality-rubric-study/.env.example

@@ -0,0 +1,12 @@
+SUPERDOCS_API_BASE=https://api.superdocs.app
+SUPERDOCS_API_KEY=sk_your-key-here
+SUPERDOCS_MODEL_TIER=pro
+SUPERDOCS_THINKING_DEPTH=balanced
+
+# Automated judge. gemini (default) uses any OpenAI-compatible endpoint.
+# JUDGE_PROVIDER=superdocs uses SuperDocs' own chat model (pilot runs found it
+# unreliable for JSON scoring - see README finding).
+JUDGE_PROVIDER=gemini
+GEMINI_API_KEY=your-gemini-key
+GEMINI_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai
+GEMINI_MODEL=gemini-3.6-flash

extensions/Nojhi3/edit-quality-rubric-study/.gitignore

@@ -0,0 +1,6 @@
+.env
+__pycache__/
+*.pyc
+.venv/
+venv/
+data/edits/*/after.docx

extensions/Nojhi3/edit-quality-rubric-study/HOW_TO_RUN.md

@@ -0,0 +1,198 @@
+# How to run — every scenario, with what to expect
+
+This is the run book for the Edit-Quality Rubric + Inter-Rater Agreement study.
+Commands are split into **zero-spend** (offline, safe to run any time) and
+**budget-spending** (real SuperDocs calls, guarded by the operation cap).
+
+All commands run from this project root unless stated otherwise.
+
+---
+
+## A. Prerequisites
+
+```bash
+pip install -r requirements.txt          # runtime
+pip install -r requirements-dev.txt      # + pytest, for the test suite
+
+cp .env.example .env                     # then fill in the values:
+# SUPERDOCS_API_KEY=sk_your-key-here     # SuperDocs key (agent self-signup: POST
+#                                        #   https://api.superdocs.app/v1/agents/signup)
+# GEMINI_API_KEY=your-gemini-key         # only needed for the automated judge
+```
+
+- **Budget cap:** default **60 operations** (see `app/config.py`,
+  `DEFAULT_BUDGET_CAP_OPS`). Override any run with `--cap N`.
+- **Small sample vs full:** `--small` = first 6 cases; `--full` = all 11.
+- Every billable call is checked against the cap **before** it starts and
+  recorded in `data/ledger.jsonl`.
+
+---
+
+## B. Zero-spend scenarios (offline, no API calls)
+
+### B1. Full test suite — 141 tests
+
+```bash
+python -m pytest
+```
+
+Expect: `141 passed`. Covers hand-derived kappa (unweighted, linear-weighted,
+band edges, bootstrap CIs), the full edit pipeline against a scripted fake
+SuperDocs client (happy path, revision loop, failures, budget cap), both
+judges against a mocked HTTP layer, report conversion, every FastAPI endpoint,
+and an integration test that pins the study's headline numbers.
+
+### B2. Integrity self-check
+
+```bash
+python scripts/selftest.py
+```
+
+Expect: `ALL CHECKS PASSED` (exit 0). Verifies the rubric schema, all 11
+cases' edit artifacts, both raters' score sheets, hand-derived kappa examples,
+report rendering, the exported PDF/DOCX files, ledger consistency, and that
+no API keys exist outside `.env`.
+
+### B3. Inspect the rubric
+
+```bash
+python -m json.tool data/rubric.json        # offline, straight from the file
+```
+
+or, with the server running (see C2):
+
+```bash
+curl -s http://127.0.0.1:8000/rubric
+```
+
+Expect: `rubric_name`, `version`, `scale`, `instructions`, and 6 `dimensions`,
+each with a 1-4 scale of written anchors, e.g.
+`faithfulness` — "Did the edit do what the instruction asked, at the scope the
+instruction implied?" with anchors for 1 (ignored the instruction) through
+4 (did exactly and only what was asked).
+
+### B4. Inspect the study data
+
+```bash
+ls data/edits/<case_id>/        # per-edit artifacts for each of the 11 cases:
+#   before.html / before.md     original source document
+#   after.html / after.md       edited document
+#   upload.json, chat_start.json, pending_changes.json, chat_result.json,
+#   job_final.json, record.json full audit of the upload->chat->approve->export flow
+#   after.docx                  exported finished file
+ls data/judgments/human/        # 11 human score sheets (one per case)
+ls data/judgments/auto/         # 11 automated-judge score sheets (blind)
+tail data/ledger.jsonl          # SuperDocs operation ledger (cap + actual spend)
+```
+
+### B5. Budget guard refusal demo — **spends 0 ops**
+
+The guard fires **before** any billable call, so this costs nothing:
+
+```bash
+python scripts/run_study.py --edits plan-01 --cap 0
+```
+
+Expect:
+
+```
+Cap stated before this run: 0 operations.
+Account: a2ab81ae-... | tier: free | remaining ops: <n> | used: <n>
+Cases selected: all
+
+>>> EDIT PIPELINE for ['plan-01']
+>>> done in 2s (0 completed, 1 failed)
+    failed: {'case_id': 'plan-01', 'error': 'budget cap exceeded'}
+[budget] after edits: spent 12/60 ops (12 billable calls)
+```
+
+The run refuses to start the billable edit call because `spent (12) + 1 > cap (0)`.
+The ledger gets **no** new entry. To see a refusal *mid*-run instead, use a cap
+large enough for one case: `python scripts/run_study.py --edits plan-01 plan-02 --cap 1`
+completes plan-01 (1 op) then refuses plan-02 (costs 1 op and re-runs plan-01).
+
+---
+
+## C. Budget-spending study scenarios
+
+Every run prints the cap **before** it starts and the actual spend **after**
+every step. Example full flow:
+
+```bash
+# 1) run the edits (small sample = 6 cases, ~1 op each)
+python scripts/run_study.py --small --edits
+
+# 2) human judging — open the UI, score every completed case
+uvicorn app.main:app
+# open http://127.0.0.1:8000/judge
+#   each card shows the INSTRUCTION, the source document, and the edited
+#   document side by side, with per-dimension 1-4 scores + an evidence box.
+
+# 3) automated judge (blind to human scores; separate Gemini token ledger)
+python scripts/run_study.py --small --judge-auto
+
+# 4) assemble the report
+python scripts/run_study.py --report
+```
+
+The full run (`--full`) covers all 11 cases instead of 6.
+
+---
+
+## D. Report export — non-billable
+
+Exports the study report through SuperDocs (0 operations). With the server
+running:
+
+```bash
+curl -o report.pdf  "http://127.0.0.1:8000/study/report/export?format=pdf"
+curl -o report.docx "http://127.0.0.1:8000/study/report/export?format=docx"
+curl -o report.html "http://127.0.0.1:8000/study/report/export?format=html"
+curl -o report.md   "http://127.0.0.1:8000/study/report/export?format=markdown"
+```
+
+Expect a valid file (PDF starts with `%PDF-`). An invalid format is rejected:
+`?format=exe` returns **422**.
+
+---
+
+## E. Server endpoint smoke checklist
+
+`uvicorn app.main:app` then:
+
+| Endpoint | Expected |
+| --- | --- |
+| `GET /` | JSON index of every endpoint |
+| `GET /rubric` | rubric JSON (6 dimensions, 1-4 anchors) |
+| `GET /cases` | the 11 cases with difficulty bands |
+| `GET /judge` | human judging UI (HTML) |
+| `GET /study/scores` | both raters' score sheets |
+| `GET /study/analysis` | kappa **0.339** (95% CI), weighted **0.405**, per-dimension bands, confusion matrices |
+| `GET /study/budget` | cap 60, spent 12, remaining 48 |
+| `GET /study/report` | the full report as markdown |
+| `GET /study/report/export?format=pdf` | report PDF (`application/pdf`) |
+
+---
+
+## F. Where the results are
+
+- **`report/study_report.md`** — the study report:
+  - §3 **Budget guard** — cap stated before the run
+  - §4 **Actual spend** — 12 of 60 ops, full per-call ledger
+  - §6 **Inter-rater agreement** — percent agreement, kappa, CIs, per-dimension
+  - §7 **Disagreement review** — the 8 judgments where the automated judge was
+    unreliable, quoting both raters' evidence
+  - §8 **Honest limitations** — small sample, single human rater, sparse human
+    evidence, ordinal ceiling, visual-blindness
+- **`report/study_report.pdf` / `.docx`** — exported copies
+- **`data/edits/<case_id>/`** — before/after material for every judgment
+- **`data/ledger.jsonl`** — the operation ledger behind the spend numbers
+
+## G. Reproduce / verify the headline numbers
+
+The integration test pins the study's numbers so a code change that alters any
+of them fails the suite:
+
+```bash
+python -m pytest tests/test_study_integration.py -v
+```

extensions/Nojhi3/edit-quality-rubric-study/README.md

@@ -0,0 +1,169 @@
+# Edit-Quality Rubric + Inter-Rater Agreement Study
+
+A rubric for scoring the quality of AI document edits, plus an inter-rater
+agreement study between a **human judge** and an **automated LLM judge** —
+with honest reporting of where the automated judge is unreliable.
+
+Built on SuperDocs' minimum four-call contract for every edit:
+`upload` -> `chat` (async, HITL) -> `approve` -> `export`.
+
+## What it does
+
+1. **Runs edits.** A corpus of 3 documents (a project plan, meeting notes, a
+   service contract with a pricing table) x a battery of instructions across
+   four difficulty bands (easy / medium / hard / ambiguous) = 11 edits. Every
+   edit goes through SuperDocs with human-in-the-loop approval of each
+   proposed change, then exports the finished file.
+2. **Scores with a rubric.** A 6-dimension ordinal rubric (1-4 with written
+   anchors) for document edit quality: faithfulness to instruction, fact
+   accuracy, content preservation, structural integrity, formatting fidelity,
+   language quality.
+3. **Measures agreement.** A human judge and an automated judge (an LLM, blind
+   to the other's scores) each score every edit. Cohen's kappa (unweighted and
+   linear-weighted), percent agreement, per-dimension confusion matrices,
+   percentile-bootstrap 95% CIs, and a case-by-case disagreement review —
+   computed in pure Python, no scipy.
+4. **Reports honestly.** The study report (markdown, plus a PDF/DOCX export
+   produced through SuperDocs' export endpoint) states the budget cap before
+   each run, reports actual operation spend, and names every judgment where
+   the automated judge was unreliable, quoting both raters' evidence.
+
+## Finding from the pilot: SuperDocs chat is an editor, not a scorer
+
+The automated judge was first implemented with SuperDocs' own chat model.
+Pilot runs showed it will not reliably score documents: it answers long
+scoring prompts with short acknowledgments ("Done.", "I'll score the edit
+now."), and with `approval_mode=ask_every_time` it only acknowledges. A JSON
+scoring turn therefore uses a general-purpose LLM (default: Gemini via any
+OpenAI-compatible endpoint, `JUDGE_PROVIDER=gemini`), recorded in its own
+token ledger (`data/ledger_gemini.jsonl`). The SuperDocs-chat judge is kept
+for comparison (`JUDGE_PROVIDER=superdocs`). This is itself a reliability
+finding about automated judging on this platform.
+
+## Budget guard
+
+- Cap stated before every run: **60 operations** (default, editable in
+  `app/budget.py`). Every billable call is checked against the cap before it
+  starts and recorded in `data/ledger.jsonl` (endpoint, billable flag, ops
+  charged, monthly remaining).
+- Small-sample mode: `--small` runs the first 6 cases; the full run is all 11.
+- Actual spend is printed after every step and published in the report.
+
+## How to run
+
+```bash
+pip install -r requirements.txt
+
+# 1) SuperDocs key (free agent self-signup, no human needed):
+#    POST https://api.superdocs.app/v1/agents/signup  {"terms_accepted": true}
+cp .env.example .env   # fill SUPERDOCS_API_KEY, GEMINI_API_KEY
+
+# 2) Run the edit pipeline (small sample first):
+python scripts/run_study.py --small --edits
+
+# 3) Human judging UI:
+uvicorn app.main:app          # open http://127.0.0.1:8000/judge
+
+# 4) Automated judge (blind to human scores):
+python scripts/run_study.py --small --judge-auto
+
+# 5) Report + export through SuperDocs:
+python scripts/run_study.py --report
+curl -o report.pdf "http://127.0.0.1:8000/study/report/export?format=pdf"
+
+# 6) Offline integrity self-check (no API calls, no spend; exit 0 = all pass):
+python scripts/selftest.py
+
+# 7) Full test suite (141 tests, offline, no spend):
+pip install -r requirements-dev.txt
+python -m pytest          # or: pytest tests/test_stats.py -v
+```
+
+### Quick recipes
+
+**Export the report** (non-billable, 0 ops):
+```bash
+python scripts/run_study.py --report
+curl -o report.pdf  "http://127.0.0.1:8000/study/report/export?format=pdf"
+curl -o report.docx "http://127.0.0.1:8000/study/report/export?format=docx"  # or html|markdown|txt
+```
+
+**Check the rubric's value** (dimensions, scale, written anchors):
+```bash
+curl -s http://127.0.0.1:8000/rubric   # live, from the API
+python -m json.tool data/rubric.json   # offline, straight from the file
+```
+
+**Start a run whose budget estimate is over the cap** — the guard refuses
+*before* any billable call, so this spends **0 ops**:
+```bash
+python scripts/run_study.py --edits plan-01 --cap 0
+# Cap stated before this run: 0 operations.
+# >>> done in 2s (0 completed, 1 failed)
+#     failed: {'case_id': 'plan-01', 'error': 'budget cap exceeded'}
+# [budget] after edits: spent 12/60 ops (12 billable calls)   <- ledger unchanged
+```
+
+Every scenario with commands and exact expected output lives in
+**[HOW_TO_RUN.md](HOW_TO_RUN.md)**.
+
+The self-check verifies the rubric schema, all 11 cases' edit artifacts,
+both raters' score sheets, the statistics on hand-derived examples (kappa =
+1.0 / 0.0 / 2/3, Landis-Koch band edges), report rendering, the exported
+DOCX/PDF files, ledger consistency, and that no API keys are committed
+outside `.env`.
+
+The pytest suite is behavioral and network-mocked. It covers: hand-derived
+kappa (unweighted, linear-weighted, band edges, bootstrap CIs, analyze() on
+synthetic sheets), the full edit pipeline against a scripted fake SuperDocs
+client (happy path, revision loop, no-op completion, failures, budget cap),
+both judges against a mocked HTTP layer (JSON extraction incl. the
+double-encoded second parse, retries, invalid-score rejection), the report
+markdown/HTML conversion (including the multi-bold regression), every FastAPI
+endpoint (validation, 404/409/422/502 paths), and — most importantly — an
+**integration test that pins the study's headline numbers** (11 cases, 66
+judgments, kappa 0.339, weighted 0.405, 8 disagreements, per-dimension
+values, score distributions, 12 ops spent). If a code change ever alters a
+study number, that test fails by design.
+
+The FastAPI service (`app/main.py`) also exposes every step as an endpoint:
+`/rubric`, `/cases`, `POST /study/edits`, `POST /study/judge/auto`, `/judge`,
+`/judge/submit`, `/study/analysis`, `/study/budget`, `/study/report`,
+`/study/report/export`.
+
+## SuperDocs features used
+
+- REST API: `POST /v1/documents/upload-base64` (upload), `POST /v1/chat/async`
+  with `approval_mode=ask_every_time` (edit + HITL review),
+  `POST /v1/chat/{session_id}/approve` (approve proposed changes, including
+  the AI's revision loop after approval), `POST /v1/documents/export` (export
+  finished documents AND the study report itself), `GET /v1/jobs/{job_id}`
+  (polling), `GET /v1/agents/whoami` (quota/usage).
+- Operation-level billing telemetry from every chat response (usage block),
+  recorded per call in `data/ledger.jsonl`.
+
+## Repo layout
+
+```
+app/            FastAPI service + client, budget guard, rubric, pipeline,
+                judges, statistics, report generation
+scripts/        run_study.py CLI (--small / --full / --edits / --judge-auto / --report),
+                selftest.py (offline integrity checks)
+tests/          pytest suite (141 tests): stats, pipeline, judges, client
+                utils, report, API endpoints, real-data integration
+data/corpus/    3 source documents (real + synthetic)
+data/rubric.json     the rubric (dimensions, anchors)
+data/instructions.json   the 11 edit cases with difficulty labels
+data/edits/<case_id>/    per-edit artifacts: upload, pending changes,
+                         approvals, job result, before/after, exported docx
+data/judgments/          human + automated score sheets
+data/ledger.jsonl        SuperDocs operation ledger (cap + actual spend)
+data/ledger_gemini.jsonl judge token ledger
+report/                  generated study report
+```
+
+## License
+
+MIT. No API keys are committed; all secrets live in a gitignored `.env`.
+
+Built for the SuperDocs engineering task (round 2) by Akshat Rai.