Run every AI step through the CAIL Gateway and improve remediation quality

[szweibel]

Phase 2 of the CAIL integration plan: every AI step runs through the CAIL Gateway with an open-weight vision model, and the remediation fixes found with the evaluation suite come with it. Merging this and deploying it on NML gives v1 users these gains before the Worker version (v2) exists.

What changes

• One model lane. Every AI step (alt text, title, front matter, table of contents, tables, forms, reading order) sends rendered page images to chat completions at LLM_BASE_URL (the Gateway). The direct Gemini path, its PDF input, the separate local-semantic lane and GEMINI_API_KEY are gone. The default model is qwen3-vl-235b-a22b-instruct, chosen in a bake-off (see evaluation.md). Structured output falls back from json_schema to json_object to plain JSON. A timed-out call is never repeated, because it may already have been paid for.

• Remediation quality. Scored on what a screen reader that follows the tags hears, each marked-content sequence read as Chrome's PDF engine (pdfium) extracts text; AI steps off. Full tables in evaluation.md:

  Measure	v1	This branch
  olmOCR-Bench tables sample (cell neighbors and headings)	10.4%	69.9%
  olmOCR-Bench multi_column sample (reading order)	41.6%	73.5%
  multi_column, as scans	51.6%	67.6%
  tables, as scans	23.0%	56.7%
  Printed books 1794–1925 (our old_print; Adobe 83.7%)	92.1%	93.9%
  Gold round trip, NID / TEDS / MHS-L	0.791 / 0.300 / 0.339	0.864 / 0.544 / 0.436
  CUNY round trip, NID / TEDS / MHS-L	0.877 / 0.512 / 0.350	0.963 / 0.683 / 0.397

  • One rule decides which element content belongs to. Each word goes to the element or table cell whose box holds it, the rule element text uses, judged in the direction the text runs. A text operator crossing elements is split, inside a string between glyphs where needed, as opendataloader-pdf does; the page renders identically. This replaced a per-page choice between Hungarian matching and run scoring (640 lines, and the scipy and rtree dependencies).
  • Elements are read from their own glyphs. /ActualText remains only for corrections, formulas, and composed accents, as the Tagged PDF Best Practice Guide advises; the element-level overlays are gone at a cost of at most 0.3 points on any whole olmOCR-Bench category.
  • The structure tree follows Docling's reading order, not drawing order: +3.5 points on the whole multi_column category (231 pages), no change on scans or CUNY documents, at a cost on the gold documents (NID 0.887 → 0.862) that are drawn in reading order already.
  • Text is recognized once. Docling reads the OCR step's text layer instead of recognizing scans again with RapidOCR (scanned multi-column sample 63.5% → 68.5%).
  • A paragraph continued across a page break is one structure element.
  • Text positions come from pdfminer's measurement of each text operator; Docling's boxes are moved into PDF user space before tagging (rotated pages and offset MediaBoxes were mistagged).
  • Element and table-cell text is the words pdfium reads inside the element's boxes, when they match Docling's letters.
  • Heading levels come from Docling's hierarchy stage, survive tagging, and sibling headings stay siblings.
  • Captions and table footnotes are tagged as text, not artifacts.
  • Pictured text on born-digital pages is OCR'd with --redo-ocr --pages, unless the page already carries an OCR layer; a picture page with no text in an otherwise born-digital document is OCR'd.
  • Every OCR line reaches the structure tree; language detection works (lingua is a dependency); ICC profiles that lack /N no longer break text extraction.

• Optional Gateway OCR (OCR_ENGINE=gateway). It helps handwriting but not printed books, so Tesseract stays the default.

• Evaluation corpus, harness and scorers. The corpus is defined in backend/eval/ and rebuilt byte for byte by scripts/fetch_eval_corpus.py: olmOCR-Bench samples and whole categories, a printed-book set with our own tests, gold and CUNY Academic Works round trips, and synthetic scans of each. Files that cannot be re-downloaded live in the private CUNY-AI-Lab/pdf-accessibility-eval-data. scripts/pipeline_bench.py keeps each page's tagger inputs, and scripts/retag_bench.py re-tags every page with other tagger code in about a minute. Adobe Auto-Tag and opendataloader-pdf are measured as peers. Ruff runs in scripts/check.sh and in CI.

• Dependencies: pypdfium2 moves to the runtime dependencies (it was already in the image through Docling); scipy, rtree, and the never-used tenacity are dropped.

• Merged from main: PR Bound docling-serve load from public readiness checks #11's readiness probe limit.

• Docs: the plan records the 2026-10-04 decisions (actual-dell as the pipeline host, the Gateway token staying on Cloudflare) and the quality work done and next. The README's NML settings name the Gateway and actual-dell's docling-serve.

Cross-system record

• Task and user action: someone uploads a PDF to https://tools.ailab.gc.cuny.edu/pdf-accessibility/, and its AI steps now call the CAIL Gateway instead of Google.
• Caller → receiver: the PDF app on NML calls the CAIL Gateway /v1/chat/completions with the Lab's own sk-cail key. This is still the anonymous v1, so all spend lands on that key, as it does on the Gemini key today.
• Deployment order:
  1. Get or mint the app's Gateway key.
  2. Upgrade actual-dell's docling-serve to the version the evaluation used: 1.35.0, with Docling 2.130. Production runs 1.12.0 with Docling 2.72. The branch asks for do_pdf_heading_hierarchy, and against 1.12 every heading came back H1. Every result in evaluation.md was measured on 1.35, the v1 column included, so v1 as deployed today already works against it.
  3. Merge this PR (CI on main).
  4. On NML, set LLM_BASE_URL, LLM_MODEL and LLM_API_KEY in .env and delete GEMINI_API_KEY and USE_DIRECT_GEMINI_PDF. Without a real LLM_API_KEY the app refuses to start.
  5. git pull && docker compose up -d --build.
  6. Check /health/ready, then upload one real PDF through the public URL.
• Boundary exercised (2026-10-04): the image built from this branch (df08f67 plus doc commits) ran locally with a personal Gateway key and actual-dell's docling-serve. A two-page born-digital PDF uploaded through /api/jobs finished complete, veraPDF reported it compliant, the alt-text step made 2 Gateway calls (the model refused json_schema and the step fell back to json_object, as designed), the chart got Gateway-written alt text, the table was tagged with TH/TD, and title and language were set. The quality commits since change only the deterministic pipeline. Every pipeline result went through run_pipeline with docling-serve 1.35 in the image built from this branch; the branch's results are a full run of de2e819 (GPU docling-serve), and its tagging layers were compared by re-tagging inputs captured from such runs.
• Local gate: ./scripts/check.sh passes on de2e819 (code; 9de3605 adds docs only): ruff is clean, 607 tests pass, and the frontend lints and builds. docker build succeeds and the image imports the changed modules.
• Still missing: the deployed NML path with the Lab key, which needs the key and Stephen's go-ahead to deploy.

Known gaps, not addressed here

• Credential refusals (401, 403, 429 quota_exceeded) are still absorbed by the AI steps' fallbacks. This is plan step 2.3.
• Reading order is the weakest measure on multi-column and printed pages, and Docling's order misreads some layouts that well-made documents draw in order.
• Word spaces the PDF makes by position are not written as space characters (ISO 32000-1, 14.8.2.5), so readers that do not infer spaces from geometry may run words together.
• Manuscripts fell from 25.9% to 24.1%: Docling's RapidOCR read some handwriting better than Tesseract.

🤖 Generated with Claude Code