Invoice Resolution Console

A runnable take-home case-study project for PS-1: Invoice processing — from PDF to decision.

The console accepts a real invoice PDF, matches it against a real purchase-order CSV, and creates an auditable decision:

• APPROVED
• NEEDS_REVIEW
• REJECTED

It uses MEDHA—the served name for cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit, running behind an OpenAI-compatible vLLM endpoint—for structured extraction when configured. PO matching and finance decisions remain deterministic in Python. A human reviewer can resolve NEEDS_REVIEW invoices from the dashboard; that resolution re-runs the same deterministic policy engine instead of trusting a free-form human verdict.

Why this architecture

• PostgreSQL is both the durable application database and the job queue — no separate broker.
• Workers safely claim jobs with FOR UPDATE SKIP LOCKED plus a lease, so a crashed worker's job is retried by another worker instead of hanging forever.
• Every stage writes an append-only invoice_events record, which is what the dashboard's live timeline is built from.
• Exact-file duplicates (same PDF bytes, via sha256) and business-level duplicates (same vendor + invoice number) are deliberately separate checks.
• The model extracts facts; Python rules make financial decisions. MEDHA is never allowed to approve or reject an invoice.
• PO-balance consumption, the result row, and job closure happen in one Postgres transaction, so two workers racing on the same PO can never over-allocate it.
• Every decision stores a frozen policy_snapshot + policy_hash of the vendor rule values it was judged under, so a later change to vendor_rules.json can never rewrite history.
• A vendor+invoice-number "identity claim" has its own lifecycle (PENDING → FINAL / RELEASED), decoupled from the job's own status, so failed/rejected runs release the identity for a corrected re-upload instead of blocking it forever.

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│ React / Vite frontend                                               │
│ Dashboard · Upload · Live run · Review · Settings                   │
│ Authenticated actor · TanStack Query polling · protected PDF preview│
└───────────────────────────────┬─────────────────────────────────────┘
                                │ HTTPS / JSON
                                ▼
┌─────────────────────────────────────────────────────────────────────┐
│ FastAPI application                                                 │
│ Auth + RBAC · secure PDF intake · jobs · review · retry · documents │
│ operations overview · Prometheus exposition · request correlation   │
└──────────────────────┬──────────────────────────┬───────────────────┘
                       │ SQL                      │ validated files
                       ▼                          ▼
┌──────────────────────────────────┐   ┌─────────────────────────────┐
│ PostgreSQL                       │   │ Managed local storage       │
│                                  │   │ Quarantine → validated PDF  │
│ invoice_documents                │   │ Server-generated keys only  │
│ invoice_jobs (durable queue)     │   └──────────────┬──────────────┘
│ invoice_events (append-only)     │                  │ PDF/pages
│ purchase_orders                  │                  │
│ po_invoice_allocations           │                  │
│ invoice_results                  │                  │
│ invoice_identity_claims          │                  │
│ invoice_review_actions           │                  │
└──────────────────┬───────────────┘                  │
                   │ `FOR UPDATE SKIP LOCKED`         │
                   │ claim + lease                    │
                   ▼                                  ▼
┌─────────────────────────────────────────────────────────────────────┐
│ Durable worker                                                      │
│ process_job() · stage events · retry/lease recovery                 │
│ PyMuPDF native text/rendering · optional PaddleOCR · MEDHA client   │
│ deterministic PO matching/policy · atomic financial finalization    │
└───────────────────────────────┬─────────────────────────────────────┘
                                │ OpenAI-compatible HTTPS
                                ▼
┌─────────────────────────────────────────────────────────────────────┐
│ MEDHA / vLLM                                                        │
│ `cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit` served as `Medha`            │
│ Text + page images → schema-constrained extraction JSON             │
└─────────────────────────────────────────────────────────────────────┘

Filesystem storage (storage/) holds the uploaded PDF bytes and rendered page-image artifacts; only the storage key is kept in Postgres.

Ownership and safety boundaries

Concern	Source of truth	Safety property
Work scheduling	invoice_jobs	Leased SKIP LOCKED claims allow concurrent workers and crash recovery
Pipeline history	invoice_events	Append-only events drive the live run and preserve stage evidence
Final outcome	invoice_results	Stores immutable model extraction plus the frozen policy snapshot/hash
PO consumption	po_invoice_allocations + purchase_orders.consumed_amount	Allocation, result, job closure, claim settlement, and audit events commit together
Business duplicate ownership	invoice_identity_claims	Partial uniqueness permits one live PENDING/FINAL vendor-invoice identity
Human actions	invoice_review_actions	Server-derived actor, corrections, note, and before/after decision remain auditable
Operations dashboard	PostgreSQL aggregates	Cross-process queue, decision, provider, and latency truth
Service telemetry	Per-process /metrics + structured logs	Low-cardinality metrics and X-Request-ID correlation without invoice PII labels

Processing flow (one invoice, start to finish)

 1. Upload PO CSV  ─────────────────────────►  purchase_orders table (upsert by po_number)

 2. Upload invoice PDF
       │
       ├─ reject if not a PDF / too large / unreadable
       ├─ sha256 hash                      → duplicate exact-file upload returns the existing job
       └─ create invoice_documents row + invoice_jobs row (status = PENDING)
              │
              ▼
 3. Worker polls, claims job (SKIP LOCKED, lease) → status = PROCESSING
              │
              ▼
 4. stage_pdf_validate      PyMuPDF opens PDF, checks page count ≤ MAX_PDF_PAGES
              │
              ▼
 5. stage_text_extract      native PyMuPDF text; if too short → render page PNGs
              │                                  (weak text also triggers OCR fallback)
              ▼
 6. stage_ocr_fallback      PaddleOCR reads the rendered pages
              │
              ▼
 7. stage_medha_extract     MEDHA reads text + page images,
              │             returns strict JSON: vendor, invoice#, PO#, amounts,
              │             line items, confidence, per-field evidence
              │             — else: deterministic heuristic_extract() fallback
              ▼
 8. stage_semantic_duplicate  claim (vendor_normalized, invoice_number_normalized)
              │               identity; a live PENDING/FINAL claim blocks/duplicates this run
              ▼
 9. stage_po_match          exact PO#, single open PO, or "only one affordable PO" match;
              │             multiple/no unambiguous match → NEEDS_REVIEW candidate list
              ▼
10. evaluate_decision()     required fields, currency, arithmetic tolerance,
              │             confidence threshold, duplicate state, PO open/vendor/
              │             currency/remaining-balance checks → policy snapshot + hash
              ▼
11. finalize_invoice_decision()   ONE transaction:
              │                   - reserve PO balance (only if APPROVED)
              │                   - write invoice_results
              │                   - close job (status = COMPLETED)
              │                   - settle identity claim (FINAL / RELEASED / stays PENDING)
              │                   - write stage_policy_validate + invoice_closed events
              ▼
12. Dashboard shows live stage timeline, evidence, rule checks, and final outcome.
        If NEEDS_REVIEW: reviewer selects a PO and/or supplies corrections;
        the server builds effective_extraction and re-runs evaluate_decision
        under the frozen policy snapshot. Attestation bypasses only the confidence
        gate; all financial and duplicate checks still run.
        If FAILED: an operator can retry, which re-queues under a new retry_generation.

Repository layout

app/
  api/routes.py            FastAPI endpoints
  core/config.py           pydantic-settings environment config
  core/schemas.py          Pydantic models (extraction, decision, requests)
  core/observability.py    bounded metrics, structured logs, request correlation
  db/pg.py                 connection pool + schema DDL (idempotent, run on startup)
  db/repository.py         all SQL: queue claim, finalize transaction, identity claims, retry
  pipeline/normalizer.py   name/invoice-number normalization, heuristic fallback extraction
  pipeline/decision.py     vendor rules, PO matching, deterministic policy evaluation
  pipeline/review.py       human resolution of NEEDS_REVIEW jobs
  pipeline/orchestrator.py process_job(): the per-invoice pipeline a worker runs
  services/pdf.py          PyMuPDF text/page extraction + optional PaddleOCR
  services/medha.py        MEDHA (OpenAI-compatible) client
  services/storage.py      local file storage adapter
  worker.py                standalone durable-queue polling worker
config/
  vendor_rules.json        per-vendor tolerance/confidence/currency/PO-requirement config
data/
  sample_invoices/
    Happy-path PDFs/       4 approvable invoices + matching PO master CSV
    Edge-Case PDFs/        ambiguous PO, exhausted balance, low-quality scan, semantic duplicate
frontend/                  React/Vite/Tailwind dashboard (shadcn/ui components)
scripts/generate_sample_invoices.py   regenerates the legacy 4-PDF demo set
tests/                     pytest suite (see "Test" below)

Quick start with Docker

cp .env.example .env
docker compose up --build

Open:

• Dashboard: http://localhost:5173
• API docs: http://localhost:8000/docs

Seed a PO master from the dashboard's Settings page, or via API:

curl -F "file=@data/sample_invoices/Happy-path PDFs/happy_path_purchase_orders.csv" \
  http://localhost:8000/api/purchase-orders/import

Then upload PDFs from data/sample_invoices/Happy-path PDFs/ and data/sample_invoices/Edge-Case PDFs/ (see Demo script below for a suggested order and what each one proves).

Without MEDHA credentials, invoices still run through the deterministic native-text fallback (heuristic_extract) so the pipeline is runnable end-to-end offline. Configure MEDHA for robust scanned-document extraction.

Configure MEDHA

Set these only in .env; never hardcode an internal address or key.

MEDHA_API_URL=https://your-medha-endpoint/v1
MEDHA_API_KEY=replace-me
MEDHA_MODEL=Medha
ENABLE_MEDHA=true

The client calls the OpenAI-compatible endpoint at:

${MEDHA_API_URL}/chat/completions

It sends page images plus available native/OCR text and requests JSON-only output (vendor, invoice number, PO number, amounts, line items, per-field evidence, confidence). The model cannot approve an invoice — app/pipeline/decision.py is the only place a verdict is produced.

Optional local open-source OCR

For scanned PDFs, install the PaddlePaddle wheel suitable for your CPU/GPU, then:

pip install -r requirements.txt -r requirements-ocr.txt

Enable it in .env:

ENABLE_PADDLE_OCR=true

Pipeline behavior:

Native PDF text available (≥ NATIVE_TEXT_MIN_CHARS) → PyMuPDF text  → MEDHA / deterministic fallback
Native text weak                                     → page render → PaddleOCR (optional) → MEDHA

Decision policy

Evaluated deterministically in app/pipeline/decision.py, per vendor rule from config/vendor_rules.json:

Check	Outcome if it fails
Required fields present (vendor, invoice number, total)	NEEDS_REVIEW
PO number required by vendor policy but missing	NEEDS_REVIEW
Currency allowed for this vendor	NEEDS_REVIEW
subtotal + tax == total within amount_tolerance	NEEDS_REVIEW
Extraction confidence ≥ vendor's minimum	NEEDS_REVIEW
Same vendor + invoice number seen before, same total, and that prior claim is FINAL	REJECTED
Same identity seen before but still PENDING (another run in flight)	NEEDS_REVIEW
Same identity, different total	NEEDS_REVIEW
Matched PO is closed	REJECTED
No unambiguous PO match (0 or >1 candidates)	NEEDS_REVIEW
PO vendor / currency mismatch, or total exceeds remaining PO balance	NEEDS_REVIEW
All checks pass	APPROVED

All money values use Python Decimal. When an invoice is approved (by the pipeline or by a reviewer), finalize_invoice_decision atomically reserves the PO amount inside the same transaction that writes the result and closes the job — two workers (or a worker and a reviewer) can never over-allocate the same balance. If the balance no longer covers the invoice at commit time, the decision is silently downgraded to NEEDS_REVIEW instead of allowed to over-allocate.

Human review flow

A NEEDS_REVIEW job is resolved from the Run Detail page (ReviewPanel), never by hand-writing an outcome:

• Approve: the reviewer may select a PO and/or correct extracted fields. The server builds an effective_extraction and re-runs evaluate_decision using the invoice's frozen policy_snapshot. Reviewer attestation bypasses only the low-confidence gate; the model's original confidence is preserved unchanged. PO status, currency, arithmetic, remaining balance, and duplicate-identity checks still apply. If validation still fails, the API returns 422 and the resolution is not committed.
• Reject: always allowed with a note; releases the identity claim so a corrected re-upload can be processed.
• A job that is no longer NEEDS_REVIEW (already resolved, or resolved concurrently by someone else) returns 409 — resolutions are not idempotent replays.
• The original model extraction in invoice_results is never overwritten. Corrections are stored in invoice_review_actions.corrections, included in the append-only stage_review_resolve audit event, and exposed through effective_extraction when reading job details.
• The reviewer and retry actor are derived from the authenticated identity. Compatibility fields such as reviewer_name and requested_by in request bodies are ignored.

Authentication and secure intake

There is no password or login flow in this service. AUTH_MODE=development uses the clearly marked local demo identity and is accepted only with ENVIRONMENT=development. Any other environment must use AUTH_MODE=jwt, provide JWT_SECRET, and send a bearer JWT with sub, exp, role, and optionally name. Supported roles are:

Role	Access
viewer	Read jobs, evidence, and documents
operator	Viewer access plus invoice upload
reviewer	Viewer access plus review resolution and failed-job retry
admin	All access, including purchase-order import and demo seed

Invoice uploads are streamed to an OS quarantine file while hashing. The server ignores the supplied media type and storage name, verifies %PDF- magic and PyMuPDF parseability, rejects encrypted files, and enforces MAX_UPLOAD_BYTES, MAX_PDF_PAGES, and MAX_RENDER_PIXELS before creating any database state. This is format validation and resource limiting, not antivirus or malware scanning.

PDFs are available only through the authenticated document API. Responses use private/no-store caching, content sniffing protection, a safe server-sanitized download name, and sandbox-oriented headers.

Observability

• GET /metrics exposes bounded, low-cardinality Prometheus metrics for HTTP requests, pipeline stages, decisions, failures, OCR usage, and MEDHA calls.
• GET /api/ops/overview?window_hours=24 returns PostgreSQL-backed operational aggregates including queue state, review backlog, decision mix, p50/p95 latency, OCR fallback rate, provider health, and top failures.
• Logs are emitted as structured JSON and correlated using X-Request-ID.
• Prometheus metrics are per process; the PostgreSQL-backed operations overview is the cross-process source of truth.

API endpoints

Endpoint	Purpose
GET /api/auth/me	Server-derived authenticated actor
POST /api/purchase-orders/import	Upload PO master CSV (upsert by po_number)
POST /api/invoices/upload	Upload invoice PDF and queue work (202)
GET /api/jobs	Dashboard history
GET /api/jobs/{job_id}	Job, event timeline, result, review actions, allocations
GET /api/jobs/{job_id}/review/candidates	Open POs for this invoice's vendor, with live remaining balance
POST /api/jobs/{job_id}/review/resolve	Approve/reject a NEEDS_REVIEW job
POST /api/jobs/{job_id}/retry	Re-queue a FAILED job (409 otherwise)
GET /api/documents/{document_id}/file	Original PDF
POST /api/demo/seed-purchase-orders	Seed data/purchase_orders.csv if present
GET /api/ops/overview	Operational queue, reliability, provider, and latency aggregates
GET /metrics	Prometheus text exposition for the current process

Test

Current verified baseline: 57 passing backend tests and a clean frontend production build.

Fast, DB-independent tests (decision policy, normalizer):

python -m pytest tests/test_decision.py tests/test_normalizer.py

Full suite, including transactional integrity tests against a real Postgres (skipped automatically if unreachable — start it with docker compose up postgres or docker compose up --build):

python -m pytest

Frontend production build:

cd frontend
npm ci
npm run build

Coverage:

File	What it proves
test_decision.py	Approval, PO-balance rejection, and exact-duplicate rejection logic
test_normalizer.py	Heuristic text extraction, name/invoice-number normalization, fenced-JSON parsing
test_finalization.py	No partial state on a failed write; two concurrent approvals cannot over-allocate a PO; a retried finalization creates exactly one allocation
test_identity.py	Identity-claim lifecycle: failure releases it, review keeps it pending, rejection releases it, approval finalizes it, a released identity can be reclaimed, a second worker cannot steal a live claim
test_review.py	Reviewer approval/rejection, double-resolution conflict (409), balance-exceeding approval refused, corrections never overwrite the stored model extraction
test_retry.py	Only FAILED jobs retry, a NEEDS_REVIEW job cannot, a retry cannot double-consume an allocation, policy snapshots survive later vendor_rules.json changes
test_corrections.py	Correction validation, immutable model extraction, effective extraction, identity migration, and approval re-evaluation
test_observability.py	Bounded metric labels, redaction, aggregate windows, OCR fallback, MEDHA outcomes, and friendly-stage timing
test_security.py	Development/JWT authentication, RBAC, upload limits, PDF validation, traversal protection, protected PDF access, actor derivation, and log redaction

Demo script

1. Seed the happy-path PO master (data/sample_invoices/Happy-path PDFs/happy_path_purchase_orders.csv).
2. Upload the four Happy-path PDFs/*.pdf invoices; each should extract cleanly and reach APPROVED against its matching PO.
3. Upload Edge-Case PDFs/semantic_duplicate_invoice.pdf (same vendor + invoice number as an already-approved one) → REJECTED with a duplicate-identity reason.
4. Seed Edge-Case PDFs/realistic_purchase_order_master.csv, then upload:
   • po_balance_exhausted_invoice.pdf → NEEDS_REVIEW, remaining PO balance insufficient.
   • ambiguous_po_match_invoice.pdf → NEEDS_REVIEW, multiple open POs match the vendor; resolve it from the Run Detail review panel by picking a PO.
   • low_quality_scanned_invoice.pdf → exercises the OCR/page-image fallback path.
5. Demonstrate retry by inducing a worker-stage failure after a valid PDF has been accepted—for example, temporarily make the configured MEDHA endpoint unavailable when fallback is disabled—then restore the dependency and retry the FAILED job.
6. Explain that model output is validated and every policy decision is deterministic, reproducible from the stored policy_snapshot/policy_hash, and reviewer resolutions re-run the same policy engine rather than trusting a free-form human verdict.

Deliberate trade-offs and boundaries

• PostgreSQL is sufficient as the durable queue for this workload and keeps queue state, leases, results, and financial locking in one transactional system. A separate broker would become useful only at substantially higher throughput or with multiple workflow types.
• Local filesystem storage keeps the take-home runnable. A multi-host production deployment should replace it with private object storage while retaining server-generated keys and authenticated access.
• Secure intake performs strict format and resource validation, but it is not antivirus scanning. A production deployment can add a quarantine scanner before promoting a file to managed storage.
• MEDHA and PaddleOCR improve extraction quality; deterministic policy evaluation remains the decision authority. Heuristic extraction keeps text-based demo invoices runnable when model access is unavailable.
• Development identity exists only for local demonstration. Non-development deployments are guarded to require JWT authentication.