feat: add reproducible RAG evaluation

.github/workflows/ci.yml

@@ -30,17 +30,48 @@ jobs:
       - name: Install locked dependencies
         run: uv sync --frozen
 
-      - name: Run tests
-        run: uv run python -m unittest discover -s tests -v
+      - name: Run tests with coverage
+        run: |
+          uv run --with coverage==7.13.4 coverage run -m unittest discover -s tests -v
+          uv run --with coverage==7.13.4 coverage report
+
+      - name: Static type check
+        if: matrix.python-version == '3.11'
+        run: uv run --with mypy==1.18.2 mypy
 
       - name: Run Ruff
         if: matrix.python-version == '3.11'
         run: |
           uvx --from ruff==0.16.1 ruff check .
           uvx --from ruff==0.16.1 ruff format --check .
 
+      - name: Audit runtime dependencies
+        if: matrix.python-version == '3.11'
+        run: |
+          uv export --frozen --no-dev --format requirements-txt \
+            --no-emit-project -o /tmp/runtime-requirements.txt
+          uvx --from pip-audit==2.10.1 pip-audit \
+            -r /tmp/runtime-requirements.txt \
+            --progress-spinner off \
+            --ignore-vuln CVE-2026-45829
+
       - name: Build and smoke-test package
         if: matrix.python-version == '3.11'
         run: |
           uv build
+          python - <<'PY'
+          import glob
+          import zipfile
+
+          wheel, = glob.glob("dist/local_ai_agent-*.whl")
+          with zipfile.ZipFile(wheel) as archive:
+              names = set(archive.namelist())
+          required = {
+              "local_ai_agent/data/rag_cases.json",
+              "local_ai_agent/data/realistic_restaurant_reviews.csv",
+          }
+          missing = required - names
+          if missing:
+              raise SystemExit(f"wheel is missing packaged assets: {sorted(missing)}")
+          PY
           uvx --from dist/local_ai_agent-*.whl local-ai-agent --help

.gitignore

@@ -5,6 +5,7 @@ build/
 dist/
 wheels/
 *.egg-info
+.coverage
 
 # Virtual environments
 .venv

README.md

@@ -16,12 +16,16 @@ By default, runtime review data, embeddings, and prompts are processed by embedd
 - **Safe offline state:** analytics still load when Ollama is unavailable, while the app shows exact setup commands instead of crashing.
 - **Adaptive CSV upload:** automatically detect common headers, manually map unfamiliar names, and isolate every dataset in content-addressed Chroma storage.
 - **Reconciled indexing:** content-derived IDs survive reordering; additions, changed records, and deletions are synchronized safely.
-- **Measured RAG:** a four-case evaluation set reports retrieval recall, citation correctness, reference-grounded faithfulness, and abstention accuracy.
+- **Measured RAG:** a versioned 30-case evaluation across nine answerable domains plus abstention compares semantic retrieval with a deterministic BM25 keyword baseline, then measures citation validity, a transparent reference-term support proxy, expected-action accuracy, answer success, and abstention recall.
 - **Installable CLI:** the packaged `local-ai-agent` command exposes status, ask, chat, and evaluate workflows.
 - **Local execution by default:** Ollama handles embeddings and answer generation unless a remote host is explicitly configured.
 
 See the [architecture diagram and boundary notes](docs/architecture.md).
 
+## Live workflow demo
+
+[Watch the silent 48-second dashboard demo](docs/assets/local-ai-agent-v0.2-demo.mp4). It shows the real local question trigger, indexing state, generated answer, validated citations, and an expanded source record using the bundled dataset.
+
 ## Requirements
 
 - [Python 3.11 or newer](https://www.python.org/downloads/)
@@ -127,6 +131,21 @@ Run the measured RAG evaluation against the configured local models:
 uv run local-ai-agent evaluate
 ```
 
+Write a reproducible machine-readable report and a Markdown summary:
+
+```bash
+uv run local-ai-agent evaluate --report-dir evaluation/results/my-run
+```
+
+The versioned case manifest is tied to the dataset SHA-256 and uses immutable
+content-derived source IDs for gold relevance. The report records model tags
+and immutable Ollama digests, dataset and case-set hashes, retrieval limit,
+runtime versions, per-case RAG and BM25 rankings, and aggregate metrics.
+Retrieval quality is reported as recall@k, hit rate@k, and MRR@k for both
+semantic search and the BM25 baseline. Relevance judgments are known-positive,
+not exhaustive. The generated report is evidence for this fixed benchmark
+configuration, not a claim about general RAG performance.
+
 The original source-tree command remains available for development:
 
 ```bash
@@ -174,18 +193,32 @@ The suite covers:
 - adaptive rating, date, and categorical filtering;
 - Ollama health states;
 - evidence-alias-to-source citation validation and model abstention;
-- all four RAG evaluation metrics;
+- the 30-case evaluation set, BM25 baseline, retrieval metrics, and decomposed
+  answer, citation, support-proxy, and abstention metrics;
+- evaluation-report serialization and credential-safe provenance;
 - deterministic upload storage;
 - Streamlit rendering without Ollama.
 
-Run lint and formatting checks:
+Run the same quality gates used by CI:
 
 ```bash
+uv run --with coverage==7.13.4 coverage run -m unittest discover -s tests -v
+uv run --with coverage==7.13.4 coverage report
+uv run --with mypy==1.18.2 mypy
 uvx --from ruff==0.16.1 ruff check .
 uvx --from ruff==0.16.1 ruff format --check .
+uv export --frozen --no-dev --format requirements-txt \
+  --no-emit-project -o /tmp/runtime-requirements.txt
+uvx --from pip-audit==2.10.1 pip-audit \
+  -r /tmp/runtime-requirements.txt --progress-spinner off \
+  --ignore-vuln CVE-2026-45829
 ```
 
-GitHub Actions runs the tests on Python 3.11 and Python 3.14.
+GitHub Actions tests Python 3.11 and Python 3.14 and reports coverage for all
+six core Python modules, including `evaluation.py`. On Python 3.11 it also runs
+static typing, Ruff, formatting, the runtime-dependency audit, package build,
+and an installed-CLI smoke test. Coverage remains visible without an arbitrary
+pass threshold.
 
 ## Project layout
 
@@ -202,14 +235,16 @@ GitHub Actions runs the tests on Python 3.11 and Python 3.14.
 │   └── data/rag_cases.json          # Curated RAG evaluation set
 ├── docs/architecture.md             # Architecture and privacy boundaries
 ├── docs/architecture.svg            # Editable architecture diagram
+├── evaluation/results/               # Versioned JSON and Markdown benchmark reports
+├── SECURITY.md                       # Reporting policy and scoped risk acceptance
 ├── tests/                            # Unit, integration, evaluation, and dashboard tests
 ├── pyproject.toml
 └── uv.lock
 ```
 
 ## Dependency audit
 
-ChromaDB is declared directly at the newest PyPI release verified during this update (`1.5.9`). `pip-audit` still reports `PYSEC-2026-311` / `CVE-2026-45829`, and the advisory currently lists no fixed release. It concerns an unauthenticated Chroma HTTP server endpoint that accepts `trust_remote_code`; this application uses embedded Chroma and does not start that server. The finding is documented rather than misrepresented as resolved.
+ChromaDB is declared directly at the newest PyPI release verified during this update (`1.5.9`). `pip-audit` still reports `PYSEC-2026-311` / `CVE-2026-45829`, and the advisory currently lists no fixed release. It concerns an unauthenticated Chroma HTTP server endpoint that accepts `trust_remote_code`; this application uses embedded Chroma and does not start that server. CI ignores only this finding and fails on every other known vulnerability. The scope and removal conditions are documented in [SECURITY.md](SECURITY.md).
 
 ## Current limitations
 
@@ -218,7 +253,7 @@ ChromaDB is declared directly at the newest PyPI release verified during this up
 - Rating, date, sentiment, restaurant, and country filters are applied after semantic ranking. This is suitable for small local datasets but not optimized for very large collections.
 - Automatic mapping is conservative. Unfamiliar or ambiguous headers require confirmation in the dashboard.
 - Sentiment labels are displayed and filtered as supplied. The application does not infer sentiment when the dataset lacks a sentiment column.
-- The faithfulness score is a transparent reference-term check against cited source text, not an LLM judge or proof that every possible claim is correct.
+- The reference-term support proxy checks expected terms against answers and cited source text; it is not an LLM judge or proof that every claim is correct.
 - Topic modeling, hybrid keyword retrieval, and reranking are not implemented.
 - The application does not provide authentication or multi-user isolation and should not be exposed directly as a shared public service.
 

RELEASE_NOTES.md

@@ -0,0 +1,69 @@
+# Local AI Agent v0.2.0
+
+## What changed
+
+- Expanded the bundled RAG evaluation from 4 to 30 cases: 25 answerable cases across nine domains and 5 abstention cases.
+- Bound evaluation relevance judgments to immutable content-derived source IDs and validated the dataset hash and row count.
+- Added a deterministic BM25 keyword baseline alongside semantic retrieval.
+- Added versioned JSON and Markdown reports with exact model digests, runtime versions, dataset/case hashes, per-case outcomes, rankings, and latency.
+- Split end-to-end behavior into explicit metrics: expected-action accuracy, answer success rate, abstention recall, citation validity, and a lexical reference-term support proxy.
+- Added stricter manifest/schema validation, duplicate-observation rejection, CLI/report tests, typing, coverage reporting, dependency auditing, package-asset checks, and installed-wheel smoke testing.
+- Added a scoped security policy for the embedded-Chroma deployment boundary.
+- Added a silent live-dashboard demo showing the real question trigger, answer, validated citations, and expanded evidence.
+
+## Verified local benchmark
+
+Runtime:
+
+- Ollama `0.32.5`
+- `llama3.2:latest` digest `a80c4f17acd55265feec403c7aef86be0c25983ab279d83f3bcd3abbcb5b8b72`
+- `mxbai-embed-large:latest` digest `468836162de7f81e041c43663fedbbba921dcea9b9fefea135685a39b2d83dd8`
+- 123 bundled restaurant reviews
+- 30 evaluation cases; retrieval limit 5
+
+Retrieval results on the 25 answerable cases:
+
+| Metric | Semantic retrieval | BM25 baseline |
+| --- | ---: | ---: |
+| Recall@5 | 0.913 | 0.770 |
+| Hit rate@5 | 1.000 | 0.880 |
+| MRR@5 | 0.960 | 0.753 |
+
+Model-dependent RAG results:
+
+| Metric | Result |
+| --- | ---: |
+| Citation validity | 0.560 |
+| Reference-term support proxy | 0.520 |
+| Answer success rate | 0.560 |
+| Abstention recall | 1.000 |
+| Expected-action accuracy | 0.633 |
+| Mean latency | 4.359 s |
+| Median latency | 2.006 s |
+
+Outcomes: 14 answered, 13 model abstentions, and 3 citation-validation rejections.
+
+Authoritative artifacts:
+
+- `evaluation/results/v0.2.0-ollama-0.32.5/evaluation-report.json`
+- `evaluation/results/v0.2.0-ollama-0.32.5/README.md`
+- `evaluation/results/v0.2.0-ollama-0.32.5/run.log`
+- `docs/assets/local-ai-agent-v0.2-demo.mp4`
+
+Report SHA-256: `5700748c2abe1b5446a850c08291c41e8bbab6aff663cd5ac783513880d02718`
+
+## Verification
+
+- 61 tests passed.
+- Ruff check and format check passed.
+- Mypy passed for six core modules.
+- Coverage reports all six core modules, including `evaluation.py`: 84% overall. No arbitrary pass threshold is imposed.
+- Wheel and source distribution passed Twine checks.
+- Wheel contains both bundled evaluation/data assets and the installed CLI smoke test passed.
+- Runtime dependency audit reported no known vulnerabilities other than one explicitly ignored advisory: `CVE-2026-45829` / `PYSEC-2026-311`.
+
+## Evidence boundaries
+
+These results apply only to the checked-in dataset, case manifest, model digests, runtime, retrieval limit, and machine recorded in the report. Known-positive relevance judgments are not exhaustive, so the semantic-versus-BM25 comparison is not evidence of general retrieval superiority. Citation validity checks whether citations resolve to retrieved evidence; the reference-term metric is a lexical support proxy, not claim-level factual-faithfulness evaluation.
+
+The Chroma advisory exception is accepted only for embedded, process-local Chroma. It must be removed if a fixed compatible release becomes available or the application exposes/uses a remote Chroma server.

SECURITY.md

@@ -0,0 +1,41 @@
+# Security policy
+
+## Supported version
+
+Security fixes target the latest release and the `main` branch.
+
+## Reporting a vulnerability
+
+Do not open a public issue for a suspected vulnerability. Use GitHub's private
+security-advisory flow for this repository so the report can be investigated
+before details are disclosed.
+
+Do not include API keys, model credentials, private review data, or other
+secrets in a report.
+
+## Runtime boundary
+
+Local AI Agent runs Chroma in embedded, process-local mode. It does not start
+or expose a Chroma HTTP server. Ollama is expected to bind to localhost unless
+the operator deliberately configures a different endpoint.
+
+Treat imported CSV files and model output as untrusted. The application parses
+CSV data, validates its schema, escapes it into structured documents, validates
+model citations against retrieved source IDs, and does not execute model or CSV
+content as code.
+
+## Accepted dependency advisory
+
+CI ignores **CVE-2026-45829 / PYSEC-2026-311** for `chromadb` because the
+advisory applies to Chroma's unauthenticated HTTP collection endpoint with
+`trust_remote_code=true`. This application uses embedded Chroma and never
+exposes that endpoint. As of 2026-07-31, pip-audit reports no fixed release.
+
+This is a scoped risk acceptance, not a claim that the dependency is generally
+safe. The exception must be removed when either:
+
+1. a fixed compatible Chroma release becomes available; or
+2. the application begins exposing a Chroma server or accepting remote Chroma
+   endpoints.
+
+The dependency audit still fails CI for every other known vulnerability.