feat: add reproducible RAG evaluation

[dk3yyyy]

Summary

• expand the bundled RAG evaluation from 4 to 30 cases, including 5 expected-abstention cases
• add a deterministic BM25 retrieval baseline and versioned JSON/Markdown reports
• record dataset and case hashes, exact Ollama model digests, per-case outcomes, retrieval rankings, and latency
• separate expected-action accuracy, answer success, abstention recall, citation validity, and lexical reference-term support
• strengthen schema validation, CI checks, package verification, documentation, and the embedded-Chroma security boundary
• include a silent live-dashboard demo showing the real trigger, cited answer, and expanded evidence

Local evaluation

The committed report was generated with Ollama 0.32.5, llama3.2:latest, and mxbai-embed-large:latest against 123 bundled reviews.

Retrieval metric	Semantic	BM25
Recall@5	0.913	0.770
Hit rate@5	1.000	0.880
MRR@5	0.960	0.753

RAG observations: 14 answered, 13 model abstentions, and 3 citation-validation rejections. Answer success was 0.560, abstention recall 1.000, citation validity 0.560, and the reference-term support proxy 0.520.

Artifacts:

• evaluation/results/v0.2.0-ollama-0.32.5/evaluation-report.json
• evaluation/results/v0.2.0-ollama-0.32.5/README.md
• docs/assets/local-ai-agent-v0.2-demo.mp4

Validation

• 61 tests passed
• Ruff check and format check passed
• mypy passed across six core modules
• coverage reports all six core modules, including evaluation.py: 84% overall
• wheel and source distribution passed Twine checks
• packaged evaluation/data assets and the installed-wheel CLI were smoke-tested
• dependency audit reported no known vulnerabilities other than the explicitly documented CVE-2026-45829 / PYSEC-2026-311 embedded-Chroma exception

Evidence boundaries

The benchmark is scoped to the checked-in dataset, case manifest, runtime, model digests, retrieval limit, and machine recorded in the report. Known-positive relevance judgments are not exhaustive, so the semantic/BM25 comparison does not establish general retrieval superiority. Citation validity checks citation resolution against retrieved evidence; the reference-term metric is a lexical support proxy rather than claim-level factual-faithfulness evaluation.

Greptile Summary

This PR expands the bundled RAG evaluation from 4 to 30 cases (including 5 abstention cases), introduces a deterministic BM25 retrieval baseline, adds versioned JSON and Markdown report generation, and refactors evaluation metrics into well-named, backward-compatible fields. It also extracts a new ollama_health.py module, adds credential-stripping for report metadata, strengthens schema validation in the loader, and wires all changes into an extended CI pipeline.

• evaluation.py gains BM25Retriever, RetrievalMetrics, build_evaluation_report, write_evaluation_report, and richer EvaluationMetrics fields, replacing the four previous metrics with six differentiated scores plus backward-compatible property aliases.
• main.py adds the evaluate --report-dir flow with a dedicated _ollama_report_metadata helper that wraps HTTPError/OSError/ResponseError into ValueError, fixing the previously flagged unhandled-exception path.
• ollama_health.py is extracted with ollama_version and model_metadata helpers; tests/test_benchmark.py and tests/test_cli.py are new test modules covering BM25, report generation, and the CLI evaluate path.

Confidence Score: 5/5

Safe to merge; the evaluation expansion, BM25 baseline, and report-writing flow are all well-tested and the previously flagged exception-handling gap is resolved in this PR.

All six core modules have mypy coverage, 61 tests pass, and the two previously flagged issues (unhandled Ollama exceptions in main and abstain-case source_labels validation) are both fixed in this PR. The only remaining note is that the CI audit ignore list names only the CVE identifier for the known Chroma advisory, not the alternate PYSEC identifier — a low-impact robustness gap that would only surface if pip-audit changes which ID it uses to report that advisory.

Files Needing Attention: .github/workflows/ci.yml — audit ignore list covers CVE-2026-45829 but not PYSEC-2026-311.

Important Files Changed

Filename	Overview
evaluation.py	Large refactor adding BM25Retriever, RetrievalMetrics, report generation helpers, and richer EvaluationMetrics. Logic is correct; abstain-case validator now properly checks relevant_titles, gold_source_ids, and reference_facts. Minor unreachable branch in _faithfulness flagged in prior review.
main.py	Adds evaluate --report-dir flow. _ollama_report_metadata now wraps HTTPError/OSError/ResponseError into ValueError, fixing the previously flagged unhandled-exception path. _safe_endpoint correctly strips credentials and query parameters.
ollama_health.py	New module extracting ollama_version and model_metadata helpers with injected request parameter for testability. Logic is clean.
.github/workflows/ci.yml	Adds audit, package smoke-test, and coverage steps. The pip-audit ignore list covers CVE-2026-45829 but omits PYSEC-2026-311, which the PR description identifies as the same vulnerability's alternate pip-audit identifier.
tests/test_benchmark.py	New test file covering BM25Retriever, retrieval metrics, report generation, and bundled evaluation set quality. Includes correct regression test for abstain case with source_labels.
tests/test_cli.py	New tests cover evaluate CLI path including report writing and late Ollama failure handling. Tests confirm _safe_endpoint strips credentials correctly.
tests/test_evaluation.py	Updated to cover new metric names and the citation-validation-rejection outcome. Backward-compatible aliases (citation_correctness, answer_faithfulness, abstention_accuracy) verified.
vector.py	Minor changes only; no functional issues introduced.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[evaluate command] --> B[load_reviews]
    B --> C[load_evaluation_cases]
    C --> D[_create_runtime\nvector_store + model]
    D --> E[run_rag_evaluation\nper-case answer_question + latency]
    E --> F[score_evaluation\nEvaluationMetrics]
    E --> G[retrieval_metrics_from_observations\nSemantic RetrievalMetrics]
    D --> H[run_bm25_baseline\nBM25Retriever.search per case]
    H --> I[retrieval_metrics_from_observations\nBM25 RetrievalMetrics]
    F & G & I --> J{--report-dir set?}
    J -- yes --> K[_ollama_report_metadata\nollama_version + model_metadata]
    K --> L[build_evaluation_report\nJSON dict]
    L --> M[write_evaluation_report\nevaluation-report.json + README.md]
    J -- no --> N[print metrics to stdout]

Loading

_{Reviews (2): Last reviewed commit: "fix: harden evaluation report validation" | Re-trigger Greptile}