Harden bounded RAG repair diagnostics

[dk3yyyy]

Summary

This follow-up hardens the RAG answer path after #11 while retaining its useful diagnostics and evaluation reporting.

• keeps citation validation fail-closed
• allows exactly one bounded repair for a clean abstention or citation failure
• rejects malformed/control-token responses without retrying them
• preserves a valid initial abstention unless repair returns a validated cited answer
• adds structured failure reasons, repair counts, and truthful outcome classifications
• excludes raw model responses from reports by default; --include-raw-responses is an explicit opt-in
• emits additive report schema v3 fields while keeping the Markdown renderer safe for older reports

Verification

Exact candidate: 1edb00e12e580352d06837851f13044639133859

• 78 unit tests passed
• 86% coverage (80% required)
• MyPy passed
• Ruff lint and formatting passed
• uv lock --check passed
• git diff --check passed
• independent exact-commit review: approved, no release blockers

Clean 30-case Ollama benchmark

Provenance recorded git_dirty=false at the exact candidate commit.

Metric	Result
Retrieval recall	0.9133
Answer success	0.88
Citation validity	0.88
Expected-action accuracy	0.90
Reference-term support proxy	0.86
Abstention recall	1.00

The abstention result covers the benchmark's five explicit unsupported topics only: parking, wheelchair accessibility, reservations, Wi-Fi, and delivery fees. It should not be generalized beyond those cases.

Raw response diagnostics were disabled in the benchmark report and no raw-response fields were serialized.

Notes

This PR does not deploy the application or change any portfolio content.

Greptile Summary

This PR hardens the RAG answer path by allowing one bounded repair attempt for both premature abstentions ("clean_abstention" added to REPAIRABLE_FAILURE_REASONS) and citation failures, while preserving valid initial abstentions when repair itself fails. Raw model responses are now excluded from evaluation reports by default and require an explicit --include-raw-responses opt-in.

• Bounded repair for abstentions: answer_question now retries INSUFFICIENT_EVIDENCE responses once with a re-evaluation prompt; if repair returns a valid cited answer the abstention is overridden, otherwise the original abstention is preserved.
• Finer-grained outcome labels: run_rag_evaluation distinguishes model_abstention_confirmed_after_repair, model_abstention_preserved_after_failed_repair, and model_abstention_after_repair for repair-path abstentions, and adds retrieved_source_missing_id as an explicit data-integrity outcome.
• Schema v3 with safe v2 fallback: build_evaluation_report emits schema_version: 3 with raw_responses_included and repair-count fields; _report_markdown uses report.get(\"diagnostics\", …) to render legacy v2 reports without crashing.

Confidence Score: 5/5

Safe to merge; the repair logic is well-bounded, the abstention-preservation path is tested, and the raw-response opt-in correctly defaults to off.

All new code paths through answer_question are exercised by unit tests, the schema v3 / v2 fallback is verified, and the --include-raw-responses gate is covered end-to-end in the CLI test. The one new finding — "model_abstention" being an unreachable branch — is a dead-code consequence of the intended design change, not a runtime defect.

Files Needing Attention: evaluation.py — the "model_abstention" outcome branch in run_rag_evaluation is now unreachable from production answer_question results; worth either removing or documenting as intentionally reserved.

Important Files Changed

Filename	Overview
agent.py	Core logic changes: "clean_abstention" added to REPAIRABLE_FAILURE_REASONS so premature abstentions now get one repair attempt; the repair prompt no longer echoes the rejected response; the combined abstention guard preserves valid initial abstentions correctly
evaluation.py	Adds three new outcome labels for repaired abstentions and retrieved_source_missing_id; raw_responses_included flag gates raw fields from serialized observations; _report_markdown gets a safe fallback for schema v2 reports; the "model_abstention" outcome branch is now unreachable dead code
tests/test_agent.py	SequenceModel refactored from variadic *args+list.pop(0) to single list[str]+iter(); new tests cover false-abstention repair, citation repair, confirmed/preserved abstention paths and benchmark topics; three prior paths remain uncovered (noted in previous review)
tests/test_evaluation.py	New tests cover retrieved_source_missing_id classification, preserved initial abstention, and confirmed-after-repair abstention; existing abstention-after-repair test is retained
tests/test_benchmark.py	Adds legacy schema v2 report rendering test and opt-in raw-response serialization assertions; coverage is thorough for new schema fields
tests/test_cli.py	Adds CLI tests for --include-raw-responses default/opt-in and verifies raw fields are absent by default and present when opted in
main.py	Adds --include-raw-responses store-true flag to the evaluate subparser and threads the value into build_evaluation_report
README.md	Documents the one-shot repair, schema v3 additive extension, --include-raw-responses opt-in, and raw-response privacy guidance

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[answer_question] --> B{matches found?}
    B -- No --> C[empty_retrieval]
    B -- Yes --> D{source IDs valid?}
    D -- No --> E[retrieved_source_missing_id]
    D -- Yes --> F[invoke model]
    F --> G{initial response valid?}
    G -- Yes --> H[answered]
    G -- No --> I{initial_failure_reason in REPAIRABLE_FAILURE_REASONS?}
    I -- No --> J[CITATION_VALIDATION_MESSAGE no repair]
    I -- Yes --> K[invoke repair model]
    K --> L{repair response valid?}
    L -- Yes --> M[answered_after_repair]
    L -- No --> N{initial OR repair == clean_abstention?}
    N -- initial AND repair == clean_abstention --> O[model_abstention_confirmed_after_repair]
    N -- initial == clean_abstention only --> P[model_abstention_preserved_after_failed_repair]
    N -- repair == clean_abstention only --> Q[model_abstention_after_repair]
    N -- No --> R[citation_validation_rejection_after_repair]

Loading

_{Reviews (2): Last reviewed commit: "test: preserve repair outcome diagnostic..." | Re-trigger Greptile}