Improve RAG answer reliability and diagnostics

[dk3yyyy]

1|## Summary
2|
3|- tighten the grounded-answer prompt so partially supported questions are answered without unnecessary abstention
4|- add one fail-closed citation-repair attempt while preserving exact clean-abstention behavior
5|- record raw and repaired model responses plus structured validation reasons in schema-v3 evaluation reports
6|- add aggregate answer diagnostics to JSON and Markdown reports
7|- document controlled answer-model comparisons without changing the retriever or benchmark set
8|
9|## Safety and validation
10|
11|- repaired responses pass through the same citation validator as first responses
12|- missing, unknown, and out-of-range citations remain rejected
13|- embedded abstention tokens and invalid remainders fail closed
14|- CLI and dashboard continue to expose only validated answers or safe fallback messages
15|- direct and post-repair abstentions remain distinct
16|
17|## Verification
18|
19|- uv run --python 3.11 --frozen python -m unittest discover -s tests — 75 passed
20|- uv run --python 3.14 --frozen python -m unittest discover -s tests — 75 passed
21|- uvx ruff check . — passed
22|- uvx mypy — passed
23|- uv build — passed
24|- built-wheel CLI smoke test — passed
25|- independent final diff review — approved with no blocker or high-severity findings
26|
27|## Runtime note
28|
29|A live Ollama benchmark was not run on the delivery host because Ollama is not installed there. The README includes commands for running the fixed 30-case comparison on the target Mac with llama3.2 and an installed 7B/8B instruct model.
30|

Greptile Summary

This PR tightens the grounded-answer prompt to reduce unnecessary abstentions, adds a one-shot citation-repair attempt for the three repairable failure reasons (missing_citations, out_of_range_citation, unknown_citation), and records per-observation repair diagnostics in a new schema-v3 evaluation report.

• agent.py introduces REPAIRABLE_FAILURE_REASONS, REPAIR_PROMPT, and _evaluate_model_response; embedded_control_token and invalid_remainder correctly bypass repair and fail closed immediately.
• evaluation.py adds five new fields to EvaluationObservation, three new outcome labels (answered_after_repair, model_abstention_after_repair, citation_validation_rejection_after_repair), and a diagnostics block to the JSON/Markdown report.
• Tests cover all new code paths with SequenceModel/EvaluationSequenceModel; one existing test (test_rejects_uncited_answer_after_removing_control_token_line) still uses FakeModel and silently exercises the repair path without asserting repair_attempted.

Confidence Score: 5/5

Safe to merge; the repair gate is well-guarded, the fail-closed cases are correctly excluded from repair, and all 73 tests pass.

The core logic changes are small and tightly scoped: a frozenset guards which failure reasons enter the repair path, embedded_control_token and invalid_remainder exit immediately, and repaired responses pass through the same validator as first responses. The only gap found is a single test that passes but does not fully assert the new repair behavior it exercises.

Files Needing Attention: tests/test_agent.py — test_rejects_uncited_answer_after_removing_control_token_line now exercises the repair path via FakeModel without asserting it

Important Files Changed

Filename	Overview
agent.py	Adds one-shot citation-repair path gated by REPAIRABLE_FAILURE_REASONS, new diagnostic fields on AnswerResult, and the REPAIR_PROMPT template; embedded_control_token and invalid_remainder correctly bypass repair
evaluation.py	Propagates repair diagnostics into EvaluationObservation, adds answered_after_repair/model_abstention_after_repair/citation_validation_rejection_after_repair outcomes, and adds a diagnostics block to the schema-v3 report
tests/test_agent.py	Adds SequenceModel helper and new repair-path tests; test_rejects_uncited_answer_after_removing_control_token_line now silently exercises the repair path via FakeModel without asserting repair_attempted
tests/test_evaluation.py	Adds EvaluationSequenceModel and three new integration tests covering answered_after_repair, model_abstention_after_repair, and citation_validation_rejection_after_repair outcomes
tests/test_benchmark.py	Updates schema_version assertion to 3 and adds a test that round-trips repair diagnostic fields through build_evaluation_report and write_evaluation_report
README.md	Documents new per-observation diagnostics fields and adds controlled model comparison instructions using separate --report-dir paths

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[answer_question called] --> B{Matches found?}
    B -- No --> C[Return NO_MATCH_MESSAGE\nfailure_reason=empty_retrieval]
    B -- Yes --> D{Source IDs valid?}
    D -- No --> E[Return CITATION_VALIDATION_MESSAGE\nfailure_reason=retrieved_source_missing_id]
    D -- Yes --> F[Invoke model → raw_response]
    F --> G[_evaluate_model_response]
    G --> H{validated?}
    H -- Success --> I[Return answered answer]
    H -- clean_abstention --> J[Return NO_MATCH_MESSAGE\nabstained=True]
    H -- embedded_control_token\nor invalid_remainder --> K[Return CITATION_VALIDATION_MESSAGE\nno repair]
    H -- missing_citations\nout_of_range_citation\nunknown_citation --> L[Invoke model again\n→ repair_response]
    L --> M[_evaluate_model_response]
    M --> N{repaired?}
    N -- Success --> O[Return answered_after_repair answer]
    N -- clean_abstention --> P[Return NO_MATCH_MESSAGE\nabstained=True\nmodel_abstention_after_repair]
    N -- any other failure --> Q[Return CITATION_VALIDATION_MESSAGE\ncitation_validation_rejection_after_repair]

Loading

_{Reviews (2): Last reviewed commit: "fix: narrow RAG response repair paths" | Re-trigger Greptile}