M2X-046: RAGAS harness, citation accuracy and abstention scoring

[yashpancholi09]

uv run m2x eval rag produces the three PRD gate numbers plus abstention.

Stacked on feature/m2x-045-rag-question-set.

The four figures are not the same kind of claim

metric	floor	computed by	what it is
context precision	0.75	RAGAS, judge LLM	a model's opinion
faithfulness	0.80	RAGAS, judge LLM	a model's opinion
citation accuracy	0.90	this module	mechanical
abstention accuracy	—	this module	mechanical

The report prints this rather than leaving it to a reader's memory. On the zero-spend stack the judge shares the answerer's weights — a judge with the answerer's blind spots inflates faithfulness exactly where the answerer is confidently wrong. --judge-model is a flag and the judge id is on every results row.

Citation accuracy is ours, and text never enters it

The ticket names the property to review: it must fail a citation whose text merely sounds right but points at the wrong segment. No similarity metric can do that — sounding right is what a similarity metric rewards.

The sanity case proving it lifts the quote verbatim from the correct passage and attaches it to the wrong chunk. Any check consulting the text passes it.

Micro not macro, so one sloppy answer with six citations cannot hide behind a careful one with one. Uncited answers counted separately, because a system that stopped citing would otherwise score 0.0 for having no denominator. The 1-based/0-based boundary is crossed in exactly one tested function.

Defect found live, which no unit test could have caught

The first run failed on 8 of 22 judged questions: 'AdapterRagasLLM' object has no attribute 'generate'. The bridge implemented the three methods BaseRagasLLM declares abstract; RAGAS's metrics call generate, the base's async wrapper.

The unit suite could not have found it — the harness takes its judge as an injected callable, so no RAGAS code ran until the live command did. A wrapper around a third-party interface is only tested by the third party.

The fix is one method. The lesson is a surface test comparing the bridge's public callables against BaseRagasLLM's — which immediately found a second gap (get_temperature) that had never reached a live run at all.

Live baseline — all four legs fail

metric	run 1 (broken judge)	run 2	floor
context precision	0.0000	0.4649	0.75
faithfulness	0.0000	0.6667	0.80
citation accuracy	0.7857	0.7857	0.90
abstention accuracy	0.4333	0.4333	—
judge failures	8	1

Citation accuracy and abstention are identical across both runs — as they must be, since neither touches the judge. A useful accident: it confirms the mechanical half is independent of RAGAS in practice, not only in the design.

single_meeting 7/20 · cross_meeting 1/5 · unanswerable 5/5

The shape matters more than the numbers. The system abstains on most answerable questions while abstaining correctly on every unanswerable one — the safe direction to fail in, but a failure. Cause: the quote check. Reference turns carry <FILL/>, <UNKNOWN/> and <PName> annotation tags, so a model quoting what it reads cannot reproduce the stored string byte for byte. day4-ask.md hit this with markdown and fixed it for emphasis; transcripts break it harder.

That is a defect in m2x ask, not in the harness, and it belongs to the Friday gate ticket. Three candidate directions recorded, none taken.

What Friday inherits, ranked — none is a harness change

1. The quote check against transcript text. Nothing else matters until it is fixed.
2. top_k = 5, never varied — five passages for a question one turn answers is four irrelevant contexts by construction. Likely most of the context-precision gap, and the cheapest experiment available.
3. max_distance = 0.48, resting on eight questions. Every results row now carries nearest_distance per question, so thirty measurements can replace the eight without another live run.
4. The judge shares the answerer's weights.

Dependency notes for review

• ragas 0.4.3 imports langchain_community.chat_models.vertexai, removed in 0.4.x — pinned langchain-community<0.4. Worth flagging that the import chain reaches a Vertex AI (Gemini) integration and Gemini is banned by CLAUDE.md; nothing calls it, but the dependency is there, and a sunset package under the gate's tree is a liability.
• ragas is an optional group. The harness injects its judge, so the mechanical half and all three ticket-mandated sanity cases run on a fresh clone without it.
• AskOutcome grew hits (additive): it recorded retrieved as a count, so learning which turns a citation pointed at needed a second index lookup that could disagree with the first.
• chunk_segments split out of chunk_transcript: a Transcript is an AdapterResult carrying provider/latency/cost, and the tiron reference was written by human annotators — there is no such call to describe.

612 tests green (27 new).