fix: prevent abstention control-token leaks

[dk3yyyy]

Summary

• reject any mixed model response containing the INSUFFICIENT_EVIDENCE control token before citation validation
• preserve token-only responses, including surrounding whitespace, as clean abstentions
• add deterministic coverage for preceded, followed, embedded, malformed, uncited, and whitespace variants
• add a Streamlit regression proving the raw token and Evidence section are never rendered for a rejected mixed response

Validation

• 65 tests passed on Python 3.11.15 with 85% measured coverage
• 65 tests passed on Python 3.14.5 with 84% measured coverage
• MyPy 1.18.2 passed
• Ruff 0.16.1 lint and formatting passed
• uv lock --check passed
• source distribution, wheel build, required packaged assets, and installed-wheel CLI smoke test passed
• two independent exact-diff reviews approved the frozen candidate

Scope

No benchmark data, evaluation results, metrics, prompts, datasets, or generated reports were changed.

Greptile Summary

This PR closes a leakage path where a model producing a mixed response — the INSUFFICIENT_EVIDENCE control token alongside real prose and citations — could have that token rendered in the Streamlit UI. A single guard (if INSUFFICIENT_EVIDENCE_TOKEN in normalized_answer) is inserted between the clean-abstention check and the citation-validation step, returning CITATION_VALIDATION_MESSAGE without abstained=True.

• agent.py: The new guard runs only when normalized_answer != INSUFFICIENT_EVIDENCE_TOKEN (the equality check fires first for token-only responses, preserving the abstained=True / NO_MATCH_MESSAGE path), so the two branches are mutually exclusive.
• tests/test_agent.py: The existing abstention test is updated to use a whitespace-padded token to exercise the .strip() boundary; three parametrized sub-tests cover "preceded", "followed", and "embedded" token positions in mixed responses.
• tests/test_dashboard.py: A new MixedTokenModel + DashboardStore fixture drives an end-to-end Streamlit regression confirming that INSUFFICIENT_EVIDENCE and #### Evidence are never rendered when the model returns a mixed response.

Confidence Score: 5/5

Safe to merge — the change is a narrowly scoped guard that plugs a specific token-leak path without touching any other logic.

The new guard is inserted at exactly the right place in the control flow: after the clean-abstention equality check and before citation validation, so the two branches remain mutually exclusive. The abstained flag, message constants, and retrieved_source_ids are all set correctly. Tests cover the three mixed-position variants at the unit level plus a full Streamlit render regression, and the PR description reports a clean 65-test run with type checking and linting.

Files Needing Attention: No files require special attention.

Important Files Changed

Filename	Overview
agent.py	Adds a 6-line guard that rejects mixed-token responses before citation validation; logic is correct and the two return branches (equality for clean abstention, substring for mixed) are mutually exclusive.
tests/test_agent.py	Updates the whitespace-abstention test to a more realistic padded token; adds a parametrized sub-test covering preceded, followed, and embedded mixed-token positions with the correct assertions on answer, sources, retrieved_source_ids, and abstained.
tests/test_dashboard.py	Adds a Streamlit regression test with a MixedTokenModel stub and DashboardStore; correctly patches create_chat_model and create_vector_store to drive the full chat path and asserts the token and Evidence section are never rendered.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[answer_question called] --> B{matches found?}
    B -- No --> C[Return NO_MATCH_MESSAGE]
    B -- Yes --> D{any missing source IDs?}
    D -- Yes --> E[Return CITATION_VALIDATION_MESSAGE]
    D -- No --> F[model.invoke prompt]
    F --> G[normalized_answer = response.strip]
    G --> H{normalized_answer == INSUFFICIENT_EVIDENCE_TOKEN?}
    H -- Yes --> I[Return NO_MATCH_MESSAGE\nabstained=True]
    H -- No --> J{INSUFFICIENT_EVIDENCE_TOKEN\nin normalized_answer?}
    J -- Yes --> K[Return CITATION_VALIDATION_MESSAGE\nabstained=False\nNEW GUARD]
    J -- No --> L[_validate_and_number_citations]
    L --> M{validated is None?}
    M -- Yes --> N[Return CITATION_VALIDATION_MESSAGE]
    M -- No --> O[Return validated answer with sources]

    style K fill:#d4edda,stroke:#28a745
    style J fill:#d4edda,stroke:#28a745

Loading

_{Reviews (1): Last reviewed commit: "fix: reject mixed abstention control tok..." | Re-trigger Greptile}