fix: preserve cited answers with standalone abstention markers

[dk3yyyy]

Summary

• preserve valid cited answers when INSUFFICIENT_EVIDENCE appears only as an exact standalone line
• remove only standalone marker lines, then run the unchanged fail-closed citation validator on the remaining answer
• continue rejecting markers embedded in prose and uncited or invalid remainders
• update Streamlit coverage to prove the cited answer and Evidence render without the raw marker

Validation

• RED reproduced on the prior strict implementation for followed, preceded, multiline-whitespace, and dashboard cases
• 67 tests passed on Python 3.11.15 with 85% measured coverage
• 67 tests passed on Python 3.14.5 with 85% measured coverage
• MyPy 1.18.2 passed
• Ruff 0.16.1 lint and formatting passed
• uv lock --check and git diff --check passed
• two independent exact-diff reviews approved the candidate

Scope

No benchmark data, evaluation results, metrics, prompts, datasets, or generated reports were changed.

Greptile Summary

This PR adds a preprocessing step (_remove_standalone_control_token) that strips INSUFFICIENT_EVIDENCE marker lines appearing on their own lines from LLM responses before citation validation, allowing mixed-output answers (where the model both hedges and provides cited content) to be surfaced rather than silently replaced with the validation error message. The citation validator and all other rejection paths are unchanged.

• Adds _remove_standalone_control_token that returns None (reject) if the token appears mid-prose, or the cleaned text (token lines removed) if it only appears as standalone lines.
• Splits the single previous test into three targeted tests: standalone-token acceptance, prose-embedded rejection, and uncited-after-removal rejection.
• Updates the dashboard integration test to assert the cited answer and evidence section render correctly instead of the old validation message.

Confidence Score: 5/5

Safe to merge — the change is narrowly scoped, the citation validator and all other rejection paths are untouched, and the new helper is fully covered by three precise tests plus a dashboard integration test.

The new _remove_standalone_control_token function correctly handles all reachable inputs: fast-path bypass when the token is absent, standalone-line removal, prose-embedded rejection, and the edge case where removal leaves an empty string (which the downstream citation validator rejects cleanly). The pure-abstention check in answer_question still runs first, so the == INSUFFICIENT_EVIDENCE_TOKEN path is unaffected. Tests cover acceptance, prose-embed rejection, uncited-after-removal rejection, and the dashboard rendering path.

Files Needing Attention: No files require special attention.

Important Files Changed

Filename	Overview
agent.py	Adds _remove_standalone_control_token helper and wires it into answer_question between the pure-abstention check and citation validation; logic is correct for all edge cases including empty-after-removal and multi-token lines.
tests/test_agent.py	Old mixed-token test split into three precise subtests (acceptance, prose-embed rejection, uncited-after-removal rejection); covers the key branches of the new helper cleanly.
tests/test_dashboard.py	Dashboard integration test updated to assert the cited answer and Evidence section render correctly; MixedTokenModel already returned the right fixture response for the new expected output.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[LLM response] --> B[answer.strip → normalized_answer]
    B --> C{normalized_answer == INSUFFICIENT_EVIDENCE_TOKEN?}
    C -- Yes --> D[AnswerResult abstained=True NO_MATCH_MESSAGE]
    C -- No --> E[_remove_standalone_control_token]
    E --> F{Token found in answer?}
    F -- No --> G[return answer as-is]
    F -- Yes --> H{For each line: line.strip == token?}
    H -- Yes --> I[skip line]
    H -- No --> J{token in line?}
    J -- Yes --> K[return None]
    J -- No --> L[retain line]
    I --> H
    L --> H
    H -- all lines processed --> M[join + strip retained lines]
    G --> N{normalized_answer is None?}
    M --> N
    K --> N
    N -- Yes --> O[AnswerResult CITATION_VALIDATION_MESSAGE]
    N -- No --> P[_validate_and_number_citations]
    P --> Q{validated is None?}
    Q -- Yes --> O
    Q -- No --> R[AnswerResult valid cited answer]

Loading

_{Reviews (1): Last reviewed commit: "fix: preserve cited answers with standal..." | Re-trigger Greptile}