Claude Code hook: flag hallucinations in agent answers

[adaamko]

Problem

Coding agents produce RAG-style answers all day: summaries of files they read, claims about API behavior, quotes from docs. Claude Code hooks are shell commands the agent harness runs at lifecycle events (e.g. Stop, when the agent finishes a reply), and their output can be surfaced back into the session — a natural, zero-friction deployment surface for a lightweight hallucination check.

Everything needed already exists in the library; this is pure integration work:

• Direct: HallucinationDetector.predict(context, answer, question, output_format="spans") (lettucedetect/models/inference.py:27-46).
• Over HTTP (avoids per-invocation model load): the FastAPI server exposes POST /v1/lettucedetect/spans (lettucedetect_api/server.py:81-99), started via python scripts/start_api.py dev, with sync and async clients providing detect_spans (lettucedetect_api/client.py:105, client.py:163).
• The integrations pattern to copy: lettucedetect/integrations/langchain/ and lettucedetect/integrations/elysia/ each ship a small module + README.md + runnable example.

There is no Claude Code integration yet.

Current behavior

Nothing exists under lettucedetect/integrations/ for Claude Code. A user who wants this today has to hand-write a hook script and work out the transcript format, the detector call, and the hook output contract themselves.

What to do

1. Create lettucedetect/integrations/claude_code/ with a hook script (e.g. check_answer.py, runnable as python -m lettucedetect.integrations.claude_code.check_answer). It reads the hook event JSON from stdin (a Stop event carries session_id and transcript_path; see the hooks docs for the exact contract), extracts the last assistant message from the transcript, and checks it against a context the user supplies (--context-file with retrieved passages, or a documented convention such as a context.md in the project). Keep transcript parsing and span formatting as pure functions.
2. Support two detector modes behind flags:
   • --api-url http://127.0.0.1:8000 → LettuceClient.detect_spans (recommended: the model stays loaded in the server, hook latency is one HTTP round trip);
   • --model-path ... → in-process HallucinationDetector (simplest setup; document the per-invocation model-load cost). Default model: KRLabsOrg/lettucedetect-base-modernbert-en-v1 (149M, 4K context — see the model table in docs/getting-started/quickstart.md:84-92); mention KRLabsOrg/lettucedect-v2-mmbert-base for code / tool-output answers (that lettucedect-v2 spelling is the actual HF id).
3. On flagged spans, print a concise report (span text, confidence, category when present) and exit with the code the hooks docs specify for feeding feedback back to the agent; exit 0 silently when clean.
4. Add README.md in the same directory with a ready-to-paste settings.json hooks block and a 5-line "try it" walkthrough, following the structure of lettucedetect/integrations/langchain/README.md.
5. Add tests for the pure functions (transcript parsing, report formatting) with a stubbed detector — no model download, no network.

Acceptance

• The hook script runs standalone against a fixture: echo '{"transcript_path": "tests/fixtures/transcript.jsonl"}' | python -m lettucedetect.integrations.claude_code.check_answer --context-file ctx.txt --model-path ... prints flagged spans for a deliberately unsupported answer and exits cleanly for a supported one.
• Pasting the README's settings.json block into a Claude Code project makes the check run on Stop (verified manually; state in the PR which Claude Code version you tested against).
• New tests live in tests/test_claude_code_hook_pytest.py (pattern required by tests/pytest.ini); python -m pytest tests/test_claude_code_hook_pytest.py -v passes; python tests/run_pytest.py stays green.
• No changes outside lettucedetect/integrations/claude_code/ and tests/.

Non-goals

• No automatic recovery of "the retrieved context" from arbitrary agent sessions — the user supplies the grounding context; keep the convention simple and documented.
• No MCP server, no plugin, no marketplace packaging — a hook script and a README.
• No changes to the detectors or the web API.
• No latency optimization work beyond choosing the API mode.

Start here

git clone https://github.com/KRLabsOrg/LettuceDetect.git
cd LettuceDetect
pip install -e ".[dev]"
python tests/run_pytest.py   # should be green before you change anything

# The code to read first:
ls lettucedetect/integrations/langchain/        # the integration pattern to copy
grep -n "def detect_spans" lettucedetect_api/client.py
sed -n '65,100p' lettucedetect_api/server.py    # the /v1/lettucedetect/spans endpoint