Retrieval: duplicate documents consume half the top-k budget on reactions #169

Measured with bin/retrieval_baseline against openai/text-embedding-3-large/reactome/Release95, 20 questions, k=10.

The number

Distinct documents actually returned, out of k=10:

Vector search on reactions returns ten results containing about five distinct reactions. For "How does TP53 regulate PTEN transcription?" the top five were two reactions repeated.

Cause

reactions.csv has one row per reaction per pathway / input / output / catalyst combination, so a reaction in several pathways becomes several rows. Those rows have different page_content, and weighted_reciprocal_rank de-duplicates on page_content, so they survive fusion as separate documents.

BM25 is unaffected — its scoring spreads across distinct text — which is why only the vector side shows it.

Effect

The LLM receives a context window roughly half-filled with restatements of the same reaction, and the effective k on the largest collection is about 5 while paying the cost of 10.

Options

• de-duplicate on st_id before fusion (RRF already accepts an id_key)
• over-fetch and collapse to k distinct stable IDs
• change the bundle so one row is one reaction, with pathways/inputs as list metadata — a data-generation change, more invasive

Independent of the LangChain version, so it can be fixed before or after the upgrade.

Reproduce:

./bin/retrieval_baseline capture --out before.json --with-selfquery

🤖 Generated with Claude Code