Retrieval: BM25 and vector results are concatenated, not fused #170

In HybridRetriever.retrieve_documents (src/retrievers/csv_chroma.py):

for i, query in enumerate(queries):
    bm25_docs = bm25_retriever.invoke(query, ...)
    vector_docs = vector_retriever.invoke(query, ...)
    doc_lists.append(bm25_docs + vector_docs)   # concatenated into ONE list
subdirectory_docs.extend(self.weighted_reciprocal_rank(doc_lists))

Each per-query list is BM25s k results followed by the vector retrievers. RRF then runs across query variants, not across retrievers. Despite the name, BM25 and vector search are never fused against each other.

Three consequences follow from that one line.

1. Vector results are systematically down-ranked

RRF scores by position in the list (weight / (rank + 60), rank from 1). With BM25 returning k=10 first, every vector result enters at rank 11+:

• BM25 top hit: 1/61
• vector top hit: 1/71

About 14% — modest, because c=60 flattens the curve — but it falls out of iteration order rather than from any decision.

2. Ties are broken by list position

Documents with equal RRF scores are ordered by first appearance in chain.from_iterable(doc_lists), because sorted is stable. So on a tie, earlier query variants win, and BM25 beats the vector store. Deterministic, but arbitrary. Pinned by tests/retrievers/test_hybrid_retriever.py::test_ties_are_broken_by_position_not_by_score.

3. The weights are inert

create_bm25_chroma_ensemble_retriever passes [1 / len(doc_lists)] * len(doc_lists). A constant multiplier across every list cannot change ordering — only absolute scores. The weighting currently does nothing. Pinned by test_weights_are_uniform_so_they_cannot_change_the_ordering.

Fix

Append BM25 and vector results as separate lists:

doc_lists.append(bm25_docs)
doc_lists.append(vector_docs)

Both top hits then score 1/61, and the weights become a real BM25-vs-vector dial — which is presumably what "weighted" was for.

This is a retrieval-quality change, not a refactor. Worth measuring with bin/retrieval_baseline before and after.

Related: #169 (duplicates eat the top-k), #168 (baseline harness).

🤖 Generated with Claude Code