Skip to content

Bench exact KNN ground truth#34

Merged
hev merged 1 commit into
mainfrom
feat/bench-exact-ground-truth
Jul 9, 2026
Merged

Bench exact KNN ground truth#34
hev merged 1 commit into
mainfrom
feat/bench-exact-ground-truth

Conversation

@hevbot

@hevbot hevbot commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

gate:impl

Closes #30

Summary

  • Change hevsearch-bench recall sweep to collect ground truth with in-engine exact: true queries on the same namespace used for indexed queries.
  • Score indexed ivf_pq recall/ndcg against those exact engine row IDs instead of external .ivecs or in-process brute force.
  • Leave the filtered ground-truth strategy as an explicit hev/search#25 seam; this implementation runs the unfiltered query set first.

Evidence

Smoke bench on isolated MinIO (search-bench-gt-minio, host port 19002) with seeded synthetic namespace:

HEVSEARCH_STORAGE_URI=s3://search-bench-gt \
HEVSEARCH_S3_ENDPOINT=http://127.0.0.1:19002 \
HEVSEARCH_S3_ACCESS_KEY=minioadmin \
HEVSEARCH_S3_SECRET_KEY=minioadmin \
HEVSEARCH_BENCH_ROWS=2000 \
HEVSEARCH_BENCH_DIM=16 \
HEVSEARCH_BENCH_QUERIES=5 \
HEVSEARCH_BENCH_PARTITIONS=8 \
HEVSEARCH_BENCH_SUB_VECTORS=1 \
HEVSEARCH_BENCH_NPROBES_SWEEP=1,4 \
HEVSEARCH_BENCH_OUT_DIR=bench/results/recall-smoke \
./scripts/cargo run -p hevsearch-bench --bin recall_sweep

Output included:

exact reference (unfiltered) qps=78.1 p50=5.4ms p95=42.7ms
unfiltered nprobes=   1  recall@1=0.2000 @10=0.2600 @100=0.3180  ndcg@10=0.3038
unfiltered nprobes=   4  recall@1=0.2000 @10=0.3600 @100=0.4740  ndcg@10=0.3791

Generated JSON recorded ground_truth_source: "in_engine_exact", same_namespace: "recall-ivfpq-1783554566869249552", and each indexed point had scored_against: "in_engine_exact".

Verification

  • ./scripts/cargo fmt --all --check
  • ./scripts/cargo check -p hevsearch-bench --bin recall_sweep
  • ./scripts/cargo test -p hevsearch-bench recall --lib
  • ./scripts/cargo clippy -p hevsearch-bench --bin recall_sweep -- -D warnings

@hevbot

hevbot commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator Author

Factory verified: bench scores indexed recall against the in-engine exact reference (RFC 0011↔0014 target state), #25 strategy seam left open as required, evidence run in body. Merge-ready pending checks.

@hev hev merged commit 427ba7d into main Jul 9, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bench: use exact KNN mode as in-engine ground truth (RFC 0011 ↔ 0014 tie-in) — blocked on the exact-mode impl

2 participants