docs: finalize RFC 0011 — recall + build benchmark harness (accept)

[hevbot]

Links issue #16.

Rendered rich diff: https://github.com/hev/search/pull/18/files?short_path=b85668e&w=false#diff-b85668e54c8a179c2eb7f25cfc3e28472f8b606f132f69822c4a5150c33cdc31

Summary

• Finalizes RFC 0011 as Accepted (2026-07-08).
• Updates the RFC against current hevsearch-bench reality: recall.rs and recall_sweep.rs already implement the source-built v1 recall lane.
• Documents PR Add bench evidence CI workflow #10's bench-evidence wrapper and calls out the provenance caveat: the workflow validates and records image_tag, but currently runs the harness from checkout source.

Open questions resolved

• Initial lane shape: source-built, in-process NamespaceService for v1, evidenced by recall_sweep.rs and PR Add bench evidence CI workflow #10's command.
• CI dataset posture: deterministic synthetic defaults, with .fvecs / .ivecs inputs available for larger runs.
• RFC 0009 gap handling: unavailable variants are recorded explicitly instead of omitted.
• Evidence inputs: required immutable image_tag and corpus_hash validation in PR Add bench evidence CI workflow #10.

Open questions for hev review

• Whether bench-evidence must run the pinned image before first publication use, or rename that field to source/toolchain provenance.
• Whether nightly should remain inert with the all-zero ECR digest placeholder until docker publishing targets mesh-account ECR.
• Lance IVF_HNSW build memory behavior and whether it stays per-partition bounded.
• Lance build scratch redirection to NVMe vs tmpfs /tmp behavior.
• Filtered ground-truth recompute strategy and stable query count.
• Whether optimize_indices folds new rows into the vector index or only scalar indexes.
• Whether future filtered/freshness axes stay in-process or gain HTTP mode.
• Where public ANN datasets live in CI.

Verification

• git diff --check
• Docs-only diff: docs/rfcs/0011-recall-and-build-benchmark-harness.md

[hevbot]

Factory verified: diff touches only docs/rfcs/0011-recall-and-build-benchmark-harness.md; rendered rich-diff linked in body; resolved questions carry evidence and the remaining ones are explicitly flagged Open question for review (harness process boundary, filtered/freshness axes). Ready for hev's docs review — merge = RFC accepted + factory files implementation issues.

[hev]

@hevbot check CI

[hevbot]

CI checked: the required fmt / clippy / test check fails at the cargo fmt step (verified via the run's job steps) — that's main's inherited fmt debt, not this diff (docs-only, single RFC file). PR #15 fixes exactly that debt (fmt + clippy + fuzzy test, currently in its integration-suite step). Once #15 merges I'll rebase this branch and the check goes green. If you want to review the content meanwhile: rendered view — https://github.com/hev/search/blob/docs/rfc-0011-finalize/docs/rfcs/0011-recall-and-build-benchmark-harness.md

[hev]

@hevbot I am going to merge but want to address these decison points.

Whether bench-evidence must run the pinned image before first publication use, or rename that field to source/toolchain provenance. - pin to a SHA. That seems simplest.
Whether nightly should remain inert with the all-zero ECR digest placeholder until docker publishing targets mesh-account ECR. the repo is public so might as well publish the image.
Lance IVF_HNSW build memory behavior and whether it stays per-partition bounded. - let' discuss. we need to ideally leverage NVME as much as possible.
Lance build scratch redirection to NVMe vs tmpfs /tmp behavior. see previous
Filtered ground-truth recompute strategy and stable query count. yeah, I am not sure how we do this. open to suggestions.
Whether optimize_indices folds new rows into the vector index or only scalar indexes. I don't understand this one.
Whether future filtered/freshness axes stay in-process or gain HTTP mode. gain HTTP I think
Where public ANN datasets live in CI. - what are reasonable options?

[hevbot]

Decisions received — here's the disposition of each, plus answers to your questions:

Recorded + actioned (issues filed with this activation):

• Pin to SHA → follow-up issue: bench-evidence workflow pins the image by digest.
• Publish the nightly image → yes, with one house-rule note: images go to mesh-account ECR, never ghcr, even for public-source repos — lyr AGENTS.md explicitly lists search's existing docker-publish.yml ghcr target as a carried-in defect to retarget. Issue filed to retarget + enable nightly publish there. Flag me if you deliberately want ghcr instead.
• Filtered/freshness axes gain HTTP mode → recorded in the resolutions docs PR.

Your questions:

• optimize_indices: after new rows land, Lance's optimize_indices() reliably folds them into scalar indexes; for the vector index it depends on version/config whether new rows merge in or sit unindexed (exact-scanned) until a rebuild. The open question was whether the freshness axis needs an explicit index-rebuild step to be measuring what we think. Proposed: bench asserts indexed-fraction after optimize, so we learn the truth empirically per release.
• Filtered ground-truth + stable query count: recommend recomputing in-engine via RFC 0014's exact-KNN mode (just finalized) against a pinned, versioned query set (fixed seed, e.g. 1k queries, checked into the repo) on each dataset bump — no external qrels artifact to drift. Alternative is precomputed qrels in the dataset bucket; happy to write up both in the discussion issue.
• Public ANN datasets in CI — options: (a) mirror to a mesh-account S3 bucket (recommended: pinned, no upstream flake, egress-controlled), (b) HuggingFace datasets at a pinned revision, (c) upstream URLs + checksum pin (flakiest).
• NVMe for IVF_HNSW builds + scratch: discussion issue filed — sketch: point Lance build scratch at the NVMe path (TMPDIR redirect on bench runners, same device class as the foyer cache), keep builds per-partition bounded, and have bench record peak RSS + scratch high-water so the discussion runs on data.