[ci] Add PR versus base vector index benchmarks

[wangzhigang1999]

Closes #96

PRs currently have no automated comparison of vector index performance against their base revision. This adds an informational benchmark check with a compact Recall/QPS report for IVF-FLAT, IVF-SQ, IVF-PQ, IVF-RQ and DiskANN.

Both revisions build the same candidate-provided benchmark driver with Rust stable on the same runner. The fixed synthetic workload uses 10,000 64D vectors, 2,048 queries and two Rayon threads. Six fresh-process samples per version/index alternate execution order; warm sequential and batch timing each run for at least one second. The report compares medians and flags QPS drops above 10%/20% or recall drops above 1/3 percentage points. Performance changes are advisory, not merge gates.

The existing benchmark gains optional warm timing and CSV fields, plus a fix for retained ground-truth allocations. Original first-pass Recall/I/O metrics are preserved. The comparison script validates samples, enforces a shared 20-minute budget, and retains raw data and failure logs.

Results are published in the Actions summary and through a separate workflow_run publisher, including external fork PRs. The publisher runs from the default branch, downloads only the triggering run/attempt's report artifact, verifies its PR against GitHub's source repository/branch/head data, and skips changed head/base revisions. It reads artifacts as text and never checks out PR code, restores PR caches or executes artifact contents.

The publisher must first exist on the target repository's default branch. This introducing PR therefore continues to expose its report through the Actions summary until merged; subsequent benchmark runs can publish comments automatically. No additional token is required.

Validation:

• Full CI passed, including Rust, C/C++, JNI and Python checks.
• Benchmark passed, followed automatically by the workflow_run publisher. The existing comment was updated without duplication. This live test used the fork repository with the publisher installed on its default branch; external-fork routing is additionally covered by publisher tests.
• Example report in the fork PR. Identical library inputs are repeatability calibration, not evidence of an index performance improvement.
• Publisher tests cover fork routing, create/update, missing reports and stale/unrelated runs.
• Eight Python tests cover calculations, advisory thresholds, invalid/incomplete samples and timeout handling.

This is a small warm-cache regression check, not a large-scale or object-storage benchmark. Production index algorithms are unchanged.

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Pull request overview

Adds an automated, informational CI benchmark that compares vector index performance between a PR merge commit and its base revision, producing a compact Recall/QPS report and retaining raw artifacts.

Changes:

• Introduces tools/benchmark_pr.py to build/run both revisions with a controlled workload and generate markdown/JSON summaries.
• Extends the Rust benchmark CSV output with optional warm “steady” timing/QPS/P95 metrics and adjusts ground-truth extraction behavior.
• Adds a GitHub Actions workflow to run the comparison on PRs and (for same-repo PRs) publish/update a single bot comment; adds Python unit tests for validation and failure modes.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tools/tests/test_benchmark_pr.py	Adds unit tests covering sample validation, delta formatting, budget timeouts, and alert thresholds.
tools/benchmark_pr.py	Implements the PR-vs-base benchmark runner, sample parsing/validation, summarization, and report rendering.
tools/README.md	Documents the workflow behavior, workload, interpretation, and local reproduction steps.
core/benches/ann_bench.rs	Adds optional warm “steady” timing metrics to CSV output; adjusts ground-truth row materialization.
.github/workflows/benchmark-pr.yml	Adds CI workflow to run benchmark comparison on PRs and publish results to summary/artifacts (and comment for same-repo PRs).

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[wangzhigang1999]

It appears to be like this: