Skip to content

Add bench evidence CI workflow#10

Merged
hev merged 2 commits into
mainfrom
feat/bench-evidence-ci
Jul 8, 2026
Merged

Add bench evidence CI workflow#10
hev merged 2 commits into
mainfrom
feat/bench-evidence-ci

Conversation

@hev

@hev hev commented Jul 8, 2026

Copy link
Copy Markdown
Owner

Closes #7.

Summary

  • Adds the bench-evidence workflow with required workflow_dispatch inputs image_tag and corpus_hash, plus a nightly scheduled lane with pinned defaults.
  • Fails fast unless image_tag is a mesh-account ECR digest ref and neither input is mutable/empty.
  • Runs the RFC 0011 recall_sweep harness against MinIO and uploads bench-evidence.md, bench-evidence.json, and the raw recall_sweep_*.json as the workflow artifact.
  • Forwards the recall-sweep env vars through ./scripts/cargo so the same pinned harness settings are usable locally through the containerized toolchain.

No Actions secrets are required for this workflow as written; it uses an in-job MinIO backend. The current RFC 0011 harness is in-process/source-built, so image_tag is validated and recorded in the artifact metadata rather than pulled.

Required evidence

1. Successful local harness run

Command invoked by the workflow: cargo run --release -p hevsearch-bench --bin recall_sweep.

Local evidence run used the same harness command with a local file:// storage root because Docker is not running on this machine, so local MinIO could not be started:

HEVSEARCH_STORAGE_URI=file:///tmp/hevsearch-bench-evidence-local \
HEVSEARCH_BENCH_ROWS=256 \
HEVSEARCH_BENCH_DIM=32 \
HEVSEARCH_BENCH_QUERIES=3 \
HEVSEARCH_BENCH_SEED=42 \
HEVSEARCH_BENCH_NPROBES_SWEEP=1,4 \
HEVSEARCH_BENCH_PARTITIONS=4 \
HEVSEARCH_BENCH_SUB_VECTORS=2 \
HEVSEARCH_BENCH_NUM_BITS=4 \
HEVSEARCH_BENCH_OUT_DIR=bench/results/recall/local-evidence \
cargo run --release -p hevsearch-bench --bin recall_sweep

Output:

Finished `release` profile [optimized] target(s) in 0.21s
Running `target/release/recall_sweep`
computing brute-force ground truth for 3 queries over 256 rows...
recall sweep: dataset=synthetic-seed42-256x32 rows=256 dim=32 queries=3 partitions=4 sub_vectors=2 nprobes=[1, 4]

--- exact baseline (linear scan): recall-exact-1783508395786550000 ---
  upsert 0.0s
  recall@100=1.0000 (must be ≈ 1.0 — the scorer self-check)

--- ivf_pq sweep: recall-ivfpq-1783508395822122000 ---
  upsert 0.0s
  index build 0.0s
  nprobes=   1  recall@1=0.0000 @10=0.1667 @100=0.2233  ndcg@10=0.1871  qps=1535.1  p50=0.3ms p95=1.4ms
  nprobes=   4  recall@1=0.0000 @10=0.3000 @100=0.6000  ndcg@10=0.3352  qps=2892.3  p50=0.3ms p95=0.4ms

wrote bench/results/recall/local-evidence/recall_sweep_synthetic-seed42-256x32.json

2. Unpinned input refused

INPUT_IMAGE_TAG='186219257916.dkr.ecr.us-east-1.amazonaws.com/hev/search:latest' \
INPUT_CORPUS_HASH=synthetic-seed42-256x32-q3-v1 \
DEFAULT_IMAGE_TAG='186219257916.dkr.ecr.us-east-1.amazonaws.com/hev/search@sha256:1111111111111111111111111111111111111111111111111111111111111111' \
DEFAULT_CORPUS_HASH=synthetic-seed42-256x32-q3-v1 \
bash -c '<workflow validation body>'

Output:

image_tag must be pinned by digest and must not use latest: 186219257916.dkr.ecr.us-east-1.amazonaws.com/hev/search:latest

3. Workflow lint

actionlint .github/workflows/bench-evidence.yml

Output: clean.

Note: fmt commit f448fd3 covers pre-existing cargo fmt debt inherited from main plus the new branch files.

@hev hev added the gate:impl Implementation PR awaiting CI + spec-review + merge (agentic-gates §2) label Jul 8, 2026
@hev

hev commented Jul 8, 2026

Copy link
Copy Markdown
Owner Author

CI rerun after fmt commit f448fd3 now passes the previous fmt gate but fails in clippy, outside this PR's changes.

Run/job: https://github.com/hev/search/actions/runs/28938399742/job/85854368138

Failures reported by cargo clippy --workspace --all-targets -- -D warnings:

  • crates/hevsearch-core/src/manager.rs:2927: clippy::type_complexity on an existing complex return type.
  • crates/hevsearch-core/src/manager.rs:3648: clippy::needless_range_loop, loop variable row used to index ids.
  • crates/hevsearch-core/src/manager.rs:3695: clippy::needless_range_loop, loop variable row used to index ids.

I checked git diff origin/main...HEAD -- crates/hevsearch-core/src/manager.rs; this PR only changes formatting in that file, not the clippy-failing regions.

@hev

hev commented Jul 8, 2026

Copy link
Copy Markdown
Owner Author

spec-review (conformance pass): ✅ approve pending rebase, with one design flag — reviewed against issue #7 / agentic-gates Phase 1.4.

  • Pinning contract implemented as specified: required image_tag + corpus_hash inputs; hard-fails on empty, :latest, non-digest refs, non-mesh-ECR refs (ghcr correctly impossible); nightly cron + workflow_dispatch; results table uploaded as artifact; needed Actions secrets documented rather than invented.
  • ⚑ Design flag for before first publication use: the job builds and runs the bench from source at the checkout SHA — the image_tag is recorded as provenance metadata but no container actually runs. A results table stamped with an image digest that didn't execute is exactly the evidence-theater failure mode the plan warns about. Before any bench-publish consumes this artifact: either make the job run the pinned image, or rename the pin to what it truly is (source git SHA + toolchain). Also: the nightly lane's DEFAULT_IMAGE_TAG is an all-zeros digest placeholder — nightly is honest-but-inert until a real digest exists (blocked on retargeting docker-publish from ghcr to mesh ECR).
  • Housekeeping: after test: refresh fuzzy typo probe #14 and Fix manager clippy debt #12 merge, rebase and drop the now-redundant fmt-only commit.

(Posted by the dispatch loop as the manual spec-review equivalent.)

@hevbot

hevbot commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Rebased onto green main (post-#15): the fmt commit dropped as already-upstream, leaving only the workflow change. CI running — factory will verify and flag for merge when green.

@hevbot

hevbot commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Factory verified post-rebase: diff is the bench-evidence workflow + scripts/cargo wrapper only (fmt commit dropped as upstream), required check green on post-#15 main. Merge-ready — merging closes #7 and completes agentic-gates Phase 1.4's CI lane.

@hev

hev commented Jul 8, 2026

Copy link
Copy Markdown
Owner Author

@hevbot deal with the merge conflicts here.

@hev hev force-pushed the feat/bench-evidence-ci branch from 7478701 to 1c93820 Compare July 8, 2026 19:01
@hevbot

hevbot commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Same as #21 — no real conflict, just behind the base after the #18 merge. Updated onto current main; CI re-running. Note: your #18 decision to pin the bench image to a SHA lands as a follow-up issue (filed with the RFC 0011 activation set), not in this PR, to keep it merge-ready as reviewed.

@hev hev merged commit 87edd67 into main Jul 8, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gate:impl Implementation PR awaiting CI + spec-review + merge (agentic-gates §2)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bench evidence CI: workflow wrapping the RFC 0011 harness with pinned inputs (agentic-gates Phase 1.4)

2 participants