strawmANN

A deliberately unfair strawman for approximate nearest neighbour search
A Qdrant-wire-compatible vector engine in Zig, built as a performance oracle: no compatibility debt, no distribution, no product.

Quick start · How it works · Comparison · Unsafe by design · Findings · Validation status

                            strawmann      qdrant 1.19      ratio
  search, one at a time         2,142            2,013      1.06x
  saturating                   27,947           14,226          -
  mixed read/write             17,800            2,017          -
  exact                           191              123      1.56x
                                                       queries/second

  sift1m, d=128, euclid — these ratios are this dimension's. Also measured: arguana, dbpedia-openai-100K-1536-angular, dbpedia-openai-1m, fiqa, h-and-m-2048-angular-filters, laion-small-clip, quora, scifact

  at equal recall (§7.4's comparison, not equal ef): 2.40x at recall 0.9625, falling to 1.34x at 0.9977, over 8 anchors

  saturating: [strawmann pass 1 -14% against two passes that agree: one pass set the spread on this row, so a band built from it is not a noise band]
  mixed read/write: [strawmann: qps over the 60.6 s of the append the search saw (19,108 over the whole search); append 3,300 points/s] [qdrant: qps over the 60.6 s of the append the search saw (2,106 over the whole search); append 3,300 points/s] [search-during-write; no recall join]

SIFT1M, both engines alone on one host, as-deployed profile, median of three passes alternated A/B/A/B. The gate verdict, conformance tier and hashes are under the full table in docs/comparison-sift1m.md; the d=1536 tier is in docs/comparison-dbpedia-openai-1m.md. Each run is one self-contained HTML page at agourlay.github.io/strawmann. One machine, one day per run.

The deliverable is the cost model and the ratio measured / modelled for each of its terms: docs/cost-model.md and docs/isa-matrix.md. The useful part is where the model was wrong.

Binary, package, flags and env vars are all strawmann.

Disclosures. The author works at Qdrant, on the engine being measured. Nothing here is a Qdrant position and nobody at Qdrant has reviewed it. Most of the harness, analysis and documents were written with Claude; bench/harness/nightrun.py runs a headless claude -p for each night's write-up. The numbers come from the gate and the load generator either way.

The tree was squashed to one commit. Report pages and docs/ cite hashes from a history this repository no longer carries; they date a number and do not resolve here.

Quick start

Zig 0.16.0, pinned in build.zig.zon and installed exactly by CI.

scripts/doctor.py                    # host setup, what it can measure, where it writes
scripts/check.py                     # the gate: format, lint, tests, both languages
zig build -Doptimize=ReleaseFast     # the only build numbers are quoted from
./zig-out/bin/strawmann --port 6334  # the banner prints build mode and ISA

Server sizing: --io-threads, --workers, --connections per I/O thread, --streams per connection, --capacity per collection. strawmann --help has the rest. The gate and the dataset fetcher are stdlib Python; the analysis tooling is a uv project under bench/.

The benchmark is one command:

bench/harness/fullrun.py --server-cpus 4-11 --client-cpus 0-3 --reps 3 --perf
bench/harness/smoke-test.sh          # sift1m, one pass per engine, ~35 min

fullrun.py runs the §7.1 gate, builds ReleaseFast, runs W0-W13 against strawmANN on the server cores with the load generator on the client cores, sweeps recall while the collections still match the ground truth, runs the mutating rows last, repeats all of it against Qdrant on the same cores, runs the §8 differ, then renders the report and regenerates the tables in this file.

--reps 3 alternates the arms A/B/A/B and folds a noise floor. One pass renders a page and publishes nothing.
--perf attaches hardware counters. Needs a native Qdrant (--qdrant-binary) and costs about 4% of throughput.
--dataset picks the corpus for the rows, the sweeps and the differ. Two datasets are never ratioed.
--segment-policy and --oversampling-policy each name one of two Qdrant experiments. Runs under different policies are never ratioed either.
--skip, --lax, --placement, --wait-for-gate: see --help.

The rows are in docs/workloads.md. smoke-test.sh is what to run after a change: one pass per engine into non-published labels, differ included.

Nothing is written outside the checkout except under one cache root, defined in bench/harness/paths.py:

$STRAWMANN_CACHE          # default ${XDG_CACHE_HOME:-~/.cache}/strawmann
  datasets/               # $STRAWMANN_DATA: ~21 GB with both tiers converted
  qdrant-storage/         # $QDRANT_STORAGE
bench/results/            # in the checkout, gitignored

Precedence, highest first: --data-dir, the env vars, ${XDG_CONFIG_HOME:-~/.config}/strawmann/config.toml, the defaults above. scripts/doctor.py prints where each root came from. --data-dir exports $STRAWMANN_DATA to every tool it spawns, so datasets can be shared with bfb and vector-db-benchmark.

The pieces run on their own:

bench/setup.py check                                             # §7.1 host gate
conformance/datasets/datasets.py fetch                           # fetch + verify
bench/harness/workloads.py run http://localhost:6334 strawmann    # W0-W13
bench/harness/compare.py strawmann qdrant                        # join two runs
bench/harness/isa_sweep.py run --reps 3                          # §7.5 ISA matrix
uv run --project bench bench/harness/report.py strawmann qdrant  # the HTML page
zig build bench                  # §7.2 hardware baseline + kernel matrix
cd conformance && cargo test     # oracle, differ tiers, metrics

How it works

flowchart TB
    client["bfb / qdrant-client<br/><i>gRPC over HTTP/2, h2c prior knowledge</i>"]

    subgraph net["net/ - one epoll thread"]
        h2["h2.zig<br/>frames, flow control, streams"]
        hpack["hpack.zig<br/>RFC 7541 + Huffman"]
        grpc["grpc.zig<br/>5-byte prefix, trailers, status"]
    end

    subgraph api["api/ - N worker threads"]
        dispatch["Rpc enum<br/><i>exhaustive switch, not a string cascade</i>"]
        handlers["handlers.zig<br/>upsert, queryBatch, scroll<br/>createFieldIndex, setPayload"]
        pred["preparePredicate<br/><i>one bit per point, preallocated (§6.3)</i>"]
    end

    subgraph core["core/"]
        coll["collection.zig<br/>arena, ids, tombstones"]
        qsearch["quantized_search.zig<br/>two-stage + rescore"]
        quant["quantized.zig<br/>SQ8, binary, PQ codes"]
        payload["payload.zig<br/>blobs as wire bytes, chunked<br/>keyword + integer posting lists<br/>must / must_not / should"]
    end

    subgraph store["core/ persistence - no parse step, no WAL"]
        persist["persist.zig<br/>save / load, 4 KiB headers"]
        storage["storage.zig<br/><b>pinned</b>: anon alloc + MADV_HUGEPAGE (VectorSpace.init)<br/><b>cached</b>: file mmap + POPULATE<br/><b>cold</b>: file mmap, faults on demand"]
        files[("vectors.bin<br/>ids.bin<br/>graph.bin")]
        persist --> storage
        storage <--> files
    end

    subgraph index["index/"]
        hnsw["hnsw.zig<br/>flat level-0, CSR upper"]
        brute["brute force<br/><i>exact path + pending tail</i>"]
    end

    subgraph kernels["dist/ - comptime-generic per lane width"]
        simd["dot, euclid, hamming, pq_adc<br/>AVX-512 VNNI / AVX2 / SSE"]
    end

    client -->|submit queue| h2
    h2 <--> hpack
    h2 --> grpc
    grpc --> dispatch
    dispatch --> handlers
    handlers --> coll
    handlers --> qsearch
    handlers -->|"filter, with_payload"| payload
    payload -->|"bitset from the<br/>cheapest must list"| pred
    payload -->|"no list answers it:<br/>evaluate per node"| pred
    pred -->|"selected² < ef·m0·n"| brute
    pred -->|otherwise| hnsw
    coll --> hnsw
    coll --> brute
    qsearch --> hnsw
    coll <--> persist
    hnsw <--> persist
    quant --> qsearch
    coll --> quant
    hnsw --> simd
    brute --> simd
    qsearch --> simd

The I/O thread never blocks and never allocates. Workers own their scratch, and every buffer on the query path is sized at startup (§6.3).

Search reads a graph published with release ordering; ingest holds a write lock. Points appended since the last build are not in the graph, so search scans that tail exhaustively and merges.

A filter becomes a predicate before the search starts. The cheapest indexed must condition fills a preallocated bitset, one bit per point; a filter no posting list answers is evaluated per node against the stored blobs. With the matching count known, the search scores the set directly when selected² < ef·m0·n and traverses under the predicate otherwise. At n=200,000, ef=128, m0=32 the crossover is 28,621 matches.

Persistence is implemented and not wired. §6.4 makes the on-disk layout the in-memory layout, and persist.save/load do exactly that, but no RPC, flag or shutdown hook calls them. The arena is anonymous memory under pinned and a file mapping under cached or cold, with 4 KiB pages on the mapped placements because huge pages on a file mapping cannot be assumed (§5.5).

The comparison

Both engines measured alone, same script, same dataset, same cores, on a host that passes §7.1 in the as-deployed profile. Full tables: docs/comparison-sift1m.md and docs/comparison-dbpedia-openai-1m.md. Rendered pages: agourlay.github.io/strawmann.

What the table does not say:

Qdrant runs in production mode since the 2026-09-26 pair. Pages before it ran Qdrant's development profile (4 search threads, audit and debug logging), which is why the saturating ratio fell from 2.24x to 1.65x on sift1m, and on dbpedia-openai-1m (the 2026-09-27 pair) from 1.38x to parity.
Only search rows carry a noise floor. Ingest and index-build rows carry no verdict. A floor from three passes is itself noisy.
W11 measures the rebuild window, not search during write. The append crosses rebuild_ratio, the graph is rebuilt, and the row mostly measures a brute-force scan. Its ratio is refused.
Both arms run at cached residency, which is not strawmANN's default. Qdrant 1.19 does not support pinned for dense vectors. --placement pinned --skip qdrant measures strawmANN's default alone.
The gate verdict is per row. A table with any row stamped FAIL is not spliced into this file.

Units are queries per second. bfb's rps counts requests, and a W5 request carries 16 queries.

How the benchmark is wired

flowchart LR
    subgraph data["datasets - pinned by upstream sha256"]
        raw["SIFT1M, dbpedia-openai-1M"]
        oracle["fp64 exhaustive oracle<br/><i>conformance/oracle</i>"]
        gt["ground truth<br/>k=100 + scores"]
        raw --> oracle --> gt
    end

    subgraph engines["engines, measured one at a time"]
        sm["strawmANN"]
        qd["Qdrant 1.19<br/><i>image by digest, or a native<br/>binary by sha256 (--qdrant-binary)</i>"]
    end

    bfb["bfb - dev @ 29240511<br/><b>load generator</b><br/>throughput, latency, tails"]
    conf["conformance - Rust<br/><b>relevance generator</b><br/>T0-T4, recall, nDCG"]
    gate["bench/setup.py<br/><b>§7.1 host gate</b><br/>governor, boost, isolation<br/>+ is the machine idle?"]
    wl["workloads.py<br/>W0-W13"]
    cmp["compare.py<br/>joins on (dataset, collection, ef)<br/>ratio only at equal recall<br/>STALE / UNLICENSED banners"]

    raw -->|fbin, same checksum| bfb
    gt --> conf
    gate -.->|stamps every result row| wl
    wl --> bfb
    bfb --> sm
    bfb --> qd
    conf --> sm
    conf --> qd
    bfb -->|qps, latency| cmp
    conf -->|recall at matched ef| cmp

bfb generates load, conformance/ generates relevance, and neither reports the other's numbers (§4.1). bfb's own recall is an id-set intersection with no tie handling; §8.6 needs ε-aware recall against the fp64 oracle, and on SIFT1M 54% of queries return the same neighbours in a different order.

W10 joins the two: bfb sweeps ef for latency, the harness measures recall at the same ef, and compare.py joins on (dataset, collection, ef). A search row gets a ratio only when both engines' recall@10 at its ef is known and within 0.01. The whole table is refused STALE when the two arms came from different harness configurations, and UNLICENSED unless the differ set licenses_comparative (§8.5 T3). compare.py --check-readme verifies this in CI.

conformance/ decides what may be published. It links the same pinned qdrant-client as bfb, runs the fp64 oracle and the T0-T4 tiers, and uses an ε calibrated from Qdrant's cross-ISA spread. No performance row is publishable without a conformance row for the same build on the same data; the results sink refuses rows whose hash it has not seen. See conformance/README.md.

The HTML report

uv run --project bench bench/harness/report.py strawmann qdrant --open

One self-contained file per run, Plotly embedded: bench/results/report-<dataset>-<a>-vs-<b>-<day>-<hhmm>.html. It carries the gate output, build and binary hashes, throughput with in-band ratios marked inconclusive, latency labelled open or closed loop, recall against the oracle with the matched-recall frontier, storage and I/O, scheduler and stall counters, and per-row ambient load. It refuses rather than estimates.

Published pages are committed under docs/reports/ and linked from agourlay.github.io/strawmann. The landing page says what each run is, which one the front page quotes, and why the pages are not a series.

Where the vectors live

Qdrant's VectorParams.memory has three values and strawmANN honours all three for the vector arena.

placement	what it is	strawmANN	Qdrant
`Pinned`	in RAM, never evicted	anonymous memory, `MADV_HUGEPAGE`	not supported for dense vectors
`Cached`	mmap, prefaulted, evictable	file-backed `MAP_SHARED` + `MAP_POPULATE`, 4 KiB pages	the default
`Cold`	mmap, faulted on demand	the same mapping without `MAP_POPULATE`, `MADV_RANDOM`	what `on_disk: true` selects

strawmann --data-dir /var/lib/... --default-placement cached   # match qdrant's default

The defaults differ: an unconfigured collection is Pinned here and Cached on Qdrant. --default-placement aligns a run.
A cold collection is not cold after an upload. Ingest writes through the page cache; fsync + fadvise(DONTNEED) empties it.
On tmpfs, cold is never cold: the page cache is the storage.
Only the vector arena has a placement. Graph and codes are always pinned, and hnsw_config.memory is refused with UNIMPLEMENTED.
on_disk: false means Cached, not Pinned, as in Qdrant.

What it does not protect you from

§1's non-goals return UNIMPLEMENTED naming the construct. This is what is present and unsafe.

Search takes no write lock. Appends are reserve, write, publish, so a reader never admits a half-written row. An in-place overwrite bumps a row version and drains the searches inside; builder threads are not drained, so the first overwrite in a collection's life can tear a row under a builder. The row is re-encoded at publish, so one graph edge can come from a torn read and no result can.
Nothing is durable. No WAL, no fsync, no recovery; persist is never called. Under cached and cold the arena's dirty pages still reach the disk, gigabytes per run, counted in the Storage and I/O table.
Deletes reclaim nothing. A tombstone bit. The row keeps its space and its place in the graph until the collection is dropped.
Capacity is fixed at startup and applied to every collection. --capacity (default 1,100,000) has no per-collection override on the wire. Above it an upsert fails with RESOURCE_EXHAUSTED. A full run creates seven collections and preallocates seven arenas.
A rebuild doubles the graph's memory. Old and new graph, each sized to capacity.
uint8 storage converts like Rust's x as u8. Saturate outside [0, 255], truncate inside, NaN to 0, and an OK response. Matched to Qdrant on purpose.
hnsw_ef above 4,096 is clamped, not refused. Nothing published went past 512.
Scroll is not stable across writes. Any upsert or delete rebuilds the id order.
No authentication, no TLS, no quotas. Do not expose it.
Two engines on one host are warned about, not stopped.

Layout

src/
  dist/     SIMD kernels, comptime-generic per lane width, + cpuid dispatch
  proto/    hand-written protobuf codecs (no codegen, no reflection)
  net/      epoll loop, HTTP/2 + HPACK, gRPC framing, fuzzers
  api/      RPC handlers, end-to-end tests over a real socket
  core/     collection, ids, arenas, quantized store, storage, payload + filters
  index/    hnsw build/search, brute force, visited set, heaps
  quant/    sq8, binary, pq, codebook training
bench/
  micro/    bandwidth, latency, MLP, TLB, kernel matrix, perf counters
  isa/      forced-ISA asm probe, dump_asm.py (docs/asm/ regeneration + --check)
  harness/  workloads.py (W0-W13), compare.py, recall.py, fullrun.py,
            nightrun.py, smoke-test.sh, isa_sweep.py, provenance.py,
            aggregate.py, procstat.py, perfstat.py,
            report.py + report_data.py + report_charts.py
  setup.py  §7.1 environment gate
conformance/  Rust, uses qdrant-client: one client, both engines (README.md)
  src/oracle/      fp64 exhaustive reference + ground-truth cache
  src/differ/      T0-T4 tiers, tolerance calibration
  src/relevance/   recall@k, MRDE, Wilson intervals (nDCG needs qrels: unrun)
  src/metamorphic/ oracle-free invariants
  src/datasets/    fbin/fvecs/parquet readers
  datasets/        datasets.json + datasets.py: fetch, verify, convert (§4.2)
scripts/      check.py (the gate), doctor.py
docs/asm/     disassembly per ISA arm, regenerated by dump_asm.py

Documents


`spec.md`	the specification everything is measured against
`findings.md`	open work, ranked by what a wrong or missing number costs
`validation.md`	what has been checked, what has not, and what would close the gap
`decisions.md`	the decision log, including the wrong calls and the measurements behind them
`comparison-sift1m.md`	the full W0-W13 table for SIFT1M, generated with the run
`comparison-dbpedia-openai-1m.md`	the same at d=1536, 1M, licensed 2026-09-24
`comparison-dbpedia-openai-100K-1536-angular.md`	the same at d=1536, 100K, measured once on 2026-08-26
`comparison-laion-small-clip.md`	the same at d=512, 100K CLIP embeddings, licensed 2026-09-27
`comparison-h-and-m-2048-angular-filters.md`	the same at d=2048, 105K product embeddings, licensed 2026-09-29
`reports/`	the rendered page for each published run, from the landing page
`cost-model.md`	the measured constants, and where the model was wrong
`datasets.md`	the descriptor, what is pinned, how to add a dataset
`ground-truth.md`	M-1: published vs recomputed ground truth
`workloads.md`	W0-W13 against the pinned bfb `dev`
`dataset-runs.md`	both tiers end to end: recall curves, scroll walk
`isa-matrix.md`	§7.5, and §6.6.1's prediction under test
`kernel-matrix-*.md`	`zig build bench` dumps per ISA arm; development-grade, not re-measured since the harness fixes in `cost-model.md`
`tolerance.md`	§8.4's calibrated ε and the spreads behind it
`conformance/README.md`	the correctness gate: the three claims, T0-T4, why it is a separate Rust binary

Licence

Apache 2.0. LICENSE, copyright in NOTICE.

The engine links no third-party code. Each page under docs/reports/ embeds Plotly.js under its MIT licence, notices inside the file.

Navigation Menu

Repository navigation

About

Repository files navigation

strawmANN

Quick start

How it works

The comparison

How the benchmark is wired

The HTML report

Where the vectors live

What it does not protect you from

Layout

Documents

Licence

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
assets		assets
bench		bench
conformance		conformance
docs		docs
scripts		scripts
src		src
.gitignore		.gitignore
CLAUDE.md		CLAUDE.md
LICENSE		LICENSE
NOTICE		NOTICE
README.md		README.md
build.zig		build.zig
build.zig.zon		build.zig.zon
ruff.toml		ruff.toml
zlint.json		zlint.json

Latest commit

History

Folders and files

Repository files navigation

strawmANN

Quick start

How it works

The comparison

How the benchmark is wired

The HTML report

Where the vectors live

What it does not protect you from

Layout

Documents

Licence

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages