-
Notifications
You must be signed in to change notification settings - Fork 20
Expand file tree
/
Copy pathindex.html
More file actions
294 lines (278 loc) · 52.7 KB
/
Copy pathindex.html
File metadata and controls
294 lines (278 loc) · 52.7 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to you under the Apache License, Version 2.0.
-->
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="Compare the five Apache Paimon Vector Index families by accuracy, latency, storage, build cost, and on-disk layout.">
<title>Index Selection Guide · Paimon Vector Index</title>
<link rel="stylesheet" href="styles.css">
<script src="docs.js" defer></script>
</head>
<body>
<a class="skip-link" href="#main">Skip to content</a>
<header class="site-header">
<div class="header-inner">
<a class="brand" href="index.html" aria-label="Paimon Vector Index documentation home">
<span class="brand-mark">VI</span>
<span>Paimon Vector Index</span>
</a>
<nav class="site-nav" data-site-nav aria-label="Documentation">
<a href="index.html" aria-current="page">Overview</a>
<a href="api.html">API</a>
<a href="development.html">Development</a>
<a href="ivf-flat.html">IVF-FLAT</a>
<a href="ivf-pq.html">IVF-PQ</a>
<a href="ivf-rq.html">IVF-RQ</a>
<a href="ivf-sq.html">IVF-SQ</a>
<a href="diskann.html">DiskANN</a>
<a href="releases.html">Releases</a>
</nav>
<div class="header-actions">
<button class="icon-button" type="button" data-theme-toggle aria-label="Switch color theme">◐</button>
<button class="nav-toggle" type="button" data-nav-toggle aria-expanded="false" aria-label="Open navigation">☰</button>
</div>
</div>
</header>
<main id="main">
<div class="page-shell">
<section class="hero">
<div class="hero-grid">
<div>
<p class="eyebrow">Index selection guide · v1</p>
<h1>Five indexes.<br>One selection map.</h1>
<p class="hero-lead">Four implementations use IVF to narrow the search space before scanning raw vectors or compact codes. DiskANN instead uses one global Vamana graph with resident PQ navigation and paged F16/F32 reranking. This guide compares recall, latency, storage, build cost, and object-store I/O.</p>
<div class="badge-row" aria-label="Shared capabilities">
<span class="badge strong">All indexes: L2 / IP / Cosine</span>
<span class="badge">Batch search</span>
<span class="badge">Roaring row filters</span>
<span class="badge">Magic-based detection</span>
</div>
</div>
<aside class="hero-note">
<strong>The short answer</strong>
Measure IVF-FLAT first. For compact indexes, start with IVF-SQ when one byte per dimension fits; compare IVF-RQ for smaller configurable codes or IVF-PQ for the smallest files. Validate each against the recall target. Choose DiskANN for an immutable collection when high recall and small local-SSD reads justify a much slower build; validate its recall independently for the production metric.
</aside>
</div>
</section>
<section class="section" id="shared-path">
<div class="section-heading">
<h2>IVF's shared two-stage search</h2>
<p>For the four IVF families, <code>nlist</code> controls the number of coarse partitions and <code>nprobe</code> controls how many partitions a query reads. Only the stored representation and scan kernel change.</p>
</div>
<div class="callout"><strong>DiskANN follows a different path</strong>It traverses one global Vamana graph using resident PQ codes, reads adjacency pages on demand, then reranks persisted F16 or F32 vectors from either the same interleaved pages or a separate compact section. It has no IVF lists; the tagged query API uses <code>l_search</code> and rejects an IVF <code>nprobe</code> override.</div>
<div class="flow" aria-label="Shared search pipeline">
<div class="flow-step"><small>01</small><strong>Preprocess vectors<br>Normalize for cosine</strong></div>
<div class="flow-step"><small>02</small><strong>Measure distance to<br>IVF coarse centroids</strong></div>
<div class="flow-step"><small>03</small><strong>Select the nearest<br><code>nprobe</code> lists</strong></div>
<div class="flow-step"><small>04</small><strong>Scan raw vectors or<br>compact codes</strong></div>
<div class="flow-step"><small>05</small><strong>Merge candidates into<br>the global top K</strong></div>
</div>
</section>
<section class="section" id="documentation">
<div class="section-heading">
<h2>Usage and development</h2>
<p>Index pages focus on algorithms and formats. Shared lifecycle, language bindings, filtering, benchmarks, and build commands live in dedicated guides so the same information is maintained only once.</p>
</div>
<div class="card-grid">
<article class="card"><h3>API and language bindings</h3><p>The Trainer / Writer / Reader lifecycle with complete Rust, C, C++, Java/JNI, and Python examples.</p><a class="card-link" href="api.html">Open the API guide →</a></article>
<article class="card"><h3>Search and filtering</h3><p>Understand automatic width, explicit IVF <code>nprobe</code>, DiskANN <code>l_search</code> calibration, warm-up, and Roaring64 filters.</p><a class="card-link" href="api.html#params">Explore the search API →</a></article>
<article class="card"><h3>Development and benchmarks</h3><p>Repository modules, Rust checks, cross-language smoke tests, ANN benchmarks, and storage compatibility.</p><a class="card-link" href="development.html">Open the development guide →</a></article>
<article class="card"><h3>DiskANN deployment and tuning</h3><p>Understand the memory, local, remote, and object-store profiles, resident-memory model, build parameters, and production-readiness boundary.</p><a class="card-link" href="diskann.html">Open the DiskANN guide →</a></article>
<article class="card"><h3>Releases and verification</h3><p>Download signed source releases, create a release candidate, or independently verify one before voting.</p><a class="card-link" href="releases.html">Open the release guides →</a></article>
</div>
</section>
<section class="section" id="comparison">
<div class="section-heading">
<h2>Core differences</h2>
<p>This table describes the implementation in this repository, not a general promise made by similarly named algorithms elsewhere. Storage estimates omit row IDs, model sections, offset tables, alignment, and headers unless noted.</p>
</div>
<div class="filter-row" aria-label="Filter indexes by priority">
<button class="filter-button" type="button" data-filter="all" aria-pressed="true">All</button>
<button class="filter-button" type="button" data-filter="recall" aria-pressed="false">Recall first</button>
<button class="filter-button" type="button" data-filter="compact" aria-pressed="false">Space first</button>
<button class="filter-button" type="button" data-filter="latency" aria-pressed="false">Latency first</button>
<button class="filter-button" type="button" data-filter="simple" aria-pressed="false">Simple baseline</button>
</div>
<div class="table-wrap">
<table class="comparison-table">
<thead><tr><th>Index</th><th>Representation</th><th>Candidate search</th><th>Main payload per vector</th><th>Accuracy profile</th><th>Build cost</th><th>Primary controls</th><th>Best fit</th></tr></thead>
<tbody>
<tr data-index-row data-fit="compact latency"><td><a href="ivf-pq.html">IVF-PQ</a></td><td>8-bit PQ codes; optional OPQ</td><td>Distance-table lookup over compact codes</td><td>About <code>m</code> bytes</td><td>PQ reconstruction error; OPQ may improve uneven subspaces</td><td>Medium to high</td><td>Automatic <code>nlist</code>/<code>nprobe</code>/<code>pq.m</code>; target-based OPQ; explicit overrides</td><td>Minimum file and selected-list bytes among IVF when the measured 0.58–0.74 recall band is sufficient</td></tr>
<tr data-index-row data-fit="compact latency simple"><td><a href="ivf-sq.html">IVF-SQ</a></td><td>8-bit scalar residual codes with pooled training bounds</td><td>SIMD code scan in probed lists</td><td>About <code>d</code> bytes</td><td>Per-coordinate scalar quantization loss</td><td>Low</td><td>Automatic <code>nlist</code>/<code>nprobe</code>; explicit overrides</td><td>Compact SIMD scans with pooled training bounds and a bounded partition cache</td></tr>
<tr data-index-row data-fit="compact recall"><td><a href="ivf-rq.html">IVF-RQ</a></td><td>Multi-bit rotated residual levels + coarse/full factors</td><td>Bounded sign-plane scan, then full bit-plane refinement</td><td>Default about <code>padded_d/2+20</code> bytes</td><td>Measured 0.82–0.91 Recall@10 across GloVe-100, SIFT1M, and GIST1M at four bits</td><td>Low to medium</td><td>Automatic <code>nlist</code>/<code>nprobe</code>; budget-based bits; explicit overrides</td><td>Configurable multi-bit compression when SQ codes are too large; measure recall and scan cost at the chosen width</td></tr>
<tr data-index-row data-fit="recall latency"><td><a href="diskann.html">DiskANN</a></td><td>Global Vamana + resident PQ + persisted rerank vectors</td><td>PQ-guided graph traversal and F32/F16 rerank</td><td><code>E·d + pq.m + 4(R+1)</code> bytes, approximately; <code>E=4</code> or <code>2</code></td><td>Approximate candidate discovery; F32-exact or F16-quantized distances for reranked candidates</td><td>Very high</td><td>Build preset + deployment/capacity objectives; calibrated automatic <code>l_search</code></td><td>Immutable L2 on local SSD when high recall and sub-MiB query reads repay the long build</td></tr>
<tr data-index-row data-fit="recall simple"><td><a href="ivf-flat.html">IVF-FLAT</a></td><td>Raw <code>f32</code> vectors</td><td>Exact distance scan in probed lists</td><td>About <code>4d</code> bytes</td><td>No quantization loss; recall mainly depends on <code>nprobe</code></td><td>Low</td><td>Automatic <code>nlist</code>/<code>nprobe</code>; explicit overrides</td><td>Recall ceiling, frequent rebuilds, IP/cosine, or production sets whose scan bytes are affordable</td></tr>
</tbody>
</table>
</div>
</section>
<section class="section" id="public-corpus-check">
<div class="section-heading">
<h2>Public-corpus benchmark results</h2>
<p><strong>IVF-SQ was refreshed on 6 September 2026 across all three corpora.</strong> Its build and local-search cells below use three-run native <code>ann_bench</code> medians with eight Rayon workers. The other four index rows retain their July 2026 measurements with 12 workers; these tables record the latest available results for each index, not a new simultaneous five-index ranking.</p>
<p>All rows use the same public corpus shapes, 1,000 independent queries, <code>nlist=1024</code>, <code>nprobe=64</code>, and Top-10. Native batch timing includes the first payload reads and cache insertion on a fresh reader. The remote-model results remain archived separately below.</p>
</div>
<details id="benchmark-setup">
<summary><strong>Historical five-index setup and reproduction parameters · July 2026</strong></summary>
<p>The following records the original 12-worker comparison. For the refreshed eight-worker IVF-SQ configuration, use the <a href="development.html#ivfsq-bench">current reproduction guide</a>.</p>
<h3>Benchmark setup</h3>
<div class="callout warning"><strong>This is not the zero-configuration benchmark</strong>Running <code>cargo bench -p paimon-vindex-core --bench ann_bench</code> without public file paths uses a 20k-vector, 64-dimensional generated smoke workload. Reproducing the results below requires the public files, recorded IVF and DiskANN search settings, a fixed worker count, and an output directory on the storage device being measured. File shape, training count, and multi-index process isolation are automatic.</div>
<p>The real-data run uses the public <a href="https://github.com/erikbern/ann-benchmarks">ANN-Benchmarks</a> SIFT1M, GIST1M, and GloVe-100 files, the first 1,000 independent test queries, and their published Top-100 exact neighbors. Recall@10 compares only the first ten published neighbors. SIFT and GIST contain one million base vectors with 128 and 960 dimensions. GloVe contains 1,183,514 vectors with 100 dimensions and angular ground truth; its base and query vectors are L2-normalized during conversion so the common L2 benchmark produces the same neighbor ordering as cosine. The benchmark supplies 65,536 base vectors to every trainer; DiskANN bounds PQ training memory with a deterministic reservoir of at most 50,000 vectors.</p>
<div class="table-wrap"><table><thead><tr><th>Source</th><th>Parameters</th><th>Recorded value</th><th>Reproduction rule</th></tr></thead><tbody>
<tr><td>Public data</td><td><code>ANN_BASE_FVECS</code>, <code>ANN_QUERY_FVECS</code>, <code>ANN_GROUND_TRUTH_IVECS</code></td><td>SIFT1M, GIST1M, or normalized GloVe-100 converted files</td><td>Set all three together; otherwise the benchmark generates synthetic vectors.</td></tr>
<tr><td>Dataset shape</td><td><code>ANN_N</code>, <code>ANN_NQ</code>, <code>ANN_D</code></td><td><code>1,000,000 / 1,183,514</code>, <code>1,000</code>, <code>128 / 960 / 100</code></td><td>Inferred from the public files. An explicit value becomes a shape assertion and fails before the full dataset is loaded if it differs.</td></tr>
<tr><td>Training input</td><td><code>ANN_TRAIN_N</code></td><td><code>65,536</code></td><td>Inferred as <code>min(N, max(65,536, 64 × nlist))</code>. DiskANN deterministically retains at most 50,000 of these vectors for bounded PQ training; the IVF trainers consume all 65,536.</td></tr>
<tr><td>IVF search</td><td><code>ANN_NLIST</code>, <code>ANN_NPROBE</code></td><td><code>1,024</code>, <code>64</code></td><td>Set explicitly; benchmark defaults are 64 and 8.</td></tr>
<tr><td>DiskANN search</td><td><code>ANN_DISKANN_L_SEARCH</code></td><td><code>100</code></td><td>May be omitted; this is the benchmark default and the automatic value for <code>k=10</code>.</td></tr>
<tr><td>Process and device isolation</td><td><code>ANN_INDEXES</code>, <code>ANN_OUTPUT_DIR</code>, <code>RAYON_NUM_THREADS</code></td><td>All five indexes, target APFS path, 12 threads</td><td>Public multi-index runs automatically spawn one child process per index. Set a subset only when needed; set the device path and worker count explicitly.</td></tr>
<tr><td>Matching defaults</td><td><code>ANN_K</code>, <code>ANN_PQ_CODE_RATIO</code>, <code>ANN_RQ_BITS</code></td><td><code>10</code>, <code>0.0625</code>, <code>4</code></td><td>May be omitted; the reproduction command pins them so a future default change cannot silently alter the comparison.</td></tr>
<tr><td>DiskANN build defaults</td><td><code>pq.bits</code>, <code>R</code>, <code>Lbuild</code>, <code>alpha</code>, memory budget, layout, raw-vector encoding, build distance</td><td><code>8</code>, <code>64</code>, <code>100</code>, <code>1.2</code>, 8 GiB, compact, F16, product-quantized</td><td>Use the benchmark defaults. Leave <code>ANN_PQ_M</code> unset so <code>pq.code-ratio</code> resolves the concrete value.</td></tr>
<tr><td>Reader/I/O model</td><td>Automatic read plan, Reader budget, simulated latency</td><td>Latency-derived local/remote/object-store plans, 4 GiB automatically partitioned budget, 0/2/20 ms per read round</td><td>Fixed by the current benchmark implementation; <code>ANN_STORAGE_CASES</code> selects a focused subset. DiskANN range reads use an I/O pool independent of query workers; all five indexes were refreshed after their current reader, storage, and parallel-scan work.</td></tr>
</tbody></table></div>
<div class="callout"><strong>Equal relative PQ budget</strong>The default <code>pq.code-ratio=0.0625</code> automatically resolves SIFT to <code>pq.m=32</code>, GIST to <code>pq.m=240</code>, and GloVe to <code>pq.m=25</code>. Every code occupies 6.25% as many bytes as its raw <code>f32</code> vector and leaves four dimensions per PQ sub-vector. The concrete value is persisted in index metadata; use explicit <code>pq.m</code> only as an expert override.</div>
<p>The cross-index run was recorded on 25 July 2026 using an Apple M4 Pro with 12 logical CPUs and 48 GiB RAM, a release build with Rust 1.95, real APFS files with warm operating-system pages, and the automatic 4 GiB DiskANN Reader budget. The IVF-RQ staged A/B and rebased IVF-PQ warm-local refresh were recorded on 30 July on the same host and toolchain. The reproduction command pins Rayon to 12 workers instead of relying on automatic host parallelism. The modeled serving profiles add 2 ms or 20 ms per positional-read round while executing all ranges in that round concurrently. DiskANN's benchmark adapter runs those ranges on a separate 12-worker I/O pool so a full query-worker pool cannot starve nested reads; this models the independent executor required of a production concurrent storage callback. For the 20 ms profile, open/optimization and sequential-query latency are computed as measured CPU/I/O time plus 20 ms per observed round; batch QPS retains literal delay injection so query overlap is measured. IVF multi-range calls are bounded to 64 MiB, so an all-query GIST batch uses 4 IVF-PQ, 15 IVF-SQ, or 59 IVF-FLAT payload rounds instead of submitting hundreds of MiB or several GiB as one unbounded call. Unified IVF Readers now reuse the 64-byte dispatch header, so opening and loading resident metadata takes two positional-read rounds rather than three. Each dataset's three DiskANN profile rows reuse the same built graph. Sequential and batch measurements use separately opened and optimized Readers, so the batch does not inherit query-dependent windows from the sequential sweep. Batch QPS measures one <code>search_batch</code> call over all 1,000 public queries; it is not concurrent-client QPS. See the <a href="development.html#ann">complete public-data command</a>.</p>
</details>
<h3 id="build-results">Build, file, and peak process memory</h3>
<div class="table-wrap"><table><thead><tr><th>Index</th><th>SIFT build</th><th>SIFT file / RSS</th><th>GIST build</th><th>GIST file / RSS</th><th>GloVe build</th><th>GloVe file / RSS</th></tr></thead><tbody>
<tr><td>IVF-PQ</td><td>8.74 s</td><td>0.032 / 0.88 GiB</td><td>55.4 s</td><td>0.230 / 5.00 GiB</td><td>7.92 s</td><td>0.030 / 0.85 GiB</td></tr>
<tr><td><strong>IVF-SQ · 6 Sep</strong></td><td>0.886 s</td><td>0.122 / 0.77 GiB</td><td>5.850 s</td><td>0.907 / 4.87 GiB</td><td>0.797 s</td><td>0.113 / 0.72 GiB</td></tr>
<tr><td>IVF-RQ</td><td>3.92 s</td><td>0.080 / 0.71 GiB</td><td>23.5 s</td><td>0.471 / 4.65 GiB</td><td>4.03 s</td><td>0.095 / 0.68 GiB</td></tr>
<tr><td>DiskANN</td><td>74.0 s</td><td>0.361 / 1.51 GiB</td><td>11 min 26 s</td><td>2.089 / 7.94 GiB</td><td>2 min 33 s</td><td>0.396 / 1.45 GiB</td></tr>
<tr><td>IVF-FLAT</td><td>4.05 s</td><td>0.479 / 1.83 GiB</td><td>24.9 s</td><td>3.582 / 12.74 GiB</td><td>4.10 s</td><td>0.443 / 1.60 GiB</td></tr>
</tbody></table></div>
<h3 id="local-results">Native local-storage results</h3>
<div class="table-wrap"><table><thead><tr><th>Index / search</th><th>SIFT Recall / P95 / batch QPS / read</th><th>GIST Recall / P95 / batch QPS / read</th><th>GloVe Recall / P95 / batch QPS / read</th></tr></thead><tbody>
<tr><td>IVF-PQ</td><td>0.7142 / 0.72 ms / 7,899 / 2.18 MiB</td><td>0.7410 / 2.47 ms / 950 / 17.84 MiB</td><td>0.5819 / 0.64 ms / 8,048 / 1.84 MiB</td></tr>
<tr><td><strong>IVF-SQ · 6 Sep</strong></td><td>0.9811 / 0.298 ms / 8,987 / 0.115 MiB</td><td>0.9400 / 1.828 ms / 998 / 0.850 MiB</td><td>0.8760 / 0.282 ms / 10,009 / 0.108 MiB</td></tr>
<tr><td>IVF-RQ</td><td>0.9148 / 1.20 ms / 3,074 / 5.54 MiB</td><td>0.9039 / 4.41 ms / 444 / 37.02 MiB</td><td>0.8203 / 1.23 ms / 2,917 / 5.89 MiB</td></tr>
<tr><td>DiskANN</td><td>0.9915 / 1.50 ms / 9,009 / 0.66 MiB</td><td>0.9336 / 1.83 ms / 4,651 / 0.83 MiB</td><td>0.8355 / 1.90 ms / 6,289 / 0.96 MiB</td></tr>
<tr><td>IVF-FLAT</td><td>0.9937 / 1.88 ms / 8,510 / 33.19 MiB</td><td>0.9549 / 11.38 ms / 875 / 283.40 MiB</td><td>0.8832 / 1.40 ms / 9,502 / 27.57 MiB</td></tr>
</tbody></table></div>
<p>The refreshed SQ reader reuses decoded partitions within its memory budget. The read column is average payload I/O per sequential query, including cache misses while that sweep warms the reader; it is not the full selected-list size or a cold-cache promise. SQ batch QPS includes the first batch's payload reads on a separate reader. The recorded July rows predate this SQ cache and training change.</p>
<p>For IVF-SQ, train / encode-add / serialize medians in SIFT / GIST / GloVe order are 252 / 538 / 93 ms; 1696 / 3613 / 563 ms; 187 / 517 / 90 ms. Stage medians are computed independently and need not sum to the median total. See the <a href="ivf-sq.html#benchmarks">same-configuration baseline comparison</a> for the optimization's measured effect.</p>
<details id="historical-implementation-notes">
<summary><strong>Historical implementation notes and earlier SQ scores · July 2026</strong></summary>
<p>These notes preserve earlier snapshots. Their performance and implementation descriptions have been superseded where the refreshed tables above provide results.</p>
<div class="callout"><strong>IVF-SQ build and scan refresh</strong>The add path now borrows L2/IP input, assigns rows once, and encodes lists in parallel with one residual scratch vector per active list task instead of materializing an additional <code>N × d</code> residual matrix. In the immediately preceding same-machine run, SIFT/GIST/GloVe peak RSS was 1.81 / 12.92 / 1.60 GiB; it is now 0.79 / 5.09 / 0.71 GiB. A Top-K threshold fast path skips hash work for candidates that cannot enter the heap: local P95 is now 0.79 / 3.56 / 0.71 ms and batch throughput is 11,082 / 1,502 / 12,962 QPS. An experimental list-major batch scan was slower on SIFT/GIST and was not retained. The blocked-code format, file size, read bytes, and measured Recall@10 remain unchanged.</div>
<div class="callout"><strong>30 July IVF-PQ batch-table reuse refresh</strong>The rebased Reader retains the v1 zero-copy/transposed-code, ordered-list, one-byte row-ID, and first-column accumulation fast paths. For large 8-bit residual-L2 batches, the default <code>Auto</code> mode now factors each distance table into reusable per-list and per-query components when the reuse heuristic and 64 MiB working-memory guard both pass; small or unsuitable batches keep the direct path, and callers may explicitly select <code>On</code> or <code>Off</code>. Six same-file runs per mode alternated execution order. SIFT/GIST/GloVe median batch throughput changed from 4,191 / 497 / 4,366 QPS with reuse disabled to 7,899 / 950 / 8,048 QPS with <code>Auto</code>, gains of 88.5% / 91.3% / 84.3%. The <code>Auto</code> medians used below are 0.72 / 2.47 / 0.64 ms P95 and 1,583 / 467 / 1,802 sequential QPS. File bytes, query bytes, and the v1 format are unchanged. GIST and GloVe Recall@10 are unchanged at four decimals; SIFT moved from 0.7143 to 0.7142 because the stable <code>f64</code> factored-table path is numerically close but not bit-identical to direct residual-table accumulation. A removed contiguous all-query table remains distinct from this bounded factorization. Faiss FastScan's 4-bit, 32-row design remains a different accuracy/format choice.</div>
<div class="callout"><strong>Latest IVF-FLAT storage and scan review</strong>The v1 writer retains only sort permutations and encoded IDs, materializes one sorted raw-vector list at a time, and reproduced all three prior public files byte-for-byte. The Reader now receives list bytes directly into an <code>f32</code>-aligned allocation; an internal prefix of at most three bytes keeps the raw-vector suffix aligned despite variable-length row IDs, so search no longer allocates and decodes a second vector payload. Together with the strict partial-L2 cutoff, the complete public rerun changed SIFT/GIST/GloVe local batch throughput from 6,570 / 559 / 6,345 to 8,510 / 875 / 9,502 QPS and P95 from 5.31 / 47.04 / 4.76 ms to 1.88 / 11.38 / 1.40 ms. Recall@10, file version, file bytes, and bytes read are unchanged; these are complete-run results, not best-of measurements.</div>
<div class="callout"><strong>Pre-release storage changes produced measurable savings</strong>The compact IVF-RQ factor layout removes one unused F32 value per row: SIFT/GIST/GloVe files fell from 90.3 / 509.8 / 106.8 MB to 86.3 / 505.8 / 102.1 MB without changing the estimator; the refreshed Recall@10 values are 0.9148 / 0.9039 / 0.8203. Changing the balanced DiskANN default from F32 to F16 rerank vectors reduced the same three files from 0.599 / 3.877 / 0.617 GiB to 0.361 / 2.089 / 0.396 GiB. The current warm-local Recall@10 values are 0.9915 / 0.9336 / 0.8355. Timing changes also include the intervening reader/cache and graph-build work.</div>
<div class="callout"><strong>30 July IVF-RQ staged scan A/B</strong>The four changes below were measured against one retained v1 index per corpus with the same 1,000 queries, <code>nlist=1,024</code>, <code>nprobe=64</code>, four stored RQ bits, 12 Rayon workers, and warm APFS pages. SIFT and GloVe use seven interleaved runs. Because GIST-960 showed thermal drift during long stage sweeps, its reported wall-clock changes use five baseline/final pairs with alternating execution order. Every staged binary returned the same Recall@10. The GIST timing A/B used the public Open VDB mirror and returned 0.9037 rather than the earlier ANN-Benchmarks file's 0.9039, so the cross-index table below retains the original recall row and this section uses timing deltas only.</div>
<div class="table-wrap"><table><thead><tr><th>IVF-RQ change</th><th>SIFT1M result</th><th>GIST1M result</th><th>GloVe-100 result</th></tr></thead><tbody>
<tr><td>Block-aggregated scan statistics</td><td>70.91% of batch candidates reached full bit-plane refinement</td><td>95.88% reached bit-plane refinement, but the complete approximate-distance bound reduced exact coarse reevaluation to 1.45%</td><td>96.03% reached full bit-plane refinement</td></tr>
<tr><td>Reuse IVF centroid distance</td><td>Removes 8.2 million repeated rotated centroid terms per 1,000-query run</td><td>Removes 61.4 million repeated terms; end-to-end movement stayed inside run variance because code scan dominates</td><td>Removes 8.2 million repeated padded terms per run</td></tr>
<tr><td>16-entry FastScan LUT + NEON/AVX2</td><td>Deliberately bypassed below padded dimension 256</td><td>Versus the optimized scalar scanner: P95 −8.3%, sequential QPS +6.6%, batch QPS +17.8% in paired medians</td><td>Deliberately bypassed below padded dimension 256</td></tr>
<tr><td>32-vector single-query seed threshold</td><td>Refinement work −4.3%; P95 −0.9% versus no seed</td><td>Exact final evaluations −33.1%; paired sequential QPS +1.3% while P95 was neutral</td><td>Refinement work −0.32%; P95 −2.3% versus no seed</td></tr>
</tbody></table></div>
<p>The first statistics prototype incremented counters inside byte-lookup loops and regressed GIST, so it was not retained. The final implementation derives lookup counts once per block or admitted candidate, keeps per-list statistics thread-local, and merges them after parallel work. Centroid reuse derives the RQ query terms from the distances already produced by IVF probing and stores only centroid norms in the Reader. FastScan quantizes two 16-entry nibble tables, evaluates 32 rows with NEON or AVX2, then uses a conservative complete-distance interval before doing exact coarse reevaluation; the final ranking still uses all persisted RQ bit planes and does not require original vectors. Small dimensions stay on the exact scalar byte-LUT path because their public A/B did not justify SIMD setup.</p>
<div class="table-wrap"><table><thead><tr><th>Same-file endpoint</th><th>SIFT P95 / sequential QPS / batch QPS</th><th>GIST P95 / sequential QPS / batch QPS</th><th>GloVe P95 / sequential QPS / batch QPS</th></tr></thead><tbody>
<tr><td>Pre-change baseline</td><td>1.101 ms / 1,020 / 2,706</td><td>5.599 ms / 203 / 325</td><td>1.071 ms / 1,041 / 2,855</td></tr>
<tr><td>Final scanner</td><td>1.081 ms / 1,042 / 2,743</td><td>5.190 ms / 218 / 399</td><td>1.046 ms / 1,057 / 2,839</td></tr>
</tbody></table></div>
<p>Against its paired baseline, the final GIST scanner improved median P95 by 8.7%, sequential QPS by 8.1%, and batch QPS by 20.0%. SIFT improved P95 by 1.8%, sequential QPS by 2.1%, and batch QPS by 1.4%. GloVe improved P95 by 2.3% and sequential QPS by 1.6%; its 0.6% batch-QPS decrease is treated as noise, not an improvement. File bytes, bytes read, storage format, and compressed-domain ranking are unchanged.</p>
<div class="callout"><strong>DiskANN read-path review</strong>DiskANN converts F16 rerank vectors and accumulates L2 directly in one AArch64 NEON loop; its local profile coalesces 16 KiB windows. The final compact-layout batch rerank groups candidate windows with a hash table, then sorts only the unique windows into deterministic I/O order. Against the immediately preceding ordered-map control, median local batch time changed from 117 / 223 / 162 ms to 111 / 215 / 159 ms on SIFT/GIST/GloVe. Local P95 remains 1.50 / 1.83 / 1.90 ms; representative batch throughput is 9,009 / 4,651 / 6,289 QPS with a separate range-I/O executor. Remote and object-store plans remain 32 / 64 KiB. A broader Vec sort/dedup replacement for graph window planners was not retained because it regressed local SIFT/GIST batch time by 7–13%.</div>
<div class="callout"><strong>Final open-source cross-check and format decision</strong><a href="https://github.com/facebookresearch/faiss/wiki/Fast-accumulation-of-PQ-and-AQ-codes-%28FastScan%29">Faiss FastScan</a> still trades down to 4-bit lookup tables and a 32-row layout, so it is not a transparent replacement for the published 8-bit IVF-PQ v1. Faiss Panorama's additional level-oriented energy data was not needed to keep the existing IVF-FLAT v1 progressive cutoff. Lance's partition prefetch and transposed PQ storage match the current batched Readers; its prepared transposed L2 target is explicitly aimed at small target sets such as PQ codebooks, not large flat lists. Faiss RaBitQ's blocked multi-bit scan remains structurally aligned with IVF-RQ, while a direct-factor RQ payload experiment regressed SIFT batch throughput by 10–15% and was removed. <a href="https://github.com/microsoft/DiskANN">DiskANN3</a>'s asynchronous provider, beam, and working-set model remains aligned with <code>SeekRead</code>, the latency-derived read planner, and the bounded caches. No measured result justified a v2 migration for IVF-PQ or IVF-FLAT, and no byte-layout change was retained for the pre-release IVF-SQ, IVF-RQ, or DiskANN formats.</div>
<p>DiskANN spends almost all build time constructing one global graph: about 18× IVF-FLAT on SIFT, 28× on GIST, and 37× on GloVe. The balanced F16 default makes its files about 42% smaller than IVF-FLAT on GIST and 11% smaller on GloVe, but they remain much larger than the compact IVF encodings because persisted rerank vectors, resident codes, and graph edges are all material. Peak RSS remains below the raw IVF writers because the DiskANN writer does not retain a second full raw-vector organization.</p>
</details>
<details id="historical-remote-results">
<summary><strong>Historical remote and object-store models · July 2026</strong></summary>
<p>All rows in this archive, including IVF-SQ, retain the original implementation and 12-worker configuration. The current pooled-bound and cached IVF-SQ has not been remeasured under these latency models.</p>
<h4>Remote cache with 2 ms per I/O round</h4>
<div class="table-wrap"><table><thead><tr><th>Index / search</th><th>SIFT Recall / P95 / batch QPS / rounds</th><th>GIST Recall / P95 / batch QPS / rounds</th><th>GloVe Recall / P95 / batch QPS / rounds</th></tr></thead><tbody>
<tr><td>IVF-PQ</td><td>0.7142 / 6.16 ms / 7,282 / 1.0</td><td>0.7410 / 7.12 ms / 1,162 / 1.0</td><td>0.5819 / 5.94 ms / 8,243 / 1.0</td></tr>
<tr><td>IVF-SQ</td><td>0.8627 / 6.38 ms / 9,345 / 1.0</td><td>0.8577 / 11.36 ms / 1,359 / 1.9</td><td>0.8036 / 6.16 ms / 10,502 / 1.0</td></tr>
<tr><td>IVF-RQ</td><td>0.9148 / 7.05 ms / 2,916 / 1.0</td><td>0.9039 / 7.80 ms / 434 / 1.0</td><td>0.8203 / 6.74 ms / 3,080 / 1.0</td></tr>
<tr><td>DiskANN</td><td>0.9808 / 15.97 ms / 3,089 / 1.7</td><td>0.8482 / 14.22 ms / 2,175 / 1.5</td><td>0.8029 / 18.18 ms / 2,203 / 2.1</td></tr>
<tr><td>IVF-FLAT</td><td>0.9937 / 7.31 ms / 6,763 / 1.0</td><td>0.9549 / 29.73 ms / 846 / 5.0</td><td>0.8832 / 6.60 ms / 7,394 / 1.0</td></tr>
</tbody></table></div>
<p>IVF-PQ and IVF-RQ load each query's selected compact lists in one concurrent multi-range round. IVF-SQ does the same on SIFT/GloVe; GIST's 960-dimensional payload crosses the 64 MiB per-call guard on 89% of queries and averages 1.9 rounds. IVF-FLAT also uses bounded concurrent multi-range reads: SIFT/GloVe fit in one round, while the 283 MiB GIST payload averages 5.0. IVF-RQ preserves the strongest compact-IVF recall with 7.05 / 7.80 / 6.74 ms P95. Parallel IVF-FLAT has much higher recall and competitive fixed-latency results on SIFT/GloVe, but transfers 28–33 MiB per query; this model does not charge bandwidth. DiskANN's adaptive coalescing and caches reduce the average sequential request count to 1.5–1.7 rounds on SIFT/GIST and 2.1 rounds on GloVe.</p>
<h4>Object store with 20 ms per I/O round</h4>
<div class="table-wrap"><table><thead><tr><th>Index / search</th><th>SIFT Recall / P95 / batch QPS / rounds</th><th>GIST Recall / P95 / batch QPS / rounds</th><th>GloVe Recall / P95 / batch QPS / rounds</th></tr></thead><tbody>
<tr><td>IVF-PQ</td><td>0.7142 / 20.70 ms / 6,831 / 1.0</td><td>0.7410 / 22.14 ms / 1,054 / 1.0</td><td>0.5819 / 20.62 ms / 8,299 / 1.0</td></tr>
<tr><td>IVF-SQ</td><td>0.8627 / 20.77 ms / 6,781 / 1.0</td><td>0.8577 / 43.50 ms / 854 / 1.9</td><td>0.8036 / 20.68 ms / 7,175 / 1.0</td></tr>
<tr><td>IVF-RQ</td><td>0.9148 / 21.05 ms / 2,688 / 1.0</td><td>0.9039 / 24.34 ms / 392 / 1.0</td><td>0.8203 / 21.04 ms / 2,767 / 1.0</td></tr>
<tr><td>DiskANN</td><td>0.9808 / 60.57 ms / 537 / 1.0</td><td>0.8483 / 41.19 ms / 1,011 / 1.0</td><td>0.8033 / 80.84 ms / 421 / 1.2</td></tr>
<tr><td>IVF-FLAT</td><td>0.9937 / 21.80 ms / 3,076 / 1.0</td><td>0.9549 / 130.49 ms / 349 / 5.0</td><td>0.8832 / 21.22 ms / 2,948 / 1.0</td></tr>
</tbody></table></div>
<p>At 20 ms per round, IVF-RQ is the strongest measured compact one-round option: it reaches 0.90-class recall on SIFT/GIST and 0.8203 on GloVe. IVF-SQ is faster when its lower recall is enough, and IVF-PQ is smaller when stronger quantization loss is acceptable. IVF-FLAT now looks competitive on one-round SIFT/GloVe in this fixed-latency model, but that result assumes 28–33 MiB transfers have no bandwidth cost; GIST's 283 MiB and five rounds expose the boundary. DiskANN averages about one modeled round after warmup, but dependent graph rounds remain visible in P95. A complete local SSD cache remains its preferred deployment.</p>
<div class="callout warning"><strong>The automatic read plan affects approximate search</strong>The latency-derived local tier uses graph beam 4 while remote and object-store tiers use beam 16, so the same <code>l_search</code> can return different approximate candidates; this is visible for both GIST and GloVe at <code>l_search=100</code>. The tiers use 16 KiB, 32 KiB, and 64 KiB coalescing windows respectively. Storage latency itself does not change ground truth. Compare indexes with the same latency and capability hints when isolating media effects.</div>
<div class="callout warning"><strong>Remote-model boundary</strong>The vectors and exact neighbors are public corpus data, but the 2 ms and 20 ms profiles are controlled I/O models rather than measurements from a production cache or object store. They add fixed latency without modeling bandwidth, cache misses, TLS, retries, throttling, request limits, or tail-latency variance.</div>
</details>
</section>
<section class="section" id="decision">
<div class="section-heading">
<h2>Choose by constraint</h2>
<p>There is no best index independent of data distribution. Narrow the field to one or two candidates, then evaluate Recall@K, P95/P99 latency, file size, build time, and object-store bytes on real queries.</p>
</div>
<div class="callout"><strong>Practical default order</strong>First reject indexes that cannot meet the measured recall target. Build IVF-FLAT to establish the corpus-specific IVF ceiling. If a compact representation is required, start with IVF-SQ when its one-byte-per-dimension codes fit; compare IVF-RQ at the required bit budget and IVF-PQ for minimum bytes. Evaluate DiskANN separately for immutable data served from local SSD; do not select it only because the collection is large or assume L2 results transfer to another metric.</div>
<h3>Measured recommendation matrix</h3>
<p>Except for the refreshed SQ results, numerical evidence below comes from the July five-index matrix. The latest SQ results cover SIFT, GIST, and GloVe; other algorithms have not been rerun in this refresh.</p>
<div class="table-wrap"><table><thead><tr><th>Production constraint</th><th>Start with</th><th>Recorded evidence</th><th>Move away when</th></tr></thead><tbody>
<tr><td>Establish a recall ceiling or debug ranking quality</td><td><a href="ivf-flat.html">IVF-FLAT</a></td><td>Highest measured recall on all three corpora: 0.9937 / 0.9549 / 0.8832, with roughly four-second SIFT/GloVe builds.</td><td>The 28–283 MiB selected-list reads or raw-vector file size exceed the serving budget.</td></tr>
<tr><td>Compact scans and repeated-query throughput</td><td><a href="ivf-sq.html">IVF-SQ</a></td><td>The <a href="#local-results">refreshed three-corpus SQ row</a> records pooled-bound recall, native P95, batch QPS, and average payload I/O. Files are about one quarter of IVF-FLAT.</td><td>The recall gate is above SQ, or one byte per dimension is still too large.</td></tr>
<tr><td>Configurable codes smaller than IVF-SQ</td><td><a href="ivf-rq.html">IVF-RQ</a></td><td>The historical four-bit run reached 0.9148 / 0.9039 / 0.8203 recall in files smaller than IVF-SQ, with one sequential read round per query in all three modeled profiles.</td><td>The chosen width misses the recall or throughput target. Compare the current SQ implementation before accepting the extra scan cost.</td></tr>
<tr><td>Minimum index file and compact-IVF scan bytes</td><td><a href="ivf-pq.html">IVF-PQ</a></td><td>The smallest files—0.032 / 0.230 / 0.030 GiB—and the smallest IVF selected-list reads at 1.84–17.84 MiB, with strong batch throughput.</td><td>0.5819–0.7410 recall is below the gate; increase the PQ budget or choose SQ/RQ instead.</td></tr>
<tr><td>High-recall immutable data on local SSD</td><td><a href="diskann.html">DiskANN</a>, checked against IVF-FLAT for the same metric</td><td>The recorded L2-equivalent run reached 0.9915 / 0.9336 / 0.8355 recall with 0.66 / 0.83 / 0.96 MiB reads; SIFT/GIST P95 is 1.50 / 1.83 ms.</td><td>Metric-specific recall misses the gate, rebuilds are frequent, the file is not locally cached, preview maturity is unacceptable, or the corpus behaves like GloVe at <code>l_search=100</code>.</td></tr>
<tr><td>Frequent rebuilds or rapidly changing snapshots</td><td><a href="ivf-flat.html">IVF-FLAT</a>, <a href="ivf-sq.html">IVF-SQ</a>, or <a href="ivf-rq.html">IVF-RQ</a></td><td>The historical five-index run favored these IVF variants for rebuild cost. Refreshed SQ builds take 0.886 s / 5.850 s / 0.797 s on SIFT / GIST / GloVe.</td><td>The serving phase dominates lifetime cost enough to justify PQ training or graph construction.</td></tr>
<tr><td>Direct 2/20 ms remote or object-store reads</td><td>Compact IVF selected by measured recall, payload bytes, and cache budget</td><td>In the historical uncached run, PQ and RQ used one sequential multi-range round; SQ did so on SIFT/GloVe and averaged 1.9 rounds on GIST. Current SQ cache hits avoid payload reads; measure miss behavior on the real adapter.</td><td>Bandwidth, request limits, or real tail latency invalidate the fixed-latency model; prefer a complete local SSD cache and rerun the benchmark.</td></tr>
<tr><td>Inner product or cosine</td><td>IVF-FLAT as the recall control; DiskANN as an additional candidate for immutable local-SSD serving</td><td>All five implementations support L2, IP, and cosine. DiskANN normalizes cosine internally and uses metric-aware graph construction and exact reranking, but the displayed public-corpus matrix was recorded through the L2-equivalent benchmark path.</td><td>The selected configuration misses its metric-specific recall gate—retune <code>nprobe</code>, representation width, OPQ, or <code>l_search</code> before deployment.</td></tr>
</tbody></table></div>
<div class="callout warning"><strong>A displayed winner can still be the wrong choice</strong>These recommendations apply to the recorded <code>nlist=1024</code>, <code>nprobe=64</code>, PQ ratio, RQ bits, and <code>l_search=100</code>. For example, the historical GloVe run does not reach 0.90 recall with any index, and historical GIST reaches 0.95 only with IVF-FLAT. The new SQ GloVe result also remains below 0.90. If a required recall threshold is not present in the table, tune and rebuild rather than choosing the closest result.</div>
<div class="card-grid">
<article class="card"><h3>I need a trustworthy baseline</h3><p>Start with IVF-FLAT. It exposes the IVF partition ceiling without quantization loss and rebuilds quickly.</p><a class="card-link" href="ivf-flat.html">Explore IVF-FLAT →</a></article>
<article class="card"><h3>I need configurable compact codes</h3><p>Try IVF-RQ when SQ is too large and tune the bit width against recall and throughput. Compare every result with the IVF-FLAT ceiling and the current SQ baseline.</p><a class="card-link" href="ivf-rq.html">Explore IVF-RQ →</a></article>
<article class="card"><h3>I need the smallest index</h3><p>Choose IVF-PQ when its corpus-specific recall passes the gate. It is the capacity-first option, not the automatic middle ground.</p><a class="card-link" href="ivf-pq.html">Explore IVF-PQ →</a></article>
<article class="card"><h3>I need compact batch speed</h3><p>Choose IVF-SQ when one byte per dimension fits. Pooled training bounds and bounded partition caching improve the measured three-corpus recall and repeated-query performance; validate the target corpus and cache budget.</p><a class="card-link" href="ivf-sq.html">Explore IVF-SQ →</a></article>
<article class="card"><h3>Raw vectors exceed RAM but fit local SSD</h3><p>Evaluate DiskANN for immutable L2, IP, or cosine data when high recall and sub-MiB query reads justify a much slower build; retain IVF-FLAT as the metric-specific accuracy control.</p><a class="card-link" href="diskann.html">Explore DiskANN →</a></article>
<article class="card"><h3>Data lives in S3, OSS, or HDFS</h3><p>Prefer durable publication plus a complete local SSD cache. For direct remote reads, start with a compact IVF index when one-round scans meet recall; use DiskANN only after measuring its corpus-dependent coalesced graph rounds.</p><a class="card-link" href="diskann.html#deployment">Compare deployment modes →</a></article>
</div>
</section>
<section class="section" id="parameters">
<div class="section-heading"><h2>How parameters interact</h2><p>Build parameters define static structures; query parameters define per-request work. For IVF, changing <code>nlist</code> usually changes the useful <code>nprobe</code> range. DiskANN instead couples graph build quality with online <code>l_search</code>.</p></div>
<div class="callout"><strong>Automate numeric work, keep semantics explicit</strong><code>index.type</code> and <code>metric</code> remain required because changing either changes persistence and result meaning. Rust callers can use <code>recommend_index</code> as an advisory starting point and must explicitly accept its result. For measured offline sweeps, <code>select_calibrated_candidate</code> chooses the smallest candidate satisfying supplied recall, byte, and build-time objectives and returns no result when the sample does not meet them.</div>
<div class="table-wrap"><table><thead><tr><th>Parameter</th><th>Stage</th><th>Indexes</th><th>Typical effect when increased</th><th>Constraint / default</th></tr></thead><tbody>
<tr><td><code>index.type</code></td><td>Build</td><td>All</td><td>Changes the persisted algorithm and its capability boundary</td><td>Required; recommendation is advisory, never silently applied</td></tr>
<tr><td><code>metric</code></td><td>Build</td><td>All</td><td>Changes training, ranking, and ground-truth semantics</td><td>Required and never inferred</td></tr>
<tr><td><code>dimension</code></td><td>Build</td><td>All</td><td>Changes representation width and distance work</td><td>Inferred by Java/Python one-shot training; required by streaming APIs</td></tr>
<tr><td><code>nlist</code></td><td>Build</td><td>IVF families</td><td>Shorter lists and more coarse centroids; a fixed <code>nprobe</code> covers less of the collection</td><td>Auto: nearest power of two around √N, with at least 64 rows of training density per list; requires <code>expected-vector-count</code></td></tr>
<tr><td><code>nprobe</code></td><td>Query</td><td>IVF families</td><td>Reads more lists; recall usually rises with latency and I/O</td><td>Auto is K-, N-, nlist-, and filter-selectivity-aware; explicit values are expert overrides</td></tr>
<tr><td><code>pq.code-ratio</code></td><td>Build</td><td>IVF-PQ, DiskANN</td><td>Raises or lowers the automatically inferred code bytes and subquantizer count</td><td>Default 0.0625; finite and positive</td></tr>
<tr><td><code>pq.m</code></td><td>Build</td><td>IVF-PQ, DiskANN</td><td>Expert override for the inferred subquantizer count; larger values often reduce quantization error but add lookup work</td><td>Optional; <code>d % m == 0</code></td></tr>
<tr><td><code>rq.bits</code></td><td>Build</td><td>IVF-RQ</td><td>More persisted bit planes improve reconstruction and usually recall while increasing file bytes, I/O, and scan work</td><td>Auto from <code>max-bytes-per-vector</code>; otherwise <code>4</code></td></tr>
<tr><td><code>use-opq</code></td><td>Build</td><td>IVF-PQ</td><td>Adds training and matrix cost; may improve PQ quality</td><td>Auto enables at <code>target-recall ≥ 0.9</code>; explicit true/false wins</td></tr>
<tr><td><code>target-recall</code></td><td>Build objective</td><td>IVF-PQ, DiskANN</td><td>Selects OPQ and a coherent DiskANN build preset</td><td>Starting policy only; validate measured recall on held-out queries</td></tr>
<tr><td><code>max-bytes-per-vector</code></td><td>Build objective and preflight bound</td><td>IVF-PQ, IVF-RQ, DiskANN</td><td>Reduces code width and may select 4-bit/F16 DiskANN storage; rejects configurations whose conservative persisted-size estimate exceeds the bound</td><td>Includes estimated row bytes and, when <code>expected-vector-count</code> is set, amortized fixed data; not an exact final-file-size promise</td></tr>
<tr><td><code>max-build-seconds</code></td><td>Offline calibration objective</td><td>Measured candidate sets</td><td>Rejects candidates whose measured build time exceeds the target</td><td>Accepted by <code>VectorIndexBuildPlan</code>; direct Trainer creation rejects it because build time cannot be safely guessed from hardware</td></tr>
<tr><td><code>diskann.build-preset</code></td><td>Build</td><td>DiskANN</td><td>Moves together across degree, construction width, encoding, and build distance</td><td><code>fast_build</code>, <code>balanced</code>, or <code>high_recall</code>; inferred from target recall</td></tr>
<tr><td><code>deployment-profile</code></td><td>Build</td><td>DiskANN</td><td>Selects interleaved layout for eligible memory/local serving and compact layout for remote/object serving</td><td>Explicit layout/encoding/build-distance overrides always win</td></tr>
<tr><td><code>estimated_random_read_latency_nanos</code></td><td>Reader input capability</td><td>DiskANN</td><td>Selects the internal read window, graph beam, and automatic cache partition without probe I/O</td><td>0 measures the mandatory header read; positive values are useful for known remote/cache latency</td></tr>
<tr><td><code>l_search</code></td><td>Query</td><td>DiskANN</td><td>Larger DiskANN candidate list, usually higher recall and latency</td><td>Auto uses calibrated 100/200/400 when available, otherwise <code>max(100, 2k)</code></td></tr>
<tr><td><code>memory_budget_bytes</code></td><td>Reader</td><td>DiskANN, IVF-SQ</td><td>Reserves resident state, then bounds DiskANN adjacency/raw-vector caches or the IVF-SQ partition cache</td><td>4 GiB; cache sub-budgets are internal</td></tr>
</tbody></table></div>
</section>
<section class="section" id="io">
<div class="section-heading"><h2>Data-lake storage and I/O</h2><p>IVF files begin with a 64-byte v1 header and use model/list sections. DiskANN uses a 256-byte header, page-aligned resident/adjacency data, and either densely packed compact vector records or interleaved page-contained records. The Reader dispatches on the first four-byte magic and uses positional reads for both layouts.</p></div>
<div class="metric-strip"><div class="metric"><span class="label">Byte order</span><span class="value">Little-endian</span></div><div class="metric"><span class="label">Type discovery</span><span class="value">First 4-byte magic</span></div><div class="metric"><span class="label">Row IDs</span><span class="value">IVF delta varints / DiskANN adaptive packed</span></div><div class="metric"><span class="label">Integrity</span><span class="value">Outer Paimon file layer</span></div></div>
<div class="callout warning"><strong>Format boundary</strong>v1 files have no footer, checksum, compression envelope, or schema registry. Roaring filters are query payloads and are not embedded. Readers reject unknown versions, required flags, non-zero reserved bytes, and malformed sections.</div>
</section>
<section class="section" id="evaluation">
<div class="section-heading"><h2>A practical evaluation order</h2><p>Fix the dataset and query set, then introduce approximation one layer at a time. This makes it possible to attribute loss to IVF selection, vector quantization, or graph traversal.</p></div>
<div class="pipeline">
<div class="pipeline-item"><span class="pipeline-index">1</span><div><h3>Build ground truth</h3><p>Generate exact top K using the production metric, realistic filters, and edge cases such as zero vectors.</p></div></div>
<div class="pipeline-item"><span class="pipeline-index">2</span><div><h3>Measure IVF-FLAT</h3><p>Establish the recall ceiling caused by probing only <code>nprobe</code> lists and record bytes read.</p></div></div>
<div class="pipeline-item"><span class="pipeline-index">3</span><div><h3>Compare compression</h3><p>Use the same <code>nlist/nprobe</code> for IVF-SQ, IVF-PQ, and IVF-RQ to isolate added approximation and storage savings.</p></div></div>
<div class="pipeline-item"><span class="pipeline-index">4</span><div><h3>Evaluate disk-backed search</h3><p>When raw data exceeds RAM, compare DiskANN on local SSD and realistic remote storage, including cold/warm caches and read rounds.</p></div></div>
<div class="pipeline-item"><span class="pipeline-index">5</span><div><h3>Apply production budgets</h3><p>Set thresholds for Recall@K, P99, file size, build time, RSS, and remote bytes rather than optimizing average latency alone.</p></div></div>
</div>
</section>
</div>
</main>
<footer class="site-footer"><div class="footer-inner"><span>Apache Paimon Vector Index · Implementation guide</span><span>Based on the current Rust core and v1 storage format</span></div></footer>
</body>
</html>