feat(gpu_hnsw): re-vendor faiss GPU tree (nq-chunked visited bitmap, OOM fix)

[devin-ai-integration]

Summary

Re-vendors the faiss GPU HNSW OOM fix (6si/faiss#9) into knowhere's in-tree thirdparty/faiss, byte-identical to 6si/faiss@devin/gpu-hnsw-oom-chunk, and adds a chunking-parity unit test.

The bug: GPU HNSW search scratch is grow-only per pool slot. The visited bitmap is nq * ceil(N/32) * 4 bytes and is never released, so high search concurrency (one bitmap per concurrent slot) on large segments exhausts device VRAM — 16 concurrent batch=512 searches on a ~538M-row segment grew the pool to ~97 of ~98 GB and OOM'd everything after (v90 Blackwell validation).

The fix (vendored): the layer-0 search processes queries in nq-chunks so the bitmap is bounded regardless of batch size / concurrency; GpuHnswSearchScratch::ensure() sizes the bitmap for one chunk. gpu_hnsw_bitmap_chunk(nq, N) derives the per-launch query count from a VRAM cap (default 256 MiB; env-tunable via GPU_HNSW_BITMAP_MB / GPU_HNSW_BITMAP_BYTES). Chunking is result-invariant (queries independent; per-query buffers indexed by chunk-local block index against caller-offset base pointers), and small segments still run a single pass. See 6si/faiss#9 for the full mechanism.

Test — tests/ut/test_gpu_search.cc [gpu_hnsw_chunk_parity]: runs each search twice and asserts element-for-element identical ids/distances:

unsetenv("GPU_HNSW_BITMAP_BYTES");            // default cap => chunk == nq (single pass)
base = gpu_idx.Search(q, json, bitset);
setenv("GPU_HNSW_BITMAP_BYTES", ~2*per_query) // force many chunks
chunked = gpu_idx.Search(q, json, bitset);
REQUIRE(chunked.ids == base.ids && chunked.dist ≈ base.dist);

Sections cover L2/IP/COSINE × {fp32 unfiltered, int8 unfiltered (DP4A: distinct int8 layer-0 vs fp32 upper-layer query offsets), fp32 filtered ~30% (two-tier beam + per-chunk worklist reset), fp32 filtered ~99% (up-front brute-force fallback)}.

Only thirdparty/faiss/faiss/gpu/impl/{GpuHnswTypes.h,GpuHnswTypes.cu,GpuHnswSearch.cuh} and the test change; no public API / index-registration changes.

Related PRs

• faiss source: fix(gpu_hnsw): bound visited-bitmap VRAM via nq-chunked layer-0 search faiss#9
• milvus pin bump: follows once this merges.

Link to Devin session: https://6sense.devinenterprise.com/sessions/55dab0bd33c346df99b5818b30059342
Requested by: @premal

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring