Skip to content

fix: bound IVF-RQ serialization allocations - #93

Open
jackylee-ch wants to merge 1 commit into
apache:mainfrom
jackylee-ch:fix/ivfrq-write-buffer-bound
Open

fix: bound IVF-RQ serialization allocations#93
jackylee-ch wants to merge 1 commit into
apache:mainfrom
jackylee-ch:fix/ivfrq-write-buffer-bound

Conversation

@jackylee-ch

Copy link
Copy Markdown

Summary

write_ivfrq_index materialized a full byte copy of each f32 section before writing it. The quantizer centroid section is nlist * d floats, bounded only by MAX_SECTION_ELEMENTS, so serializing a large index could allocate several GiB transiently on top of the resident centroids. Stream those sections through one reusable 64 MiB write buffer, mirroring #77 for IVF-Flat.

Testing

  • New ivfrq_chunked_writer_preserves_format_and_bounds asserts byte-identical output and a bounded maximum write. Without the buffer the centroid section goes out as one 32768-byte write and the test fails.
  • cargo test --release --workspace (503 passed), including storage_format_fixtures
  • cargo fmt --all -- --check, cargo +1.98.0 clippy --release --all-targets --workspace -- -D warnings

Notes

No public API, index option, or file-format change; output is byte-for-byte identical and independent of chunk boundaries. The per-list code and factor buffers built by block_list are unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant