feat: add configurable IVF and PQ training sample limits - #98
Draft
jerry-024 wants to merge 2 commits into
Draft
Conversation
jerry-024
marked this pull request as draft
September 9, 2026 05:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Allow callers to configure IVF and PQ training sample limits independently through the shared option-map API. Both options default to the existing limit of 256 points per centroid.
Changes
ivf.train.max-points-per-centroidpq.train.max-points-per-centroidTesting
cargo test -p paimon-vindex-core training_max_points_per_centroid --lib— 3 passed; covers defaults, validation, configured training across all five index types, and OPQ.cargo test -p paimon-vindex-core config_ --lib— 14 passed.cargo test -p paimon-vindex-core --test storage_format_fixtures— 2 passed; fixture generator ignored.cargo clippy --all-targets --workspace -- -D warningscargo fmt --all -- --checkdocs/*.htmlinto a temporary Cargo example and compiled them withcargo check --offline -p paimon-vindex-core --example docs_training_config_check.Notes
VectorIndexConfigenum literals must provide the new training-limit fields. Existing constructors andtrainmethods retain their defaults.fields.<field-name>.prefix. Paimon integrations need a separate option-filter change to forward these keys.