Skip to content

feat: add configurable IVF and PQ training sample limits - #98

Draft
jerry-024 wants to merge 2 commits into
apache:mainfrom
jerry-024:feat/train-max-points-per-centroid
Draft

feat: add configurable IVF and PQ training sample limits#98
jerry-024 wants to merge 2 commits into
apache:mainfrom
jerry-024:feat/train-max-points-per-centroid

Conversation

@jerry-024

@jerry-024 jerry-024 commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Allow callers to configure IVF and PQ training sample limits independently through the shared option-map API. Both options default to the existing limit of 256 points per centroid.

Changes

Option Applies to
ivf.train.max-points-per-centroid Coarse K-means training for IVF-FLAT, IVF-SQ, IVF-RQ, and IVF-PQ, including hierarchical clustering stages
pq.train.max-points-per-centroid PQ training for IVF-PQ and DiskANN, including PQ training inside OPQ
  • Pass the configured limits through the trainer to the corresponding clustering stages.
  • Reject non-positive or malformed values, sample-count overflow, and options unsupported by the selected index type.
  • Document the options in the API and per-index parameter tables, fix Rust configuration examples, and add migration guidance.

Testing

  • cargo test -p paimon-vindex-core training_max_points_per_centroid --lib — 3 passed; covers defaults, validation, configured training across all five index types, and OPQ.
  • cargo test -p paimon-vindex-core config_ --lib — 14 passed.
  • cargo test -p paimon-vindex-core --test storage_format_fixtures — 2 passed; fixture generator ignored.
  • cargo clippy --all-targets --workspace -- -D warnings
  • cargo fmt --all -- --check
  • Extracted all 9 Rust configuration examples from docs/*.html into a temporary Cargo example and compiled them with cargo check --offline -p paimon-vindex-core --example docs_training_config_check.

Notes

  • Rust source compatibility: Direct VectorIndexConfig enum literals must provide the new training-limit fields. Existing constructors and train methods retain their defaults.
  • Existing trainer reservoirs, OPQ's input cap, and DiskANN's memory budget still constrain the available training data. Increasing these options does not raise those limits.
  • These options affect training only and do not change the index file format.
  • Native option keys have no fields.<field-name>. prefix. Paimon integrations need a separate option-filter change to forward these keys.

@jerry-024 jerry-024 changed the title feat: configure IVF and PQ training points per centroid feat: add configurable IVF and PQ training sample limits Sep 9, 2026
@jerry-024
jerry-024 marked this pull request as draft September 9, 2026 05:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant