Skip to content

feat(lumina): support Lumina tag filtering#404

Open
lxy-9602 wants to merge 3 commits into
alibaba:mainfrom
lxy-9602:update-lumina-tag
Open

feat(lumina): support Lumina tag filtering#404
lxy-9602 wants to merge 3 commits into
alibaba:mainfrom
lxy-9602:update-lumina-tag

Conversation

@lxy-9602

@lxy-9602 lxy-9602 commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Purpose

This PR upgrades the bundled Lumina dependency to v0.3.0 and adds Lumina tag-aware build/search support for Paimon C++ global vector indexes.

Main changes:

  • Add GlobalIndexer::GetExtraFieldNames() so index implementations can request extra table fields during global index construction.
  • Persist extra_field_ids through GlobalIndexIOMeta / GlobalIndexMeta, allowing Lumina tag fields to be recorded in index metadata.
  • Build Lumina indexes with tag data when lumina.extension.build.tag.tag_schema is configured.
  • Support VectorSearch.predicate for Lumina tag filtering by converting Paimon predicates to Lumina TagFilter.
  • Support tag predicates including =, range comparisons, IN, and compound AND / OR.
  • Combine Lumina tag predicates with existing pre-filter logic.
  • Keep non-tag Lumina indexes on the existing search path.
  • Update third_party/lumina headers, liblumina.so, version metadata, and reference docs to Lumina v0.3.0.

Tests

Added and updated tests covering:

  • Lumina tag write/read with string, integer, double, list, null values.
  • Predicate and pre-filter orthogonality.
  • Nested compound predicates.
  • Unsupported / invalid tag predicates, including NOT_IN, unknown tag fields, and literal type mismatches.
  • Tag schema validation failures, including invalid range + string usage and Arrow field type incompatibility.
  • Compatible Arrow tag types: int8, int16, int32, int64, float, and double.
  • GetExtraFieldNames() behavior.
  • Integration coverage for Lumina global index metadata persistence and tag-filtered query.

API and Format

API:

  • Adds GlobalIndexer::GetExtraFieldNames() returning Result<std::optional<std::vector<std::string>>>.
  • Extends GlobalIndexIOMeta with optional extra_field_ids.

Format:

  • GlobalIndexMeta.extra_field_ids is now populated for Lumina tag fields.
  • Lumina tag schema is stored in the index metadata when tag filtering is enabled.
  • Bundled Lumina dependency is updated to v0.3.0.

Compatibility:

  • Indexers that do not need extra fields continue returning std::nullopt.
  • Non-tag Lumina indexes continue to use the existing build/search behavior.

Documentation

  • Updated bundled Lumina reference docs under third_party/lumina/reference.
  • Added Lumina filtering parameter documentation.
  • Updated Lumina release/version metadata for v0.3.0.

Generative AI tooling

Generated-by: Codex (GPT-5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant