Skip to content

feat(shared-shredding): Expose shared-shredding utilities#405

Open
dalingmeng wants to merge 1 commit into
alibaba:mainfrom
dalingmeng:feature/shared-shredding-schema-utils
Open

feat(shared-shredding): Expose shared-shredding utilities#405
dalingmeng wants to merge 1 commit into
alibaba:mainfrom
dalingmeng:feature/shared-shredding-schema-utils

Conversation

@dalingmeng

@dalingmeng dalingmeng commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #342

Expose shared-shredding schema metadata utilities for downstream integrations.

This change introduces MapSharedShreddingSchemaUtils as a public schema-level utility for shared-shredding MAP layout. It provides:

  • Physical sub-column naming contract, including __field_mapping, __overflow, and __col_i.
  • AttachMetadataToSchema to attach serialized shared-shredding metadata to Arrow fields.
  • ExtractMetadataFromField to parse shared-shredding metadata from an Arrow field.

The internal MapSharedShreddingDefine continues to own metadata keys, while public callers can reuse the schema contract and metadata read/write helpers without depending on internal src/ headers.

Tests

  • MapSharedShreddingSchemaUtilsTest.*

API and Format

This change affects public API in the include dir.

New public header:

  • include/paimon/shredding/map_shared_shredding_schema_utils.h

New public API:

  • MapSharedShreddingSchemaUtils::AttachMetadataToSchema
  • MapSharedShreddingSchemaUtils::ExtractMetadataFromField

This change does not introduce a new storage format or protocol. It exposes existing shared-shredding schema/metadata conventions through public utilities.

Documentation

No.

Generative AI tooling

Generated-by: Qoder

@dalingmeng dalingmeng changed the title Expose shared-shredding metadata utilities feat(shared-shredding)Expose shared-shredding metadata utilities Jul 9, 2026
@dalingmeng dalingmeng changed the title feat(shared-shredding)Expose shared-shredding metadata utilities feat(shared-shredding)Expose shared-shredding utilities Jul 9, 2026
@lxy-9602 lxy-9602 changed the title feat(shared-shredding)Expose shared-shredding utilities feat(shared-shredding): Expose shared-shredding utilities Jul 9, 2026
@dalingmeng dalingmeng force-pushed the feature/shared-shredding-schema-utils branch 3 times, most recently from a61fdd1 to dac41bc Compare July 9, 2026 10:29
@dalingmeng dalingmeng force-pushed the feature/shared-shredding-schema-utils branch from dac41bc to ae43061 Compare July 9, 2026 10:32

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

zjw1111

This comment was marked as duplicate.

@zjw1111 zjw1111 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-ups on the newly exposed schema utils. See inline comments for the type-guard and test-gap items.

One doc fix on the PR description:

  • The new public header path is include/paimon/data/shredding/map_shared_shredding_schema_utils.h, not include/paimon/shredding/....
  • "New public API" omits LogicalToPhysicalSchema, which is also exposed publicly.

Please update the description to match.

PAIMON_ASSIGN_OR_RAISE_FROM_ARROW(std::shared_ptr<arrow::Schema> arrow_logical_schema,
arrow::ImportSchema(logical_schema.get()));
std::shared_ptr<arrow::Schema> physical_schema =
MapSharedShreddingUtils::LogicalToPhysicalSchema(arrow_logical_schema,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LogicalToPhysicalSchema is now a public API, but the underlying impl does static_pointer_cast<arrow::MapType>(field->type()) without checking the field is actually a MAP. Internal callers only pass known MAP columns, but an external caller passing a non-MAP field name in field_to_num_columns would hit UB instead of a clean error. Please add a type guard (e.g. field->type()->id() == arrow::Type::MAP) and return Status::Invalid otherwise.

ASSERT_EQ(parsed_meta, tags_meta);
}

TEST(MapSharedShreddingSchemaUtilsTest, LogicalToPhysicalSchemaNestedListValue) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test gap: all cases pass valid MAP columns. Please add a case for LogicalToPhysicalSchema with a non-MAP field name in field_to_num_columns (covers the type-guard above), and for ExtractMetadataFromField where the field exists but has no shredding metadata.

@zjw1111

zjw1111 commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

The description uses close #342, which auto-closes the issue on merge. This PR only exposes the utilities and shouldn't close #342 — please change it to a non-closing reference (e.g. Related to #342).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Support columnar-extend storage layout for MAP columns

3 participants