feat(shared-shredding): Expose shared-shredding utilities#405
feat(shared-shredding): Expose shared-shredding utilities#405dalingmeng wants to merge 1 commit into
Conversation
a61fdd1 to
dac41bc
Compare
dac41bc to
ae43061
Compare
zjw1111
left a comment
There was a problem hiding this comment.
Follow-ups on the newly exposed schema utils. See inline comments for the type-guard and test-gap items.
One doc fix on the PR description:
- The new public header path is
include/paimon/data/shredding/map_shared_shredding_schema_utils.h, notinclude/paimon/shredding/.... - "New public API" omits
LogicalToPhysicalSchema, which is also exposed publicly.
Please update the description to match.
| PAIMON_ASSIGN_OR_RAISE_FROM_ARROW(std::shared_ptr<arrow::Schema> arrow_logical_schema, | ||
| arrow::ImportSchema(logical_schema.get())); | ||
| std::shared_ptr<arrow::Schema> physical_schema = | ||
| MapSharedShreddingUtils::LogicalToPhysicalSchema(arrow_logical_schema, |
There was a problem hiding this comment.
LogicalToPhysicalSchema is now a public API, but the underlying impl does static_pointer_cast<arrow::MapType>(field->type()) without checking the field is actually a MAP. Internal callers only pass known MAP columns, but an external caller passing a non-MAP field name in field_to_num_columns would hit UB instead of a clean error. Please add a type guard (e.g. field->type()->id() == arrow::Type::MAP) and return Status::Invalid otherwise.
| ASSERT_EQ(parsed_meta, tags_meta); | ||
| } | ||
|
|
||
| TEST(MapSharedShreddingSchemaUtilsTest, LogicalToPhysicalSchemaNestedListValue) { |
There was a problem hiding this comment.
Test gap: all cases pass valid MAP columns. Please add a case for LogicalToPhysicalSchema with a non-MAP field name in field_to_num_columns (covers the type-guard above), and for ExtractMetadataFromField where the field exists but has no shredding metadata.
|
The description uses |
Purpose
Linked issue: close #342
Expose shared-shredding schema metadata utilities for downstream integrations.
This change introduces
MapSharedShreddingSchemaUtilsas a public schema-level utility for shared-shredding MAP layout. It provides:__field_mapping,__overflow, and__col_i.AttachMetadataToSchemato attach serialized shared-shredding metadata to Arrow fields.ExtractMetadataFromFieldto parse shared-shredding metadata from an Arrow field.The internal
MapSharedShreddingDefinecontinues to own metadata keys, while public callers can reuse the schema contract and metadata read/write helpers without depending on internalsrc/headers.Tests
MapSharedShreddingSchemaUtilsTest.*API and Format
This change affects public API in the
includedir.New public header:
include/paimon/shredding/map_shared_shredding_schema_utils.hNew public API:
MapSharedShreddingSchemaUtils::AttachMetadataToSchemaMapSharedShreddingSchemaUtils::ExtractMetadataFromFieldThis change does not introduce a new storage format or protocol. It exposes existing shared-shredding schema/metadata conventions through public utilities.
Documentation
No.
Generative AI tooling
Generated-by: Qoder