Add ArrayPlugin to enable serde for new DecimalBytePartsArray format - #9810
Add ArrayPlugin to enable serde for new DecimalBytePartsArray format#9810mhk197 wants to merge 4 commits into
ArrayPlugin to enable serde for new DecimalBytePartsArray format#9810Conversation
Merging this PR will improve performance by 11.7%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | Simulation | allocate_drop_arrow[0] |
456.9 ns | 402.7 ns | +13.45% |
| ⚡ | WallTime | mul_u32_nonnull_avx512 |
6.3 µs | 5.6 µs | +11.29% |
| ⚡ | Simulation | allocate_drop_bytes[0] |
575.7 ns | 521.6 ns | +10.39% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing mk/dbp-serde (03a13f8) with mk/dbp-array (7283d4f)
Footnotes
-
224 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
b822fb6 to
e13f457
Compare
a68b2ae to
b970a8f
Compare
ArrayPlugin to enable serde for new DecimalBytePartsArray format
Use one ArrayPlugin for the frozen single-part format and the new wide format. Preserve frozen files with wider physical storage and add wire contract tests plus an opt-in compatibility fixture. Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Move the plugin and serde coverage into plugin.rs while preserving metadata and frozen-format VTable serde. Share wide decimal test fixtures and exercise frozen compatibility through both registration paths. Include the v2 compatibility fixture in the default suite without enabling unstable encodings. Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Add serialization for
DecimalBytePartsArraywith lower parts while preserving the frozen legacy format (no lower parts). The in-memoryDecimalBytePartsArraycan now use either wire format, selected from its child layout:vortex.decimal_byte_partsvortex.decimal_byte_parts_v2Both formats deserialize into the same array representation. The metadata records the MSP's physical integer type and the lower-part count. Decoding preserves valid physical storage that is wider than the minimum required by the decimal's precision.
DecimalBytePartsPluginregisters both serialized IDs and selects the appropriate one when writing. On read, it enforces each ID's contract: the frozen format must have no lower parts, and v2 must have at least one.Both vtable serde and plugin serde remain supported:
DecimalBytePartscontinues to read and write the frozen format. It rejects lower parts, so existing callers cannot accidentally write a wider layout under the old ID. Keeping this path avoids breaking callers that use direct vtable registration or its serde methods.DecimalBytePartsPluginsupports both formats. The plugin can return a distinct serialized ID, which the vtable's metadata-only serializer cannot do.vortex_decimal_byte_parts::initializeregisters this plugin.Add a separate v2 compatibility fixture covering positive and negative wide
i128values and nullablei256values. The existing frozen fixture is unchanged.