Skip to content

Add ArrayPlugin to enable serde for new DecimalBytePartsArray format - #9810

Open
mhk197 wants to merge 4 commits into
mk/dbp-arrayfrom
mk/dbp-serde
Open

Add ArrayPlugin to enable serde for new DecimalBytePartsArray format#9810
mhk197 wants to merge 4 commits into
mk/dbp-arrayfrom
mk/dbp-serde

Conversation

@mhk197

@mhk197 mhk197 commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Add serialization for DecimalBytePartsArray with lower parts while preserving the frozen legacy format (no lower parts). The in-memory DecimalBytePartsArray can now use either wire format, selected from its child layout:

Array layout Serialized ID
Signed most significant part (MSP) only vortex.decimal_byte_parts
Signed MSP plus one to three unsigned 64-bit lower parts vortex.decimal_byte_parts_v2

Both formats deserialize into the same array representation. The metadata records the MSP's physical integer type and the lower-part count. Decoding preserves valid physical storage that is wider than the minimum required by the decimal's precision.

DecimalBytePartsPlugin registers both serialized IDs and selects the appropriate one when writing. On read, it enforces each ID's contract: the frozen format must have no lower parts, and v2 must have at least one.

Both vtable serde and plugin serde remain supported:

  • Vtable serde preserves the existing API. Directly registering DecimalByteParts continues to read and write the frozen format. It rejects lower parts, so existing callers cannot accidentally write a wider layout under the old ID. Keeping this path avoids breaking callers that use direct vtable registration or its serde methods.
  • Plugin serde adds format selection. Registering DecimalBytePartsPlugin supports both formats. The plugin can return a distinct serialized ID, which the vtable's metadata-only serializer cannot do. vortex_decimal_byte_parts::initialize registers this plugin.

Add a separate v2 compatibility fixture covering positive and negative wide i128 values and nullable i256 values. The existing frozen fixture is unchanged.

@codspeed-hq

codspeed-hq Bot commented Sep 8, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 11.7%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 3 improved benchmarks
✅ 2202 untouched benchmarks
⏩ 224 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation allocate_drop_arrow[0] 456.9 ns 402.7 ns +13.45%
WallTime mul_u32_nonnull_avx512 6.3 µs 5.6 µs +11.29%
Simulation allocate_drop_bytes[0] 575.7 ns 521.6 ns +10.39%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing mk/dbp-serde (03a13f8) with mk/dbp-array (7283d4f)

Open in CodSpeed

Footnotes

  1. 224 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@mhk197
mhk197 force-pushed the mk/dbp-serde branch 2 times, most recently from b822fb6 to e13f457 Compare September 9, 2026 02:43
@mhk197
mhk197 force-pushed the mk/dbp-serde branch 2 times, most recently from a68b2ae to b970a8f Compare September 9, 2026 04:15
@mhk197
mhk197 marked this pull request as ready for review September 9, 2026 04:49
@mhk197 mhk197 changed the title Register versioned decimal byte-part serialization Add ArrayPlugin to enable serde for new DecimalBytePartsArray format Sep 9, 2026
@mhk197
mhk197 removed this pull request from stack #9811 September 9, 2026 15:09
@mhk197
mhk197 added this pull request to stack #9813 September 9, 2026 15:09
Use one ArrayPlugin for the frozen single-part format and the new wide
format. Preserve frozen files with wider physical storage and add wire
contract tests plus an opt-in compatibility fixture.

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Move the plugin and serde coverage into plugin.rs while preserving metadata and frozen-format VTable serde. Share wide decimal test fixtures and exercise frozen compatibility through both registration paths.

Include the v2 compatibility fixture in the default suite without enabling unstable encodings.

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
@mhk197 mhk197 added the changelog/skip Do not list PR in the changelog label Sep 9, 2026
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/skip Do not list PR in the changelog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant