Skip to content

Support wide decimals in DecimalBytePartsArray and kernels - #9809

Open
mhk197 wants to merge 3 commits into
mk/dbp-partsfrom
mk/dbp-array
Open

Support wide decimals in DecimalBytePartsArray and kernels#9809
mhk197 wants to merge 3 commits into
mk/dbp-partsfrom
mk/dbp-array

Conversation

@mhk197

@mhk197 mhk197 commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

DecimalBytePartsArray previously stored the entire unscaled decimal value in one signed integer child, limiting it to values that fit in 64 bits. It now supports wide decimals by representing each value as integer parts that can be compressed independently, while preserving the decimal's logical precision, scale, and nullability.

The array has a signed most significant part (MSP) and up to three unsigned 64-bit lower parts, ordered most significant first. Splitting canonical decimal storage produces:

Decimal storage Children
i8 / i16 / i32 / i64 Signed MSP only; shares the original value buffer
i128 i64 MSP + one u64 lower part
i256 i64 MSP + three u64 lower parts

Only the MSP carries validity. Every lower part must be a non-nullable u64 array with the same length, and splitting wide decimals zeroes the parts at null positions. All children remain ArrayRefs, so their individual encodings are independent of the decimal representation.

execute::<DecimalArray> executes the children and reassembles their bit patterns, sign-extending the MSP. MSP-only arrays reuse their value buffer; wide layouts produce i128 or i256 buffers. Scalar access reconstructs the same value from the corresponding parts. Physical storage can be wider than the minimum implied by decimal precision, so the part layout determines reconstruction while the logical dtype retains its precision and scale.

The constructors validate the part types, lengths, count, and nullability. Row operations keep all parts aligned, while changes to validity only need to update the MSP.

take with nullable indices is not yet supported by the DecimalByteParts kernel for arrays with lower parts; it falls back to canonical execution. Taking each part directly would make the lower parts nullable, violating the representation's invariant. The frozen serializer also continues to reject arrays with lower parts.

@codspeed-hq

codspeed-hq Bot commented Sep 8, 2026

Copy link
Copy Markdown

Merging this PR will regress 1 benchmark

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
❌ 1 regressed benchmark
✅ 2209 untouched benchmarks
🆕 12 new benchmarks
⏩ 194 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation random_i8[0.5] 67.1 µs 91.1 µs -26.31%
Simulation random_i8[0.8] 99 µs 69.5 µs +42.37%
🆕 Simulation assemble[i128] N/A 2 ms N/A
🆕 Simulation assemble[i256] N/A 6 ms N/A
🆕 Simulation assemble[i64] N/A 31.1 µs N/A
🆕 Simulation all_valid[i128] N/A 3.3 ms N/A
🆕 Simulation all_valid[i256] N/A 6.6 ms N/A
🆕 Simulation all_valid[i64] N/A 7.7 µs N/A
🆕 Simulation clustered_nulls[i128] N/A 1.7 ms N/A
🆕 Simulation clustered_nulls[i256] N/A 4.4 ms N/A
🆕 Simulation clustered_nulls[i64] N/A 10.2 µs N/A
🆕 Simulation mixed_nulls[i128] N/A 4.2 ms N/A
🆕 Simulation mixed_nulls[i256] N/A 6.2 ms N/A
🆕 Simulation mixed_nulls[i64] N/A 8.3 µs N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing mk/dbp-array (32a80b1) with mk/dbp-parts (d93444d)

Open in CodSpeed

Footnotes

  1. 194 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@mhk197
mhk197 force-pushed the mk/dbp-array branch 3 times, most recently from f7dfebe to a566a28 Compare September 9, 2026 02:43
@mhk197 mhk197 changed the title Support multi-part decimal arrays and kernels Support wide decimals in DecimalBytePartsArray and kernels Sep 9, 2026
@mhk197
mhk197 marked this pull request as ready for review September 9, 2026 04:17
Represent wide decimals with a signed high part and up to three unsigned
low parts. Add validation, execution, kernel support, property tests, and
assembly benchmarks while keeping serialization on the frozen format.

Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Move array helpers onto a crate-private extension trait, preserve decimal
precision and scale when replacing the MSP, and group slicing with the
other compute operations. Inline canonical execution and select scalar
storage directly from the lower-part count.

Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Cover i64, i128, and i256 assembly through DecimalArray execution. Add split benchmarks for all-valid, mixed-null, and clustered-null inputs, with shared fixtures and setup outside the timed region.

Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
@mhk197
mhk197 removed this pull request from stack #9811 September 9, 2026 15:09
@mhk197
mhk197 added this pull request to stack #9813 September 9, 2026 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant