Support wide decimals in DecimalBytePartsArray and kernels - #9809
Conversation
Merging this PR will regress 1 benchmark
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | random_i8[0.5] |
67.1 µs | 91.1 µs | -26.31% |
| ⚡ | Simulation | random_i8[0.8] |
99 µs | 69.5 µs | +42.37% |
| 🆕 | Simulation | assemble[i128] |
N/A | 2 ms | N/A |
| 🆕 | Simulation | assemble[i256] |
N/A | 6 ms | N/A |
| 🆕 | Simulation | assemble[i64] |
N/A | 31.1 µs | N/A |
| 🆕 | Simulation | all_valid[i128] |
N/A | 3.3 ms | N/A |
| 🆕 | Simulation | all_valid[i256] |
N/A | 6.6 ms | N/A |
| 🆕 | Simulation | all_valid[i64] |
N/A | 7.7 µs | N/A |
| 🆕 | Simulation | clustered_nulls[i128] |
N/A | 1.7 ms | N/A |
| 🆕 | Simulation | clustered_nulls[i256] |
N/A | 4.4 ms | N/A |
| 🆕 | Simulation | clustered_nulls[i64] |
N/A | 10.2 µs | N/A |
| 🆕 | Simulation | mixed_nulls[i128] |
N/A | 4.2 ms | N/A |
| 🆕 | Simulation | mixed_nulls[i256] |
N/A | 6.2 ms | N/A |
| 🆕 | Simulation | mixed_nulls[i64] |
N/A | 8.3 µs | N/A |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing mk/dbp-array (32a80b1) with mk/dbp-parts (d93444d)
Footnotes
-
194 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
f7dfebe to
a566a28
Compare
DecimalBytePartsArray and kernels
Represent wide decimals with a signed high part and up to three unsigned low parts. Add validation, execution, kernel support, property tests, and assembly benchmarks while keeping serialization on the frozen format. Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Move array helpers onto a crate-private extension trait, preserve decimal precision and scale when replacing the MSP, and group slicing with the other compute operations. Inline canonical execution and select scalar storage directly from the lower-part count. Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Cover i64, i128, and i256 assembly through DecimalArray execution. Add split benchmarks for all-valid, mixed-null, and clustered-null inputs, with shared fixtures and setup outside the timed region. Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
DecimalBytePartsArraypreviously stored the entire unscaled decimal value in one signed integer child, limiting it to values that fit in 64 bits. It now supports wide decimals by representing each value as integer parts that can be compressed independently, while preserving the decimal's logical precision, scale, and nullability.The array has a signed most significant part (MSP) and up to three unsigned 64-bit lower parts, ordered most significant first. Splitting canonical decimal storage produces:
i8/i16/i32/i64i128i64MSP + oneu64lower parti256i64MSP + threeu64lower partsOnly the MSP carries validity. Every lower part must be a non-nullable
u64array with the same length, and splitting wide decimals zeroes the parts at null positions. All children remainArrayRefs, so their individual encodings are independent of the decimal representation.execute::<DecimalArray>executes the children and reassembles their bit patterns, sign-extending the MSP. MSP-only arrays reuse their value buffer; wide layouts producei128ori256buffers. Scalar access reconstructs the same value from the corresponding parts. Physical storage can be wider than the minimum implied by decimal precision, so the part layout determines reconstruction while the logical dtype retains its precision and scale.The constructors validate the part types, lengths, count, and nullability. Row operations keep all parts aligned, while changes to validity only need to update the MSP.
takewith nullable indices is not yet supported by the DecimalByteParts kernel for arrays with lower parts; it falls back to canonical execution. Taking each part directly would make the lower parts nullable, violating the representation's invariant. The frozen serializer also continues to reject arrays with lower parts.