Compress wide decimals when the writer permits the v2 format - #9759
Compress wide decimals when the writer permits the v2 format#9759mhk197 wants to merge 16 commits into
Conversation
DecimalByteParts
Merging this PR will degrade performance by 2.84%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | arrow_checked_add_u32_neon[16384] |
13.4 µs | 20.3 µs | -34.3% |
| ❌ | Simulation | decompress[u64, (4000, 1024)] |
70.8 µs | 86.1 µs | -17.77% |
| ⚡ | WallTime | arrow_checked_add_u32_avx2[16384] |
21.3 µs | 17.6 µs | +20.79% |
| ⚡ | WallTime | filtered_owned_i64_avx512[OneNullInEight] |
26.3 µs | 22.4 µs | +17.47% |
| ⚡ | Simulation | chunked_bool_canonical_into[(1000, 10)] |
30.6 µs | 27.1 µs | +12.95% |
| 🆕 | Simulation | assemble[(I128, 1024)] |
N/A | 47.3 µs | N/A |
| 🆕 | Simulation | assemble[(I128, 65536)] |
N/A | 2 ms | N/A |
| 🆕 | Simulation | assemble[(I128, 8192)] |
N/A | 265.6 µs | N/A |
| 🆕 | Simulation | assemble[(I256, 1024)] |
N/A | 109 µs | N/A |
| 🆕 | Simulation | assemble[(I256, 65536)] |
N/A | 6 ms | N/A |
| 🆕 | Simulation | assemble[(I256, 8192)] |
N/A | 764.5 µs | N/A |
| 🆕 | Simulation | assemble[(I64, 1024)] |
N/A | 9.3 µs | N/A |
| 🆕 | Simulation | assemble[(I64, 65536)] |
N/A | 7.4 µs | N/A |
| 🆕 | Simulation | assemble[(I64, 8192)] |
N/A | 7.4 µs | N/A |
| 🆕 | Simulation | all_null[(I128, 1024)] |
N/A | 62.7 µs | N/A |
| 🆕 | Simulation | all_null[(I128, 65536)] |
N/A | 2.4 ms | N/A |
| 🆕 | Simulation | all_null[(I128, 8192)] |
N/A | 328.9 µs | N/A |
| 🆕 | Simulation | all_null[(I256, 1024)] |
N/A | 108.3 µs | N/A |
| 🆕 | Simulation | all_null[(I256, 65536)] |
N/A | 4.8 ms | N/A |
| 🆕 | Simulation | all_null[(I256, 8192)] |
N/A | 639.2 µs | N/A |
| ... | ... | ... | ... | ... | ... |
ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing mk/dbp-v2 (52a1cb5) with develop (e3b8eb2)2
Footnotes
-
224 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
mk/compressor-serialized-ids(b28f0b6) during the generation of this report, sodevelop(e3b8eb2) was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
DecimalBytePartsDecimalByteParts
DecimalBytePartsDecimalByteParts and guarantee compat with editions
35300e3 to
c48b00b
Compare
3d1237a to
fc7f432
Compare
DecimalByteParts and guarantee compat with editions416873c to
2c4016d
Compare
Introduce typed splitting and reassembly for i128 and i256 decimals, including sign extension and boundary tests. Route existing single-part canonicalization through the same assembly helper without changing its wire representation. Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Resolve validity once for wide storage and populate only valid rows in zero-initialized part buffers, so arbitrary null payloads do not inflate lower-part compression. Preserve the all-valid loops and narrow zero-copy path. Cover sliced, empty, nullable, and wider-than-precision storage. Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Keep i256 words in most-significant-first order and assemble the two 128-bit halves directly. Dispatch on lower-part count, validate signed MSPs and equal child lengths, and cover word order and sign extension. Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Keep the splitting and assembly helpers independent of the array changes in the next PR. Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Measure split_decimal and assemble_decimal directly across storage widths and input sizes, with fixtures outside the timed calls. Cover all-valid, all-null, random-null, and clustered-null split inputs. Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Represent wide decimals with a signed high part and up to three unsigned low parts. Add validation, execution, kernel support, property tests, and assembly benchmarks while keeping serialization on the frozen format. Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Move array helpers onto a crate-private extension trait, preserve decimal precision and scale when replacing the MSP, and group slicing with the other compute operations. Inline canonical execution and select scalar storage directly from the lower-part count. Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Use one ArrayPlugin for the frozen single-part format and the new wide format. Preserve frozen files with wider physical storage and add wire contract tests plus an opt-in compatibility fixture. Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Move the plugin and serde coverage into plugin.rs while preserving metadata and frozen-format VTable serde. Share wide decimal test fixtures and exercise frozen compatibility through both registration paths. Include the v2 compatibility fixture in the default suite without enabling unstable encodings. Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
CascadingCompressor carries the snapshot of serialized IDs the writer may emit, filled by the file writer from the enabled editions through BtrBlocksCompressorBuilder::allow_serialized_ids. A scheme whose encoding has more than one wire format picks its compression mode from it with allows_serialized_id, the newest permitted one; without a restriction every ID is allowed. No scheme consults the set yet. This is the mechanism docs/specs/editions.md describes under compression with replacement encodings (#9779). Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Scheme::produced_encodings now names the serialized IDs a scheme may write its output under, oldest first. BtrBlocksCompressorBuilder::allow_serialized_ids replaces retain_allowed_encodings: it keeps a scheme when at least one of those IDs is permitted and hands the set to the compressor, so the writer makes one call from the serialized IDs its editions permit instead of mapping them back to in-memory encodings, which could not tell two wire formats of one encoding apart. Signed-off-by: Matt Katz <mhkatz97@gmail.com>
The compressor seeds each root CompressorContext with its permitted serialized IDs and every descent inherits them, so a scheme asks compress_ctx.allows_serialized_id both while estimating and while compressing and picks the same mode in both. The per-compressor accessor goes; allowed_serialized_ids remains for inspection. Signed-off-by: Matt Katz <mhkatz97@gmail.com>
…ed ids Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Restore produced_encodings in feature-gated schemes and fix stale trait links. Document scheme ID resolution for Clippy. Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Register DecimalSchemeV2 with the frozen scheme as its predecessor, so format permissions select a scheme before decimal splitting. Cover compression fallback and writer round trips for both format versions. Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Enable wide decimal compression through
DecimalSchemeV2, with the existingDecimalSchemeas its predecessor. The compressor selects the permitted version before matching, estimating, or splitting values. Default editions continue to use the frozen scheme; allowing both byte-parts IDs enables splitting and compression of every part.The v2 scheme requires both IDs because narrow values still serialize under v1. Tests cover edition fallback, exclusions, nullable i128/i256 inputs, and file round trips. Writer tests verify that pre-encoded lower parts survive when v2 is allowed and become canonical decimals when targeting the frozen core edition.
Stack, in merge order: #9770 → #9808 (math helpers) → #9809 (array behavior) → #9810 (versioned serde) → #9759 (compression/writer integration). Each PR targets the branch immediately before it. This is layer 4 of 4.
Supersedes #9276.
Validation:
vortex-compressor,vortex-btrblocks,vortex-decimal-byte-parts,vortex-file, andvortex-compat, with all features enabled.cargo clippy --all-targets --all-featurespassed. The build emitted the existing missing-clang-formatwarning for a generated DuckDB header.cargo +nightly fmt --allandgit diff --checkpassed.