Skip to content

Compress wide decimals when the writer permits the v2 format - #9759

Draft
mhk197 wants to merge 16 commits into
mk/compressor-serialized-idsfrom
mk/dbp-v2
Draft

Compress wide decimals when the writer permits the v2 format#9759
mhk197 wants to merge 16 commits into
mk/compressor-serialized-idsfrom
mk/dbp-v2

Conversation

@mhk197

@mhk197 mhk197 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Enable wide decimal compression through DecimalSchemeV2, with the existing DecimalScheme as its predecessor. The compressor selects the permitted version before matching, estimating, or splitting values. Default editions continue to use the frozen scheme; allowing both byte-parts IDs enables splitting and compression of every part.

The v2 scheme requires both IDs because narrow values still serialize under v1. Tests cover edition fallback, exclusions, nullable i128/i256 inputs, and file round trips. Writer tests verify that pre-encoded lower parts survive when v2 is allowed and become canonical decimals when targeting the frozen core edition.

Stack, in merge order: #9770#9808 (math helpers) → #9809 (array behavior) → #9810 (versioned serde) → #9759 (compression/writer integration). Each PR targets the branch immediately before it. This is layer 4 of 4.

Supersedes #9276.

Validation:

  • 451 tests passed (1 skipped) across vortex-compressor, vortex-btrblocks, vortex-decimal-byte-parts, vortex-file, and vortex-compat, with all features enabled.
  • Workspace cargo clippy --all-targets --all-features passed. The build emitted the existing missing-clang-format warning for a generated DuckDB header.
  • Doctests passed for the encoding, compressor, file, and compatibility crates.
  • cargo +nightly fmt --all and git diff --check passed.
  • The complete stack was validated after the readability refactor and downstream helper updates. CUDA runtime tests were not run.

@mhk197 mhk197 changed the title Wide decimals in DecimalByteParts, serialized as vortex.decimal_byte_parts_v2 behind preview2026.09.0 Wide decimals in DecimalByteParts Sep 3, 2026
@mhk197
mhk197 marked this pull request as draft September 3, 2026 18:40
@codspeed-hq

codspeed-hq Bot commented Sep 3, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 2.84%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 3 improved benchmarks
❌ 2 regressed benchmarks
✅ 2182 untouched benchmarks
🆕 45 new benchmarks
⏩ 224 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
WallTime arrow_checked_add_u32_neon[16384] 13.4 µs 20.3 µs -34.3%
Simulation decompress[u64, (4000, 1024)] 70.8 µs 86.1 µs -17.77%
WallTime arrow_checked_add_u32_avx2[16384] 21.3 µs 17.6 µs +20.79%
WallTime filtered_owned_i64_avx512[OneNullInEight] 26.3 µs 22.4 µs +17.47%
Simulation chunked_bool_canonical_into[(1000, 10)] 30.6 µs 27.1 µs +12.95%
🆕 Simulation assemble[(I128, 1024)] N/A 47.3 µs N/A
🆕 Simulation assemble[(I128, 65536)] N/A 2 ms N/A
🆕 Simulation assemble[(I128, 8192)] N/A 265.6 µs N/A
🆕 Simulation assemble[(I256, 1024)] N/A 109 µs N/A
🆕 Simulation assemble[(I256, 65536)] N/A 6 ms N/A
🆕 Simulation assemble[(I256, 8192)] N/A 764.5 µs N/A
🆕 Simulation assemble[(I64, 1024)] N/A 9.3 µs N/A
🆕 Simulation assemble[(I64, 65536)] N/A 7.4 µs N/A
🆕 Simulation assemble[(I64, 8192)] N/A 7.4 µs N/A
🆕 Simulation all_null[(I128, 1024)] N/A 62.7 µs N/A
🆕 Simulation all_null[(I128, 65536)] N/A 2.4 ms N/A
🆕 Simulation all_null[(I128, 8192)] N/A 328.9 µs N/A
🆕 Simulation all_null[(I256, 1024)] N/A 108.3 µs N/A
🆕 Simulation all_null[(I256, 65536)] N/A 4.8 ms N/A
🆕 Simulation all_null[(I256, 8192)] N/A 639.2 µs N/A
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing mk/dbp-v2 (52a1cb5) with develop (e3b8eb2)2

Open in CodSpeed

Footnotes

  1. 224 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. No successful run was found on mk/compressor-serialized-ids (b28f0b6) during the generation of this report, so develop (e3b8eb2) was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@mhk197 mhk197 changed the title Wide decimals in DecimalByteParts Support wide decimals in DecimalByteParts Sep 4, 2026
@mhk197 mhk197 changed the title Support wide decimals in DecimalByteParts Wide decimals in DecimalByteParts, serialized as vortex.decimal_byte_parts_v2 Sep 4, 2026
@mhk197 mhk197 changed the title Wide decimals in DecimalByteParts, serialized as vortex.decimal_byte_parts_v2 Support wide decimals in DecimalByteParts and guarantee compat with editions Sep 4, 2026
@mhk197
mhk197 force-pushed the mk/dbp-v2 branch 2 times, most recently from 35300e3 to c48b00b Compare September 8, 2026 15:10
@mhk197
mhk197 changed the base branch from develop to mk/compressor-serialized-ids September 8, 2026 15:17
@mhk197
mhk197 force-pushed the mk/dbp-v2 branch 3 times, most recently from 3d1237a to fc7f432 Compare September 8, 2026 20:35
@mhk197 mhk197 changed the title Support wide decimals in DecimalByteParts and guarantee compat with editions Compress wide decimals when the writer permits the v2 format Sep 8, 2026
@mhk197
mhk197 changed the base branch from mk/compressor-serialized-ids to mk/dbp-serde September 8, 2026 20:38
@mhk197
mhk197 force-pushed the mk/dbp-v2 branch 2 times, most recently from 416873c to 2c4016d Compare September 9, 2026 03:46
Introduce typed splitting and reassembly for i128 and i256 decimals, including
sign extension and boundary tests. Route existing single-part canonicalization
through the same assembly helper without changing its wire representation.

Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Resolve validity once for wide storage and populate only valid rows in
zero-initialized part buffers, so arbitrary null payloads do not inflate
lower-part compression. Preserve the all-valid loops and narrow zero-copy
path. Cover sliced, empty, nullable, and wider-than-precision storage.

Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Keep i256 words in most-significant-first order and assemble the two
128-bit halves directly. Dispatch on lower-part count, validate signed
MSPs and equal child lengths, and cover word order and sign extension.

Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Keep the splitting and assembly helpers independent of the array changes in the next PR.

Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
@mhk197
mhk197 removed this pull request from stack #9811 September 9, 2026 15:09
@mhk197
mhk197 changed the base branch from mk/dbp-serde to mk/compressor-serialized-ids September 9, 2026 15:09
@mhk197
mhk197 added this pull request to stack #9813 September 9, 2026 15:09
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Measure split_decimal and assemble_decimal directly across storage widths and input sizes, with fixtures outside the timed calls. Cover all-valid, all-null, random-null, and clustered-null split inputs.

Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Represent wide decimals with a signed high part and up to three unsigned
low parts. Add validation, execution, kernel support, property tests, and
assembly benchmarks while keeping serialization on the frozen format.

Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Move array helpers onto a crate-private extension trait, preserve decimal
precision and scale when replacing the MSP, and group slicing with the
other compute operations. Inline canonical execution and select scalar
storage directly from the lower-part count.

Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Use one ArrayPlugin for the frozen single-part format and the new wide
format. Preserve frozen files with wider physical storage and add wire
contract tests plus an opt-in compatibility fixture.

Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Move the plugin and serde coverage into plugin.rs while preserving metadata and frozen-format VTable serde. Share wide decimal test fixtures and exercise frozen compatibility through both registration paths.

Include the v2 compatibility fixture in the default suite without enabling unstable encodings.

Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
CascadingCompressor carries the snapshot of serialized IDs the writer may emit, filled by the file writer from the enabled editions through BtrBlocksCompressorBuilder::allow_serialized_ids. A scheme whose encoding has more than one wire format picks its compression mode from it with allows_serialized_id, the newest permitted one; without a restriction every ID is allowed. No scheme consults the set yet. This is the mechanism docs/specs/editions.md describes under compression with replacement encodings (#9779).

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Scheme::produced_encodings now names the serialized IDs a scheme may write its output under, oldest first. BtrBlocksCompressorBuilder::allow_serialized_ids replaces retain_allowed_encodings: it keeps a scheme when at least one of those IDs is permitted and hands the set to the compressor, so the writer makes one call from the serialized IDs its editions permit instead of mapping them back to in-memory encodings, which could not tell two wire formats of one encoding apart.
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
The compressor seeds each root CompressorContext with its permitted serialized IDs and every descent inherits them, so a scheme asks compress_ctx.allows_serialized_id both while estimating and while compressing and picks the same mode in both. The per-compressor accessor goes; allowed_serialized_ids remains for inspection.

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
…ed ids

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Restore produced_encodings in feature-gated schemes and fix stale trait links. Document scheme ID resolution for Clippy.

Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Register DecimalSchemeV2 with the frozen scheme as its predecessor, so
format permissions select a scheme before decimal splitting. Cover
compression fallback and writer round trips for both format versions.

Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant