[ntuple] Support multiple column representations in the merger by silverweed · Pull Request #22017 · root-project/root

silverweed · 2026-04-22T15:07:43Z

This Pull request:

Significantly reworks the innards of the RNTupleMerger to support fast merging of fields with different but compatible column representations.
Basically it does two things:

turns all L3 merging cases into L2/L1.
no longer rejects merging fields with different column representations (previously this was only supported for representations that were the split/unsplit version of each other, and only via L3 merging).

A potentially negative consequence that we might want to revisit is that now the merger won't ever adapt the columns' splitness to the output compression (e.g. if merging changes the source compression from 0 to 505 it will still encode the columns as unsplit, and vice-versa). This will probably be readded in a future PR.

In order to achieve this, some new internal functionality had to be added, most notably RPagePersistentSink::AddColumnRepresentation.

Note that this PR is independent on #21740, which in fact might not be needed at all.

IMPORTANT

This PR introduces our first feature flag and thus the first bump to the specs' major version (1.1.0.0). This means we can now start producing RNTuples which cannot be read by older ROOT versions.

TODO

check if we need a feature flag for the changes in AddExtendedColumnRanges
add a test for merging of Real32Trunc/Quant columns with different bit width/value range
properly split the big merger commit
update Merging.md

Checklist:

tested changes locally
updated the docs (if necessary)

github-actions · 2026-04-22T17:16:23Z

Test Results

23 files 23 suites 3d 20h 30m 41s ⏱️
3 871 tests 3 871 ✅ 0 💤 0 ❌
78 841 runs 78 841 ✅ 0 💤 0 ❌

Results for commit 753d8de.

♻️ This comment has been updated with latest results.

jblomer

LGTM. (But see one tiny formatting issue.)

Perhaps add one test merging std::string with different index column types (to test multi-column fields).

Instead of calling continue multiple times in the AddColumnFromField loop, just early return in case of projected fields.

We are currently serializing columns per-field, but in case of late column extension this might result in inconsistent sorting of the columns in the serialized footer. e.g. assume you have fields "A" and "B", both late model extended, both with a single column: - col 0 -> field A, repr 0 - col 1 -> field B, repr 0 Now you add a new column representation to field "A"; this new column has id 2: - col 2 -> field A, repr 1 When serializing this RNTuple, all columns are written in the footer by RNTupleSerialize::SerializeColumnsForFields(). Before this change, they would end up on disk in order: [0, 2, 1]. This would corrupt the data by swapping the pages for columns 2 and 1. After this change, they get written as [0, 1, 2] which is the correct order. Note that this exact case is tested in ntuple_merger in the unit test MergeDeferredAdvanced.

silverweed · 2026-07-22T15:40:20Z

Last things missing (hopefully):

~~improve AddColumnRepresentation test to actually write data (multiple clusters)~~ -> not easily doable without introducing new bespoke APIs and not worth it for just the test for now.
~~properly test FirstElementIndex in AddColumnRepresentation~~ -> will come in a future PR

- RPagePersistentSink::AddColumnRepresentation - RPagePersistentSink::AddAliasColumn Internal functionality to be used by the Merger. This entails 2 additional changes: - AddExtendedColumnRanges needs to be updated to handle the case where a column representation is added to a field during writing after some clusters have already been written; - ShiftAliasColumns needs to properly shift the ids of extended alias columns when called, otherwise a mismatch may happen when serializing the footer

Two field descriptors might legitimately have different numbers of elements in their LogicalColumnIds vectors due to having a different number of representations. The proper check to do is the one on the column cardinality, which makes sure that the same number of columns is actually used at a time.

This test checks two things: 1. that the fix applied by 14ade04 works properly (this is already checked by MergeDeferredAdvanced but that might not be the case anymore if we update the merger to change the column encodings depending on the output compression - in which case that test would not be adding a late extended column anymore); 2. that the merger properly handles fields with the same cardinality but different number of column representations

…ation Also set the anchor version in InitFromDescriptor().

clarify that it is only valid for the 0th representation index

silverweed requested a review from jblomer as a code owner April 22, 2026 15:07

silverweed marked this pull request as draft April 22, 2026 15:07

silverweed changed the title ~~Ntuple merge colrep2~~ [ntuple] Support multiple column representations in the merger Apr 22, 2026

silverweed added the in:RNTuple label Apr 22, 2026

silverweed self-assigned this Apr 22, 2026

silverweed force-pushed the ntuple_merge_colrep2 branch 3 times, most recently from b2ae5fc to 1db6b5e Compare April 22, 2026 15:22

silverweed force-pushed the ntuple_merge_colrep2 branch 9 times, most recently from 82e5299 to d3efcfc Compare April 30, 2026 09:45

silverweed force-pushed the ntuple_merge_colrep2 branch 2 times, most recently from 857d425 to 60b1bde Compare May 4, 2026 13:36

silverweed mentioned this pull request May 4, 2026

[ntuple] Some changes/fixes to better handle feature flags #22135

Merged

2 tasks

silverweed force-pushed the ntuple_merge_colrep2 branch 3 times, most recently from 1d462ee to a249301 Compare May 5, 2026 07:42

silverweed force-pushed the ntuple_merge_colrep2 branch from a249301 to ab08930 Compare May 13, 2026 15:05

silverweed marked this pull request as ready for review May 15, 2026 06:48

silverweed requested review from enirolf, hahnjo, pcanal and vepadulano May 15, 2026 06:48

silverweed force-pushed the ntuple_merge_colrep2 branch 3 times, most recently from 30a4004 to bcf53b2 Compare July 1, 2026 13:20

silverweed added the clean build Ask CI to do non-incremental build on PR label Jul 2, 2026

silverweed force-pushed the ntuple_merge_colrep2 branch 3 times, most recently from 2c49e98 to df617cd Compare July 7, 2026 08:21

silverweed force-pushed the ntuple_merge_colrep2 branch 2 times, most recently from 89db1e7 to 7531fbf Compare July 9, 2026 07:04

jblomer approved these changes Jul 20, 2026

View reviewed changes

Comment thread tree/ntuple/test/ntuple_storage.cxx Outdated

Comment thread tree/ntuple/src/RPageStorage.cxx

Comment thread tree/ntuple/src/RPageStorage.cxx Outdated

Comment thread tree/ntuple/inc/ROOT/RPageStorage.hxx

Comment thread tree/ntuple/src/RPageStorage.cxx Outdated

silverweed added 2 commits July 22, 2026 16:14

[ntuple] Small improvement in RNTupleMerger

7242cb3

Instead of calling continue multiple times in the AddColumnFromField loop, just early return in case of projected fields.

silverweed force-pushed the ntuple_merge_colrep2 branch from 7531fbf to 4499bd5 Compare July 22, 2026 15:38

silverweed added 13 commits July 24, 2026 09:58

[ntuple] Add multiple column representation support in the Merger

83f8e74

[ntuple] Merger: extract column repr matching into a separate function

c574eb0

[ntuple] support merging column with metadata (and different col types)

e9b5984

[ntuple] Support merging columns with same type and different metadata

242c721

[ntuple][NFC] update Merging.md

00af5fe

[ntuple] ensure valid descriptor in AddAliasColumn/AddColumnRepresent…

85e3259

…ation Also set the anchor version in InitFromDescriptor().

[ntuple] Derive nCommonColumnsInCluster instead of passing it

4d3fa21

[ntuple] Add clusterOffset to AddColumnRepresentation

cfbc7f9

[ntuple] update comment in AddExtendedColumnRanges

6d78e75

clarify that it is only valid for the 0th representation index

[ntuple] Add more tests

753d8de

silverweed force-pushed the ntuple_merge_colrep2 branch from 4499bd5 to 753d8de Compare July 24, 2026 07:59

silverweed merged commit 1989411 into root-project:master Jul 27, 2026
35 checks passed

silverweed deleted the ntuple_merge_colrep2 branch July 27, 2026 06:39

Navigation Menu

Uh oh!

[ntuple] Support multiple column representations in the merger - #22017

[ntuple] Support multiple column representations in the merger#22017
silverweed merged 15 commits into
root-project:masterfrom
silverweed:ntuple_merge_colrep2

silverweed commented Apr 22, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Apr 22, 2026 •

edited

Loading

Uh oh!

jblomer left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

silverweed commented Jul 22, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

silverweed commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

This Pull request:

IMPORTANT

TODO

Checklist:

Uh oh!

github-actions Bot commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

jblomer left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

silverweed commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

silverweed commented Apr 22, 2026 •

edited

Loading

github-actions Bot commented Apr 22, 2026 •

edited

Loading

jblomer left a comment •

edited

Loading

silverweed commented Jul 22, 2026 •

edited

Loading