RDataFrame in MultiThread with RNTuples

[fraorlnd]

Check duplicate issues.

• Checked for duplicates

Description

Dear ROOT developers,

We have an analysis framework for CMS data that uses ROOT RDataFrame in MultiThread. In view of the expected adoption of RNTuples, we tried to use an RDataFrame script to convert CMS data stored in ROOT files as flat ntuples (NanoAOD, TTree structure) to RNTuples, which we can then read with our usual framework.

However, when running on the RNTuples the program opens a number of files equal to the number of available threads, waits for all of these files to be processed, and only then opens the next batch. As a result, the execution is limited by the slowest file-thread combination of the batch.

We have prepared a reproducer [linked below with instructions and comments] that generates a set of files as RNTuples, processes the events, and reproduces the issue. We have observed the same behavior on lxplus, on our institute's cluster and our laptop.

On lxplus and our institute cluster we also get the warning

Warning in <RIoUring>: io_uring is unexpectedly not available because:
Error initializing io_uring: Operation not permitted
Warning in <RRawFileUnix>: io_uring setup failed, falling back to blocking I/O in ReadV

Thank you in advance for your help!

Francesco Orlandi and Federico Ferri (@ferriff)

Reproducer

On lxplus:

1. copy reproduce_problem.cc and j.json in a folder
2. compile: clang++ reproduce_problem.cc $(root-config --cflags --libs) -o reproduce_problem
3. execute: ./reproduce_problem -g -n 3 -j j.json

Options:

• g (activate generation of files, necessary only the first time)
• n (number of threads for MT execution)
• j (json file name containing the input files name)

The program generates 100 files: the odd-numbered files contain 9 entries, while the even-numbered files contain 10000 entries. During the execution, we use CPU time by performing some mathematical computations for each entry. For each “batch”, we print the file names and the number of entries contained in each file. Together with the progress bar from RDataFrame, this shows the problem.

reproduce_problem.cc.txt

j.json.txt

ROOT version

6.40.02, built on Jun 20 2026

Installation method

lxplus

Operating system

Linux

Additional context

No response

[vepadulano]

Dear @fraorlnd ,

Thank you for the report of the issue and the reproducer! I have started working on it and I believe the root cause is the same one that I was trying to address recently with a PR I opened as a draft #22631 .

I wonder, would it be possible for you to build a custom version of ROOT from the branch of my PR? If so, I would appreciate if you could run your tests again and confirm that the performance improves.

Otherwise, I will continue the work on my side with your reproducer and let you know of any updates.

[ferriff]

Dear @vepadulano ,

Thanks, I tested the PR cherry-picking your commit on top of 6-40-04 and it does fix the problem of idle threads waiting for the slowest busy one before reading another bunch of files. Tested on the reproducer as well as on real-life RNTuples.

However, interestingly, even if the threads are all busy, the total processing time in the new fixed version is comparable with the old version (actually a little slower). The displayed event rate is consistent with this: for the old version, it gets lower when the threads are waiting but the peaks are significantly higher than in the new version, which displays a constant rate as expected. I varied the number of processing threads in the new version, trying to find a sweet spot that would reach the peak rates of the old version, but with no luck.

Hope this helps. Feel free to contact us directly on mattermost if you wish more feedback.