CppInterOp based cppyy migration - #21261
Open
aaronj0 wants to merge 99 commits into
Open
Conversation
aaronj0
requested review from
bellenot,
guitargeek and
vepadulano
as code owners
February 12, 2026 16:05
aaronj0
marked this pull request as draft
February 12, 2026 16:07
aaronj0
force-pushed
the
cppyy-interop-migration
branch
from
February 12, 2026 16:41
6ddc3ce to
42586b9
Compare
Test Results 21 files 21 suites 3d 4h 24m 35s ⏱️ For more details on these failures, see this check. Results for commit 8ee352a. ♻️ This comment has been updated with latest results. |
aaronj0
force-pushed
the
cppyy-interop-migration
branch
from
February 18, 2026 13:34
54c4fb0 to
45f4502
Compare
aaronj0
force-pushed
the
cppyy-interop-migration
branch
2 times, most recently
from
March 12, 2026 13:15
6d03e1a to
986d7ac
Compare
Contributor
Author
|
Just rebased, windows builds failed with something related to libafterimage |
aaronj0
force-pushed
the
cppyy-interop-migration
branch
from
March 13, 2026 14:40
986d7ac to
c124bcf
Compare
aaronj0
force-pushed
the
cppyy-interop-migration
branch
3 times, most recently
from
March 28, 2026 20:49
9336443 to
66eb416
Compare
Contributor
|
Thanks @aaronj0 for the great progress so far! Just to record where we stand:
|
aaronj0
force-pushed
the
cppyy-interop-migration
branch
3 times, most recently
from
March 31, 2026 08:01
66fbcd0 to
d3cdb55
Compare
aaronj0
force-pushed
the
cppyy-interop-migration
branch
from
April 26, 2026 18:39
70ef9c3 to
3d9b1de
Compare
Contributor
|
Great progress! Similar to #21261 (comment), here the summary:
|
aaronj0
force-pushed
the
cppyy-interop-migration
branch
2 times, most recently
from
April 28, 2026 09:08
3e50982 to
65d8ca1
Compare
Contributor
Author
|
Update: with the last two [tmp] commits: net roottests: |
aaronj0
force-pushed
the
cppyy-interop-migration
branch
4 times, most recently
from
April 30, 2026 15:14
ee3eb5c to
b6e1a3f
Compare
Contributor
Author
|
Update: alma10: 185 (-8) Note: these updates are regression tested, no new failures/early error |
ROOT-specific patches on top of the compres fork cppyy/master baseline: __init__.py: - set gInterpereter to gbl.TInterpreter.Instance() - load_library(): use gSystem.Load/FindDynamicLibrary, add Windows winmode=0 for search path - add_library_path(): use gSystem.AddDynamicPath - Drop the CPPYY_API_PATH apipath_extra dispatcher-headers (ROOT installs CPyCppyy API headers) _cpython_cppyy.py: - Wrap `from cppyy_backend import loader` in try/except, fall back to c = None (ROOT does not ship cppyy_backend.so) - Platform: `import libcppyy` vs `import cppyy.libcppyy` (standalone cppyy uses the former; ROOT exposes it as cppyy.libcppyy) - load_reflection_info(): use gSystem.Load ROOT/_facade.py: - Store gInterpreter and gPad in __dict__
…es are rejected [upstream]
…[upstream]
At import time, the cppyy frontend declares on Windows an explicit
specialization of std::endl<char> so that JITed code binds to the C++
runtime's exported symbol instead of instantiating its own. In a ROOT
build, headers in the PCH loaded on startup (e.g. RooAbsDataStore.h)
have already implicitly instantiated std::endl<char> at that point,
making the declaration ill-formed:
error: explicit specialization of 'endl<char, ...>' after instantiation
so cppdef raised SyntaxError and "import cppyy" itself failed, taking
down essentially every PyROOT test on the Windows CI. Catch the error
and continue: when the instantiation already exists, JITed code can use
it directly and the workaround is unnecessary.
cppyy.cppexec relied solely on the return code of Cpp::Process to decide whether to raise SyntaxError. That return code does not cover errors raised while executing a wrapped expression: with cling's dynamic scopes enabled, a statement like "(doesnotexist)" compiles successfully and fails at runtime in EvaluateDynamicExpression, which reports the failure by throwing cling::CompilationException from the JITed wrapper frame. On Linux the exception happens to propagate and cppexec's exception handler turns it into SyntaxError, but on Windows the exception cannot unwind through the JITed frame, nothing reaches Python, and the return code stays 0 (cppyy test_fragile.py test22). Check the captured diagnostics as well, the way the reference cppyy implementation's _cling_report does.
…T-patch] Four rounds of expectation changes to the vendored cppyy test suite, made while bringing up the CppInterOp-based bindings. Markers removed because the tests now pass: a large set of xfails that CppInterOp fixes outright (advancedcpp, eigen, lowlevel, numba, operators, regression, stltypes, templates), including three xfail(run=False) crash markers in test_concurrent; and the ROOT-side Windows xfails for datatypes test39_aggregates and test42_mixed_complex_arithmetic, doc-features test03_use_of_ctypes_and_enum and stltypes test09_string_as_str_bytes, which are stable on the Windows CI across all five ctest --rerun-failed retries of the windows_11 run and whose strict markers would otherwise turn into XPASS suite failures. Markers added: - test_numba test13_std_vector_dot_product is skipped outright, as it compares execution times and is therefore sporadic. - Five tests are marked xfail(run=False) because they do not merely fail but take the process down, which would abort the whole suite: three in test_numba hit the cling assertion "Transaction.cpp:98 ... Must not nest within unloading transaction", and two in test_stltypes (TestSTLVECTOR, TestSTLPAIR) segfault. Those five are open regressions of the new bindings rather than long-standing breakage: they are disabled to keep the suite runnable and still need to be fixed.
…OOT-patch] _generic.pythonize_generic skips the pretty-printer for std::string because ToString returns a quoted ""x"" form, and CPyCppyy already pythonizes std::string with the unquoted shape. With CppInterOp, klass.__cpp_name__ is the canonical fully-qualified template form, so the typedef "std::string" no longer matched the exclude check
… vector [ROOT-patch]
…OT-patch]
Names produced by the backend (proxy __name__/__cpp_name__, docstring
signatures) used clang's printer style: "a, b" in template argument
lists, "T *"/"T &", and literal suffixes on non-type arguments
("array<unsigned int, 3UL>"). CPyCppyy itself constructs names in the
upstream cppyy convention - no space after commas, pointers and
references attached, no suffixes - e.g. when looking up cached template
instantiations by name. The mismatch made e.g.
gbl.__dict__['multiply<int,int,int>'] fail after mul(1, 2) cached the
instantiation under 'multiply<int, int, int>', and setattr-based
overriding of instantiations impossible.
Normalize comma spacing and pointer/reference attachment in
GetFinalName, GetScopedFinalName, GetFullName and the per-argument
types of GetMethodSignature, and print non-type template arguments
without literal suffixes. Name-matching sites that had been adapted to
the clang spellings (std::string/std::string_view pythonizations, the
pretty-printing exclusion, the NPOS pythonization) recognize the
canonical spelling as well, and the rdataframe_asnumpy expectation
returns to the ROOT-master form "array<unsigned int,3>".
Fixes the test_doc_strings and test09_templated_function cases of
cppyy-test-doc-features, the test01_using case of cppyy-test-templates
and the npos handling in test09_string_as_str_bytes of
cppyy-test-stltypes.
…cppyy [ROOT-patch] With the CppInterOp-based cppyy backend, instantiating a function template with the shortened class name AtlasLikeDataVector<CustomStruct> no longer fails: resolving the name autoloads the class dictionary, after which the compiler sees the default second template argument and resolves the correct specialization. The old test pinned the previous backend's limitation, where the first instantiation attempt had to raise and the caller had to retry with a TClass alternative name. Update the test to assert the new behaviour, and strengthen it: foo now returns the demangled typeid name so the test checks that the shortened name really instantiates AtlasLikeDataVector<CustomStruct, DataModel_detail::NoBase>, not merely that the call does not throw. The autoload side effect of registering the alternative class names is still covered, since the REntry pythonization keeps relying on it as a fallback (_try_getptr in _rntuple.py).
… [ROOT-patch] The Python version of this tutorial sporadically crashes on CI (e.g. on opensuse16) with memory corruption detected in cling's JIT at interpreter shutdown, as a segfault in llvm::Value::~Value() / ReplaceableMetadataImpl::resolveAllUses() while ~IncrementalJIT destroys the compiled modules. It is the same overlapping-heap-ownership problem that was worked around for rf619_discrete_profiling.py in 49fb353: with MALLOC_PERTURB_ poisoning enabled the crash becomes deterministic, and the crashing reads then find the fresh-malloc fill pattern in live LLVM metadata, i.e. the allocator handed the same memory to two owners. An ASan-runtime preload run shows no double free, and disabling the glibc tcache makes the crash disappear, so the corruption plausibly enters through a stale write into the tcache header of a freed chunk. The nll/pll pair used for the likelihood surface in pad 3 is not needed after that plot. Deleting it right there tears down the NLL evaluation machinery at a well-defined point instead of at interpreter shutdown, where the order of cleanups is less controlled. With this change the previously deterministic poisoned repro (4/4 crashing) passes 3/3, and plain and ctest invocations pass as well. This is a workaround at the tutorial level; the underlying overlapping heap ownership still deserves an AddressSanitizer investigation (note the CI ASan job currently excludes tutorials).
…T-patch] Fixes a sporadic crash of the `pyunittests-roottest-python-distrdf-backends-all` test (reported by CTest as "Subprocess aborted", i.e. the main pytest process dies from a fatal signal, most often surfacing at `TestInitialization::test_initialization_method[dask]`). The problem was that the DistRDF client (the user's main process) drives ROOT from more than one thread without ROOT thread safety being enabled.
bindexplib deliberately excluded all read-only data from the generated .def files, so const globals in ACLiC-built libraries could never be resolved at runtime. The interpreter needs exactly that: cppyy's advancedcpp test accesses 'extern const char my_global_string2[]' whose definition lives in the ACLiC library, and with the symbol unexported the lookup fails and there is no way to reconstruct the value (the initializer is not in the AST either). Export '?'-prefixed (C++-mangled) external read-only data symbols as DATA. User const globals are what the interpreter needs to resolve; restricting to C++-mangled names excludes both the compiler-generated symbols (vtables ??_7, RTTI ??_R, string literals ??_C, deleting dtors ??_G/??_E) and MSVC's constant pools (__real@/__xmm@) and exception data (_CT*/_TI*). The latter's names get garbled by the cdecl '@'-truncation into non-existent symbols such as '_real', which would otherwise break every ACLiC library with a floating-point literal (LNK2001/LNK1120, 242 CI failures). This only affects runtime lookup: compile-time clients never linked against these symbols (without dllimport they could not), so no existing link can change behavior.
Running TPython from an interpreted macro requires code JIT-compiled in the cling session to resolve the CppInternal::DispatchRaw::MakeFunctionCallable_interp dispatch slot. On Windows that fails: the slot is a data symbol (a function-pointer variable), which the bindexplib-generated export tables do not cover, so the IncrementalExecutor reports it unresolved and the test fails deterministically. Properly exporting writable data across a DLL boundary (or better, registering the dispatch slots by value in the JIT instead of relying on symbol lookup, an upstream CppInterOp change) is not worth blocking the migration for this niche path. Disable the test on MSVC following the existing precedent in this file, behind the usual win_broken_tests escape hatch.
… bit [ROOT-patch]
The fit JIT-compiles its formula through cling on top of an already
large Python + ROOT process, and on the win32 CI the JIT runs out of
32-bit address space during TH1::Fit:
LLVM ERROR: out of memory
Buffer allocation failed
error code: Exit code 0xc0000409
The baseline does not fail, so this is a memory-footprint effect of the
larger per-method JIT wrapper volume of the CppInterOp-based bindings,
which the 32-bit address space no longer absorbs for this tutorial.
Veto it on MSVC x86 following the existing 32-bit veto precedents,
behind the usual win_broken_tests escape hatch.
Combined for CI validation; to be split into a [cppinterop] [upstream] API commit and a [cppyy-backend] [ROOT-patch] wiring commit before merge. Under runtime_cxxmodules=OFF, autoloading a dictionary registers only a forward declaration, so GetActualClass saw an incomplete derived scope and dropped the cast. IsComplete's plain-class branch is a passive hasDefinition() query that never asks Sema to complete the type, so ROOT's on-demand autoparse is never triggered. GetOrForceDefinition completes it through Sema (RequireCompleteType -> autoparse), generalizing over any entity with a getDefinition() accessor (tags, functions, variables).
aaronj0
force-pushed
the
cppyy-interop-migration
branch
from
August 20, 2026 10:49
9284b12 to
7ff6861
Compare
aaronj0
force-pushed
the
cppyy-interop-migration
branch
from
September 4, 2026 11:31
b76eb87 to
8ee352a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bringing cppjit to ROOT, the successor to the cppyy project, for Python/C++ bindings.
This PR migrates all of ROOT's cppyy codebase to use CppInterOp directly.
The development workflow here is designed to have commits in the following fashion:
A typical update of this PR involves first upgrading CppInterOp in ROOT, independently. Then rebase on top and rework the commits to the above stencil. This allows us to cycle our development periods in the most efficient manner. At the final stage, we will have the new "cppjit" codebase in (See the new layout in https://github.com/compiler-research/cppjit) with the legacy mode turned on to continue supporting the Python
cppyy.gblnamespace.