Skip to content

⚡️ Optimize release wheels with SDK/Core PGO - #2476

Open
burgholzer wants to merge 24 commits into
codex/fix-mlir-exception-boundariesfrom
codex/optimized-release-toolchain
Open

⚡️ Optimize release wheels with SDK/Core PGO#2476
burgholzer wants to merge 24 commits into
codex/fix-mlir-exception-boundariesfrom
codex/optimized-release-toolchain

Conversation

@burgholzer

@burgholzer burgholzer commented Sep 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

🤖 AI text below 🤖

Description

Build Linux and macOS release wheels with assertion-free SDKs and combined SDK/Core PGO. Linux uses the manylinux container's matched Clang packages, full LTO, and BOLT; macOS uses Apple Clang, ThinLTO, and a 13.3 deployment target. Windows keeps its normal build.

Core owns SDK rebuilding and trains with the existing C++ MLIR suite plus one shared installed-wheel exercise. Common cibuildwheel hooks cover both Unix platforms and both Python ABIs. Linux repair validates the optimized wheel and installed Clang/GCC consumers.

Depends on SDK #94, its September 15 release, Setup #255, and Workflows #464. Stacked on #2545, which owns the exception-boundary fixes.

Validation: the Linux ARM64 joint run passed both ABIs, PGO/BOLT, and installed Clang/GCC consumers. Each ABI passed 2,800 MLIR training tests; the stable wheel passed 1,396 Python tests (two skips), and the free-threaded wheel passed its configured import check. Lints and the source-archive check passed. Hosted wheels await the SDK/installer dependencies; Linux x86 also hit a registry HTTP 502. AI assisted implementation, review, and validation.

Checklist

  • The pull request only contains commits that are focused and relevant to this change.
  • I have added appropriate tests that cover the new/changed functionality.
  • I have updated the documentation to reflect these changes.
  • The changes follow the project's style guidelines and introduce no new warnings.
  • The changes are fully tested and pass the CI checks.
  • I have reviewed my own code changes.

If PR contains AI-assisted content:

  • Any agent that created, edited, or submitted GitHub content was explicitly authorized for that scope, as required by our AI Usage Guidelines.
  • Every agent-authored or agent-edited public text body begins with the visible disclosure 🤖 *AI text below* 🤖 (titles are exempt).
  • I have disclosed AI assistance in the PR description.
  • I confirm that I have personally reviewed and understood all AI-generated content, and accept full responsibility for it.

@burgholzer burgholzer self-assigned this Sep 8, 2026
@burgholzer burgholzer added continuous integration Anything related to the CI setup enhancement Improvement of existing feature labels Sep 8, 2026
This was referenced Sep 8, 2026
Open
Open
Open
@codecov

codecov Bot commented Sep 8, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@burgholzer burgholzer changed the title ⚡ Optimize wheel links and release SDK selection ⚡ Optimize release wheels with matched LTO and BOLT Sep 8, 2026
@burgholzer burgholzer changed the title ⚡ Optimize release wheels with matched LTO and BOLT ⚡ Optimize release wheels with LTO and BOLT Sep 9, 2026
@mergify mergify Bot added the conflict label Sep 10, 2026
@burgholzer burgholzer added this to the v4.1.0 - QDMI 1.4 / MQSF milestone Sep 10, 2026
@burgholzer
🐛 Return QDMI and benchmark errors before MLIR
bb0ae82 
Decode QDMI properties as values or diagnostics and reuse that decoder in
both C++ interfaces. Convert remaining client and benchmark exceptions
inside their owning libraries, before returning to MLIR.

Keep the adapter, benchmark generator, and CLI free of exceptions and RTTI.
Remove platform flags and the split CLI entry point. Preserve existing
throwing C++ APIs and require invalid CLI input to return a diagnostic
with exit status 1.

Assisted-by: GPT-6 via Codex
@burgholzer
⚡ Train native SDK and Core release wheels together
f1eeefe 
Use the measured Linux Clang 22, full Core LTO, combined PGO and BOLT
recipe. Keep SDK library rebuilding in its owning repository and train
fresh profiles per Python ABI before configuring the final wheel build.

Preserve assertion-enabled development defaults, retain native SDK tools,
and split release ABI jobs to fit the hosted resource budget. Adapt the
training corpus to the current typed compiler and QIR result contracts.

Assisted-by: Codex
@burgholzer
burgholzer force-pushed the codex/optimized-release-toolchain branch from 706fd8f to f1eeefe Compare September 12, 2026 13:58
@burgholzer burgholzer changed the title ⚡ Optimize release wheels with LTO and BOLT ⚡ Train and optimize native SDK release wheels Sep 12, 2026
@mergify mergify Bot removed the conflict label Sep 12, 2026
@burgholzer
🧪 Check repaired wheels with CMake consumers
78edfb9 
Build and run the installed C++ package with the release compiler and
manylinux GCC after BOLT and repair. Preserve the source-package inputs
and enforce the measured Linux CPU and memory limits.

Assisted-by: Codex
@burgholzer
🐛 Stop release provisioning on SDK download errors
3a3139f 
Run SDK installation inside the checked provisioning script instead of
a curl-to-shell pipeline. Accept an already installed assertion-free
23.1.1 SDK so retained trial artifacts can qualify the production hooks.

Assisted-by: Codex
@burgholzer
🐛 Hash release sources with the Python runtime
59157db 
Manylinux does not provide Perl shasum. Use the already required Python
standard library for source archive identities on Linux and macOS.

Assisted-by: Codex
@burgholzer
🔧 Pin the qualified manylinux release images
0fcd550 
Use the measured image revisions for release wheels. Let standalone
qualification enable C++ tests with the final PGO configuration; the
wheel backend already disables those tests for instrumented builds.

Assisted-by: GPT-6
@burgholzer
⚡ Prepare macOS wheels with native SDK PGO
7d31073 
Use the native finalist selected by both paired macOS cohorts: ThinLTO
for Core and combined SDK/Core PGO. Reuse the existing preparation hooks
and select Clang and profile tools from one Xcode installation. Keep the
three-worker bound and the SDK/Core deployment targets.

The final matched-SDK gate and integrated wheel qualification remain
pending.

Assisted-by: Codex
@burgholzer
🔧 Keep PGO out of compiler feature checks
1a8514a 
Core profiles contain main functions that collide with unrelated CMake
probe programs under -Werror. This rejects supported PIC flags and can
produce invalid TLS relocations in the final LTO shared library.

Apply final LTO and PGO flags as directory compile and link options so
compiler probes remain independent of workload profiles.

Assisted-by: Codex
@burgholzer
🧪 Link driver tests with the dynamic loader
37c7ce6 
The driver tests call dlopen, dlsym, and dlclose directly. Link their own
platform loader dependency so the shared-library test build works on
manylinux glibc 2.28, where those symbols are not part of libc.

Assisted-by: Codex
@burgholzer
📦 Keep build paths out of installed wheels
1691559 
Apply the driver device search path only to consumers using build RPATHs.
Wheel targets use install RPATHs and must not retain a build-directory
lookup that can mask missing packaged dependencies. Reject absolute ELF
runtime paths during release processing.

All thirteen rebuilt wheel binaries have relative runtime paths. Installed
numerical, compiler, QIR, CLI, and DD checks pass with the build tree hidden.
The path check rejects the three affected libraries in the earlier wheel.

Assisted-by: Codex
@burgholzer
📝 Record final release qualification
1023d06 
Retain the selected native SDK recipes and completed platform checks.
Keep trial validation distinct from production release dependencies.

Assisted-by: GPT-6 via Codex
@burgholzer
👷 Include the wheel consumer in C++ lint
3dc5d03 
Add its compile-only target to the test build so clang-tidy resolves the
same library headers as the standalone installed-wheel check. Exclude the
target from normal builds. Pin the workflow ABI environment repair.

Validation: whole-file C++ lint and repository lint pass; cibuildwheel
identifier probes preserve default, cp311, and cp315t selection.

Assisted-by: Codex
@burgholzer
♻️ Integrate the selected release optimization recipe
49247ff 
Use fixed native-SDK PGO recipes for Linux and macOS, package SDK helpers with the toolchain, and keep BOLT processing in wheel repair. Remove experiment settings, evaluation code, and phase-report machinery while preserving the study in history.

Assisted-by: Codex
@burgholzer
🔀 Integrate current main into release optimization
629180b 
Validate the final release recipe with the latest parameter identity, Qiskit import, and dependency updates.

Assisted-by: Codex
@burgholzer
🧪 Cover QDMI provider query failures
5bc048f 
Exercise device, site, operation, and site-calibration failures through the existing sessionless provider. Require compiler diagnostics and successful recovery after restoring the provider.

Assisted-by: Codex
@burgholzer
🔧 Strip bindings after release optimization
b6ca7d4 
Let installation and Linux wheel repair strip Python modules. Nanobind link-time stripping conflicts with the relocations required by BOLT; remove the obsolete conditional build option.

Assisted-by: Codex
@burgholzer
👷 Use the complete SDK release manifest
828fbf0 
Pin the setup and wheel workflows that select the complete 2026.09.14 SDK release. Record that the immutable prior release cannot accept companion archives and that Linux installation requires the manifest on setup-mlir main.

Assisted-by: Codex
@mergify mergify Bot added conflict and removed conflict labels Sep 14, 2026
@burgholzer
🔀 Merge current release documentation policy
29f4802 
Adopt main's release-time changelog policy and remove this PR's unpublished changelog entries. Retain the selected wheel recipe and its documentation while clearing the merge conflict that blocked CI.

Assisted-by: Codex
@burgholzer burgholzer mentioned this pull request Sep 14, 2026
Open
10 tasks
@burgholzer
♻️ Simplify release wheel preparation
3443517 
Use the container toolchain, train PGO with the MLIR tests, and run
both Python ABIs in one cibuildwheel job. Keep SDK rebuilding in
Core and share the installed-wheel training exercise with BOLT.

Assisted-by: GPT-6 via Codex
@burgholzer
burgholzer force-pushed the codex/optimized-release-toolchain branch from cc686e4 to 3443517 Compare September 14, 2026 18:18
@burgholzer burgholzer changed the title ⚡ Train and optimize native SDK release wheels ⚡️ Optimize release wheels with SDK/Core PGO Sep 14, 2026
@burgholzer
burgholzer changed the base branch from main to codex/fix-mlir-exception-boundaries September 14, 2026 18:18
@burgholzer
🐛 Build PGO tests inside the wheel environment
8d48541 
Build the MLIR training targets before uv removes its isolated
Python environment, so later CMake regeneration cannot reference
a deleted interpreter.

Assisted-by: GPT-6 via Codex
@burgholzer
🐛 Keep PGO training binaries runnable
9677f4f 
Honor the native CMake RPATH option so instrumented builds can run
shared-library tests from the build tree. Retain install RPATHs by
default and build mqt-cc for the MLIR suite's CLI tests.

Assisted-by: GPT-6 via Codex
@burgholzer
♻️ Merge PGO profiles during training
f293590 
Use LLVM's per-binary raw-profile merging so the CTest suite does
not keep a separate SDK profile for every test process.

Assisted-by: GPT-6 via Codex
@burgholzer
🐛 Place runtime assets beside shared QDMI drivers
8d686a8 
Reuse the runtime-copy helper on Unix shared builds. Registry
discovery starts beside the driver library, so assets copied only
beside an executable cannot register bundled devices there.

Assisted-by: GPT-6 via Codex
@burgholzer
📦 Honor ignore rules for test sources
fd7a7a5 
Include test sources without overriding the normal ignore rules, so source
archives do not retain cached bytecode from removed tests.

Assisted-by: GPT-6 via Codex
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@burgholzer burgholzer

Labels

continuous integration Anything related to the CI setup enhancement Improvement of existing feature

Projects

None yet

Milestone

v4.1.0 - QDMI 1.4 / MQSF

Development

Successfully merging this pull request may close these issues.

1 participant

@burgholzer