GPU build fails: nvc++ can't find cublas_v2.h / cusolverDn.h (NVHPC env not forwarded to pixi build sandbox)

[joshkamm]

Summary

Building slatergpu as a pixi source dependency (git-main, currently a7f971f) fails to compile the GPU library because nvc++ cannot find cublas_v2.h / cusolverDn.h:

"…/include/cuda_util.h", line 11: catastrophic error: cannot open source file "cublas_v2.h"
  #include <cublas_v2.h>
                        ^
1 catastrophic error detected in the compilation of "…/src/integrals/cuda_util.cpp".
Compilation terminated.

Every GPU-path translation unit (cuda_util.cpp, scf_util.cpp, integrals*.cpp, …) fails the same way. include/cuda_util.h:11 pulls in <cublas_v2.h> / <cusolverDn.h> under #if USE_ACC.

Reported by multiple group members recently while building "the same way as before" — consistent with the umbrella in #95 (git-main build fails, local-path workaround). This issue documents the cuBLAS/cuSolver facet with a fully-traced root cause.

Root cause — environment, not the repo

nvc++ locates the CUDA math-library headers via the NVHPC module environment (module load nvhpc sets CPATH=…/math_libs/include). The pixi build sandbox runs nvc++ with that environment stripped, so the headers are unreachable. Confirmed both directions on athena (NVHPC 25.5, nvc++ 25.5-0; pixi 0.70.2):

Condition	cuda_util.cpp compile
Interactive shell, nvhpc module loaded (CPATH set)	✅ compiles
Env stripped (simulates pixi sandbox)	❌ cannot open source file "cublas_v2.h"
Env stripped + CPATH=…/math_libs/include re-added	✅ compiles
Env stripped + -cudalib=cublas,cusolver	✅ compiles
Env stripped + -I …/math_libs/include	✅ compiles

The header itself is present (…/nvidia_hpc_sdk/Linux_x86_64/25.5/math_libs/include/cublas_v2.h); the build just isn't told where to look.

Why a git bisect of the repos won't isolate it

The in-repo wiring hasn't changed: the <cublas_v2.h> include is long-standing, and SlaterGPU has never passed -cudalib (git log -S cudalib is empty). What flipped is whether the pixi build sandbox forwards the NVHPC/CPATH environment to nvc++ — i.e. the pixi / pixi-build-cmake toolchain, not SlaterGPU source. The useful bisect is to hold both repos fixed and vary the pixi version (0.70 ↔ 0.71), tying in with #96.

Proposed fix (repo-side, environment-independent)

Make the GPU build supply the CUDA math-lib headers itself, so it no longer depends on every developer's shell having CPATH set. Options (both validated under the stripped env above):

1. Add -cudalib=cublas,cusolver to the GPU target_compile_options (nvc++ then provides both include + link). Minimal, nvc++-native.
2. find_package(CUDAToolkit) and link CUDA::cublas / CUDA::cusolver — also closes Replace deprecated FindCUDA module with find_package(CUDAToolkit) in Config.cmake.in #93 (FindCUDA → CUDAToolkit modernization).

Environment

• athena; NVHPC SDK 25.5 (nvc++ 25.5-0), CUDA math_libs 12.9
• pixi 0.70.2, pixi-build-cmake backend, USE_ACC=ON (default GPU build)
• SlaterGPU git-main @ a7f971f

Related

• Build reliability: slatergpu git-main builds fail (path-dep workaround); confirmed facet = find_package(MPI REQUIRED) #95 — umbrella: git-main builds fail for multiple members (this is the cuBLAS/cuSolver facet)
• Replace deprecated FindCUDA module with find_package(CUDAToolkit) in Config.cmake.in #93 — FindCUDA → find_package(CUDAToolkit) (option 2 above closes it)
• pixi 0.71 breaks build-backend resolution — pixi-build-cmake <=0.3.8 cap too low (recurring; all pixi-build repos) #96 — pixi 0.71 build-backend break (same "toolchain version changed the build env" axis)

[joshkamm]

Root cause pinned down — it's the build backend, not pixi (and not a blanket env strip)

Bisected on Perlmutter (login node, NVHPC 25.5, all builds truly cold: fresh PIXI_CACHE_DIR + wiped .pixi), varying the pixi CLI and pixi-build-cmake backend versions independently:

pixi	backend	CPATH seen by CMake*	cublas_v2.h
0.70.2	0.3.14	empty	❌ fails
0.66.0	0.3.14	—	❌ fails
0.70.2	0.3.12 / 0.3.13	empty	❌ fails
0.70.2	0.3.9 – 0.3.11	inherited from shell	✅ found
0.70.2	0.3.8	inherited from shell	✅ found
0.66.0	0.3.8	inherited	✅ found
0.60.0	0.3.6	inherited	✅ found

* probed with a temporary message(STATUS "...$ENV{CPATH}...") in CMakeLists.txt; compile flags are byte-identical across backends, so the env is the only variable.

The flip is at backend 0.3.11 → 0.3.12, and the pixi CLI version is irrelevant — so the pixi 0.70 ↔ 0.71 bisect proposed above would have found nothing.

Mechanism

rattler-build 0.62.0 (2026-04-13) enabled strict environment isolation by default ("we now clean environment variables before executing the build script"; --env-isolation strict|conda-build|none). It entered pixi-build-cmake at 0.3.12 (the tag where the monorepo pins rattler_build_core past 0.62). PATH is on the forwarding allowlist — which is why nvc++ from the module env is found at all — but CPATH is not, so the module's math_libs/include never reaches the compiler. The backend hardcodes env_isolation: Default::default() with no config knob, so consumers can't opt out. (Two corrections to the issue text: build_env.sh is a red herring — it never lists CPATH in any era, old backends simply ran the build in the full inherited env. And strict isolation also remaps HOME into the work dir, which is where the stray .bashrc/.zshrc files appearing in checkouts since July come from.)

Why "the same way as before" broke in July

SlaterGPU's <=0.3.8 backend cap shielded everyone from the April change until #97 bumped the cap to <=0.4.2 (merged 2026-07-01) to fix the pixi-0.71 solve — silently upgrading the backend to 0.3.14 (pixi 0.70) / 0.4.2 (pixi 0.71). The #96 fix exposed this issue. My 6/30 note on #96 calling the cublas error "pre-existing" was wrong in an interesting way: those verification runs were themselves using the widened cap being tested.

One loose end: the athena trace in this issue's text claims git-main a7f971f (old cap → backend 0.3.8 → env inherited → shouldn't hit this). Worth grepping that athena build log for its Instantiated backend pixi-build-cmake@… line — I suspect the checkout was actually newer than the quoted SHA.

Fix

PR #100 implements option 2 (find_package(CUDAToolkit) + CUDA::cublas/CUDA::cusolver, also closing #93). Two findings from validating it under the sanitizing backend:

• Option 1 (-cudalib) on the SlaterGPU target alone is not sufficient — the io target also compiles with -acc=gpu and pulls cuda_util.h via scf_util.h/gauss.h. The imported-targets approach handles this transitively (one PUBLIC link on SlaterGPU).
• With the cublas facet fixed, Perlmutter login-node builds hit a separate, pre-existing NVHPC-25.5 codegen failure in integrals.cpp — filed as OpenACC kernel names truncated for long source paths, causing duplicate device functions (NVVM_ERROR_COMPILATION) #99 (likely the uncaptured Perlmutter facet of Build reliability: slatergpu git-main builds fail (path-dep workaround); confirmed facet = find_package(MPI REQUIRED) #95). Fixing this issue alone won't make those builds green.

[joshkamm]

Correction to my comment above: #99's root cause is now identified as an NVHPC OpenACC kernel-name truncation collision triggered by long source paths (details in #99 (comment)) — it only bit because this investigation ran in a deep .claude/worktrees/... checkout. So the earlier caveat "fixing #98 alone won't make Perlmutter login-node builds green" applies only to unusually deep checkout paths. Normal checkouts, athena, and CI are unaffected by #99, and PR #100's CI validation of the #98 fix is not impacted.