Repository navigation
2D cumsum throwing GPU Kernel Exception #742
Description
Several recent builds for KomaMRI.jl have begun failing with AMDGPU on Julia 1.10. Examples:
https://buildkite.com/julialang/komamri-dot-jl/builds/1418#0195b5f6-5b8f-446e-9800-f59c29ffe098
https://buildkite.com/julialang/komamri-dot-jl/builds/1420#0195ba9c-a682-4918-8cba-97d030849721
https://buildkite.com/julialang/komamri-dot-jl/builds/1417#0195b5d5-72cc-4352-b06e-489fd9865dbf
The line where it fails is here: https://github.com/JuliaHealth/KomaMRI.jl/blob/master/KomaMRICore/src/simulation/SimMethods/BlochDict/BlochDict.jl#L53
This line is just calling cumsum on a 1D ROCArray of Float32 values, and the array is also a view within a larger array. Without having access to an AMD GPU, I can't investigate much further. I wonder if this would be enough to reproduce the issue:
using AMDGPU
A = ROCArray(rand(Float32, 1000))
B = view(A, 500:600)
C = cumsum(B)
Activity
pxl-th commentedon Mar 22, 2025
Hm... Interesting. I cannot reproduce the error with MWE that you provided (my machine also runs as part of the CI).
Can you maybe output the array before
cumsumto get the exact values?Maybe right before this line: https://github.com/JuliaHealth/KomaMRI.jl/blob/dc943ed3c657d9d37fbe62d706536dc4c3ea18ec/KomaMRICore/src/simulation/SimMethods/BlochDict/BlochDict.jl#L54
rkierulf commentedon Mar 23, 2025
It looks like the array is just
Float32[1.0f-14, 1.0f-14]: https://buildkite.com/julialang/komamri-dot-jl/builds/1421#0195c05a-b862-499f-beda-971831f858a9. There is also a warning before about global hostcalls.pxl-th commentedon Mar 23, 2025
Ah... My bad, the error is not related to
cumsumbecause they are asynchronous and are checked before every kernel launch, so the fact that it errors before callingcumsummeans that it appeared earlier.Can you add environment variable
HIP_LAUNCH_BLOCKING=1here to synchronize after every kernel launch immediately and then we'll see what's causing it.Additionally, the fact that
mallochostcalls are launched means that some kernels emit exception related code that captures original value that (e.g. during rounding, conversion).In this case it's better to use gpu-friendly functions (to avoid emitting such code).
For example:
You can also compare
@code_llvmto see how fewer things it does vs the originalfloor, ceil, cld.rkierulf commentedon Mar 23, 2025
Ok, it appears the issue is with a different cumsum here: https://github.com/JuliaHealth/KomaMRI.jl/blob/master/KomaMRIBase/src/timing/TrapezoidalIntegration.jl#L49. This is a 2D cumsum of a matrix across the second dimension. Let me know if you are unable to reproduce on your machine and I can try printing the matrix values beforehand.
pxl-th commentedon Mar 24, 2025
Still cannot reproduce... If you can print out the values maybe that will help
rkierulf commentedon Mar 26, 2025
This build has the values printed: https://buildkite.com/julialang/komamri-dot-jl/builds/1428#0195d002-435b-400a-9d1b-1df5624de035. The matrix before the call to cumsum where it crashes has shape 1 x 548 and consists of all zero Float32 values. I also noticed the result is assigned to the same matrix the cumsum is computed on:
y = cumsum(y, dims=2), not sure if that affects anything. And this is happening inside tests so could be affected by--check-boundswhich I think is always set to yes in package test environments.If you still can't reproduce, I don't think this is a major issue for KomaMRI.jl since it doesn't affect the default
Blochsimulation method (@cncastillo feel free to weigh in) , so should be ok to treat as lower-priority.[-]1D cumsum throwing GPU Kernel Exception[/-][+]2D cumsum throwing GPU Kernel Exception[/+]pxl-th commentedon Jul 16, 2025
I see latest KomaMRI AMDGPU tests pass, so I assume this is no longer a problem.
Feel free to re-open if needed.
cncastillo commentedon Jul 16, 2025
Yes, this does in fact work perfectly now! Thanks!
We are still "hacking" a solution for oneAPI.jl. Related PR: JuliaGPU/GPUArrays.jl#568. But no real urgency for that one, most people will probably use CUDA, AMDGPU or Metal.