Repository navigation
Validate atomics for the GCN target, and work around two LLVM atomic bugs - #997
Merged
Merged
Conversation
Atomics reach the GCN back-end from AMDGPU.jl, UnsafeAtomics and Atomix, Enzyme and Julia's intrinsics, so check them on the IR like the PTX target does. Julia's LLVM compiles atomics wider than 64 bits or under-aligned ones to libatomic calls, which only fail when the code object is loaded (hipErrorNoBinaryForGpu); the external back-end rejects them without pointing at the Julia code that performs them. Atomics on constant, region or buffer resource memory and unknown synchronization scopes fail in the back-end or abort it. The cluster scope is accepted when the LLVM that generates code knows it (LLVM 22 and later), which for the external back-end is the version of AMDGPU_LLVM_Backend_jll rather than Julia's.
Since LLVM 22, the AMDGPU back-end selects 32-bit `atomicrmw usub_sat` natively on gfx10.3 and later, but it has no pattern for flat memory, and on local memory only for gfx12, so it fails with "Cannot select" (llvm/llvm-project#229442). This breaks UnsafeAtomics' saturating subtraction on the external back-end (LLVM 23): on local memory for gfx10.3 and gfx11, and on flat memory for gfx12, or for gfx10.3 and gfx11 once the operation carries !amdgpu.no.fine.grained.memory. Expand these operations to compare-exchange loops after optimization, keyed on the version of the LLVM that generates code. Flat memory is expanded on every target, where LLVM otherwise chooses between the instruction and a loop based on scope and metadata. This runs whether or not the IR is validated.
Before LLVM 21, AMDGPUAtomicOptimizer broadcasts the result of an 8- or 16-bit atomic read-modify-write on a uniform address with an illegal readfirstlane (fixed by llvm/llvm-project#128388). With Julia's own LLVM this aborts the process (LLVM 20, Julia 1.13) or crashes it (LLVM 18, Julia 1.12), e.g. for an Atomix `+=` on an Int8 array element whose result is used. The optimizer can only be disabled through a process-wide option, so expand the operations it handles to compare-exchange loops when their result is used, without predicting uniformity. This only affects the in-process back-end; the external one uses LLVM 23.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #997 +/- ##
==========================================
+ Coverage 88.62% 88.73% +0.11%
==========================================
Files 30 30
Lines 6443 6498 +55
==========================================
+ Hits 5710 5766 +56
+ Misses 733 732 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
AMDGPU_LLVM_Backend_jll 23.1.3 carries the fix for llvm/llvm-project#229442, so require it and keep expanding 32-bit usub_sat to a compare-exchange loop only for the in-process back-end. On gfx12 flat memory this now selects the native flat_atomic_sub_clamp_u32.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Atomics reach the GCN back-end from AMDGPU.jl, UnsafeAtomics and Atomix, Enzyme and Julia's own intrinsics, and some of them don't compile or crash the compiler without pointing at the Julia code. Like #996 does for PTX, this validates them on the IR, and works around two LLVM bugs.
Validation rejects, with an
InvalidIRErrorand a Julia backtrace:__atomic_compare_exchange_16), which only fail when the code object is loaded (hipErrorNoBinaryForGpu); the external back-end rejects them without Julia frames;clusteris accepted when the LLVM that generates code knows it (LLVM 22+), which for the external back-end isAMDGPU_LLVM_Backend_jll's version rather than Julia's.The workarounds expand atomics to compare-exchange loops after optimization, also when the IR isn't validated:
atomicrmw usub_sat(UnsafeAtomics' saturating subtraction) natively on gfx10.3+, but has no pattern for flat memory, and for local memory only on gfx12, so it fails with "Cannot select" ([AMDGPU] Cannot select AtomicLoadUSubSat on LDS (gfx10.3/gfx11) and flat memory (gfx12, or with no.fine.grained.memory) llvm/llvm-project#229442).AMDGPU_LLVM_Backend_jll23.1.3 carries the fix (AMDGPU: Fix selection of atomicrmw usub_sat for LDS and flat llvm/llvm-project#229471), so this requires that version and only expands 32-bitusub_satwith Julia's own LLVM (22+, currently nightly): on flat memory, and on local memory for gfx10.3 and gfx11. With the fixed back-end, gfx12 flat memory now uses the nativeflat_atomic_sub_clamp_u32.readfirstlanefor 8- and 16-bit read-modify-writes whose result is used (AMDGPU: Fix creating illegally typed readfirstlane in atomic optimizer llvm/llvm-project#128388), which aborts or crashes Julia 1.12 and 1.13 with the in-process back-end, e.g. for an Atomix+=on anInt8array. The optimizer can only be turned off process-wide, so this expands the operations it handles when their result is used.The tests check each rejection and, for the workarounds, the compile failure or crash without them (in a subprocess) and that the cases LLVM handles stay native. The GCN tests pass on Julia 1.10 through 1.13, and the full suite on 1.13 (at d1d5546). The
AMDGPU_LLVM_Backend_jll23.1.3 change (37e1d20) was checked with the GCN tests on Julia 1.13 and nightly (LLVM 22, which exercises the in-process workaround).