Skip to content

Pad allocations and shared memory to whole 32-bit words - #3353

Open
maleadt wants to merge 1 commit into
mainfrom
tb/atomic-padding
Open

maleadt wants to merge 1 commit into
mainfrom
tb/atomic-padding

Conversation

@maleadt

@maleadt maleadt commented Oct 6, 2026

Copy link
Copy Markdown
Member

PTX has few 8- and 16-bit atomics, so LLVM implements them as a compare-and-swap on the containing 32-bit word. When the value sits in the last partial word of an array, that word extends past the end of the allocation. This never faults (allocations are 256-byte aligned), but compute-sanitizer reports it as an out-of-bounds access and aborts the kernel, leaving a sticky error:

========= Invalid __global__ read of size 4 bytes
=========     Access to 0xb00000000 is out of bounds
=========     and is inside the nearest allocation at 0xb00000000 of size 2 bytes

This hits any sub-word atomic that goes through LLVM: Int8/Int16 read-modify-writes and CAS from UnsafeAtomics, Atomix or KernelAbstractions on every architecture (current LLVM never emits atom.cas.b16), and CUDA.jl's own 16-bit atomics before sm_70 or BFloat16 add before sm_90. It applies to global memory and to static and dynamic shared memory alike. NVCC doesn't run into this because it emits atom.cas.b16, which ptxas widens after the sanitizer's view of the access; libcu++'s cuda::atomic_ref<short> has the same problem we do.

Like Metal.jl did in JuliaGPU/Metal.jl#977, this pads pool allocations, static shared memory arrays and the dynamic shared memory size of a launch to a whole number of 32-bit words. That costs no memory: the stream-ordered pool hands out 512-byte blocks even for 2-byte requests, and shared memory is allocated in units of at least 128 bytes. The new tests make the sanitizer job abort without this change.

Caveats:

  • Memory that doesn't come from the pool isn't padded (unsafe_wrap, the low-level CUDA.alloc, device-side malloc). The kernel programming docs now say so.
  • Memory statistics now count the padded size, e.g. CUDA.@allocated CuArray{Int8}(undef, 1) reports 4 bytes.
  • A launch whose dynamic shared memory size equals an odd-sized FUNC_ATTRIBUTE_MAX_DYNAMIC_SHARED_SIZE_BYTES limit will now exceed it by up to 3 bytes.

With this in place, #3350 shouldn't need its native atom.cas.b16 paths to keep the sanitizer happy, and 8-bit atomics (which #3350 enables through UnsafeAtomics and which have no native fallback) are covered.

LLVM implements 8- and 16-bit atomics as a compare-and-swap on the
containing 32-bit word, which can extend past the end of an array. That
never faults, since allocations are at least 256-byte aligned, but
compute-sanitizer reports it as an out-of-bounds access and aborts the
kernel. This affects any sub-word atomic going through LLVM: Int8 and
Int16 operations from UnsafeAtomics, Atomix or KernelAbstractions on
every architecture, and 16-bit operations before sm_70.

Round pool allocations, static shared memory arrays and the dynamic
shared memory size up to a multiple of 4 bytes, like Metal.jl does. The
pool already hands out much larger blocks, so this costs no memory.
Memory CUDA.jl did not allocate, such as pointers passed to unsafe_wrap,
can't be padded; the docs now say so.
@maleadt maleadt added bugfix This gets something working again. cuda kernels Stuff about writing CUDA kernels. labels Oct 6, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix This gets something working again. cuda kernels Stuff about writing CUDA kernels.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant