Conversation
A DMA whose chunk indexing depends on a thread-level parallel variable (e.g. dma.copy rhs.chunkat(k_tile, q#n_tile) => shared) was lowered to a single block-common shared buffer loaded under __CHOREO_BLOCK_SINGLE__, so only thread 0's slice was staged and every thread computed from it — a silent miscompile. Give such buffers union-of-slices semantics: a new analysis marks shared buffers staged per-thread, mem_reuse scales their footprint by the thread count, codegen offsets each thread into its own slice (vtid * slice bytes), and the copies and initializers run per-thread instead of under the block-single guard. Group-level (warp) variance and shared->local distributed reads keep their existing lowering.
choreo.h falls back to a __co_abort__ macro when the target provides none, and choreo_cute.h defined an unguarded __co_abort__ function that the macro expanded into a choreo::__builtin_trap declaration, making every device abort call ambiguous. Guard the function definition, and make the fallback macro device-aware: __trap() under __CUDA_ARCH__ (__builtin_trap is host-only under nvcc), __builtin_trap() otherwise.
gxf
force-pushed
the
fix/thread-sliced-shared
branch
from
September 15, 2026 14:57
4c2f613 to
fc3ce48
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix silent miscompile of thread-varying DMA into shared storage
Problem
On the CuTe target, a DMA whose chunk indexing depends on a thread-level
parallel variable — e.g.
was lowered to one block-common shared buffer whose copy was emitted under
if (__CHOREO_BLOCK_SINGLE__)(thread 0 only). Thread 0 staged its own slice;all 10 threads then computed from thread 0's slice. Every thread's per-thread
results were silently wrong. The same applied to the S2G writeback and to
sharedbuffer initializers (only thread 0's slice was initialized).Nothing ever caught this because the benchmark harness gates its reference
check behind
-D__CHECK__, which nothing enables by default: the affectedkernels "ran fine" while producing wrong results. Reproducer (correctness
check enabled):
benchmark/choreo/matmul/11_dynamic_...and14_efficientnet_...in svn-artifacts both fail their ownTEST2/TEST1assertions before this fix; the
local-storage variants pass.Fix: union-of-slices semantics
A shared buffer staged per-thread now gets one slice per thread, mirroring
what the source expresses:
lib/thread_sliced_shared.hpp, scheduled right beforeMemoryReuse): marks a shared buffer thread-sliced when a DMA stagesper-thread data into it (thread-varying source) or writes per-thread
results out of it to distinct global regions (S2G with thread-varying
destination), and records the enclosing thread-level parallel-by bound.
mem_reuse): the buffer footprint is scaled by the threadcount, so the shared spm (static and JIT-heap-sim paths) holds the union of
all slices. The existing shared-memory budget checks now account the true
footprint — oversized cases are rejected at compile time instead of
silently miscomputing.
addend
__choreo_vtid_x * <slice bytes>; all uses flow through it.per-thread naive copies (no
__CHOREO_BLOCK_SINGLE__guard, no cooperativetiled copy), and
sharedinitializers run per-thread. A block-commonproducer into a sliced buffer degrades to a per-thread broadcast copy, so
mixed producers stay correct.
Out of scope (existing behavior unchanged, tests pass): group/warp-level
(
GROUP/GROUPx4) variance — that is the MMA machinery's domain — andshared→local distributed reads, which are per-thread reads of block-common
data and need no slicing. Nested or multi-dimensional thread parallel-bys and
non-static thread bounds are rejected with a clear error instead of a silent
miscompile.
Validation
tests/gpu/codegen/cute/thread_sliced_shared.co(scaled footprint, per-thread addend, unguarded copies/init).
tests/gpu: 272/272 pass (1 pre-existing expected failure);tests/check: 151 pass + 1 expected failure;tests/cli: 18/18.-D__CHECK__:matmul/11_dynamic_32xSx768_768x768_32xSx768.co(migrated toshared,retiled) — passes (previously: launch failure, then wrong results);
matmul/12_dynamic_...— passes;matmul/3_cnn_...(local pattern,untouched) — still passes.
eurosys27/e2e, BERT-base encoderblock, dynamic sequence length): all 11 stages build and execute,
including the previously unlaunchable dynamic matmul and softmax stages.
Second commit:
__co_abort__macro/function collision (upstream regression)Exposed while re-running the graph's embedding stage:
choreo.hfalls backto a
__co_abort__()macro (recentc9177d0), andchoreo_cute.hdefinedan unguarded
__co_abort__()function that the macro expanded into achoreo::__builtin_trapdeclaration — every device abort call becameambiguous under nvcc (CUDA 13), so any kernel with device-side assertions
failed to compile. The commit guards the function definition and makes the
fallback macro device-aware (
__trap()under__CUDA_ARCH__—__builtin_trapis host-only under nvcc). Verified:embedding/10_dynamic_...now compiles and passes its built-in check.source problems to be handled in svn-artifacts):
matmul/13_dynamic_...fails its own reference check even in itsoriginal
localform on the unmodified compiler — the case itself isbuggy;
matmul/14_efficientnet_...has a buffer-type[4,80]vs DMA tile[5,80]inconsistency (320B under-allocation) that was harmless withover-provisioned static arrays but faults once the spm is sized
correctly;
matmul/15_gpt_...(and similar large-tile cases) now fail the sharedbudget at compile time — their true union-of-slices footprint exceeds the
limit, so they must be retiled.