Skip to content

Fix silent miscompile of thread-varying DMA into shared storage - #23

Open
gxf wants to merge 7 commits into
mainfrom
fix/thread-sliced-shared
Open

gxf wants to merge 7 commits into
mainfrom
fix/thread-sliced-shared

Conversation

@gxf

@gxf gxf commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Fix silent miscompile of thread-varying DMA into shared storage

Problem

On the CuTe target, a DMA whose chunk indexing depends on a thread-level
parallel variable — e.g.

parallel p by 4, q by 10 {
  ...
  l1_b = dma.transp<1,0> rhs.chunkat(k_tile, q#n_tile) => shared;

was lowered to one block-common shared buffer whose copy was emitted under
if (__CHOREO_BLOCK_SINGLE__) (thread 0 only). Thread 0 staged its own slice;
all 10 threads then computed from thread 0's slice. Every thread's per-thread
results were silently wrong. The same applied to the S2G writeback and to
shared buffer initializers (only thread 0's slice was initialized).

Nothing ever caught this because the benchmark harness gates its reference
check behind -D__CHECK__, which nothing enables by default: the affected
kernels "ran fine" while producing wrong results. Reproducer (correctness
check enabled): benchmark/choreo/matmul/11_dynamic_... and
14_efficientnet_... in svn-artifacts both fail their own TEST2/TEST1
assertions before this fix; the local-storage variants pass.

Fix: union-of-slices semantics

A shared buffer staged per-thread now gets one slice per thread, mirroring
what the source expresses:

  • New analysis (lib/thread_sliced_shared.hpp, scheduled right before
    MemoryReuse): marks a shared buffer thread-sliced when a DMA stages
    per-thread data into it (thread-varying source) or writes per-thread
    results out of it to distinct global regions (S2G with thread-varying
    destination), and records the enclosing thread-level parallel-by bound.
  • Sizing (mem_reuse): the buffer footprint is scaled by the thread
    count, so the shared spm (static and JIT-heap-sim paths) holds the union of
    all slices. The existing shared-memory budget checks now account the true
    footprint — oversized cases are rejected at compile time instead of
    silently miscomputing.
  • Addressing (codegen): the buffer base pointer carries a per-thread
    addend __choreo_vtid_x * <slice bytes>; all uses flow through it.
  • Copies / init: DMAs touching a thread-sliced buffer are forced to
    per-thread naive copies (no __CHOREO_BLOCK_SINGLE__ guard, no cooperative
    tiled copy), and shared initializers run per-thread. A block-common
    producer into a sliced buffer degrades to a per-thread broadcast copy, so
    mixed producers stay correct.

Out of scope (existing behavior unchanged, tests pass): group/warp-level
(GROUP/GROUPx4) variance — that is the MMA machinery's domain — and
shared→local distributed reads, which are per-thread reads of block-common
data and need no slicing. Nested or multi-dimensional thread parallel-bys and
non-static thread bounds are rejected with a clear error instead of a silent
miscompile.

Validation

  • New lit regression test tests/gpu/codegen/cute/thread_sliced_shared.co
    (scaled footprint, per-thread addend, unguarded copies/init).
  • tests/gpu: 272/272 pass (1 pre-existing expected failure);
    tests/check: 151 pass + 1 expected failure; tests/cli: 18/18.
  • End-to-end correctness on RTX 5060 Ti (sm_120) with -D__CHECK__:
    • matmul/11_dynamic_32xSx768_768x768_32xSx768.co (migrated to shared,
      retiled) — passes (previously: launch failure, then wrong results);
    • matmul/12_dynamic_...passes; matmul/3_cnn_... (local pattern,
      untouched) — still passes.
  • End-to-end model graph (svn-artifacts eurosys27/e2e, BERT-base encoder
    block, dynamic sequence length): all 11 stages build and execute,
    including the previously unlaunchable dynamic matmul and softmax stages.

Second commit: __co_abort__ macro/function collision (upstream regression)

Exposed while re-running the graph's embedding stage: choreo.h falls back
to a __co_abort__() macro (recent c9177d0), and choreo_cute.h defined
an unguarded __co_abort__() function that the macro expanded into a
choreo::__builtin_trap declaration — every device abort call became
ambiguous under nvcc (CUDA 13), so any kernel with device-side assertions
failed to compile. The commit guards the function definition and makes the
fallback macro device-aware (__trap() under __CUDA_ARCH__
__builtin_trap is host-only under nvcc). Verified:
embedding/10_dynamic_... now compiles and passes its built-in check.

  • Known pre-existing issues this fix exposes (not caused by it; benchmark
    source problems to be handled in svn-artifacts):
    • matmul/13_dynamic_... fails its own reference check even in its
      original local form on the unmodified compiler — the case itself is
      buggy;
    • matmul/14_efficientnet_... has a buffer-type [4,80] vs DMA tile
      [5,80] inconsistency (320B under-allocation) that was harmless with
      over-provisioned static arrays but faults once the spm is sized
      correctly;
    • matmul/15_gpt_... (and similar large-tile cases) now fail the shared
      budget at compile time — their true union-of-slices footprint exceeds the
      limit, so they must be retiled.

gxf added 7 commits September 15, 2026 22:48
A DMA whose chunk indexing depends on a thread-level parallel variable
(e.g. dma.copy rhs.chunkat(k_tile, q#n_tile) => shared) was lowered to a
single block-common shared buffer loaded under __CHOREO_BLOCK_SINGLE__,
so only thread 0's slice was staged and every thread computed from it —
a silent miscompile.

Give such buffers union-of-slices semantics: a new analysis marks shared
buffers staged per-thread, mem_reuse scales their footprint by the
thread count, codegen offsets each thread into its own slice
(vtid * slice bytes), and the copies and initializers run per-thread
instead of under the block-single guard.  Group-level (warp) variance
and shared->local distributed reads keep their existing lowering.
choreo.h falls back to a __co_abort__ macro when the target provides
none, and choreo_cute.h defined an unguarded __co_abort__ function that
the macro expanded into a choreo::__builtin_trap declaration, making
every device abort call ambiguous.  Guard the function definition, and
make the fallback macro device-aware: __trap() under __CUDA_ARCH__
(__builtin_trap is host-only under nvcc), __builtin_trap() otherwise.
@gxf
gxf force-pushed the fix/thread-sliced-shared branch from 4c2f613 to fc3ce48 Compare September 15, 2026 14:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant