Skip to content

feat(musa): 13 reduction/normalization mudnn kernels + unbreak MUSA build - #63

Open
lvyufeng wants to merge 3 commits into
mainfrom
feat/musa-p2-reductions
Open

feat(musa): 13 reduction/normalization mudnn kernels + unbreak MUSA build#63
lvyufeng wants to merge 3 commits into
mainfrom
feat/musa-p2-reductions

Conversation

@lvyufeng

@lvyufeng lvyufeng commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

Second mudnn batch: 13 reduction/normalization operators, taking MUSA coverage
92 -> 105 kernels. Follows #60 and uses the same table-driven route -- 8 new
templates in scripts/codegen_mudnn.py, no framework changes.

This re-targets main. #61 was opened against #60's branch, and #60 was
squash-merged, so #61's content never reached main -- these commits are rebased onto
main instead.

New ops:

  • softmax family: _log_softmax _softmax_backward_data _log_softmax_backward_data
  • reductions: amax amin prod.dim_int any.dim all.dim
  • statistics: var.correction std.correction linalg_vector_norm
  • normalization: native_layer_norm native_layer_norm_backward

Softmax::RunBwd's operands are (gradInput, output, gradOutput), measured against
CPU, and the numbers match aten's y*(g - sum(g*y)) exactly. input_dtype only
records what the forward input was; when it differs from the gradient's dtype aten
wants a converting backward, which mudnn does not express, so that combination goes
to the host rather than silently computing in the wrong precision.

The Reduce fix (2nd commit)

The existing guard only covered multi-dim reductions, where mudnn v3300 raises
SIGFPE on an input that is a broadcast of one element. A single-dim reduce over
the same input has a second, quieter failure: it intermittently writes only out[0]
and leaves the rest of the output untouched, so the answer is whatever the caching
allocator last left in that block.

It surfaces as a wrong bias gradient. linear(x, w, b).sum().backward() reduces a
grad_output that autograd builds as ones.expand(...), whose storage is a single
float, and the gradient came back as [4, 34, 38, 42, 46] -- element 0 correct,
elements 1.. left over from a previous op.

Two properties worth knowing before reviewing:

  • It does not reproduce outside the allocator. Probed standalone against a
    deliberately poisoned output buffer, mudnn is correct every time. The trigger needs
    a recycled non-zero block underneath, so a C++ probe will report the library fine.
  • A pure-tensor reduce cannot serve as the regression test, for the same reason.
    The test added here goes through the bias-gradient path, which is what actually
    leaves recycled data in the output block.

MudnnReduceWouldFault becomes MudnnReduceNeedsContiguous, dropping the dim-count
argument and keying only on "fully 0-strided". The materialization it forces reads one
element, so the cost is negligible.

The guard.h fix (3rd commit)

Unrelated to the kernels, but MUSA cannot build without it. #55 added a gate around
#include <c10/cuda/CUDAStream.h> in csrc/runtime/guard.h listing ASCEND,
TSINGMICRO, DCU and GCU -- but not MUSA. The Moore Threads toolkit ships no CUDA
runtime, so the header is absent and every MUSA build has failed since that merge
with cuda_runtime_api.h: No such file or directory while compiling guard.cc.
Confirmed by building flagos/main unmodified on an MTT S5000: it fails, and builds
with this one-line change. hooks.h and copy_ops.cc already carry
!defined(USE_MUSA) on the equivalent gate.

Test plan

Environment: torch==2.10.0+cpu, 8x MTT S5000, --no-build-isolation.

  • Numerics vs CPU for all 13 ops across float32/float16, including keepdim both
    ways, negative dims, multi-dim reductions and correction=0/1 for var/std.
  • Reduce regression: the bias-gradient path, repeated after ops that dirty the
    allocator, now returns [4, 4, 4, 4, 4] every time. Added as a test.
  • Suite: pytest tests/ --ignore=tests/manual/metax gives 814 passed / 203
    skipped / 4 xpassed / 24 failed. The same run on flagos/main at a02f62b (plus
    only the guard.h fix, since main otherwise does not build here) gives 810 passed /
    24 failed
    -- the failure sets are byte-identical, so passed goes up by 4 and
    nothing regresses. Those 24 are pre-existing environment limits on this box:
    profiler tests need CUPTI, test_compile.py needs inductor's CUDA backend. The 11
    errors are the known missing transformers.
  • tests/manual/metax is excluded because test_qwen3_fsdp2_metax.py (added by feat(metax): FSDP2 full feature parity + Qwen3 training match #62)
    runs argparse at import time and consumes pytest's own argv, which aborts
    collection for the whole run with INTERNALERROR> SystemExit: 2. That is unrelated
    to this PR and reproduces on unmodified main.
  • ruff==0.15.12 check + format clean.
  • Regenerating with python scripts/codegen_mudnn.py at each commit reproduces the
    checked-in generated files exactly.

🤖 Generated with Claude Code

lvyufeng and others added 3 commits August 6, 2026 17:35
Extends the table-driven mudnn codegen from 90 to 103 kernels (105 m.impl
lines), covering the reductions and normalization a training step leans on
after the P1 elementwise batch.

New categories, each probed against CPU before entering the table:

  reduce_dims_plain (2)  amax, amin.
  reduce_prod (1)        prod.dim_int.
  reduce_bool (2)        any.dim, all.dim -- Reduce::AND/OR take a BOOL
                         input and write BOOL; aten's "nonzero" semantics
                         become a `!= 0` cast first.
  reduce_correction (2)  var.correction, std.correction.
  reduce_norm (1)        linalg_vector_norm via SetNormOrd.
  softmax_bwd (2)        _softmax_backward_data,
                         _log_softmax_backward_data. RunBwd's operands are
                         (gradInput, output, gradOutput), matching aten's
                         y*(g - sum(g*y)) exactly.
  layer_norm (2)         native_layer_norm and its backward. Run emits
                         (out, mean, inv_var) in one call and RunBwd emits
                         (dX, dGamma, dBeta), so both map 1:1 onto aten's
                         tuples.

_log_softmax needed no new template -- the existing softmax forward already
carries the mode, so it is one table entry.

Three measured facts the headers do not give away:

  - Reduce::Mode::MUL returns SUCCESS and writes all zeros. PROD is the
    working product mode. Reduced [[1,2,3,4],[5,6,7,8]] over dim 1: MUL gave
    0 0, PROD gave the correct 24 1680. The plausibly-named mode is the
    broken one.
  - Reduce's VARIANCE/STD default to correction=1, not 0 -- var of 1..4 came
    back 1.66667 (unbiased), not the biased 1.25. That matches aten's
    default, but correction=0 must be set explicitly.
  - LayerNorm's third output is inv_var (1/sqrt(var+eps) = 0.89442 for
    var=1.25, eps=1e-5), which is exactly aten's rstd -- no conversion.

Verification on 8x MTT S5000 / mudnn v3300, torch 2.10.0+cpu:
  - 63 op/dtype cases vs CPU pass. The one fp16 log_softmax delta is a
    rounding artifact, not an error: device and CPU are equidistant from an
    fp64 reference (both 1.895e-3) and their gap (1.95e-3) is below the fp16
    spacing at that magnitude (2.62e-3) -- they straddle the true value on
    adjacent representable values.
  - Autograd end-to-end matches CPU for layer_norm x/w/b across 2-D, 3-D and
    two-axis normalized shapes, and for softmax/log_softmax backward.
  - Every kernel beats the cpu_fallback it replaces at 2048x2048, from 2.8x
    (layer_norm backward) to 132x (layer_norm forward).
  - pytest tests/unit tests/integration/ops: 344 passed / 209 skipped /
    3 xpassed / 0 failed, matching the pre-change baseline exactly.
  - ruff 0.15.12 check + format clean.

Left on cpu_fallback deliberately: vector_norm with ord 0 or +-inf (aten
defines those as a nonzero-count and a max/min of |x|, which NORM does not
express), half_to_float softmax variants, and non-float layer_norm (aten
requires float32 mean/rstd even for a half input, which mudnn will not do).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Reduce guard only covered multi-dim reductions, where mudnn v3300 raises
SIGFPE on an input that is a broadcast of one element. A single-dim reduce over
the same input has a second, quieter failure: it intermittently writes only
out[0] and leaves the rest of the output untouched, so the answer is whatever
the caching allocator last left in that block.

It surfaces as a wrong bias gradient. `linear(x, w, b).sum().backward()` reduces
a grad_output that autograd builds as `ones.expand(...)`, whose storage is a
single float, and the gradient came back as `[4, 34, 38, 42, 46]` -- element 0
correct, elements 1.. left over from a previous op. The same reduce on a
materialized copy is always right, and mudnn is correct when probed standalone
against a poisoned output buffer, so the trigger needs a recycled non-zero block
underneath and does not reproduce outside the allocator.

MudnnReduceWouldFault therefore becomes MudnnReduceNeedsContiguous, dropping the
dim-count argument and keying only on "fully 0-strided". The materialization it
forces reads one element, so the cost is negligible.

Adds the bias-gradient path as a regression test, since a pure-tensor reduce
cannot stand in for it -- the bug only shows when the output buffer holds
recycled data.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#55 added a gate around `#include <c10/cuda/CUDAStream.h>` listing ASCEND,
TSINGMICRO, DCU and GCU, but not MUSA. The Moore Threads toolkit ships no CUDA
runtime, so the header is simply absent and every MUSA build has failed since
that merge with `cuda_runtime_api.h: No such file or directory` while compiling
runtime/guard.cc.

hooks.h and copy_ops.cc already carry `!defined(USE_MUSA)` on the equivalent
gate; this brings guard.h in line. Verified: `flagos/main` alone fails to build
with ACCELERATOR=musa, and builds with this one-line change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@lvyufeng
lvyufeng force-pushed the feat/musa-p2-reductions branch from 24d9efe to 2fab784 Compare August 6, 2026 09:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant