feat(musa): 13 reduction/normalization mudnn kernels + unbreak MUSA build - #63
Open
lvyufeng wants to merge 3 commits into
Open
feat(musa): 13 reduction/normalization mudnn kernels + unbreak MUSA build#63lvyufeng wants to merge 3 commits into
lvyufeng wants to merge 3 commits into
Conversation
Extends the table-driven mudnn codegen from 90 to 103 kernels (105 m.impl
lines), covering the reductions and normalization a training step leans on
after the P1 elementwise batch.
New categories, each probed against CPU before entering the table:
reduce_dims_plain (2) amax, amin.
reduce_prod (1) prod.dim_int.
reduce_bool (2) any.dim, all.dim -- Reduce::AND/OR take a BOOL
input and write BOOL; aten's "nonzero" semantics
become a `!= 0` cast first.
reduce_correction (2) var.correction, std.correction.
reduce_norm (1) linalg_vector_norm via SetNormOrd.
softmax_bwd (2) _softmax_backward_data,
_log_softmax_backward_data. RunBwd's operands are
(gradInput, output, gradOutput), matching aten's
y*(g - sum(g*y)) exactly.
layer_norm (2) native_layer_norm and its backward. Run emits
(out, mean, inv_var) in one call and RunBwd emits
(dX, dGamma, dBeta), so both map 1:1 onto aten's
tuples.
_log_softmax needed no new template -- the existing softmax forward already
carries the mode, so it is one table entry.
Three measured facts the headers do not give away:
- Reduce::Mode::MUL returns SUCCESS and writes all zeros. PROD is the
working product mode. Reduced [[1,2,3,4],[5,6,7,8]] over dim 1: MUL gave
0 0, PROD gave the correct 24 1680. The plausibly-named mode is the
broken one.
- Reduce's VARIANCE/STD default to correction=1, not 0 -- var of 1..4 came
back 1.66667 (unbiased), not the biased 1.25. That matches aten's
default, but correction=0 must be set explicitly.
- LayerNorm's third output is inv_var (1/sqrt(var+eps) = 0.89442 for
var=1.25, eps=1e-5), which is exactly aten's rstd -- no conversion.
Verification on 8x MTT S5000 / mudnn v3300, torch 2.10.0+cpu:
- 63 op/dtype cases vs CPU pass. The one fp16 log_softmax delta is a
rounding artifact, not an error: device and CPU are equidistant from an
fp64 reference (both 1.895e-3) and their gap (1.95e-3) is below the fp16
spacing at that magnitude (2.62e-3) -- they straddle the true value on
adjacent representable values.
- Autograd end-to-end matches CPU for layer_norm x/w/b across 2-D, 3-D and
two-axis normalized shapes, and for softmax/log_softmax backward.
- Every kernel beats the cpu_fallback it replaces at 2048x2048, from 2.8x
(layer_norm backward) to 132x (layer_norm forward).
- pytest tests/unit tests/integration/ops: 344 passed / 209 skipped /
3 xpassed / 0 failed, matching the pre-change baseline exactly.
- ruff 0.15.12 check + format clean.
Left on cpu_fallback deliberately: vector_norm with ord 0 or +-inf (aten
defines those as a nonzero-count and a max/min of |x|, which NORM does not
express), half_to_float softmax variants, and non-float layer_norm (aten
requires float32 mean/rstd even for a half input, which mudnn will not do).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Reduce guard only covered multi-dim reductions, where mudnn v3300 raises SIGFPE on an input that is a broadcast of one element. A single-dim reduce over the same input has a second, quieter failure: it intermittently writes only out[0] and leaves the rest of the output untouched, so the answer is whatever the caching allocator last left in that block. It surfaces as a wrong bias gradient. `linear(x, w, b).sum().backward()` reduces a grad_output that autograd builds as `ones.expand(...)`, whose storage is a single float, and the gradient came back as `[4, 34, 38, 42, 46]` -- element 0 correct, elements 1.. left over from a previous op. The same reduce on a materialized copy is always right, and mudnn is correct when probed standalone against a poisoned output buffer, so the trigger needs a recycled non-zero block underneath and does not reproduce outside the allocator. MudnnReduceWouldFault therefore becomes MudnnReduceNeedsContiguous, dropping the dim-count argument and keying only on "fully 0-strided". The materialization it forces reads one element, so the cost is negligible. Adds the bias-gradient path as a regression test, since a pure-tensor reduce cannot stand in for it -- the bug only shows when the output buffer holds recycled data. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#55 added a gate around `#include <c10/cuda/CUDAStream.h>` listing ASCEND, TSINGMICRO, DCU and GCU, but not MUSA. The Moore Threads toolkit ships no CUDA runtime, so the header is simply absent and every MUSA build has failed since that merge with `cuda_runtime_api.h: No such file or directory` while compiling runtime/guard.cc. hooks.h and copy_ops.cc already carry `!defined(USE_MUSA)` on the equivalent gate; this brings guard.h in line. Verified: `flagos/main` alone fails to build with ACCELERATOR=musa, and builds with this one-line change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
lvyufeng
force-pushed
the
feat/musa-p2-reductions
branch
from
August 6, 2026 09:42
24d9efe to
2fab784
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Second mudnn batch: 13 reduction/normalization operators, taking MUSA coverage
92 -> 105 kernels. Follows #60 and uses the same table-driven route -- 8 new
templates in
scripts/codegen_mudnn.py, no framework changes.This re-targets main. #61 was opened against #60's branch, and #60 was
squash-merged, so #61's content never reached main -- these commits are rebased onto
main instead.
New ops:
_log_softmax_softmax_backward_data_log_softmax_backward_dataamaxaminprod.dim_intany.dimall.dimvar.correctionstd.correctionlinalg_vector_normnative_layer_normnative_layer_norm_backwardSoftmax::RunBwd's operands are(gradInput, output, gradOutput), measured againstCPU, and the numbers match aten's
y*(g - sum(g*y))exactly.input_dtypeonlyrecords what the forward input was; when it differs from the gradient's dtype aten
wants a converting backward, which mudnn does not express, so that combination goes
to the host rather than silently computing in the wrong precision.
The Reduce fix (2nd commit)
The existing guard only covered multi-dim reductions, where mudnn v3300 raises
SIGFPE on an input that is a broadcast of one element. A single-dim reduce over
the same input has a second, quieter failure: it intermittently writes only
out[0]and leaves the rest of the output untouched, so the answer is whatever the caching
allocator last left in that block.
It surfaces as a wrong bias gradient.
linear(x, w, b).sum().backward()reduces agrad_outputthat autograd builds asones.expand(...), whose storage is a singlefloat, and the gradient came back as
[4, 34, 38, 42, 46]-- element 0 correct,elements 1.. left over from a previous op.
Two properties worth knowing before reviewing:
deliberately poisoned output buffer, mudnn is correct every time. The trigger needs
a recycled non-zero block underneath, so a C++ probe will report the library fine.
The test added here goes through the bias-gradient path, which is what actually
leaves recycled data in the output block.
MudnnReduceWouldFaultbecomesMudnnReduceNeedsContiguous, dropping the dim-countargument and keying only on "fully 0-strided". The materialization it forces reads one
element, so the cost is negligible.
The guard.h fix (3rd commit)
Unrelated to the kernels, but MUSA cannot build without it. #55 added a gate around
#include <c10/cuda/CUDAStream.h>incsrc/runtime/guard.hlisting ASCEND,TSINGMICRO, DCU and GCU -- but not MUSA. The Moore Threads toolkit ships no CUDA
runtime, so the header is absent and every MUSA build has failed since that merge
with
cuda_runtime_api.h: No such file or directorywhile compilingguard.cc.Confirmed by building
flagos/mainunmodified on an MTT S5000: it fails, and buildswith this one-line change.
hooks.handcopy_ops.ccalready carry!defined(USE_MUSA)on the equivalent gate.Test plan
Environment:
torch==2.10.0+cpu, 8x MTT S5000,--no-build-isolation.keepdimbothways, negative dims, multi-dim reductions and
correction=0/1for var/std.allocator, now returns
[4, 4, 4, 4, 4]every time. Added as a test.pytest tests/ --ignore=tests/manual/metaxgives 814 passed / 203skipped / 4 xpassed / 24 failed. The same run on
flagos/mainata02f62b(plusonly the guard.h fix, since main otherwise does not build here) gives 810 passed /
24 failed -- the failure sets are byte-identical, so passed goes up by 4 and
nothing regresses. Those 24 are pre-existing environment limits on this box:
profiler tests need CUPTI,
test_compile.pyneeds inductor's CUDA backend. The 11errors are the known missing
transformers.tests/manual/metaxis excluded becausetest_qwen3_fsdp2_metax.py(added by feat(metax): FSDP2 full feature parity + Qwen3 training match #62)runs
argparseat import time and consumes pytest's own argv, which abortscollection for the whole run with
INTERNALERROR> SystemExit: 2. That is unrelatedto this PR and reproduces on unmodified main.
ruff==0.15.12check + format clean.python scripts/codegen_mudnn.pyat each commit reproduces thechecked-in generated files exactly.
🤖 Generated with Claude Code