feat(musa): 13 reduction/normalization mudnn kernels - #61
Merged
Conversation
Extends the table-driven mudnn codegen from 90 to 103 kernels (105 m.impl
lines), covering the reductions and normalization a training step leans on
after the P1 elementwise batch.
New categories, each probed against CPU before entering the table:
reduce_dims_plain (2) amax, amin.
reduce_prod (1) prod.dim_int.
reduce_bool (2) any.dim, all.dim -- Reduce::AND/OR take a BOOL
input and write BOOL; aten's "nonzero" semantics
become a `!= 0` cast first.
reduce_correction (2) var.correction, std.correction.
reduce_norm (1) linalg_vector_norm via SetNormOrd.
softmax_bwd (2) _softmax_backward_data,
_log_softmax_backward_data. RunBwd's operands are
(gradInput, output, gradOutput), matching aten's
y*(g - sum(g*y)) exactly.
layer_norm (2) native_layer_norm and its backward. Run emits
(out, mean, inv_var) in one call and RunBwd emits
(dX, dGamma, dBeta), so both map 1:1 onto aten's
tuples.
_log_softmax needed no new template -- the existing softmax forward already
carries the mode, so it is one table entry.
Three measured facts the headers do not give away:
- Reduce::Mode::MUL returns SUCCESS and writes all zeros. PROD is the
working product mode. Reduced [[1,2,3,4],[5,6,7,8]] over dim 1: MUL gave
0 0, PROD gave the correct 24 1680. The plausibly-named mode is the
broken one.
- Reduce's VARIANCE/STD default to correction=1, not 0 -- var of 1..4 came
back 1.66667 (unbiased), not the biased 1.25. That matches aten's
default, but correction=0 must be set explicitly.
- LayerNorm's third output is inv_var (1/sqrt(var+eps) = 0.89442 for
var=1.25, eps=1e-5), which is exactly aten's rstd -- no conversion.
Verification on 8x MTT S5000 / mudnn v3300, torch 2.10.0+cpu:
- 63 op/dtype cases vs CPU pass. The one fp16 log_softmax delta is a
rounding artifact, not an error: device and CPU are equidistant from an
fp64 reference (both 1.895e-3) and their gap (1.95e-3) is below the fp16
spacing at that magnitude (2.62e-3) -- they straddle the true value on
adjacent representable values.
- Autograd end-to-end matches CPU for layer_norm x/w/b across 2-D, 3-D and
two-axis normalized shapes, and for softmax/log_softmax backward.
- Every kernel beats the cpu_fallback it replaces at 2048x2048, from 2.8x
(layer_norm backward) to 132x (layer_norm forward).
- pytest tests/unit tests/integration/ops: 344 passed / 209 skipped /
3 xpassed / 0 failed, matching the pre-change baseline exactly.
- ruff 0.15.12 check + format clean.
Left on cpu_fallback deliberately: vector_norm with ord 0 or +-inf (aten
defines those as a nonzero-count and a max/min of |x|, which NORM does not
express), half_to_float softmax variants, and non-float layer_norm (aten
requires float32 mean/rstd even for a half input, which mudnn will not do).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Reduce guard only covered multi-dim reductions, where mudnn v3300 raises SIGFPE on an input that is a broadcast of one element. A single-dim reduce over the same input has a second, quieter failure: it intermittently writes only out[0] and leaves the rest of the output untouched, so the answer is whatever the caching allocator last left in that block. It surfaces as a wrong bias gradient. `linear(x, w, b).sum().backward()` reduces a grad_output that autograd builds as `ones.expand(...)`, whose storage is a single float, and the gradient came back as `[4, 34, 38, 42, 46]` -- element 0 correct, elements 1.. left over from a previous op. The same reduce on a materialized copy is always right, and mudnn is correct when probed standalone against a poisoned output buffer, so the trigger needs a recycled non-zero block underneath and does not reproduce outside the allocator. MudnnReduceWouldFault therefore becomes MudnnReduceNeedsContiguous, dropping the dim-count argument and keying only on "fully 0-strided". The materialization it forces reads one element, so the cost is negligible. Adds the bias-gradient path as a regression test, since a pure-tensor reduce cannot stand in for it -- the bug only shows when the output buffer holds recycled data. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zhaoyinglia
approved these changes
Aug 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Second mudnn batch: 13 reduction/normalization operators, taking MUSA coverage
90 -> 105 kernels. Same table-driven route as #60 -- 8 new templates, no framework
changes. Stacked on #60; review that one first.
New ops:
_log_softmax_softmax_backward_data_log_softmax_backward_dataamaxaminprod.dim_intany.dimall.dimvar.correctionstd.correctionlinalg_vector_normnative_layer_normnative_layer_norm_backwardSoftmax::RunBwd's operands are(gradInput, output, gradOutput), measured, and thenumbers match aten's
y*(g - sum(g*y))exactly.input_dtypeonly records what theforward input was; when it differs from the gradient's dtype aten wants a converting
backward, which mudnn does not express, so that combination goes to the host instead
of silently computing in the wrong precision.
The Reduce fix (2nd commit)
The pre-existing guard only covered multi-dim reductions, where mudnn v3300 raises
SIGFPE on an input that is a broadcast of one element. A single-dim reduce over the
same input has a second, quieter failure: it intermittently writes only
out[0]andleaves the rest of the output untouched, so the answer is whatever the caching
allocator last left in that block.
It surfaces as a wrong bias gradient.
linear(x, w, b).sum().backward()reduces agrad_outputthat autograd builds asones.expand(...), whose storage is a singlefloat, and the gradient came back as
[4, 34, 38, 42, 46]-- element 0 correct,elements 1.. left over from a previous op.
Two properties make this one worth reading carefully:
deliberately poisoned output buffer, mudnn is correct every time. The trigger needs a
recycled non-zero block underneath, so a C++ probe will tell you the library is fine.
The test added here goes through the bias-gradient path, which is what actually
leaves recycled data in the output block.
MudnnReduceWouldFaulttherefore becomesMudnnReduceNeedsContiguous, dropping thedim-count argument and keying only on "fully 0-strided". The materialization it forces
reads one element, so the cost is negligible.
Note the fix lands here rather than in #60: it rewrites the guard in five templates,
and three of them (
T_REDUCE_CORRECTION,T_REDUCE_DIMS_PLAIN,T_REDUCE_NORM) areintroduced by this PR, so it cannot be applied to the P1 tree.
Test plan
Environment:
torch==2.10.0+cpu, 8x MTT S5000, built with--no-build-isolation.keepdimbothways, negative dims, multi-dim reductions, and
correction=0/1for var/std.allocator, now returns
[4, 4, 4, 4, 4]every time. Added as a test.pytest tests/gives 482 passed / 209 skipped / 3 xpassed vs a478-passed baseline. The 4 failures are
tests/manual/metaxvendor testsAttributeError-ing on a MUSA box; the 9 errors are the known missing
transformers.ruff==0.15.12check + format clean.Regenerating with
python scripts/codegen_mudnn.pyat each of the two commitsreproduces the checked-in generated files exactly.
🤖 Generated with Claude Code