Skip to content

feat(musa): 13 reduction/normalization mudnn kernels - #61

Merged
zhaoyinglia merged 2 commits into
feat/musa-p1-hot-pathfrom
feat/musa-p1-training-ops
Aug 6, 2026
Merged

feat(musa): 13 reduction/normalization mudnn kernels#61
zhaoyinglia merged 2 commits into
feat/musa-p1-hot-pathfrom
feat/musa-p1-training-ops

Conversation

@lvyufeng

@lvyufeng lvyufeng commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

Second mudnn batch: 13 reduction/normalization operators, taking MUSA coverage
90 -> 105 kernels. Same table-driven route as #60 -- 8 new templates, no framework
changes. Stacked on #60; review that one first.

New ops:

  • softmax family: _log_softmax _softmax_backward_data _log_softmax_backward_data
  • reductions: amax amin prod.dim_int any.dim all.dim
  • statistics: var.correction std.correction linalg_vector_norm
  • normalization: native_layer_norm native_layer_norm_backward

Softmax::RunBwd's operands are (gradInput, output, gradOutput), measured, and the
numbers match aten's y*(g - sum(g*y)) exactly. input_dtype only records what the
forward input was; when it differs from the gradient's dtype aten wants a converting
backward, which mudnn does not express, so that combination goes to the host instead
of silently computing in the wrong precision.

The Reduce fix (2nd commit)

The pre-existing guard only covered multi-dim reductions, where mudnn v3300 raises
SIGFPE on an input that is a broadcast of one element. A single-dim reduce over the
same input has a second, quieter failure: it intermittently writes only out[0] and
leaves the rest of the output untouched, so the answer is whatever the caching
allocator last left in that block.

It surfaces as a wrong bias gradient. linear(x, w, b).sum().backward() reduces a
grad_output that autograd builds as ones.expand(...), whose storage is a single
float, and the gradient came back as [4, 34, 38, 42, 46] -- element 0 correct,
elements 1.. left over from a previous op.

Two properties make this one worth reading carefully:

  • It does not reproduce outside the allocator. Probed standalone against a
    deliberately poisoned output buffer, mudnn is correct every time. The trigger needs a
    recycled non-zero block underneath, so a C++ probe will tell you the library is fine.
  • A pure-tensor reduce cannot serve as the regression test for the same reason.
    The test added here goes through the bias-gradient path, which is what actually
    leaves recycled data in the output block.

MudnnReduceWouldFault therefore becomes MudnnReduceNeedsContiguous, dropping the
dim-count argument and keying only on "fully 0-strided". The materialization it forces
reads one element, so the cost is negligible.

Note the fix lands here rather than in #60: it rewrites the guard in five templates,
and three of them (T_REDUCE_CORRECTION, T_REDUCE_DIMS_PLAIN, T_REDUCE_NORM) are
introduced by this PR, so it cannot be applied to the P1 tree.

Test plan

Environment: torch==2.10.0+cpu, 8x MTT S5000, built with --no-build-isolation.

  • Numerics vs CPU for all 13 ops across float32/float16, including keepdim both
    ways, negative dims, multi-dim reductions, and correction=0/1 for var/std.
  • Reduce regression: the bias-gradient path, repeated after ops that dirty the
    allocator, now returns [4, 4, 4, 4, 4] every time. Added as a test.
  • Suite: pytest tests/ gives 482 passed / 209 skipped / 3 xpassed vs a
    478-passed baseline. The 4 failures are tests/manual/metax vendor tests
    AttributeError-ing on a MUSA box; the 9 errors are the known missing transformers.
  • ruff==0.15.12 check + format clean.
  • Only MUSA-only files change, so cuda/gcu are unaffected by construction.

Regenerating with python scripts/codegen_mudnn.py at each of the two commits
reproduces the checked-in generated files exactly.

🤖 Generated with Claude Code

lvyufeng and others added 2 commits August 4, 2026 18:07
Extends the table-driven mudnn codegen from 90 to 103 kernels (105 m.impl
lines), covering the reductions and normalization a training step leans on
after the P1 elementwise batch.

New categories, each probed against CPU before entering the table:

  reduce_dims_plain (2)  amax, amin.
  reduce_prod (1)        prod.dim_int.
  reduce_bool (2)        any.dim, all.dim -- Reduce::AND/OR take a BOOL
                         input and write BOOL; aten's "nonzero" semantics
                         become a `!= 0` cast first.
  reduce_correction (2)  var.correction, std.correction.
  reduce_norm (1)        linalg_vector_norm via SetNormOrd.
  softmax_bwd (2)        _softmax_backward_data,
                         _log_softmax_backward_data. RunBwd's operands are
                         (gradInput, output, gradOutput), matching aten's
                         y*(g - sum(g*y)) exactly.
  layer_norm (2)         native_layer_norm and its backward. Run emits
                         (out, mean, inv_var) in one call and RunBwd emits
                         (dX, dGamma, dBeta), so both map 1:1 onto aten's
                         tuples.

_log_softmax needed no new template -- the existing softmax forward already
carries the mode, so it is one table entry.

Three measured facts the headers do not give away:

  - Reduce::Mode::MUL returns SUCCESS and writes all zeros. PROD is the
    working product mode. Reduced [[1,2,3,4],[5,6,7,8]] over dim 1: MUL gave
    0 0, PROD gave the correct 24 1680. The plausibly-named mode is the
    broken one.
  - Reduce's VARIANCE/STD default to correction=1, not 0 -- var of 1..4 came
    back 1.66667 (unbiased), not the biased 1.25. That matches aten's
    default, but correction=0 must be set explicitly.
  - LayerNorm's third output is inv_var (1/sqrt(var+eps) = 0.89442 for
    var=1.25, eps=1e-5), which is exactly aten's rstd -- no conversion.

Verification on 8x MTT S5000 / mudnn v3300, torch 2.10.0+cpu:
  - 63 op/dtype cases vs CPU pass. The one fp16 log_softmax delta is a
    rounding artifact, not an error: device and CPU are equidistant from an
    fp64 reference (both 1.895e-3) and their gap (1.95e-3) is below the fp16
    spacing at that magnitude (2.62e-3) -- they straddle the true value on
    adjacent representable values.
  - Autograd end-to-end matches CPU for layer_norm x/w/b across 2-D, 3-D and
    two-axis normalized shapes, and for softmax/log_softmax backward.
  - Every kernel beats the cpu_fallback it replaces at 2048x2048, from 2.8x
    (layer_norm backward) to 132x (layer_norm forward).
  - pytest tests/unit tests/integration/ops: 344 passed / 209 skipped /
    3 xpassed / 0 failed, matching the pre-change baseline exactly.
  - ruff 0.15.12 check + format clean.

Left on cpu_fallback deliberately: vector_norm with ord 0 or +-inf (aten
defines those as a nonzero-count and a max/min of |x|, which NORM does not
express), half_to_float softmax variants, and non-float layer_norm (aten
requires float32 mean/rstd even for a half input, which mudnn will not do).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Reduce guard only covered multi-dim reductions, where mudnn v3300 raises
SIGFPE on an input that is a broadcast of one element. A single-dim reduce over
the same input has a second, quieter failure: it intermittently writes only
out[0] and leaves the rest of the output untouched, so the answer is whatever
the caching allocator last left in that block.

It surfaces as a wrong bias gradient. `linear(x, w, b).sum().backward()` reduces
a grad_output that autograd builds as `ones.expand(...)`, whose storage is a
single float, and the gradient came back as `[4, 34, 38, 42, 46]` -- element 0
correct, elements 1.. left over from a previous op. The same reduce on a
materialized copy is always right, and mudnn is correct when probed standalone
against a poisoned output buffer, so the trigger needs a recycled non-zero block
underneath and does not reproduce outside the allocator.

MudnnReduceWouldFault therefore becomes MudnnReduceNeedsContiguous, dropping the
dim-count argument and keying only on "fully 0-strided". The materialization it
forces reads one element, so the cost is negligible.

Adds the bias-gradient path as a regression test, since a pure-tensor reduce
cannot stand in for it -- the bug only shows when the output buffer holds
recycled data.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants