Skip two SDPA fallback tests on ROCm, where a stale AOTInductor kernel cache breaks them - #23383
Merged
shoumikhin merged 1 commit intoOct 3, 2026
Merged
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23383
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 2dc206a with merge base 0b3d26d ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
…them
The tests for SDPA calls the Triton kernel does not accept export each call
through the regular AOTInductor lowering. On ROCm, two of those exports fail
inside AOTInductor's C++ wrapper codegen, after the replacement pass has
correctly left the call alone:
AssertionError: ([torch.bfloat16, torch.float32, torch.float32, Integer],
['in_ptr0', 'out_ptr0', 'xnumel'])
The calls are SDPA with an additive float mask and SDPA with dropout. A
generated kernel's argument types and names differ in length. The same
exports pass on CUDA, and the other regular-lowering cases pass on ROCm.
Skip just those two tests on ROCm, as the file already does for split-K.
The root cause is tracked in pytorch/pytorch#199619, and the skip carries a
TODO to remove it once that is fixed.
shoumikhin
force-pushed
the
rocm-skip-sdpa-regular-lowering
branch
from
October 3, 2026 02:09
bc0338b to
2dc206a
Compare
Gasoonjia
approved these changes
Oct 3, 2026
shoumikhin
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is wrong today
unittest-rocm-gfx950-rocm7.2fails on main since #23349. Two of its new tests fail on ROCm:test_float_mask_is_not_replacedtest_dropout_is_not_replacedBoth export an SDPA call the Triton kernel does not accept, so the replacement pass leaves it to the regular AOTInductor lowering. The pass does that correctly, and the log shows "Skipping SDPA replacement" and "Replaced 0 nodes". The export then fails later, inside AOTInductor's C++ wrapper codegen:
The cause is in AOTInductor, not in these exports. AOTInductor saves each compiled Triton kernel's launch data in one cache for the whole process, keyed only by the kernel name. When a later compile reuses a kernel with the same source, it does not save that data again. So if a different kernel with the same name was compiled in between, its launch data stays in the cache. These tests run in one process after other tests that compile a kernel with the same name, so they get the wrong launch data.
On ROCm, Inductor gives that earlier kernel and these tests' kernel the same name. On CUDA the names differ, which is why these tests pass there. When two models do produce the same kernel name, the same error happens on CUDA too: compiling model B, then model A, then model B again gives exactly this assertion with torch 2.14 on a CUDA GPU.
What this change does
Skips just those two tests on ROCm. This matches how the same file already skips its split-K tests on ROCm. The behavior on CUDA does not change.
The root cause is tracked in pytorch/pytorch#199619. The skip carries a TODO to remove it once that is fixed.
What was tested
torch.version.hipset, both tests skip.unittest-rocm-gfx950-rocm7.2passes: 270 passed, 8 skipped, 0 failed, with both tests skipped. On main the same job fails these two tests.