Skip to content

Bump AITER to 4fa508ef - #31

Draft
lauri9 wants to merge 5 commits into
mainfrom
feature/979-bump-aiter
Draft

Bump AITER to 4fa508ef#31
lauri9 wants to merge 5 commits into
mainfrom
feature/979-bump-aiter

Conversation

@lauri9

@lauri9 lauri9 commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Bump AITER to 4fa508ef and refresh CI patches for PyTorch 2.9 compatibility.

Closes #979

Background

Part of plan to remove triton attention.

Upstream AITER is bumped forward to 4fa508ef, which brings in the MHAv4 API and changes file layout and type annotations.

On the current PyTorch 2.9 stack, AITER's nested _validate_cu helper in aiter/ops/mha.py uses PEP 604 (X | None) syntax that causes a Dynamo graph break under torch.compile. This is resolved in the PR with a temporary patch.

Goals

  • Update AITER to add support for mha_v4 API
  • Keep the FP8 sparse ASM kernel patch applying cleanly against the bumped AITER revision.
  • Restore torch.compile compatibility for AITER MHA on PyTorch 2.9 with temporary patch

Tasks

  • Bump AITER_COMMIT to 4fa508ef2935110ff99adf2743ea93807dbd9c67 in docker/Dockerfile.ci.
  • Regenerate patches/aiter_fp8_sparse_asm_kernel.patch for the new AITER layout (retargeted hunks, updated mutates_args typing).
  • Add patches/aiter_mha_pep604_dynamo.patch and apply it in the CI Dockerfile to replace PEP 604 annotations with Optional[...] in _validate_cu.

Tests

  • CI core image build succeeds with bumped AITER and refreshed patches.
  • Validate performance and visual quality on gfx942 & gfx950 - CI run
  • Validate inference / torch.compile paths that exercise AITER MHA on PyTorch 2.9.

Performance on MI350X - no significant changes

model AITER bump Baseline Ratio
flux.usp 0.84 0.85 101.0%
flux.single_gpu 2.38 2.41 101.3%
flux.usp_2k 2.22 2.25 101.1%
flux.single_gpu_2k 12.03 12.16 101.1%
flux2.default 4.48 4.52 100.8%
flux2.quantgemm.gfx950 2.45 2.47 100.7%
flux2.quantgemm_sageattn.gfx950 2.57 2.61 101.6%
flux2.ti2i_2k 12.57 12.53 99.7%
flux2.t2i_1k 2.90 2.91 100.3%
flux2.t2i_2k 10.75 10.72 99.8%
flux2.t-multi-i2i_1k 5.99 5.95 99.3%
flux2.t-multi-i2i_2k 15.04 15.02 99.9%
flux2_klein.default 0.49 0.49 100.2%
flux_kontext.default 1.50 1.50 100.2%
hunyuanvideo.default 104.91 104.06 99.2%
hunyuanvideo.fp8attn 66.04 66.39 100.5%
hunyuanvideo.sageattn.gfx950 87.62 88.00 100.4%
hunyuanvideo_1_5.default 116.42 116.71 100.2%
hunyuanvideo_1_5.distilled.gfx950 54.80 55.18 100.7%
hunyuanvideo_1_5.distilled_sparse.gfx950 45.52 46.04 101.2%
ltx2_3.default 16.14 16.17 100.2%
qwen_image.default 4.83 4.89 101.2%
qwen_image_edit.default 14.97 15.12 101.1%
stablediffusion_3_5.default 0.81 0.81 100.1%
wan2_1.default 73.83 74.54 101.0%
wan2_2.default 75.72 75.99 100.4%
wan2_2.quantgemm.gfx950 67.83 68.76 101.4%
wan2_2.quantgemm_fp8attn.gfx950 40.85 41.17 100.8%
wan2_2.quantgemm_sageattn.gfx950 53.30 54.72 102.7%
wan2_2.quantgemm_spargeattn.gfx950 30.79 31.03 100.8%
z_image.default 2.65 2.67 101.1%

Performance on MI300X - no significant changes

model AITER bump Baseline Ratio
flux.usp 1.29 1.29 99.6%
flux.single_gpu 3.80 3.81 100.3%
flux.usp_2k 3.77 3.78 100.1%
flux.single_gpu_2k 20.02 20.03 100.1%
flux2.default 7.67 7.74 100.9%
flux2.quantgemm.gfx942 5.47 5.51 100.7%
flux2.quantgemm_sageattn.gfx942 5.74 5.78 100.7%
flux2.ti2i_2k 22.54 22.58 100.2%
flux2.t2i_1k 5.04 5.05 100.2%
flux2.t2i_2k 18.58 18.59 100.1%
flux2.t-multi-i2i_1k 9.99 10.00 100.1%
flux2.t-multi-i2i_2k 25.60 25.60 100.0%
flux2_klein.default 0.79 0.79 99.1%
flux_kontext.default 2.46 2.46 100.1%
hunyuanvideo.default 179.92 178.85 99.4%
hunyuanvideo.fp8attn 129.91 131.46 101.2%
hunyuanvideo.sageattn.gfx942 147.56 149.88 101.6%
hunyuanvideo_1_5.default 218.58 218.53 100.0%
hunyuanvideo_1_5.distilled.gfx942 105.35 107.52 102.1%
hunyuanvideo_1_5.distilled_sparse.gfx942 81.15 81.74 100.7%
ltx2_3.default 27.47 27.80 101.2%
qwen_image.default 8.18 8.16 99.8%
qwen_image_edit.default 22.60 22.57 99.9%
stablediffusion_3_5.default 1.25 1.26 100.3%
wan2_1.default 133.40 134.02 100.5%
wan2_2.default 133.30 133.80 100.4%
wan2_2.quantgemm.gfx942 121.27 121.78 100.4%
wan2_2.quantgemm_fp8attn.gfx942 88.18 88.46 100.3%
wan2_2.quantgemm_sageattn.gfx942 99.57 102.01 102.4%
wan2_2.quantgemm_spargeattn.gfx942 73.92 75.18 101.7%
z_image.default 4.52 4.53 100.2%

Other

  • The Dynamo workaround patch is intentionally temporary and documented inline in docker/Dockerfile.ci.

lauri9 and others added 3 commits August 25, 2026 12:54
Track upstream AITER at the commit validated in the aiter-bump test branch.

Co-authored-by: Cursor <cursoragent@cursor.com>
Regenerate against 4fa508ef: drop stale Union import hunk, use Optional[...]
annotations for infer_schema, and retarget hunks to the new file layout.

Co-authored-by: Cursor <cursoragent@cursor.com>
Nested _validate_cu in aiter/ops/mha.py breaks torch.compile when annotated
with PEP 604 syntax. Apply aiter_mha_pep604_dynamo.patch in Dockerfile.ci;
remove it after PyTorch is bumped above 2.9.

Co-authored-by: Cursor <cursoragent@cursor.com>
@lauri9
lauri9 marked this pull request as ready for review August 27, 2026 08:57
@lauri9
lauri9 requested review from a team as code owners August 27, 2026 08:57
@lauri9
lauri9 marked this pull request as draft August 28, 2026 10:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant