fix(rocm): keep the MiniMax H3 encoder conv3d decomposition on HIP >= 7.2 - #187
Merged
Conversation
… 7.2 #168 retired both conv2d decompositions on HIP >= 7.2 from Wan measurements (native conv3d fast there, decomposition corrupting Wan decodes); the H3 encoder was never re-timed. It should have been: on a W7900 with torch 2.13.0+rocm7.2 (HIP 7.2.53211) one 17-frame 768x448 reference chunk takes 208 s fp32 / 222 s fp16-autocast on native MIOpen conv3d, peak 9.2 GiB (the Im3d2Col column buffer), against 3.6 s / 2.7 s decomposed, peak 6.7 GiB. That is the same ~50x Im3d2Col fallback the decomposition exists for - a 10 s Ref2VA reference encode of ~50 minutes instead of ~40 s. Drop the HIP-version gate for the H3 patch only; the Wan gate is untouched. INVOKEAI_ROCM_CONV3D=native (the shared diagnostic override) still leaves the stock forward in place on any HIP version. Gate test updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A
lstein
requested review from
JPPhoto,
Pfannkuchensack and
blessedcoolant
as code owners
August 30, 2026 14:02
lstein
added a commit
that referenced
this pull request
Aug 30, 2026
…umerically Runs every mode on the stock conv3d and then on the conv2d decomposition in one process and reports the drift between the two paths, so a partial revert of the HIP >= 7.2 retirement (#187) can be checked for the allocator-state corruption that motivated it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A
lstein
enabled auto-merge
August 30, 2026 15:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#168 retired both ROCm conv2d decompositions on HIP ≥ 7.2 based on Wan VAE measurements (native conv3d fast there; the decomposition corrupted Wan decodes). The MiniMax H3 encoder was never re-timed on the new stack, and it should have been. Measured with
scripts/benchmark_minimax_h3_vae_encode.py(from #162) on a W7900, torch 2.13.0+rocm7.2, HIP 7.2.53211, one 17-frame 768×448 reference chunk:The native numbers are the same ~50×
Im3d2Colfallback the decomposition was written for (the extra ~5 GiB is its column buffer). Warm-up equals steady state, so this is not kernel search. In practice: a 10 s Ref2VA reference encode of ~50 minutes instead of ~40 s, and the same penalty on every FL2VA keyframe encode.Changes
invokeai/backend/minimax_h3/rocm_causal_conv3d.py:patch_minimax_h3_causal_conv3d_for_rocmapplies on every HIP version (the Wan gate from fix: retire the ROCm conv3d decompositions on HIP >= 7.2 #168 is untouched).INVOKEAI_ROCM_CONV3D=native— the shared diagnostic override — still leaves the stock forward in place. Docstrings carry the measurements.tests/backend/minimax_h3/test_rocm_causal_conv3d.py: gate test updated (HIP 7.2 and 7.1 both patch; non-HIP never;nativeopts out).On the #168 corruption concern
The heisenbug that drove the retirement was in Wan decodes on non-square canvases. The H3 encoder ran the decomposition on a non-square 768×448 clip in the measurement above with finite output and 1.8e-3 relative drift between its fp32 and fp16 encodes (the expected autocast spread). A direct native-vs-decomposed numeric comparison on the rig is still worth running before merge; the benchmark script's
--conv3d both(in #162) reports it.Testing
--conv3d bothat 256×256 (1×1 tiles, fp32, W7900 / HIP 7.2.53211): native 26.7 s/chunk, peak 9.11 GiB; decomposed 0.45 s/chunk, peak 6.61 GiB — 59× faster. Numeric drift decomposed vs native: rel-norm 3.4e-5, max 0.002 on rows with std 1.05, all finite. That is fp32 summation-order noise pushed through the fp16-rounding step of the keyframe recipe (an fp16 ulp at magnitude ~2 is 0.002) — no sign of the allocator-state corruption seen in Wan decodes.🤖 Generated with Claude Code
https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A