Skip to content

fix(rocm): keep the MiniMax H3 encoder conv3d decomposition on HIP >= 7.2 - #187

Merged
lstein merged 2 commits into
mainfrom
fix/h3-encoder-rocm-decomposition
Aug 30, 2026
Merged

fix(rocm): keep the MiniMax H3 encoder conv3d decomposition on HIP >= 7.2#187
lstein merged 2 commits into
mainfrom
fix/h3-encoder-rocm-decomposition

Conversation

@lstein

@lstein lstein commented Aug 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

#168 retired both ROCm conv2d decompositions on HIP ≥ 7.2 based on Wan VAE measurements (native conv3d fast there; the decomposition corrupted Wan decodes). The MiniMax H3 encoder was never re-timed on the new stack, and it should have been. Measured with scripts/benchmark_minimax_h3_vae_encode.py (from #162) on a W7900, torch 2.13.0+rocm7.2, HIP 7.2.53211, one 17-frame 768×448 reference chunk:

conv path fp32 fp16 autocast peak allocated
native MIOpen conv3d 208 s 222 s 9.2 GiB
conv2d decomposition 3.6 s 2.7 s 6.7 GiB

The native numbers are the same ~50× Im3d2Col fallback the decomposition was written for (the extra ~5 GiB is its column buffer). Warm-up equals steady state, so this is not kernel search. In practice: a 10 s Ref2VA reference encode of ~50 minutes instead of ~40 s, and the same penalty on every FL2VA keyframe encode.

Changes

  • invokeai/backend/minimax_h3/rocm_causal_conv3d.py: patch_minimax_h3_causal_conv3d_for_rocm applies on every HIP version (the Wan gate from fix: retire the ROCm conv3d decompositions on HIP >= 7.2 #168 is untouched). INVOKEAI_ROCM_CONV3D=native — the shared diagnostic override — still leaves the stock forward in place. Docstrings carry the measurements.
  • tests/backend/minimax_h3/test_rocm_causal_conv3d.py: gate test updated (HIP 7.2 and 7.1 both patch; non-HIP never; native opts out).

On the #168 corruption concern

The heisenbug that drove the retirement was in Wan decodes on non-square canvases. The H3 encoder ran the decomposition on a non-square 768×448 clip in the measurement above with finite output and 1.8e-3 relative drift between its fp32 and fp16 encodes (the expected autocast spread). A direct native-vs-decomposed numeric comparison on the rig is still worth running before merge; the benchmark script's --conv3d both (in #162) reports it.

Testing

  • Decomposition + Wan tests pass (15); ruff 0.11.2 clean.
  • Rig, --conv3d both at 256×256 (1×1 tiles, fp32, W7900 / HIP 7.2.53211): native 26.7 s/chunk, peak 9.11 GiB; decomposed 0.45 s/chunk, peak 6.61 GiB — 59× faster. Numeric drift decomposed vs native: rel-norm 3.4e-5, max 0.002 on rows with std 1.05, all finite. That is fp32 summation-order noise pushed through the fp16-rounding step of the keyframe recipe (an fp16 ulp at magnitude ~2 is 0.002) — no sign of the allocator-state corruption seen in Wan decodes.
  • Rig, same check non-square (384×256, 1×2 tiles — the canvas class where the Wan decomposition tore): native 51.6 s/chunk, decomposed 0.90 s/chunk (57×); drift rel-norm 3.0e-5, max 0.002, all finite. Same clean result.

🤖 Generated with Claude Code

https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A

… 7.2

#168 retired both conv2d decompositions on HIP >= 7.2 from Wan measurements
(native conv3d fast there, decomposition corrupting Wan decodes); the H3
encoder was never re-timed. It should have been: on a W7900 with torch
2.13.0+rocm7.2 (HIP 7.2.53211) one 17-frame 768x448 reference chunk takes
208 s fp32 / 222 s fp16-autocast on native MIOpen conv3d, peak 9.2 GiB (the
Im3d2Col column buffer), against 3.6 s / 2.7 s decomposed, peak 6.7 GiB.
That is the same ~50x Im3d2Col fallback the decomposition exists for - a
10 s Ref2VA reference encode of ~50 minutes instead of ~40 s.

Drop the HIP-version gate for the H3 patch only; the Wan gate is untouched.
INVOKEAI_ROCM_CONV3D=native (the shared diagnostic override) still leaves
the stock forward in place on any HIP version. Gate test updated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A
lstein added a commit that referenced this pull request Aug 30, 2026
…umerically

Runs every mode on the stock conv3d and then on the conv2d decomposition
in one process and reports the drift between the two paths, so a partial
revert of the HIP >= 7.2 retirement (#187) can be checked for the
allocator-state corruption that motivated it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A
@lstein
lstein enabled auto-merge August 30, 2026 15:25
@lstein
lstein merged commit a0e0b86 into main Aug 30, 2026
19 checks passed
@lstein
lstein deleted the fix/h3-encoder-rocm-decomposition branch August 30, 2026 15:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant