feat(backend): MiniMax H3 Ref2VA generation — packing, conditioning, invocations - #162
Open
lstein wants to merge 8 commits into
Open
feat(backend): MiniMax H3 Ref2VA generation — packing, conditioning, invocations#162lstein wants to merge 8 commits into
lstein wants to merge 8 commits into
Conversation
lstein
requested review from
JPPhoto,
Pfannkuchensack and
blessedcoolant
as code owners
August 27, 2026 12:29
github-actions
Bot
added
python
python-tests
backend
services
frontend
invocations
labels
lstein
force-pushed
the
feat/minimax-h3-ref2va-backend
branch
2 times, most recently
from
August 27, 2026 13:14
9ba19bf to
66d3950
Compare
…invocations Ports the Ref2VA task from the merged diffusers MiniMax-H3 integration (PR #14355, commit f53d552036a0d1bd5570782a39cd40cfabf112bc), matched operation for operation: Backend (invokeai/backend/minimax_h3/): - packing: build_ref2va_packed_sequence - the [text | reference blocks | target audio | target video] layout with the shared per-reference rotary clock (image +1.0; audio +latents on the target width grid; video soundtrack-before-frames on its own grid, advancing by max(audio span, SEQUENTIALLY-summed video span)); reference caps and kind validation. - sampling: build_ref2va_denoise_state - per-visual-reference condition noise draws (packed order, before the generated draws), visual rows noise-augmented to t=0.999, soundtrack rows concatenated CLEAN (pinned at timestep 1.0). FL2VA's build_denoise_state is untouched and now golden-frozen. - reference_conditioning (new): normalization (image 2048/match short edge; video ffmpeg-fps-filter resample to 24 fps, truncation, own-aspect canvas, with a bit-identical streaming variant; audio truncate-at-native -rate-then-resample) and encoding (keyframe recipe; video refs snapped DOWN to 17n+5, chunked host-to-device encode bit-identical to vae._encode; soundtracks take the audio posterior MEAN). - audio_resample (new): dependency-free port of torchaudio's windowed-sinc resample (BSD-2), verified bit-exact against torchaudio 2.11 and pinned by baked golden vectors. - text_conditioning: encode_prompt_ref2va - per-modality-numbered labels in request order (<Audio j> before its video's <Video k>), 2 fps video sampling with round-half-even block timestamps, per-modality-batched vision tensors, prompt last. encode_prompt (FL2VA) untouched. App: - extract_audio_pcm gains float_pcm=True (pcm_f32le; upstream conditions on float samples, not int16-quantized ones). - New nodes: minimax_h3_image_reference / minimax_h3_video_reference (ordered descriptors: conditioning selector video_audio/video/audio, trim, image detail max/match) and minimax_h3_reference_conditioning (all VAE work; decoder fully drained before GPU work; encode working- memory hints). - minimax_h3_text_encoder 1.1.0: references + num_frames inputs, mutually exclusive with keyframes; stores the per-reference signature in the conditioning info. - minimax_h3_denoise 1.4.0: reference_conditioning input; signature and num_frames cross-checks against the prompt side; task-variant guard (references on FL2VA weights, or Ref2VA weights without references, fail fast). - minimax_h3_model_loader 1.4.0: stamps the transformer's task variant onto MiniMaxH3TransformerField (replaces the interim hard rejection). - minimax_h3_frame_conditioning 1.0.1: passes the encode working-memory hint it previously omitted. - Bundled H3 workflows re-stamped to the new node versions; openapi.json + schema.ts regenerated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A
lstein
force-pushed
the
feat/minimax-h3-ref2va-backend
branch
from
August 27, 2026 13:50
66d3950 to
7709468
Compare
…ices Reference and keyframe encodes ran the 180M-parameter 3D-conv encoder in full fp32 on every platform, with no autocast at all - unlike decode, which main runs under fp16 autocast over the fp32-pinned weights (PR #156). A video reference is encoded one 17-frame chunk at a time, spatially tiled, so a 10 s reference is 14 full encoder passes and three references are 42. `vae_encode_autocast(device)` (keyframe_conditioning.py) applies the decode recipe - float16 autocast, cache_enabled=False - on `device.type == "cuda"` (NVIDIA and ROCm alike) and is a no-op on MPS/CPU. It wraps the three encode sites; the moments are cast back to float32 before the seeded posterior sample so the fp16-rounding contract sees the same dtypes as before. Measured on a 5060 Ti, one 17-frame 768x448 chunk (8 tiles): 4.6 s -> 3.2 s with TF32 convs (NVIDIA default), 9.2 s -> 3.2 s without TF32 (ROCm has no TF32 path); peak VRAM 4.10 -> 3.57 GiB. The working-memory estimator keys off the fp32 weights, so the reservation only gains headroom. Numerics: 1.5e-3 relative RMS drift from the fp32 encode. Visual conditioning rows are noise-augmented to t = 0.999 before the transformer sees them, so the clean encode is 0.1% of each row and the drift lands at ~1e-6 of the row magnitude. Chunked encode stays bit-identical to `vae._encode` under the same precision (verified on the released VAE). ROCm wall-time not yet measured on the rig (encoder runs the conv2d decomposition there on HIP < 7.2). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A
Times the real `encode_reference_video` path both ways on a synthetic clip (--vae folder or --root discovery), applying the loader's ROCm conv3d handling and printing which conv path is active; reports s/chunk, peak memory, the numeric drift between the two encodes, and the projected cost of a 10 s reference. For measuring the autocast recipe on the ROCm rig. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A
Each mode was silent until its warm-up and every timed repeat finished, which on a slow fp32 path reads as a hang. Print the warm-up and each repeat as they complete. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A
…nchmark Matches the server startup default; a standalone script otherwise spends its warm-up in MIOpen's exhaustive kernel search on ROCm. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A
…e benchmark Lets the ROCm conv2d decomposition be A/B'd against native MIOpen conv3d on the same HIP version; the loader retired the decomposition on HIP >= 7.2 from Wan measurements and the H3 encoder was never re-timed there. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A
…umerically Runs every mode on the stock conv3d and then on the conv2d decomposition in one process and reports the drift between the two paths, so a partial revert of the HIP >= 7.2 retirement (#187) can be checked for the allocator-state corruption that motivated it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Second of three stacked PRs (on #161): the Ref2VA numerics port and graph nodes. Ported operation-for-operation from the merged diffusers MiniMax-H3 integration (PR huggingface/diffusers#14355 @
f53d552), cross-checked against ComfyUI.Backend (
invokeai/backend/minimax_h3/)packing.build_ref2va_packed_sequence: the[text | reference blocks | target audio | target video]layout — shared per-reference rotary clock (image +1.0 on its own grid; standalone audio on the target width grid; a video's soundtrack rows immediately before its frames on its own grid, advancing bymax(audio span, sequentially-summed video span)).sampling.build_ref2va_denoise_state: per-visual-reference condition noise draws in packed order before the generated draws; visual rows noise-augmented to t=0.999; soundtrack rows concatenated clean (pinned at timestep 1.0). FL2VA'sbuild_denoise_stateis untouched and now golden-frozen (bit-exact checksum test).reference_conditioning(new): normalization (image 2048/match short edge; video ffmpeg-fps-filter resample to 24 fps → truncate → own-aspect canvas, with a bit-identical streaming variant; audio truncate-at-native-rate-then-resample) and encoding (keyframe recipe; video refs snapped down to 17n+5, chunked host→device encode bit-identical tovae._encode; soundtracks take the audio posterior mean).audio_resample(new): dependency-free port of torchaudio's windowed-sinc resample (BSD-2), verified bit-exact against torchaudio 2.11 and pinned by baked golden vectors.text_conditioning.encode_prompt_ref2va: per-modality-numbered labels in request order (<Audio j>:before its video's<Video k>:), 2 fps sampling with round-half-even timestamps, per-modality vision batching, prompt last.App
extract_audio_pcm(float_pcm=True):pcm_f32ledecode pinned to-map 0:a:0(upstream conditions on float samples of the first audio stream; probe and decode must describe the same stream — regression-tested with a two-track fixture).minimax_h3_image_reference/minimax_h3_video_reference(ordered descriptors: conditioningvideo_audio|video|audio, trim, image detailmax|match) andminimax_h3_reference_conditioning(all VAE work; decoder fully drained before GPU work; encode working-memory hints sized to the 17-frame streaming chunk).minimax_h3_text_encoder1.1.0 (references + num_frames; signature stored in conditioning info),minimax_h3_denoise1.4.0 (reference input; signature/num_frames/canvas cross-checks; task-variant guard with actionable messages),minimax_h3_model_loader1.4.0 (stamps the transformer's task variant),minimax_h3_frame_conditioning1.0.1 (adds the encode working-memory hint it omitted).openapi.json/schema.tsregenerated.Testing
tests/backend/minimax_h3(147), invocations, model-manager, workflow registry.torch.equalto whole-stack_encodefor 6 frame counts, and torchaudio bit-parity live. All confirmed findings fixed in this PR.Stack
🤖 Generated with Claude Code
https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A