QVAC-18026: Wan video Metal support — bump ggml to IM2COL_3D + PAD branch, add test infra - #4
Closed
aegioscy wants to merge 3 commits into
Closed
QVAC-18026: Wan video Metal support — bump ggml to IM2COL_3D + PAD branch, add test infra#4aegioscy wants to merge 3 commits into
aegioscy wants to merge 3 commits into
Conversation
…video generation Made-with: Cursor
Bumps the ggml submodule to tetherto/qvac-ext-ggml@feature/metal-pr-16669-clean,
which adds the two Metal ops actually required by Wan video diffusion:
- IM2COL_3D (3D im2col for Wan VAE/transformer 3D convs)
- PAD left-padding support (Wan input preprocessing)
Without this, the Metal backend crashes during Wan inference with:
ggml_metal_op_encode_impl: error: unsupported op 'IM2COL_3D'
Why no DIAG_MASK_INF Metal kernel
---------------------------------
The original CLDawes patch (ggml-org/llama.cpp#16669) bundled a Metal
DIAG_MASK_INF kernel along with IM2COL_3D + PAD. It is intentionally
not pulled in here because:
- GGML_OP_DIAG_MASK_INF is being deprecated upstream in favor of
generic ops (ggml-org/llama.cpp#17654), and ggerganov has steered
contributors away from new DIAG_MASK_INF backend kernels
(ggml-org/ggml#1395).
- leejet#1230 (already in our base, merged
Jan 2026) refactored every CLIP / SD / MMDiT / RoPE / VAE
attention site to build explicit -INFINITY mask tensors and
removed all `diag_mask_inf=true` calls. There are zero
ggml_diag_mask_inf call sites left in app code.
- All ggml_ext_attention_ext(...) calls in src/wan.hpp pass
mask = nullptr, so a Metal DIAG_MASK_INF kernel would be dead
code on this stack.
Other changes
-------------
- Point .gitmodules at tetherto/qvac-ext-ggml so clones can fetch
the patched submodule directly.
- Ignore test-wan/output/, test-wan/models/, and test-wan/cache/
so generated videos, downloaded weights, and model caches are
not accidentally committed.
Verified on Apple M3 Ultra: Wan2.1 1.3B txt2video runs end-to-end
on Metal with the host process at 0-1.5% CPU throughout.
Made-with: Cursor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Enables Metal GPU acceleration for Wan2.1 / Wan2.2 video generation on Apple Silicon by bumping the ggml submodule to a patched branch that adds the two Metal ops Wan actually needs, plus the test infrastructure to exercise it from `sd-cli`.
Changes
Why no Metal DIAG_MASK_INF kernel
The earlier iteration of this PR also pulled in a Metal `DIAG_MASK_INF` kernel (from CLDawes' ggml-org/llama.cpp#16669). That has been intentionally dropped from the new ggml branch because:
Net effect: smaller, future-proof patch.
Build (Apple Silicon)
```bash
make build-metal
```
This produces `build/bin/sd-cli` with Metal enabled and the patched ggml ops linked in.
Test (Wan2.1 1.3B)
```bash
cd test-wan
./test-wan-models.sh wan2-1-t2v-small
```
Expected on Apple M3 Ultra: ~432 s for a 33-frame 832×480 clip; host process at 0–1.5 % CPU throughout (no CPU fallback for ops in the hot path).
Dependencies
All commits are signed with an SSH ED25519 key.
Made with Cursor