refactor: unify the processing of the attention mask - #1230
Merged
Conversation
This was referenced Jan 25, 2026
gianni-cor
pushed a commit
to gianni-cor/ggml
that referenced
this pull request
May 14, 2026
Adds Metal backend support for ops required by Wan video diffusion models on Apple Silicon. Without this patch the Metal backend crashes with: ggml_metal_op_encode_impl: error: unsupported op 'IM2COL_3D'. Changes ------- - IM2COL_3D Metal kernel implementation (f32 and f16 variants) - Pipeline getter and op handler for IM2COL_3D dispatch - PAD kernel updated to support non-zero left padding (lp0..lp3) - ggml_metal_kargs_pad extended with left-padding fields - supports_op for PAD relaxed to allow any padding values DIAG_MASK_INF intentionally omitted ----------------------------------- The original CLDawes patch (ggml-org/llama.cpp#16669) also added a Metal DIAG_MASK_INF kernel, but it is not included here because: - GGML maintainers are deprecating GGML_OP_DIAG_MASK_INF in favor of generic ops (see ggml-org/llama.cpp#17654) - leejet/stable-diffusion.cpp#1230 (merged Jan 2026) refactored the attention path to build explicit -INFINITY mask tensors and no longer calls ggml_diag_mask_inf in app code - Wan attention sites pass mask=nullptr and never request DIAG_MASK_INF, so the kernel would be dead code on this stack Adapted from CLDawes' patch (https://github.com/CLDawes/ggml/tree/patch-qwen-image), updated to use the current ggml_metal_pipeline_with_params API. Verified on Apple M3 Ultra: Wan2.1 1.3B txt2video runs end-to-end on Metal with the host process at 0-1.5% CPU throughout. Made-with: Cursor
aegioscy
added a commit
to tetherto/qvac-ext-ggml
that referenced
this pull request
Jun 29, 2026
Adds Metal backend support for ops required by Wan video diffusion models on Apple Silicon. Without this patch the Metal backend crashes with: ggml_metal_op_encode_impl: error: unsupported op 'IM2COL_3D'. Changes ------- - IM2COL_3D Metal kernel implementation (f32 and f16 variants) - Pipeline getter and op handler for IM2COL_3D dispatch - PAD kernel updated to support non-zero left padding (lp0..lp3) - ggml_metal_kargs_pad extended with left-padding fields - supports_op for PAD relaxed to allow any padding values DIAG_MASK_INF intentionally omitted ----------------------------------- The original CLDawes patch (ggml-org/llama.cpp#16669) also added a Metal DIAG_MASK_INF kernel, but it is not included here because: - GGML maintainers are deprecating GGML_OP_DIAG_MASK_INF in favor of generic ops (see ggml-org/llama.cpp#17654) - leejet/stable-diffusion.cpp#1230 (merged Jan 2026) refactored the attention path to build explicit -INFINITY mask tensors and no longer calls ggml_diag_mask_inf in app code - Wan attention sites pass mask=nullptr and never request DIAG_MASK_INF, so the kernel would be dead code on this stack Adapted from CLDawes' patch (https://github.com/CLDawes/ggml/tree/patch-qwen-image), updated to use the current ggml_metal_pipeline_with_params API. Verified on Apple M3 Ultra: Wan2.1 1.3B txt2video runs end-to-end on Metal with the host process at 0-1.5% CPU throughout. Made-with: Cursor
aegioscy
added a commit
to tetherto/qvac-ext-ggml
that referenced
this pull request
Aug 11, 2026
Adds Metal backend support for ops required by Wan video diffusion models on Apple Silicon. Without this patch the Metal backend crashes with: ggml_metal_op_encode_impl: error: unsupported op 'IM2COL_3D'. Changes ------- - IM2COL_3D Metal kernel implementation (f32 and f16 variants) - Pipeline getter and op handler for IM2COL_3D dispatch - PAD kernel updated to support non-zero left padding (lp0..lp3) - ggml_metal_kargs_pad extended with left-padding fields - supports_op for PAD relaxed to allow any padding values DIAG_MASK_INF intentionally omitted ----------------------------------- The original CLDawes patch (ggml-org/llama.cpp#16669) also added a Metal DIAG_MASK_INF kernel, but it is not included here because: - GGML maintainers are deprecating GGML_OP_DIAG_MASK_INF in favor of generic ops (see ggml-org/llama.cpp#17654) - leejet/stable-diffusion.cpp#1230 (merged Jan 2026) refactored the attention path to build explicit -INFINITY mask tensors and no longer calls ggml_diag_mask_inf in app code - Wan attention sites pass mask=nullptr and never request DIAG_MASK_INF, so the kernel would be dead code on this stack Adapted from CLDawes' patch (https://github.com/CLDawes/ggml/tree/patch-qwen-image), updated to use the current ggml_metal_pipeline_with_params API. Verified on Apple M3 Ultra: Wan2.1 1.3B txt2video runs end-to-end on Metal with the host process at 0-1.5% CPU throughout. Made-with: Cursor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix #857, #1040.