Skip to content

HIP: branch-free SWAR for __vsub4 / __vcmpne4 / __vcmpeq4 - #28616

Draft
SimonTeixidor wants to merge 1 commit into
ggml-org:masterfrom
SimonTeixidor:hip-swar-byte-intrinsics
Draft

SimonTeixidor wants to merge 1 commit into
ggml-org:masterfrom
SimonTeixidor:hip-swar-byte-intrinsics

Conversation

@SimonTeixidor

Copy link
Copy Markdown
Contributor

Overview

ggml/src/ggml-cuda/vendors/hip.h emulates these 3 intrinsics with a loop, this PR
replaces those loops with a branch-free SWAR.

Additional information

__vsub4 was previously forwarded to __vsubss4, which saturates, and is now
a wrapping subtract instead. That should be fine for two reasons. First it
matches CUDA's __vsub4, and second it passes test-backend-ops -o MUL_MAT
(1288/1288).

End-to-end benchmark (Strix Halo / gfx1151, llama-batched-bench, S_TG)

The percentage after each quant is its share of decode weight bytes in the affected types.

npl Qwen3.8-27B-UD-IQ4_XS (19%) Qwen3.8-27B-AD-IQ2_S (86%)
4 +5.4% +23.1%
6 +2.1% +21.6%

The five types that use these intrinsics are IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: I noticed that certain quants were faster/slower than others, at the same file size. Claude Opus found this (and a few other) issue and helped me implement a fix.

Replace the scalar loop emulation of these 3 intrinsics in
ggml/src/ggml-cuda/vendors/hip.h with a branch-free SWAR.
@github-actions github-actions Bot added ggml changes relating to the ggml tensor library for machine learning CUDA Related to the CUDA backend labels Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CUDA Related to the CUDA backend ggml changes relating to the ggml tensor library for machine learning

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant