ggml : add native INT8 convrot support

[leejet]

Overview

This PR adds native execution support for tensorwise INT8 ConvRot linear layers without converting the stored INT8 weights to another format.

It introduces:

• A normalized grouped H4 Hadamard transform.
• Runtime activation rotation and row-wise INT8 quantization.
• INT8 × INT8 → INT32 matrix multiplication with per-row activation scales, per-output weight scales, and optional bias.
• CPU reference implementations.
• A CUDA H256 path using fused activation rotation/quantization and cuBLAS INT8 GEMM.
• Vulkan packed INT8 dot-product and cooperative-matrix paths, with scalar handling for non-tile-aligned rows.
• Backend capability checks, graph metadata support, and correctness tests.
• An RPC protocol patch-version bump for the two new operation enums.

The intended downstream use case is loading ComfyUI int8_tensorwise checkpoints with convrot metadata directly, while preserving the original INT8 weights. Check stable-diffusion.cpp pr leejet/stable-diffusion.cpp#1857.

Additional information

The runtime activation representation is a transient I8 tensor containing the quantized activation rows followed by one F32 scale per logical row. Keeping both outputs in one tensor avoids recomputing the Hadamard transform and row maximum in separate graph operations, and allows the packed activation to be cached and reused by linear layers sharing the same input.

The tensorwise INT8 helper retains GGML_OP_MUL_MAT because the logical operation and output shape remain a matrix multiplication. It uses:

• src[0]: INT8 weights
• src[1]: F32 input or prequantized INT8 activation
• src[2]: F32 weight scales
• src[3]: optional F32 bias
• op_params[2]: ConvRot group size

The dedicated CUDA and Vulkan activation paths currently target H256. Unsupported configurations and devices are rejected by backend capability checks so that the scheduler can select a fallback backend. HIP and MUSA do not currently expose the dedicated ConvRot quantization path.

This functionality is currently consumed by stable-diffusion.cpp; no llama.cpp model path uses these operations yet.

Validation performed:

• CPU correctness test on an Intel Core i7-10700K.
• CUDA correctness test on an NVIDIA GeForce RTX 4090 (compute capability 8.9).
• Vulkan correctness tests on the RTX 4090, covering packed INT8 dot product, cooperative matrix execution, bias/no-bias cases, and scalar tail rows.
• Backend results are checked against the quantized reference calculation with a relative tolerance of 1e-5.
• A complete Krea2 INT8 ConvRot 1024×1024 generation was tested through stable-diffusion.cpp.

On the RTX 4090 Vulkan backend, cooperative-matrix execution reached approximately 73.5 TOPS in the synthetic large-GEMM test. Handling 4096 aligned rows with cooperative matrices and the remaining 18 Krea2 text-token rows with the scalar path reduced steady-state Krea2 execution from approximately 9.47 s/step to 1.72 s/step. The scalar-only and mixed-dispatch runs produced byte-identical final PNG output.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes, I use AI to help with debugging and porting code from ggml to llama.cpp.

[ggml-gh-bot]

Hi @leejet, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Multiple backend changes in one PR: When adding support for a new model or feature, focus on CPU support only in the initial PR. Add support for other backends like CUDA in follow-up PRs. If you have a good reason to modify multiple backends in one PR, please explain it.

• AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.

• Large PR: Large changes require prior discussion (e.g. an issue or RFC) and maintainers may not be able to review this PR as-is. Consider splitting it into smaller, focused PRs.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[dfriehs]

I rewrote the CUDA rowwise quantization kernel to work without large shared memory (leejet#1) and ended up at something very similar to fwht_cuda. It only does H256 and might take a few more shuffles compared but works with float4 coalesced loads.

Similar optimizations could probably be made to the group_amax/reduce_row_amax/group_quantize path as well.

[noctrex]

@leejet
I've verified the INT8 convrot ops on HIP and opened a PR against your int8_convrot branch with the enablement changes: leejet/ggml#6

Verified on gfx1100 (Windows, ROCm 7.1): tests/test-int8-convrot passes, CPU-vs-GPU bit-exact at real model shapes (K=3840/10240, N<=11520), krea2_turbo int8 convrot generates fully on my 7900XTX.