[osabnis]

Details:

Improve GGUF quantization accuracy in the llama.cpp OpenVINO backend by preserving exact zero points for compressed FC layers on the oneDNN path.

GGUF Q4_K, Q4_1, Q5_1, and Q5_K weights use w = d * q - m. Representing this as scale * (q - zp) requires zp = m / d, which is generally non-integer. Rounding zp introduces systematic weight errors. This change uses oneDNN's quantization_mode::static_sbzp to compute w = scale * q - zp', with zp' = zp * scale, preserving exact dequantization.

Status: Draft; depends on oneDNN. Requires quantization_mode::static_sbzp: RFC uxlfoundation/oneDNN#6218 (approved) and implementation uxlfoundation/oneDNN#6306 (in review). This branch does not build against OpenVINO's current oneDNN version; it can merge once #6306 is merged and the oneDNN submodule is updated. Related llama.cpp change: ravi9/llama.cpp#341.

Accuracy

KL divergence against the llama.cpp CPU backend during decode at context 1024, measured on Intel Arc B390 (Panther Lake), Windows. Lower is better.

Model	Current OpenVINO	Exact zero points	Vulkan
Gemma-4 E2B	0.3634	0.0196	0.0187
Llama-3.2 1B	0.0406	0.0018	0.0014
SmolLM2 1.7B	0.0728	0.0101	0.0092
Phi-3.5 mini	0.0312	0.0048	0.0040
Phi-4 mini	0.0357	0.0108	0.0030
Qwen3.5 4B	0.0250	0.0033	0.0039

KL divergence improves by 3–23×, bringing OpenVINO within 1.3× of Vulkan on five of six models. Qwen3.5 4B performs better than Vulkan on this metric.

Phi-4 mini remains 3.6× above Vulkan. The remaining gap comes from the output layer, which llama.cpp's OpenVINO backend requantizes: keeping that layer in its original GGUF type reduces KL divergence to 0.0031. This is outside the scope of this change.

Implementation

• prepare_quantization.cpp: For compressed FC layers with constant f16/bf16 scales and zero points, u4/i4/u8 weights, and at least two K groups, precompute zp * scale on the host and set decompression_zero_point_after_scale. Apply this only when oneDNN is enabled and supported on the device.
• fully_connected_onednn: Accept floating-point zero points when flagged and configure static_sbzp.
• OpenCL registry: Reject primitives with this flag; behavior remains unchanged when oneDNN is disabled.
• Primitive metadata: Include the flag in hashing, comparison, and serialization.

Scaled zero points are written directly in the layout consumed by the FC: [N, groups] parameters in fbyx, while [groups, N] parameters retain their layout. This avoids a reorder and temporary copy during program build.

The transformation belongs in prepare_quantization, rather than a graph transformation, because zp * scale is specific to the oneDNN implementation. Backend selection depends on device capabilities (including IMMAD support) and oneDNN availability. Applying it at graph level would also change the values consumed by OpenCL kernels, which subtract the zero point before scaling.

Dynamic quantization

FCs with floating-point weight zero points are already excluded from dynamic quantization (DynamicQuantizeFullyConnected skips an unsupported weight zero-point type), so the FCs this change applies to keep f16/bf16 activations, which static_sbzp requires. The check on dynamic_quantized_activation in prepare_quantization is only a guard. The prefill numbers below include this: prefill stays within 2% of rounded integer zero points, which are eligible for dynamic quantization. Integer activations with static_sbzp are left for a later revision of the oneDNN RFC, driven by performance data.

Tests

New coverage in fully_connected_gpu_test.cpp, compressed_int{4,8}_f16_zp_*, includes:

• u4/u8 weights, group-32 f16 scales, and non-integer f16 zero points.
• Single-token and batched inputs, static/dynamic shapes, and 2D/3D inputs.
• Cached networks and scale/zero-point constants shared across two FC layers.
• OpenCL execution with oneDNN disabled.
• Backend selection, flag propagation, and accuracy against exact dequantization, requiring at least 4× lower error than with rounded zero points.
• Verification that oneDNN reads scaled zero points directly from fbyx constants without an intervening reorder, including shared constants.

Validation

Tested on Intel Arc B390 (Panther Lake), Windows, using master 38ff87c030 with the oneDNN static_sbzp implementation (uxlfoundation/oneDNN#6306, prototype version at the time) in the submodule.

• *f16_zp*: 15/15 passed.
• *fully_connected*:*FullyConnected*: 1,282 passed, 0 failed.
• Re-run with the final code alongside [GPU] Upload grouped FC decompression scales directly in fbyx to reduce peak GPU memory #38995: identical results; [GPU] Upload grouped FC decompression scales directly in fbyx to reduce peak GPU memory #38995's fully_connected_decompression_params test also passed 64/64.
• Gemma-4 decode with exact zero points: all 261 FC executions used oneDNN jit:gemm.
• Accuracy: Llama-3.2 1B and Gemma-4 E2B measured on the rebased build; the other four models measured on the same change before rebase.
• Vulkan reference: llama.cpp b11236, using the same reference and settings.

Performance

llama-bench median tokens/s across three runs, rounded → exact zero points:

Model	Decode, depth 0	Decode, depth 4096	Prefill, pp4096
Llama-3.2 1B	96.18 → 95.19	75.34 → 75.31	8,590 → 8,637
Gemma-4 E2B	51.88 → 49.98	43.92 → 43.16	4,033 → 3,940
Qwen3.5 4B	27.04 → 26.61	24.75 → 24.86	1,869 → 1,867

Decode at depth 0 is 1–4% slower, partly because f16 zero points add about 8% to Q4_K weight bytes. Deep-context decode and prefill remain within 2% of the original performance. Without this change, exact zero points made prefill 2.2–2.9× slower in measurements taken before rebase. Run-to-run variation is approximately ±2% for deep-context decode and ±1% for prefill.

Peak GPU memory

Exact zero points increase peak GPU memory during decode at depth 4096 because f16 zero points are larger than u4 zero points, and original and scaled values coexist during program build.

Model	Exact zero points alone	With #38995
Llama-3.2 1B	+50 MB	+6 MB
Gemma-4 E2B	+147 MB	+47 MB
Qwen3.5 4B	+224 MB	+26 MB

#38995 uploads grouped FC scales in their final layout, reducing the additional memory required by this change.

Related changes

• [GPU] Refactor for FC Compressed  #38697: Compressed FC refactor introducing transpose_b_scale and transpose_b_zp for [N, groups] versus [groups, N] parameters. This branch is rebased on that refactor. New tests cover [groups, N] parameters (compressed_int4_f16_zp_groups_major) with static and dynamic shapes.
• [GPU] Upload grouped FC decompression scales directly in fbyx to reduce peak GPU memory #38995: Both changes skip the reorder for parameters already in fbyx in prepare_quantization.cpp; whichever merges second should not conflict on that change.

Tickets:

• CVS-196698

AI Assistance:

• AI assistance was used for analysis, test execution, measurements.