[GPU] Use oneDNN static_sbzp for floating-point decompression zero points

[osabnis]

Details

• Draft; depends on oneDNN: Requires quantization_mode::static_sbzp from uxlfoundation/oneDNN#6218. This does not build with OpenVINO's current oneDNN version and can merge only after the upstream change is accepted and the submodule is updated. Related llama.cpp change: SDPA in f32 ravi9/llama.cpp#7.

• Problem: oneDNN's default static_sazp mode requires integer zero points and computes w = scale * (q - zp). As a result, compressed FC layers with f16/bf16 zero points fall back to OpenCL. With exact zero points enabled (GGML_OPENVINO_EXACT_ZERO_POINT=1), 130 of 261 FC executions in a Gemma-4 decode used fully_connected_gpu_gemv instead of oneDNN.

• Motivation: GGUF Q4_K, Q4_1, Q5_1, and Q5_K use w = d * q - m, corresponding to a generally non-integer zero point m / d. Rounding this value introduces a systematic weight error and is a major contributor to the accuracy gap between the llama.cpp OpenVINO and CPU backends.

• Design: Use oneDNN's static_sbzp mode, which computes w = scale * q - zp'. Setting zp' = zp * scale preserves the exact dequantized weights.

  • prepare_quantization: For eligible compressed FC layers with constant f16/bf16 scales and zero points, u4/i4/u8 weights, and at least two K-groups, precompute zp * scale on the host and set decompression_zero_point_after_scale. This is enabled only when oneDNN will be used on the device. Write the scaled zero points directly in the FC's fbyx layout to avoid an additional reorder and temporary copy during program build.
  • fully_connected_onednn: Accept floating-point zero points when flagged and configure static_sbzp.
  • OpenCL registry: Reject flagged primitives; behavior is unchanged when oneDNN is disabled.
  • Primitive: Include the flag in hashing, comparison, and serialization.
  • Naming: The flag could instead be represented as a quantization-mode enum; feedback is welcome.

• Tests: fully_connected_gpu_test.cpp, compressed_int{4,8}_f16_zp_*.

  • Cover u4/u8 weights, group-32 f16 scales, non-integer f16 zero points, single-token and batched inputs, static/dynamic shapes, 2D/3D inputs, cached networks, shared scale/zero-point constants across two FC layers, and OpenCL with oneDNN disabled.
  • Verify implementation selection, flag propagation, and output accuracy against exact dequantization, requiring at least 4x lower error than a rounded zero point.
  • Verify that the oneDNN path reads scaled zero points directly from an fbyx constant, with no intervening reorder, including when constants are shared between FC layers.

• Validation: Intel Arc B390 (Panther Lake), master a54783c84e with the rfcs: add a scales-before-zero-points quantization mode for matmul weights uxlfoundation/oneDNN#6218 prototype applied to the submodule. The branch is rebased onto 405eb32bba; there are no GPU plugin changes between the two revisions.

  • Tests: *f16_zp*: 13/13 passed; *fully_connected*:*FullyConnected*: 1,195 passed, 31 skipped, 0 failed.

  • Gemma-4: All 261 FC executions use oneDNN jit:gemm with GGML_OPENVINO_EXACT_ZERO_POINT=1.

  • KL divergence vs. llama.cpp CPU backend (decode, context 1024):

    Model	Rounded zero point	Exact zero point
    Gemma-4 E2B	0.363	0.0196
    Llama-3.2 1B	0.041	0.0018
    SmolLM2 1.7B	0.073	0.0101
    Phi-3.5 mini	0.031	0.0048
    Phi-4 mini	0.036	0.0108
    Qwen3.5 4B	0.025	0.0031

  • Performance: llama.cpp llama-bench, flash attention, median of 5 runs, tokens/s; all builds use a54783c84e. Each cell is rounded zero points (today's default) / exact zero points without this change / exact zero points with this change:

    Model	Decode tg128, depth 0	Decode tg128, depth 4096	Prefill pp4096
    Gemma-4 E2B	52.05 / 48.83 / 50.86	45.23 / 42.63 / 44.50	3,747 / 1,456 / 3,874
    Llama-3.2 1B	98.92 / 80.59 / 94.88	77.92 / 72.75 / 78.42	8,749 / 3,024 / 8,618
    SmolLM2 1.7B	73.57 / 66.23 / 72.60	42.64 / 40.01 / 42.42	5,322 / 1,826 / 5,230
    Phi-3.5 mini	33.68 / 31.62 / 33.16	21.51 / 20.57 / 21.29	2,239 / 1,023 / 2,247
    Phi-4 mini	32.37 / 30.39 / 31.95	26.37 / 24.81 / 26.00	2,573 / 1,081 / 2,564
    Qwen3.5 4B	26.99 / 25.32 / 26.57	25.32 / 23.71 / 24.94	1,868 / 848 / 1,859

    This change brings exact zero points back to the speed of rounded ones: prefill 2.2–2.9x faster than without it, within 4% of rounded zero points, and decode at depth 4096 within 2%. Decode at depth 0 remains 1–4% slower than with rounded zero points, because f16 zero points add about 8% to the Q4_K weight bytes.

  • Peak GPU memory: At decode depth 4096, writing scaled zero points directly in their final layout reduces peak GPU memory by 52 MB for Llama-3.2 1B, 100 MB for Gemma-4 E2B, and 66 MB for Qwen3.5 4B. Exact zero points still require more memory than rounded zero points (1,292 vs. 1,242 MB for Llama-3.2 1B) because f16 zero points are larger than u4 zero points, and both the original and scaled values coexist while the program is built.

Tickets

• N/A

AI Assistance

• AI assistance used: yes
• AI assistance used for running and drafting tests.