Q4_K/Q6_K parity tests compare the ggml-faithful int8 (block_q8_K) native kernels against exact-float scalar references — divergence is intended activation-quant loss, not a bug

[michalharakal]

Found while validating the Android JNI bridge (#920): with block data whose FP16 scales are pinned to 1.0 AND the packed 6-bit sub-block scale bytes set to a fixed pattern, the C kernels (q4k_matmul.c/q6k_matmul.c, host gcc build via FFM — same archive the K/N tests link) and the commonMain Kotlin scalar references diverge far beyond float noise. The divergence is bit-identical across compilers and flags (with/without -ffast-math, NDK clang x86_64 and host gcc produce the same deltas) — i.e. a deterministic decode disagreement, not reassociation.

Host reproducer (JVM, NativeQ*MatmulKernel FFM vs ScalarQ*_KMatmulKernel), 256×16, kotlin.random.Random(42) block bytes and Random(42+i).nextFloat()-0.5f inputs:

Q4_K — pin d=1.0f16 @0-1, dMin=1.0f16 @2-3, scale bytes 4..15 = 0x21:

row 0 scalar=315.83618 ffm=317.25122 diff=1.415
row 1 scalar=30.482384 ffm=36.86566  diff=6.383   <- 21% relative
row 2 scalar=667.81525 ffm=667.81464 diff=0.0006  <- agrees
row 3 scalar=-152.33876 ffm=-148.93115 diff=3.408

Q6_K — pin int8 sub-block scales @192-207 = 1, d=1.0f16 @208-209:

row 0 scalar=81.53886 ffm=81.18377 diff=0.355
row 1 scalar=118.43294 ffm=117.88452 diff=0.548
row 2 scalar=97.66035 ffm=98.00478 diff=0.344
row 3 scalar=35.540615 ffm=36.182453 diff=0.642

The per-row pattern (some rows agree to 1e-4, others off by units) points at the 6-bit scale/min bit-extraction — Q4_K packs 8 scales + 8 mins across bytes 4..15 with the upper sub-blocks split across high bits, exactly the kind of layout where two implementations can disagree for specific bit patterns while random data averages the error into invisibility.

Why existing tests miss it: the NativeKn*/FFM parity suites randomize the scale bytes, which inflates row sums to thousands — the systematic delta (units) then hides under rel < 1e-4 … and when it doesn't, the absolute tolerances (5e-2..2e0) absorb it. The suites validate statistically typical data, not adversarial scale patterns.

What's needed:

1. Decide which side is ggml-correct (llama.cpp dequantize_row_q4_K/q6_K is ground truth) — the answer decides whether real Q4_K_M/Q6_K checkpoints decode slightly wrong via the fast path today.
2. Fix the wrong side; add the pinned-pattern reproducers above to the parity suites (both FFM/jvmTest and K/N nativeTest) so the class of bug stays caught.
3. Audit Q5_K's unpacking for the same class of issue (same packing scheme as Q4_K).

Happy to help bisect against the llama.cpp reference.

Refs #920

[michalharakal]

Sharper evidence from the Android JNI bridge validation (x86_64 emulator, NDK clang build of the same C):

1. The disagreement is bigger than first reported and hits typical data, not just adversarial patterns. With the exact NativeKn* generator recipe (d/dMin pinned, everything else random), per-row divergences between NDK-clang-built C and the ART-executed Kotlin scalar reach double digits: e.g. Q6_K row 13: scalar=73.83 vs C=101.44 (37%), Q4_K row 23: 358.01 vs 431.75 (21%).
2. A compiler dimension exists on top of the data dimension. Q4_K (inputDim=1024, outputDim=64, seed=123) is exactly the NativeKnQ4KMatmulKernelParityTest.multi_block_multi_row combination that passes at 5e-2 abs / 1e-4 rel with the gcc-built archive — the NDK clang build of the same source fails it at ~1% (row 0) and far worse on other rows. Since q8_0/q4_0 (pure uint8 handling) agree perfectly across all of this, a prime suspect is implementation-defined signed-char/shift semantics in the K-format 6-bit scale extraction differing between gcc and clang codegen — worth checking the C for bare char arithmetic or sign-extension assumptions in q4k_matmul.c/q6k_matmul.c.
3. Consequence in-tree: the JNI parity tests for Q4_K/Q6_K are @Ignored referencing this issue (bridge mechanics are proven by the q8_0/q4_0/smoke parity, which is exact); the FFM/K-N suites' current green is partly seed luck.

This upgrades the priority: if clang-built kernels decode K-formats differently from gcc-built ones, correctness on real checkpoints depends on which toolchain produced the shipped binary.

[michalharakal]

Bisection result: not a scale-decode bug — it's the block_q8_K activation quantization

I traced this against the llama.cpp reference and it is not a bug in either kernel, and not compiler-dependent. My earlier "gcc passes / clang fails" comment was a red herring — that was a data-magnitude artifact (pinned vs random inner scales), not codegen. Correcting the record:

Root cause

The C K-quant kernels (q4k_matmul.c, q6k_matmul.c) and the Kotlin scalar references implement different algorithms, and the scale decode (get_scale_min_k4) is byte-for-byte identical between them:

• C kernels quantize the FP32 activation row to int8 first (skainet_q8_quantize_block / skainet_q6k_q8_quantize_block, d_in = maxabs/127) and do integer dot-products — this is ggml's block_q8_K fast path, and it's exactly what ggml_vec_dot_q4_K_q8_K / _q6_K_q8_K do (llama.cpp quantizes y to block_q8_K before the dot). The kernels' own comments say so ("the source of the small error vs the exact float kernel").
• ScalarQ*_KMatmulKernel does exact float math on the raw activations — an idealized reference ggml itself does not match.

Proof

A Kotlin transcription of the C algorithm (quantize activations to int8, integer dot) matches the C FFM kernel exactly — |kotlinQ8 − ffm| = 0.0 on every row — while the exact scalar reference is the outlier (Q4_K, 1024×8, seed 123, d=dMin=1.0):

row 0: exact=...  |exact − kotlinQ8|=1.598   |kotlinQ8 − ffm|=0.0
row 1:            |exact − kotlinQ8|=13.06   |kotlinQ8 − ffm|=0.0
row 7:            |exact − kotlinQ8|=13.75   |kotlinQ8 − ffm|=0.0

Same algorithm → identical output regardless of compiler; the divergence is entirely the activation-quantization error. It looks larger on pinned-scale (d=dMin=1.0) blocks because the row sums are small, so the same absolute error is a bigger fraction — which is why my JNI parity tests (pinned scales) surfaced it while the NativeKn* tests (random scales, large sums, loose abs tol) hid it.

So which side is "correct"?

Both, for their contract — but they answer different questions. The C kernel is faithful to llama.cpp (what real inference does); the scalar is exact-float. Comparing them within float-noise tolerance is the actual mistake.

Recommended fix (test, not kernel)

1. Give the K-format parity tests a q8_K-quantized reference (quantize the activation the same way, then exact integer dot) and hold that to tight tolerance — it isolates genuine kernel bugs (offsets, layout, scale decode) from the intended activation-quant loss. My transcription above is ready to contribute.
2. Keep an exact-vs-native check too, but with an activation-quant-aware tolerance (it's data-dependent: sub-percent on typical GGUF weight magnitudes, a few percent on adversarial small-magnitude blocks), and document that the native K-quant path is lossy-by-design.
3. The JNI q4k/q6k parity tests I @Ignored (PR fix(jni): eager library init; emulator-validated parity suite; gate q4k/q6k on #944 (#920) #945) should be re-enabled against the q8_K reference from (1), not the exact one.

Optional follow-up worth its own issue: the Q4KMatmulKernel/Q6KMatmulKernel SPI has two conforming implementations that differ numerically (exact scalar vs lossy native). If deterministic cross-provider output ever matters, the scalar reference could adopt the same q8_K activation quant so every provider agrees — at a small accuracy cost that matches ggml.

[michalharakal]

Implemented the remaining recommendation (item 1 — a q8-activation-quantized Kotlin reference for tight-tolerance isolation, distinct from the RMS-energy loosening already landed in c8966c08/#945): #980.