Summary
chooseQuantizedMatmulHeap (skainet-backend-cpu/src/commonMain/.../DefaultCpuOps.kt) and chooseQuantizedMatmul (skainet-backend-cpu/src/jvmMain/.../DefaultCpuOpsJvm.kt) both gate on a.shape.rank >= 2, returning null (i.e. "not a case I handle, fall through") for a rank-1 activation. In practice, a real forward pass produces a rank-1 hidden-state vector for every incremental decode step once it moves past the initial batched prefill (which is rank-2+). That rank-1 case then falls all the way through to matmulGeneric's untyped per-element TensorData.get() access, which — for a packed-quant weight (including one wrapped by SKaiNET-transformers' PreTransposedWeight marker types, e.g. PreTransposedQ4_K) — returns the raw packed byte rather than a dequantized Float, throwing:
java.lang.ClassCastException: class java.lang.Byte cannot be cast to class java.lang.Float
at sk.ainet.exec.tensor.ops.DefaultCpuOpsBase.matmulGeneric$lambda$1(DefaultCpuOps.kt:755)
at sk.ainet.exec.tensor.ops.DefaultCpuOpsBase.matmulGeneric(DefaultCpuOps.kt:704)
at sk.ainet.exec.tensor.ops.DefaultCpuOpsBase.matmul(DefaultCpuOps.kt:623)
at sk.ainet.lang.nn.transformer.LinearProjectionKt.linearProject(LinearProjection.kt:40)
at sk.ainet.lang.nn.transformer.MultiHeadAttention.attentionImpl(MultiHeadAttention.kt:298)
...
Related to #991 / #992 (same dispatch functions, same crash signature) but a distinct root cause: #991/#992 was about the activation operand's TensorData subtype being too strict (FloatArrayTensorData-only) for a rank-2 input; this one is about rank, and reproduces even after #991/#992's fix is applied.
Repro
End-to-end against a real Llama-3.2-1B-Instruct-Q4_K_M.gguf via SKaiNET-transformers' KLlamaJava.loadGGUF (DIRECT mode): the first forward pass (prefill, rank-2 batched input) succeeds; the crash hits on the first post-prefill decode step, where the runtime passes a rank-1 [hidden_dim] activation.
Fix
Pushed as an additional commit to #992 (fix/q6k-memseg-matmul-dispatch), since it touches the exact same dispatch functions:
- Both guards changed from
a.shape.rank < 2 to a.shape.rank < 1 — the existing leading-dim-flattening logic (added for rank>2 attention batches) already generalizes correctly to rank-1 (empty leading dims, flatBatch=1, [in] promotes to [1, in], result squeezes back to [out]) with no further changes needed.
matmulGeneric hardened as defense in depth: a lazily-materialized copyToFloatArray() fallback if a TensorData's generic get() ever returns something other than the tensor's own dtype, instead of an unconditional unsafe as Float cast.- New regression test: rank-1 FP32 activation against a Q4_K-packed weight, asserting no throw and finite output.
Summary
chooseQuantizedMatmulHeap(skainet-backend-cpu/src/commonMain/.../DefaultCpuOps.kt) andchooseQuantizedMatmul(skainet-backend-cpu/src/jvmMain/.../DefaultCpuOpsJvm.kt) both gate ona.shape.rank >= 2, returningnull(i.e. "not a case I handle, fall through") for a rank-1 activation. In practice, a real forward pass produces a rank-1 hidden-state vector for every incremental decode step once it moves past the initial batched prefill (which is rank-2+). That rank-1 case then falls all the way through tomatmulGeneric's untyped per-elementTensorData.get()access, which — for a packed-quant weight (including one wrapped by SKaiNET-transformers'PreTransposedWeightmarker types, e.g.PreTransposedQ4_K) — returns the raw packed byte rather than a dequantizedFloat, throwing:Related to #991 / #992 (same dispatch functions, same crash signature) but a distinct root cause: #991/#992 was about the activation operand's
TensorDatasubtype being too strict (FloatArrayTensorData-only) for a rank-2 input; this one is about rank, and reproduces even after #991/#992's fix is applied.Repro
End-to-end against a real
Llama-3.2-1B-Instruct-Q4_K_M.ggufvia SKaiNET-transformers'KLlamaJava.loadGGUF(DIRECT mode): the first forward pass (prefill, rank-2 batched input) succeeds; the crash hits on the first post-prefill decode step, where the runtime passes a rank-1[hidden_dim]activation.Fix
Pushed as an additional commit to #992 (
fix/q6k-memseg-matmul-dispatch), since it touches the exact same dispatch functions:a.shape.rank < 2toa.shape.rank < 1— the existing leading-dim-flattening logic (added for rank>2 attention batches) already generalizes correctly to rank-1 (empty leading dims,flatBatch=1,[in]promotes to[1, in], result squeezes back to[out]) with no further changes needed.matmulGenerichardened as defense in depth: a lazily-materializedcopyToFloatArray()fallback if aTensorData's genericget()ever returns something other than the tensor's own dtype, instead of an unconditional unsafeas Floatcast.