fix(cpu): packed ternary weights reach dispatch through the Wᵀ marker (#1136) - #1181
Merged
Merged
Conversation
…ch path isHeapPackedWeight was a hardcoded list of the seven Q-format interfaces, so the packed ternary types (BitNetB158TensorData, BitNetPlanesTensorData) fell into transpose's dense decode-copy — whose get() returns ternary CODES, not values: ClassCastException (Byte → Float) on the first linearProject over a packed BitNet weight, found wiring the packed GGUF path in SKaiNET-transformers (transformers#337). The predicate becomes `data is PackedBlockStorage` — the #1136 physiology stated once more: the Wᵀ-marker path that reaches KernelDispatch selects by storage format, not by an enumeration of formats. The JVM tier keeps its own narrower isHeapPackedWeightForJvm for the formats its relayout-and-cache kernels actually read; every other packed encoding lands in matmulWeightTransposedViaViews, which serves any encoding through dispatch (exact ternary keys included, when the packs are installed). Pinned by TernaryWeightTransposeDispatchTest: matmul(x, transpose(W)) against the decoded matmul for both ternary formats — the consumer shape every dense layer's linearProject produces. Refs #1136 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found wiring the packed BitNet GGUF path in SKaiNET-transformers (transformers#337):
linearProject'sops.matmul(x, ops.transpose(W))threwClassCastException(Byte → Float) for a packedBITNET_B1_58weight.Root cause
isHeapPackedWeight— the predicate gating bothtranspose's Wᵀ-marker andmatmulWeightTransposed's views→KernelDispatchpath — was a hardcoded list of the seven Q-format interfaces. The ternary types weren't in it, sotransposedense-copied them through a decodingget()that returns ternary codes, not values.Fix
The predicate becomes
data is PackedBlockStorage— format-driven, like everything else in #1136. The JVM tier keeps its narrowerisHeapPackedWeightForJvm(its relayout-and-cache kernels read specific formats); all other packed encodings land inmatmulWeightTransposedViaViews, which serves any encoding through dispatch — the exact ternary keys included when the packs are installed.Tests
TernaryWeightTransposeDispatchTest:matmul(x, transpose(W))vs the decoded matmul forBITNET_B1_58andBITNET_PLANES— the consumer shape every dense layer produces.backend-cpujvm suite green.OptimizedLLMRuntimeforward) matches the FP32-widened baseline, with and without the native packs.🤖 Generated with Claude Code