From b46e3c7ca3323272a486818c08c2e6a6043824f2 Mon Sep 17 00:00:00 2001 From: Michal Harakal Date: Sat, 29 Aug 2026 22:54:23 +0200 Subject: [PATCH] docs(#1205): document why a native GROUP_128/GROUP_64 decode kernel was closed, not built #1205 (native decode kernel for BitNet.cpp's grouped I2_S layout) is closed as not planned: AOT conversion (I2sAotConverter, #1207) already reaches the same zero-copy mmap path (#1203) for a fraction of the ongoing cost of maintaining a second decode variant per SIMD backend, and the layout is a property of which pipeline quantized the file, not which device runs it -- so a native kernel would still need runtime dispatch across every layout regardless of build target. Full reasoning and what a kernel would need if a real use case someday can't tolerate an AOT step: #1205's closing comment. - I2sGgufLayout.GROUP_128/GROUP_64 doc comments point here instead of leaving the decision as tribal knowledge in a closed GitHub issue only. - ternary-getting-started.adoc gains a "Loading a BitNet.cpp-quantized GGUF" section covering the same ground for someone hitting the grouped layout for the first time, plus a Status and roadmap update. --- .../tutorials/ternary-getting-started.adoc | 34 +++++++++++++++++++ .../kotlin/sk/ainet/io/gguf/I2sRepack.kt | 17 ++++++++-- 2 files changed, 49 insertions(+), 2 deletions(-) diff --git a/docs/modules/ROOT/pages/tutorials/ternary-getting-started.adoc b/docs/modules/ROOT/pages/tutorials/ternary-getting-started.adoc index 51c00b08..5a869893 100644 --- a/docs/modules/ROOT/pages/tutorials/ternary-getting-started.adoc +++ b/docs/modules/ROOT/pages/tutorials/ternary-getting-started.adoc @@ -41,6 +41,31 @@ payload = ceil(n·k / 4) bytes, then 4 bytes FP32 per-tensor scale `TernaryCodec.encodeBitNet` writes this layout; the native kernel reads the payload directly and the dispatcher applies the scale to the output. +=== Loading a BitNet.cpp-quantized GGUF (not this sequential layout) + +A GGUF's I2_S tensor isn't always already in the sequential order above. BitNet.cpp's own +quantizer (`quantize_i2_s`) packs a different, grouped layout instead — 128-element blocks on its +x86/AVX pipeline, 64-element blocks on ARM/NEON — which the kernel above cannot read directly. +SKaiNET's loader detects this and repacks it into the sequential layout automatically at load +time; the result is correct, but that repack is a real cost paid on every load unless the file is +converted once, ahead of time. + +**If you're preparing a model for repeated loads (a shipped app, a benchmark harness), convert it +once instead of paying the repack cost every time**: `I2sAotConverter` (GGUF → GGUF, +xref:contributing/build-from-source.adoc[built from source] like the rest of the toolchain) or the +IREE-facing equivalent in +https://github.com/SKaiNET-developers/SKaiNET-IREE-tools[SKaiNET-IREE-tools] re-encode the grouped +layout into the sequential one ahead of time, so the load path never repacks at all and gets the +same zero-copy mmap treatment every other packed format has. + +A *native kernel that decodes the grouped layout directly* — skipping the repack (and any AOT +conversion) entirely — was considered and explicitly **not built** +(https://github.com/SKaiNET-developers/SKaiNET/issues/1205[#1205]): AOT conversion already gets a +converted file to the same zero-copy fast path with far less ongoing kernel-maintenance cost (one +layout to decode, not three, across every SIMD backend). See #1205's closing comment for the full +reasoning and what a grouped-layout kernel would actually need if a real workload someday can't +tolerate an AOT step at all. + == Step 1 — Define and train the model in FP32 Nothing about the architecture changes for ternary. This is the same @@ -194,3 +219,12 @@ https://github.com/anjaustin/neogpu/issues/1[neogpu#1]). Landed in 0.49.0: (https://github.com/SKaiNET-developers/SKaiNET/issues/1141[#1141]). * BitNet model support in SKaiNET-transformers (https://github.com/SKaiNET-developers/SKaiNET-transformers/issues/335[transformers#335]). +* Off-heap storage, zero-copy mmap for sequential-layout files, and an AOT GGUF converter + (https://github.com/SKaiNET-developers/SKaiNET/issues/1198[#1198], + https://github.com/SKaiNET-developers/SKaiNET/issues/1202[#1202], + https://github.com/SKaiNET-developers/SKaiNET/issues/1203[#1203], + https://github.com/SKaiNET-developers/SKaiNET/issues/1207[#1207]) — landed in 0.51.0; see + "Loading a BitNet.cpp-quantized GGUF" above. +* A native decode kernel for BitNet.cpp's grouped layout was considered and closed, not deferred + (https://github.com/SKaiNET-developers/SKaiNET/issues/1205[#1205]) — AOT conversion covers the + real use case at a fraction of the kernel-maintenance cost. diff --git a/skainet-io/skainet-io-gguf/src/commonMain/kotlin/sk/ainet/io/gguf/I2sRepack.kt b/skainet-io/skainet-io-gguf/src/commonMain/kotlin/sk/ainet/io/gguf/I2sRepack.kt index 6297f5d9..f06e4a63 100644 --- a/skainet-io/skainet-io-gguf/src/commonMain/kotlin/sk/ainet/io/gguf/I2sRepack.kt +++ b/skainet-io/skainet-io-gguf/src/commonMain/kotlin/sk/ainet/io/gguf/I2sRepack.kt @@ -23,10 +23,23 @@ package sk.ainet.io.gguf * `BITNET_B1_58` payload. All three agree on the code mapping `{0,1,2} → {-1,0,+1}`. */ public enum class I2sGgufLayout(internal val blockElements: Int) { - /** BitNet.cpp file quantized with the x86/AVX pipeline (`QK_I2_S = 128`, 32-byte blocks). The common case for published GGUFs. */ + /** + * BitNet.cpp file quantized with the x86/AVX pipeline (`QK_I2_S = 128`, 32-byte blocks). The + * common case for published GGUFs. + * + * No kernel decodes this layout directly — [toSequentialPayload] always repacks it to + * [SEQUENTIAL] first, on every load. A native decode kernel that reads `GROUP_128`/[GROUP_64] + * in place, skipping the repack, was considered and deliberately **not built** + * (issue #1205 on SKaiNET-developers/SKaiNET, closed): converting the file once ahead of time + * with `sk.ainet.io.gguf.export.I2sAotConverter` reaches the same zero-copy mmap path (#1203) + * for a fraction of the ongoing cost of maintaining a second decode variant per SIMD backend. + * See the ternary getting-started tutorial's "Loading a BitNet.cpp-quantized GGUF" section, + * and #1205's closing comment for what a grouped-layout kernel would need if some future + * workload genuinely can't tolerate an AOT step. + */ GROUP_128(128), - /** BitNet.cpp file quantized with the ARM/NEON pipeline (`QK_I2_S = 64`, 16-byte blocks). */ + /** BitNet.cpp file quantized with the ARM/NEON pipeline (`QK_I2_S = 64`, 16-byte blocks). Same no-native-kernel note as [GROUP_128]. */ GROUP_64(64), /** NeoGPU's converter: sequential 4-per-byte, low bit-pair first — already the `BITNET_B1_58` payload order. */