From 4a976199bd52b90ca0c8484ff1d7cea535b0d02c Mon Sep 17 00:00:00 2001 From: Michal Harakal Date: Sat, 29 Aug 2026 21:03:11 +0200 Subject: [PATCH 1/2] docs(#1198): 0.51.0 changelog, README highlights, regenerated reference docs The release entry for the ternary/off-heap-storage + AOT-conversion arc (#1202/#1203/#1207/#1211): CHANGELOG gets the full technical writeup; README's "What's New" is trimmed to a short, current-release-only highlight list pointing to CHANGELOG.md for full history instead of carrying the growing "Previously, in ..." cascade inline (it had grown back to 0.38.0). BOM coordinate and the Android classifier tutorial's dependency snippet bumped to 0.51.0; docs/antora.yml's skainet_version likewise (both had drifted at 0.49.0 across the 0.50.0 release). Kernel-support matrix and operator reference regenerated at 0.51.0. --- CHANGELOG.md | 72 +++++++++++++++ README.md | 87 +++++-------------- docs/antora.yml | 2 +- .../reference/kernel-support-matrix.adoc | 2 +- .../reference/operators/generated/index.adoc | 2 +- .../pages/reference/ops-status-matrix.adoc | 2 +- .../android-classifier-getting-started.adoc | 12 +-- 7 files changed, 105 insertions(+), 74 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7d10c4af..7980d427 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,78 @@ ## [Unreleased] +## [0.51.0] - 2026-08-29 + +Headline: **ternary/BitNet weights join the memory-mapped weight story 0.50.0 started for every +other quant format.** 0.50.0 shipped mapped staging for every GGML block format but left ternary +(`BITNET_B1_58`) heap-staging, flagged then as needing the work tracked in +[#1198](https://github.com/SKaiNET-developers/SKaiNET/issues/1198). That work is done: off-heap +storage removes the Android ART heap-cap OOM risk a repacked ternary weight used to carry, and a +`SEQUENTIAL`-layout (NeoGPU-converted) GGUF now gets a true zero-copy mmap load with no repack at +all. A new AOT converter lets a build that owns its model pipeline pay that repack cost once, +offline, instead of on every load — its IREE-facing counterpart lives in a new home, +[SKaiNET-IREE-tools](https://github.com/SKaiNET-developers/SKaiNET-IREE-tools), on the +architectural grounds that compiled-target-specific conversion belongs beside the compiler +toolchain it targets, not inside core (see [#1207](https://github.com/SKaiNET-developers/SKaiNET/issues/1207)). + +### Added + +- **Off-heap storage for packed ternary/quantized weights** + ([#1202](https://github.com/SKaiNET-developers/SKaiNET/issues/1202), + [#1206](https://github.com/SKaiNET-developers/SKaiNET/pull/1206)): `Storage` gains + `copyInto`/`copyFrom` bulk byte primitives implemented for every concrete storage kind (`Heap`, + `SegmentStorage`, `MappedFileStorage`, `DirectBufferStorage`, `MappedBufferStorage`, + `NativeMallocStorage`, `NativeMappedStorage`). `PackedBlockStorage` gains a `packedStorage` + property (default: wraps `packedData` in `Storage.Heap`, so every existing quantized format is + unaffected); `packedView` reads from it instead of always re-wrapping the raw array. + `BitNetB158TensorData` gains a `Storage`-backed constructor/`fromStorage()` factory — the + per-element accessors lazily snapshot off-heap storage only if actually touched, so inference + through the GEMV kernels never materializes a heap copy. `StreamingGgufParametersLoader` + allocates off-heap for repacked I2_S payloads at or above `PlannerProfile.OFF_HEAP_THRESHOLD` + (256 KB) instead of a permanent `ByteArray`. +- **Zero-copy native gemv for off-heap ternary weights** + ([#1202](https://github.com/SKaiNET-developers/SKaiNET/issues/1202), + [#1206](https://github.com/SKaiNET-developers/SKaiNET/pull/1206)): `TernaryF32GemvNative` gains + `gemvPackedStorage()`, letting the JVM/FFM face hand a `SegmentStorage`'s `MemorySegment` + straight to the native downcall — the weight is never copied, not even once, where the + previous `gemvPacked` path re-copied the *entire* weight matrix into a fresh arena on every + row of every call, independent of storage kind. `NativeTernaryF32ViewKernel` now dispatches on + the weight's storage kind instead of only accepting `Storage.Heap`, so an off-heap ternary + weight keeps the fast NEON/FFM path instead of silently falling back to the slow reference + kernel. +- **Zero-copy mmap for `SEQUENTIAL`-layout I2_S tensors** + ([#1203](https://github.com/SKaiNET-developers/SKaiNET/issues/1203), + [#1208](https://github.com/SKaiNET-developers/SKaiNET/pull/1208)): `I2sRepack.toSequentialPayload` + no longer copies a `SEQUENTIAL` buffer that's already exactly the target payload (the common + NeoGPU-converted case). More significantly, the loader's mmap-eligibility branch — previously + keyed on a hardcoded encoding whitelist that didn't include `I2_S` — now maps a `SEQUENTIAL` + I2_S tensor directly off the file whenever its trailing bytes are provably the real scale (no + companion `_scale` tensor overriding them), giving it the same zero-copy path Q4_K/Q8_0/etc. + already had. `GROUP_128`/`GROUP_64` payloads and companion-scored `SEQUENTIAL` files correctly + keep repacking. +- **`I2sAotConverter`: AOT GGUF → GGUF conversion for I2_S** + ([#1207](https://github.com/SKaiNET-developers/SKaiNET/issues/1207), + [#1210](https://github.com/SKaiNET-developers/SKaiNET/pull/1210)): reads an arbitrary GGUF, + repacks I2_S tensors into `SEQUENTIAL`+trailer order ahead of time, drops the now-redundant + companion scale tensor, and passes every other tensor and all KV metadata through unchanged — + the converted file always takes the new zero-copy mmap path above, with no on-device cost at + all. `GgufTensorEntry` (the writer's tensor-entry type) gains a `rawBytes` passthrough mode to + support this without going through the element-indexed `TensorFlatten` path. +- **GGUF I2_S → `.irpa` conversion** (IREE-facing counterpart, in + [SKaiNET-IREE-tools#1](https://github.com/SKaiNET-developers/SKaiNET-IREE-tools/pull/1), not + this repo): a standalone Python tool producing an IREE parameter archive directly from a + GGUF's ternary tensors, for `iree-compile --iree-opt-import-parameters=`. + +### Fixed + +- **Scoped dense-FP32 activations silently fell out of the quantized matmul chooser** + ([#1211](https://github.com/SKaiNET-developers/SKaiNET/pull/1211)): `chooseQuantizedMatmul2D` + accepted only `FloatArrayTensorData`/`MemorySegmentBackedData` activations; a + `ScopedExecutionContext` forward's slab-backed `StorageFloatTensorData` fell through to + `matmulGeneric`, whose per-element `get()` on a `Q8MemorySegmentTensorData` weight returns the + raw quantization byte, not the value — silently wrong logits (the #993 class of bug). Any dense + activation (`encoding == null`) is now accepted via its own offset-aware `copyToFloatArray()`. + ## [0.50.0] - 2026-08-28 Headline: **model size on Android is now a page-cache question, not a heap question — and decode is diff --git a/README.md b/README.md index 18219456..28a48ad5 100644 --- a/README.md +++ b/README.md @@ -51,7 +51,7 @@ Add the core dependencies (Gradle Kotlin DSL): ```kotlin dependencies { // Recommended: import the umbrella BOM and drop versions on the engine modules. - implementation(platform("sk.ainet:skainet-bom:0.49.0")) + implementation(platform("sk.ainet:skainet-bom:0.51.0")) implementation("sk.ainet.core:skainet-lang-core") implementation("sk.ainet.core:skainet-backend-cpu") @@ -296,69 +296,22 @@ val withoutLabel = dataPipeline() --- -## What's New in 0.49.0 - -The **SKEEP-jump release**: 0.40.1 → 0.49.0, ~100 merged PRs — the SKEEP-003 memory & storage -architecture complete, from accepted proposal to shipped system. This is the release downstream -repositories (SKaiNET-transformers, the IREE conformity pipeline) should build on. It **removes -every façade the architecture replaced** — see the Breaking-changes section of -[CHANGELOG.md](CHANGELOG.md) for the migration map (`QuantPolicy`/`StagingPolicy`/`WeightOrientation` -→ `WeightForm`; `@Place`/`@Weights`/`StorageSpec`/old `MemoryPlanner` → `AllocationResolver`). - -- **One storage model** — `Storage` / `Scope` / `Format` / `Layout` / `TensorView`: enforced ownership - (use-after-free throws, loudly), scoped lifetimes, `prepack()` as the visible relayout, - `materialize()` as the single copy point. -- **Decisions are resolved, not declared** — `WeightFormResolver` picks a weight's in-memory form from - *file × profile × kernels*; `AllocationResolver` picks domain and scope, and `explain()` says why, - per tensor, before a byte of payload is read. You always outrank the resolver - (per-tensor `weightFormFor` > uniform `weightForm` > resolver). -- **Flat-memory decode** — `ctx.forwardScope(slabFloats) { … }` recycles one slab per step; creation - *and op outputs* draw from it, and the FP32 fast paths + JVM Panama vector kernels are offset-aware, - so scoped tensors keep SIMD speed. Steady-state decode allocates zero new slab bytes per step. -- **"Will it fit?" in seconds, any format** — header-only footprint plans for GGUF, **safetensors and - ONNX** (external-data sidecars priced correctly), with `PlannerProfile.EDGE` for embedded devices: - `skainet-plan model.onnx --profile edge --budget 2.1G`, exit code 0/1. -- **The compile lane carries what the runtime decides** — tensor identity, structural encodings - (`skainet.tensor_layouts`: block sizes and bit widths as integers, not names) and block order flow - into the exported MLIR and `.irpa`; `HloGenerator.generate(target = …)` runs the first layout pass - on the production path. -- **BitNet / ternary compute** — `BITNET_PLANES` multi-plane packing, i2s GGUF import, and the vendored - NeoGPU ternary f32 NEON kernel through FFM, JNI and Kotlin/Native, including a fused lm_head kernel. -- **Docs that cannot rot** — the Android classifier and ternary getting-started tutorials are compiled - *and executed* in CI (the Iris training loop asserts held-out accuracy ≥ 0.80), plus the - virtual-tensors explanation and SKEEP-003a resolution record. - -### Previously, in 0.40.1 - -- **Correctness hotfix: packed-quant `transpose()` was silently wrong, not crashing.** `ops.matmul(x, ops.transpose(W))` on a packed-quantized weight (Q4_0/Q5_0/Q5_1/Q8_0/Q4_K/Q5_K/Q6_K) with more than one quant block per row produced silently incorrect output — sometimes all-zero — across the scalar, Panama-vector, *and* native (FFM/JNI) kernel tiers, with no exception raised. `transpose()` now performs a real block-grid byte permutation instead of a shape-only relabel; a misaligned packed tensor now throws instead of silently truncating. Closes [#968](https://github.com/SKaiNET-developers/SKaiNET/issues/968). **Upgrading is strongly recommended** for anyone calling `ops.transpose()` on packed-quantized weights. - -### Previously, in 0.40.0 - -- **Android models grow past the ART heap cap.** Off-heap/mmap tensor storage shares the JVM's memory-mapped weight loading with Android — dense F32 tensors serve as zero-heap mapped views, and weight bytes live in OS-paged file-backed pages instead of the managed heap. A 640 MB dense model now loads with **1.4 MB** of heap allocation. -- **GGUF `DEQUANTIZE_TO_FP32` no longer over-allocates.** A 1.1B Q4_K_M GGUF transiently needed >12 GB heap against a ~4.4 GB dense-FP32 floor. Three compounding allocation sources in the loader and K-quant kernels are fixed, bringing peak live allocation to ~1.05x of the dense FP32 size. -- **Q5_0/Q5_1 packed matmul reaches the native tier.** New NEON C kernels for both formats are wired into the FFM (JVM), Kotlin/Native, and Android JNI providers — unblocking the packed Q5_1 path for Q5_K_M checkpoints under `NATIVE_OPTIMIZED`. -- **SKaiNET reaches iOS and macOS natively.** `skainet-backend-native-cpu` now publishes `iosArm64`, `iosSimulatorArm64`, and `macosArm64` Kotlin/Native targets with embedded kernel archives; a single Apple arm64 archive dispatches FEAT_DotProd at runtime, so one build serves A12 through M-series. - -### Previously, in 0.39.1 - -- **Eager CPU ops run primitive FP32 fast paths.** The generic per-element paths (index-array allocations, boxed accessors, dtype dispatch) dominated on-device LLM decode — 83% of end-to-end SmolLM2-135M decode time on a Pixel 8a was non-matmul overhead even with the NEON backend. Hot ops (arithmetic, activations, unary math, softmax/logSoftmax, reductions, concat, reshape) now run flat primitive loops over the dense `FloatArray` buffer, benefiting every non-JVM target — Android, Kotlin/Native, JS/Wasm. `DirectCpuExecutionContext.ops` is also cached instead of rebuilt per access. -- **README points LLM users to SKaiNET-transformers.** A callout under "Start in 5 minutes" makes clear that LLM inference lives in the SKaiNET-transformers repository — this repo is the engine underneath. - -### Previously, in 0.39.0 - -- **On-device AI on Android — a NEON kernel backend.** New `skainet-backend-jni-cpu` module: the hand-tuned ARM matmul kernels reach Android through a JNI bridge (ART has no `java.lang.foreign`, so the FFM provider can never run there). Two `.so` tiers are built from the same sources and selected at load time from `/proc/cpuinfo` — a baseline `armv8-a` build that runs on every 64-bit core, and an `armv8.2-a+dotprod` build for the `vdotq_s32` Q4_K/Q6_K paths — so a single artifact is safe from Cortex-A53 up. Measured on a Pixel 8a: **~24 tok/s** SmolLM2-135M Q8_0 decode versus ~3.8 scalar (6.4x), clearing the on-device usability bar. The provider auto-registers via `ServiceLoader`; an app just adds the AAR. -- **Android GGUF loading no longer OOMs.** `createRandomAccessSource` returned `null` on Android, forcing every model load through a full-file heap read that exhausted the ART heap on real devices. It now streams via positional `FileChannel` reads across `skainet-io-gguf` / `-safetensors` / `-onnx`. -- **Published Kotlin/Native kernel klibs are linkable.** The static kernel archive is now embedded into the cinterop klib, so downstream K/N consumers of `skainet-backend-native-cpu` (`-linuxx64` / `-linuxarm64`, and the path future Apple targets will use) link with no manual setup. A NEON body was also added for the Q4_0 matmul kernel. -- **Tensor-storage correctness pass.** Fail-fast on unsupported GGUF quant types instead of silently dropping weights; truthful ownership labels and real byte counts in the storage layer; a materializable `FileBacked`/`Aliased` transfer path; and a rank-safe default `copyToFloatArray`. - -### Previously, in 0.38.0 - -- **Streaming KV-cache decode (dynamic dimensions)** — a first-class `Dim` vocabulary makes "dynamic extent" explicit instead of an overloaded `-1`, and the StableHLO emitter renders it as an MLIR `?`. One compiled vmfb now serves every autoregressive decode step with a growing cache, instead of one fixed cache length. Verified end-to-end: the full FunctionGemma `with_past` decode graph and the Moonshine v2 decoder (dynamic self *and* cross caches) self-compile from the DSL to a CPU vmfb — graphs that could not be compiled before. Static graphs are emitted byte-for-byte unchanged. -- **Narrow-float (BF16 + FP16) weights kept packed** — SafeTensors F16 and GGUF F16/BF16 weights load `KEEP_NATIVE`, two bytes per element at rest instead of widening to FP32, and reach format-specific matmul kernels still packed. Narrow floats are a storage width only: kernels widen to f32 lanes and accumulate in f32. -- **Both narrow formats now beat the FP32 SGEMM** — BF16 by 1.8–1.9x, FP16 by 1.5–1.7x on a 4096x11008 projection. Getting there took a zero-copy transpose for input-major weights (the per-token transpose previously widened the tensor elementwise, 4.4 s per projection), a native FFM FP16 kernel to match the existing BF16 one, and tiling both kernels so the weight is read once per matmul rather than once per input row. -- **Allocation-free shape-only tracing** — `VoidTensorOps` propagates shapes through a `ShapeOnlyTensorData` that allocates no backing buffer, so a dynamic extent flows through a whole decode trace instead of throwing on a negative-size allocation. - -See [CHANGELOG.md](CHANGELOG.md) for details and the full release history. +## What's New in 0.51.0 + +Ternary/BitNet weights join the memory-mapped weight story 0.50.0 started for every other quant +format: + +- **Off-heap ternary storage** — `BitNetB158TensorData` no longer risks the Android ART heap-cap + OOM; `Storage.copyInto`/`copyFrom` give every storage kind one shared bulk-copy primitive. +- **True zero-copy mmap** for `SEQUENTIAL`-layout (NeoGPU-converted) GGUFs, and a zero-copy + native gemv path for off-heap ternary weights on the JVM/FFM kernel. +- **`I2sAotConverter`** (GGUF → GGUF): convert I2_S tensors ahead of time so a controlled model + pipeline never pays a runtime repack. The IREE-facing counterpart lives in + [SKaiNET-IREE-tools](https://github.com/SKaiNET-developers/SKaiNET-IREE-tools). +- **Correctness fix** — scoped dense-FP32 activations no longer silently fall out of the + quantized matmul chooser (was producing wrong logits under `ScopedExecutionContext`). + +See [CHANGELOG.md](CHANGELOG.md) for full release notes, including every prior release. --- @@ -382,6 +335,12 @@ We love contributions! Whether it's a new operator, documentation, or a bug fix: Browse the full codebase documentation on [DeepWiki](https://deepwiki.com/SKaiNET-developers/SKaiNET). +### Contributors (0.51.0) + +- **Michal Harakal** ([@michalharakal](https://github.com/michalharakal)) — off-heap ternary + storage, zero-copy mmap for `SEQUENTIAL` I2_S, the AOT GGUF converter and its IREE-facing + counterpart in SKaiNET-IREE-tools, and the scoped dense-FP32 activation matmul-chooser fix + ### Contributors (0.49.0) - **Michal Harakal** ([@michalharakal](https://github.com/michalharakal)) — the SKEEP-003 memory & storage architecture end to end: M0/M1/M2 milestones, the weight-form and placement-resolution arcs, scope-recycled execution, multi-format footprint analysis, the compile-lane carriage arc, the BitNet/ternary kernel track, and the release docs diff --git a/docs/antora.yml b/docs/antora.yml index 9fbd0f10..b4948094 100644 --- a/docs/antora.yml +++ b/docs/antora.yml @@ -15,7 +15,7 @@ asciidoc: framework_name: SKaiNET # Current SKaiNET release — bump once per release; referenced as # {skainet_version} in dependency snippets (blocks need subs="attributes+"). - skainet_version: 0.49.0 + skainet_version: 0.51.0 ksp_version: 2.2.21-2.0.5 dokka_version: 2.1.0 asciidoctorj_version: 3.0.0 diff --git a/docs/modules/ROOT/pages/reference/kernel-support-matrix.adoc b/docs/modules/ROOT/pages/reference/kernel-support-matrix.adoc index 2f48adfc..37f85d44 100644 --- a/docs/modules/ROOT/pages/reference/kernel-support-matrix.adoc +++ b/docs/modules/ROOT/pages/reference/kernel-support-matrix.adoc @@ -1,7 +1,7 @@ = Kernel × platform support matrix :description: Which compute-kernel provider serves each weight format on each KMP target. -Generated from `kernel-support.json` (version `0.49.0`) by `KernelSupportMatrixTest` — registry introspection of the registered `KernelProvider` implementations. Do not edit by hand; run `./gradlew generateKernelMatrix` to refresh. +Generated from `kernel-support.json` (version `0.51.0`) by `KernelSupportMatrixTest` — registry introspection of the registered `KernelProvider` implementations. Do not edit by hand; run `./gradlew generateKernelMatrix` to refresh. Each cell is the best (highest-priority) provider that serves `Float32 × format` `matmul` on that platform: *native-ffm* (100) → *panama-vector* (50) → *scalar* (0). An empty cell (`—`) means no provider carries a kernel there (the format is dequant-to-FP32 only). diff --git a/docs/modules/ROOT/pages/reference/operators/generated/index.adoc b/docs/modules/ROOT/pages/reference/operators/generated/index.adoc index 763ab549..0b5d0fab 100644 --- a/docs/modules/ROOT/pages/reference/operators/generated/index.adoc +++ b/docs/modules/ROOT/pages/reference/operators/generated/index.adoc @@ -1,6 +1,6 @@ = AI-NET Operators Reference -Generated from version `0.49.0` on 2026-08-26 +Generated from version `0.51.0` on 2026-08-29 == Operators by Modality diff --git a/docs/modules/ROOT/pages/reference/ops-status-matrix.adoc b/docs/modules/ROOT/pages/reference/ops-status-matrix.adoc index cff069a6..2dba6c47 100644 --- a/docs/modules/ROOT/pages/reference/ops-status-matrix.adoc +++ b/docs/modules/ROOT/pages/reference/ops-status-matrix.adoc @@ -1,7 +1,7 @@ = Operator Coverage Matrix :description: Cross-backend status for every operator function in SKaiNET. -Generated from `operators.json` version `0.49.0` on 2026-08-26. +Generated from `operators.json` version `0.51.0` on 2026-08-29. Rows are `Operator.function` pairs. The `Validated` column shows whether the function's documentation has been DARC-validated by a reviewer (see xref:contributing/darc-workflow.adoc[DARC workflow]). Remaining columns are backends that appear in any function's `statusByBackend` map — a missing entry means the backend makes no claim about the function (treat it as "unknown", not "not supported"). diff --git a/docs/modules/ROOT/pages/tutorials/android-classifier-getting-started.adoc b/docs/modules/ROOT/pages/tutorials/android-classifier-getting-started.adoc index e809719d..2b49f6c0 100644 --- a/docs/modules/ROOT/pages/tutorials/android-classifier-getting-started.adoc +++ b/docs/modules/ROOT/pages/tutorials/android-classifier-getting-started.adoc @@ -19,12 +19,12 @@ An Android project with Kotlin. Add the SKaiNET modules: [source,kotlin] ---- dependencies { - implementation("sk.ainet.core:skainet-lang-core:0.49.0") // tensors, DSL, training - implementation("sk.ainet.core:skainet-backend-cpu:0.49.0") // CPU ops - implementation("sk.ainet.core:skainet-compile-dag:0.49.0") // autograd (training context) - implementation("sk.ainet.core:skainet-data-api:0.49.0") // Dataset / DataBatch - implementation("sk.ainet.core:skainet-data-simple:0.49.0") // embedded Iris - runtimeOnly("sk.ainet.core:skainet-backend-jni-cpu:0.49.0") // NEON kernels (see below) + implementation("sk.ainet.core:skainet-lang-core:0.51.0") // tensors, DSL, training + implementation("sk.ainet.core:skainet-backend-cpu:0.51.0") // CPU ops + implementation("sk.ainet.core:skainet-compile-dag:0.51.0") // autograd (training context) + implementation("sk.ainet.core:skainet-data-api:0.51.0") // Dataset / DataBatch + implementation("sk.ainet.core:skainet-data-simple:0.51.0") // embedded Iris + runtimeOnly("sk.ainet.core:skainet-backend-jni-cpu:0.51.0") // NEON kernels (see below) } ---- From 4bf88a5bbd9eb880bea8bd846277a9661665cfda Mon Sep 17 00:00:00 2001 From: Michal Harakal Date: Sat, 29 Aug 2026 21:03:18 +0200 Subject: [PATCH 2/2] release: 0.51.0 --- gradle.properties | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gradle.properties b/gradle.properties index e93f88cc..9d44d63c 100644 --- a/gradle.properties +++ b/gradle.properties @@ -1,5 +1,5 @@ GROUP=sk.ainet.core -VERSION_NAME=0.50.0 +VERSION_NAME=0.51.0 POM_DESCRIPTION=SKaiNET POM_URL=https://github.com/SKaiNET-developers/skainet/