diff --git a/CHANGELOG.md b/CHANGELOG.md index 7d10c4af..7980d427 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,78 @@ ## [Unreleased] +## [0.51.0] - 2026-08-29 + +Headline: **ternary/BitNet weights join the memory-mapped weight story 0.50.0 started for every +other quant format.** 0.50.0 shipped mapped staging for every GGML block format but left ternary +(`BITNET_B1_58`) heap-staging, flagged then as needing the work tracked in +[#1198](https://github.com/SKaiNET-developers/SKaiNET/issues/1198). That work is done: off-heap +storage removes the Android ART heap-cap OOM risk a repacked ternary weight used to carry, and a +`SEQUENTIAL`-layout (NeoGPU-converted) GGUF now gets a true zero-copy mmap load with no repack at +all. A new AOT converter lets a build that owns its model pipeline pay that repack cost once, +offline, instead of on every load — its IREE-facing counterpart lives in a new home, +[SKaiNET-IREE-tools](https://github.com/SKaiNET-developers/SKaiNET-IREE-tools), on the +architectural grounds that compiled-target-specific conversion belongs beside the compiler +toolchain it targets, not inside core (see [#1207](https://github.com/SKaiNET-developers/SKaiNET/issues/1207)). + +### Added + +- **Off-heap storage for packed ternary/quantized weights** + ([#1202](https://github.com/SKaiNET-developers/SKaiNET/issues/1202), + [#1206](https://github.com/SKaiNET-developers/SKaiNET/pull/1206)): `Storage` gains + `copyInto`/`copyFrom` bulk byte primitives implemented for every concrete storage kind (`Heap`, + `SegmentStorage`, `MappedFileStorage`, `DirectBufferStorage`, `MappedBufferStorage`, + `NativeMallocStorage`, `NativeMappedStorage`). `PackedBlockStorage` gains a `packedStorage` + property (default: wraps `packedData` in `Storage.Heap`, so every existing quantized format is + unaffected); `packedView` reads from it instead of always re-wrapping the raw array. + `BitNetB158TensorData` gains a `Storage`-backed constructor/`fromStorage()` factory — the + per-element accessors lazily snapshot off-heap storage only if actually touched, so inference + through the GEMV kernels never materializes a heap copy. `StreamingGgufParametersLoader` + allocates off-heap for repacked I2_S payloads at or above `PlannerProfile.OFF_HEAP_THRESHOLD` + (256 KB) instead of a permanent `ByteArray`. +- **Zero-copy native gemv for off-heap ternary weights** + ([#1202](https://github.com/SKaiNET-developers/SKaiNET/issues/1202), + [#1206](https://github.com/SKaiNET-developers/SKaiNET/pull/1206)): `TernaryF32GemvNative` gains + `gemvPackedStorage()`, letting the JVM/FFM face hand a `SegmentStorage`'s `MemorySegment` + straight to the native downcall — the weight is never copied, not even once, where the + previous `gemvPacked` path re-copied the *entire* weight matrix into a fresh arena on every + row of every call, independent of storage kind. `NativeTernaryF32ViewKernel` now dispatches on + the weight's storage kind instead of only accepting `Storage.Heap`, so an off-heap ternary + weight keeps the fast NEON/FFM path instead of silently falling back to the slow reference + kernel. +- **Zero-copy mmap for `SEQUENTIAL`-layout I2_S tensors** + ([#1203](https://github.com/SKaiNET-developers/SKaiNET/issues/1203), + [#1208](https://github.com/SKaiNET-developers/SKaiNET/pull/1208)): `I2sRepack.toSequentialPayload` + no longer copies a `SEQUENTIAL` buffer that's already exactly the target payload (the common + NeoGPU-converted case). More significantly, the loader's mmap-eligibility branch — previously + keyed on a hardcoded encoding whitelist that didn't include `I2_S` — now maps a `SEQUENTIAL` + I2_S tensor directly off the file whenever its trailing bytes are provably the real scale (no + companion `_scale` tensor overriding them), giving it the same zero-copy path Q4_K/Q8_0/etc. + already had. `GROUP_128`/`GROUP_64` payloads and companion-scored `SEQUENTIAL` files correctly + keep repacking. +- **`I2sAotConverter`: AOT GGUF → GGUF conversion for I2_S** + ([#1207](https://github.com/SKaiNET-developers/SKaiNET/issues/1207), + [#1210](https://github.com/SKaiNET-developers/SKaiNET/pull/1210)): reads an arbitrary GGUF, + repacks I2_S tensors into `SEQUENTIAL`+trailer order ahead of time, drops the now-redundant + companion scale tensor, and passes every other tensor and all KV metadata through unchanged — + the converted file always takes the new zero-copy mmap path above, with no on-device cost at + all. `GgufTensorEntry` (the writer's tensor-entry type) gains a `rawBytes` passthrough mode to + support this without going through the element-indexed `TensorFlatten` path. +- **GGUF I2_S → `.irpa` conversion** (IREE-facing counterpart, in + [SKaiNET-IREE-tools#1](https://github.com/SKaiNET-developers/SKaiNET-IREE-tools/pull/1), not + this repo): a standalone Python tool producing an IREE parameter archive directly from a + GGUF's ternary tensors, for `iree-compile --iree-opt-import-parameters=`. + +### Fixed + +- **Scoped dense-FP32 activations silently fell out of the quantized matmul chooser** + ([#1211](https://github.com/SKaiNET-developers/SKaiNET/pull/1211)): `chooseQuantizedMatmul2D` + accepted only `FloatArrayTensorData`/`MemorySegmentBackedData` activations; a + `ScopedExecutionContext` forward's slab-backed `StorageFloatTensorData` fell through to + `matmulGeneric`, whose per-element `get()` on a `Q8MemorySegmentTensorData` weight returns the + raw quantization byte, not the value — silently wrong logits (the #993 class of bug). Any dense + activation (`encoding == null`) is now accepted via its own offset-aware `copyToFloatArray()`. + ## [0.50.0] - 2026-08-28 Headline: **model size on Android is now a page-cache question, not a heap question — and decode is diff --git a/README.md b/README.md index 18219456..28a48ad5 100644 --- a/README.md +++ b/README.md @@ -51,7 +51,7 @@ Add the core dependencies (Gradle Kotlin DSL): ```kotlin dependencies { // Recommended: import the umbrella BOM and drop versions on the engine modules. - implementation(platform("sk.ainet:skainet-bom:0.49.0")) + implementation(platform("sk.ainet:skainet-bom:0.51.0")) implementation("sk.ainet.core:skainet-lang-core") implementation("sk.ainet.core:skainet-backend-cpu") @@ -296,69 +296,22 @@ val withoutLabel = dataPipeline() --- -## What's New in 0.49.0 - -The **SKEEP-jump release**: 0.40.1 → 0.49.0, ~100 merged PRs — the SKEEP-003 memory & storage -architecture complete, from accepted proposal to shipped system. This is the release downstream -repositories (SKaiNET-transformers, the IREE conformity pipeline) should build on. It **removes -every façade the architecture replaced** — see the Breaking-changes section of -[CHANGELOG.md](CHANGELOG.md) for the migration map (`QuantPolicy`/`StagingPolicy`/`WeightOrientation` -→ `WeightForm`; `@Place`/`@Weights`/`StorageSpec`/old `MemoryPlanner` → `AllocationResolver`). - -- **One storage model** — `Storage` / `Scope` / `Format` / `Layout` / `TensorView`: enforced ownership - (use-after-free throws, loudly), scoped lifetimes, `prepack()` as the visible relayout, - `materialize()` as the single copy point. -- **Decisions are resolved, not declared** — `WeightFormResolver` picks a weight's in-memory form from - *file × profile × kernels*; `AllocationResolver` picks domain and scope, and `explain()` says why, - per tensor, before a byte of payload is read. You always outrank the resolver - (per-tensor `weightFormFor` > uniform `weightForm` > resolver). -- **Flat-memory decode** — `ctx.forwardScope(slabFloats) { … }` recycles one slab per step; creation - *and op outputs* draw from it, and the FP32 fast paths + JVM Panama vector kernels are offset-aware, - so scoped tensors keep SIMD speed. Steady-state decode allocates zero new slab bytes per step. -- **"Will it fit?" in seconds, any format** — header-only footprint plans for GGUF, **safetensors and - ONNX** (external-data sidecars priced correctly), with `PlannerProfile.EDGE` for embedded devices: - `skainet-plan model.onnx --profile edge --budget 2.1G`, exit code 0/1. -- **The compile lane carries what the runtime decides** — tensor identity, structural encodings - (`skainet.tensor_layouts`: block sizes and bit widths as integers, not names) and block order flow - into the exported MLIR and `.irpa`; `HloGenerator.generate(target = …)` runs the first layout pass - on the production path. -- **BitNet / ternary compute** — `BITNET_PLANES` multi-plane packing, i2s GGUF import, and the vendored - NeoGPU ternary f32 NEON kernel through FFM, JNI and Kotlin/Native, including a fused lm_head kernel. -- **Docs that cannot rot** — the Android classifier and ternary getting-started tutorials are compiled - *and executed* in CI (the Iris training loop asserts held-out accuracy ≥ 0.80), plus the - virtual-tensors explanation and SKEEP-003a resolution record. - -### Previously, in 0.40.1 - -- **Correctness hotfix: packed-quant `transpose()` was silently wrong, not crashing.** `ops.matmul(x, ops.transpose(W))` on a packed-quantized weight (Q4_0/Q5_0/Q5_1/Q8_0/Q4_K/Q5_K/Q6_K) with more than one quant block per row produced silently incorrect output — sometimes all-zero — across the scalar, Panama-vector, *and* native (FFM/JNI) kernel tiers, with no exception raised. `transpose()` now performs a real block-grid byte permutation instead of a shape-only relabel; a misaligned packed tensor now throws instead of silently truncating. Closes [#968](https://github.com/SKaiNET-developers/SKaiNET/issues/968). **Upgrading is strongly recommended** for anyone calling `ops.transpose()` on packed-quantized weights. - -### Previously, in 0.40.0 - -- **Android models grow past the ART heap cap.** Off-heap/mmap tensor storage shares the JVM's memory-mapped weight loading with Android — dense F32 tensors serve as zero-heap mapped views, and weight bytes live in OS-paged file-backed pages instead of the managed heap. A 640 MB dense model now loads with **1.4 MB** of heap allocation. -- **GGUF `DEQUANTIZE_TO_FP32` no longer over-allocates.** A 1.1B Q4_K_M GGUF transiently needed >12 GB heap against a ~4.4 GB dense-FP32 floor. Three compounding allocation sources in the loader and K-quant kernels are fixed, bringing peak live allocation to ~1.05x of the dense FP32 size. -- **Q5_0/Q5_1 packed matmul reaches the native tier.** New NEON C kernels for both formats are wired into the FFM (JVM), Kotlin/Native, and Android JNI providers — unblocking the packed Q5_1 path for Q5_K_M checkpoints under `NATIVE_OPTIMIZED`. -- **SKaiNET reaches iOS and macOS natively.** `skainet-backend-native-cpu` now publishes `iosArm64`, `iosSimulatorArm64`, and `macosArm64` Kotlin/Native targets with embedded kernel archives; a single Apple arm64 archive dispatches FEAT_DotProd at runtime, so one build serves A12 through M-series. - -### Previously, in 0.39.1 - -- **Eager CPU ops run primitive FP32 fast paths.** The generic per-element paths (index-array allocations, boxed accessors, dtype dispatch) dominated on-device LLM decode — 83% of end-to-end SmolLM2-135M decode time on a Pixel 8a was non-matmul overhead even with the NEON backend. Hot ops (arithmetic, activations, unary math, softmax/logSoftmax, reductions, concat, reshape) now run flat primitive loops over the dense `FloatArray` buffer, benefiting every non-JVM target — Android, Kotlin/Native, JS/Wasm. `DirectCpuExecutionContext.ops` is also cached instead of rebuilt per access. -- **README points LLM users to SKaiNET-transformers.** A callout under "Start in 5 minutes" makes clear that LLM inference lives in the SKaiNET-transformers repository — this repo is the engine underneath. - -### Previously, in 0.39.0 - -- **On-device AI on Android — a NEON kernel backend.** New `skainet-backend-jni-cpu` module: the hand-tuned ARM matmul kernels reach Android through a JNI bridge (ART has no `java.lang.foreign`, so the FFM provider can never run there). Two `.so` tiers are built from the same sources and selected at load time from `/proc/cpuinfo` — a baseline `armv8-a` build that runs on every 64-bit core, and an `armv8.2-a+dotprod` build for the `vdotq_s32` Q4_K/Q6_K paths — so a single artifact is safe from Cortex-A53 up. Measured on a Pixel 8a: **~24 tok/s** SmolLM2-135M Q8_0 decode versus ~3.8 scalar (6.4x), clearing the on-device usability bar. The provider auto-registers via `ServiceLoader`; an app just adds the AAR. -- **Android GGUF loading no longer OOMs.** `createRandomAccessSource` returned `null` on Android, forcing every model load through a full-file heap read that exhausted the ART heap on real devices. It now streams via positional `FileChannel` reads across `skainet-io-gguf` / `-safetensors` / `-onnx`. -- **Published Kotlin/Native kernel klibs are linkable.** The static kernel archive is now embedded into the cinterop klib, so downstream K/N consumers of `skainet-backend-native-cpu` (`-linuxx64` / `-linuxarm64`, and the path future Apple targets will use) link with no manual setup. A NEON body was also added for the Q4_0 matmul kernel. -- **Tensor-storage correctness pass.** Fail-fast on unsupported GGUF quant types instead of silently dropping weights; truthful ownership labels and real byte counts in the storage layer; a materializable `FileBacked`/`Aliased` transfer path; and a rank-safe default `copyToFloatArray`. - -### Previously, in 0.38.0 - -- **Streaming KV-cache decode (dynamic dimensions)** — a first-class `Dim` vocabulary makes "dynamic extent" explicit instead of an overloaded `-1`, and the StableHLO emitter renders it as an MLIR `?`. One compiled vmfb now serves every autoregressive decode step with a growing cache, instead of one fixed cache length. Verified end-to-end: the full FunctionGemma `with_past` decode graph and the Moonshine v2 decoder (dynamic self *and* cross caches) self-compile from the DSL to a CPU vmfb — graphs that could not be compiled before. Static graphs are emitted byte-for-byte unchanged. -- **Narrow-float (BF16 + FP16) weights kept packed** — SafeTensors F16 and GGUF F16/BF16 weights load `KEEP_NATIVE`, two bytes per element at rest instead of widening to FP32, and reach format-specific matmul kernels still packed. Narrow floats are a storage width only: kernels widen to f32 lanes and accumulate in f32. -- **Both narrow formats now beat the FP32 SGEMM** — BF16 by 1.8–1.9x, FP16 by 1.5–1.7x on a 4096x11008 projection. Getting there took a zero-copy transpose for input-major weights (the per-token transpose previously widened the tensor elementwise, 4.4 s per projection), a native FFM FP16 kernel to match the existing BF16 one, and tiling both kernels so the weight is read once per matmul rather than once per input row. -- **Allocation-free shape-only tracing** — `VoidTensorOps` propagates shapes through a `ShapeOnlyTensorData` that allocates no backing buffer, so a dynamic extent flows through a whole decode trace instead of throwing on a negative-size allocation. - -See [CHANGELOG.md](CHANGELOG.md) for details and the full release history. +## What's New in 0.51.0 + +Ternary/BitNet weights join the memory-mapped weight story 0.50.0 started for every other quant +format: + +- **Off-heap ternary storage** — `BitNetB158TensorData` no longer risks the Android ART heap-cap + OOM; `Storage.copyInto`/`copyFrom` give every storage kind one shared bulk-copy primitive. +- **True zero-copy mmap** for `SEQUENTIAL`-layout (NeoGPU-converted) GGUFs, and a zero-copy + native gemv path for off-heap ternary weights on the JVM/FFM kernel. +- **`I2sAotConverter`** (GGUF → GGUF): convert I2_S tensors ahead of time so a controlled model + pipeline never pays a runtime repack. The IREE-facing counterpart lives in + [SKaiNET-IREE-tools](https://github.com/SKaiNET-developers/SKaiNET-IREE-tools). +- **Correctness fix** — scoped dense-FP32 activations no longer silently fall out of the + quantized matmul chooser (was producing wrong logits under `ScopedExecutionContext`). + +See [CHANGELOG.md](CHANGELOG.md) for full release notes, including every prior release. --- @@ -382,6 +335,12 @@ We love contributions! Whether it's a new operator, documentation, or a bug fix: Browse the full codebase documentation on [DeepWiki](https://deepwiki.com/SKaiNET-developers/SKaiNET). +### Contributors (0.51.0) + +- **Michal Harakal** ([@michalharakal](https://github.com/michalharakal)) — off-heap ternary + storage, zero-copy mmap for `SEQUENTIAL` I2_S, the AOT GGUF converter and its IREE-facing + counterpart in SKaiNET-IREE-tools, and the scoped dense-FP32 activation matmul-chooser fix + ### Contributors (0.49.0) - **Michal Harakal** ([@michalharakal](https://github.com/michalharakal)) — the SKEEP-003 memory & storage architecture end to end: M0/M1/M2 milestones, the weight-form and placement-resolution arcs, scope-recycled execution, multi-format footprint analysis, the compile-lane carriage arc, the BitNet/ternary kernel track, and the release docs diff --git a/docs/antora.yml b/docs/antora.yml index 9fbd0f10..b4948094 100644 --- a/docs/antora.yml +++ b/docs/antora.yml @@ -15,7 +15,7 @@ asciidoc: framework_name: SKaiNET # Current SKaiNET release — bump once per release; referenced as # {skainet_version} in dependency snippets (blocks need subs="attributes+"). - skainet_version: 0.49.0 + skainet_version: 0.51.0 ksp_version: 2.2.21-2.0.5 dokka_version: 2.1.0 asciidoctorj_version: 3.0.0 diff --git a/docs/modules/ROOT/pages/reference/kernel-support-matrix.adoc b/docs/modules/ROOT/pages/reference/kernel-support-matrix.adoc index 2f48adfc..37f85d44 100644 --- a/docs/modules/ROOT/pages/reference/kernel-support-matrix.adoc +++ b/docs/modules/ROOT/pages/reference/kernel-support-matrix.adoc @@ -1,7 +1,7 @@ = Kernel × platform support matrix :description: Which compute-kernel provider serves each weight format on each KMP target. -Generated from `kernel-support.json` (version `0.49.0`) by `KernelSupportMatrixTest` — registry introspection of the registered `KernelProvider` implementations. Do not edit by hand; run `./gradlew generateKernelMatrix` to refresh. +Generated from `kernel-support.json` (version `0.51.0`) by `KernelSupportMatrixTest` — registry introspection of the registered `KernelProvider` implementations. Do not edit by hand; run `./gradlew generateKernelMatrix` to refresh. Each cell is the best (highest-priority) provider that serves `Float32 × format` `matmul` on that platform: *native-ffm* (100) → *panama-vector* (50) → *scalar* (0). An empty cell (`—`) means no provider carries a kernel there (the format is dequant-to-FP32 only). diff --git a/docs/modules/ROOT/pages/reference/operators/generated/index.adoc b/docs/modules/ROOT/pages/reference/operators/generated/index.adoc index 763ab549..0b5d0fab 100644 --- a/docs/modules/ROOT/pages/reference/operators/generated/index.adoc +++ b/docs/modules/ROOT/pages/reference/operators/generated/index.adoc @@ -1,6 +1,6 @@ = AI-NET Operators Reference -Generated from version `0.49.0` on 2026-08-26 +Generated from version `0.51.0` on 2026-08-29 == Operators by Modality diff --git a/docs/modules/ROOT/pages/reference/ops-status-matrix.adoc b/docs/modules/ROOT/pages/reference/ops-status-matrix.adoc index cff069a6..2dba6c47 100644 --- a/docs/modules/ROOT/pages/reference/ops-status-matrix.adoc +++ b/docs/modules/ROOT/pages/reference/ops-status-matrix.adoc @@ -1,7 +1,7 @@ = Operator Coverage Matrix :description: Cross-backend status for every operator function in SKaiNET. -Generated from `operators.json` version `0.49.0` on 2026-08-26. +Generated from `operators.json` version `0.51.0` on 2026-08-29. Rows are `Operator.function` pairs. The `Validated` column shows whether the function's documentation has been DARC-validated by a reviewer (see xref:contributing/darc-workflow.adoc[DARC workflow]). Remaining columns are backends that appear in any function's `statusByBackend` map — a missing entry means the backend makes no claim about the function (treat it as "unknown", not "not supported"). diff --git a/docs/modules/ROOT/pages/tutorials/android-classifier-getting-started.adoc b/docs/modules/ROOT/pages/tutorials/android-classifier-getting-started.adoc index e809719d..2b49f6c0 100644 --- a/docs/modules/ROOT/pages/tutorials/android-classifier-getting-started.adoc +++ b/docs/modules/ROOT/pages/tutorials/android-classifier-getting-started.adoc @@ -19,12 +19,12 @@ An Android project with Kotlin. Add the SKaiNET modules: [source,kotlin] ---- dependencies { - implementation("sk.ainet.core:skainet-lang-core:0.49.0") // tensors, DSL, training - implementation("sk.ainet.core:skainet-backend-cpu:0.49.0") // CPU ops - implementation("sk.ainet.core:skainet-compile-dag:0.49.0") // autograd (training context) - implementation("sk.ainet.core:skainet-data-api:0.49.0") // Dataset / DataBatch - implementation("sk.ainet.core:skainet-data-simple:0.49.0") // embedded Iris - runtimeOnly("sk.ainet.core:skainet-backend-jni-cpu:0.49.0") // NEON kernels (see below) + implementation("sk.ainet.core:skainet-lang-core:0.51.0") // tensors, DSL, training + implementation("sk.ainet.core:skainet-backend-cpu:0.51.0") // CPU ops + implementation("sk.ainet.core:skainet-compile-dag:0.51.0") // autograd (training context) + implementation("sk.ainet.core:skainet-data-api:0.51.0") // Dataset / DataBatch + implementation("sk.ainet.core:skainet-data-simple:0.51.0") // embedded Iris + runtimeOnly("sk.ainet.core:skainet-backend-jni-cpu:0.51.0") // NEON kernels (see below) } ---- diff --git a/gradle.properties b/gradle.properties index e93f88cc..9d44d63c 100644 --- a/gradle.properties +++ b/gradle.properties @@ -1,5 +1,5 @@ GROUP=sk.ainet.core -VERSION_NAME=0.50.0 +VERSION_NAME=0.51.0 POM_DESCRIPTION=SKaiNET POM_URL=https://github.com/SKaiNET-developers/skainet/