Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,24 @@

## [Unreleased]

## [0.39.1] - 2026-08-11

Headline: **the packed Q5 gap is closed, and eager overhead off the JVM is gone.**
Native Q5_0/Q5_1 matmul kernels land in all three native tiers — FFM (JVM),
Kotlin/Native, and the Android JNI bridge — unblocking the packed `NATIVE_OPTIMIZED`
path for Q5_1-quantized checkpoints such as `functiongemma-270m`. On top of that,
the eager CPU ops gain primitive FP32 fast paths, removing the per-element
allocation/boxing overhead that dominated on-device LLM decode (83% of end-to-end
time on a Pixel 8a even with NEON matmul).

### Documentation

- **README points LLM users to SKaiNET-transformers**
([#923](https://github.com/SKaiNET-developers/SKaiNET/issues/923)): a callout under
"Start in 5 minutes" says plainly that LLM inference lives in the
SKaiNET-transformers repository — this repo is the engine underneath — and names
the `sk.ainet.transformers` artifacts and BOM to depend on.

### Added

- **Native Q5_0 / Q5_1 packed matmul kernels (FFM, Kotlin/Native, JNI).** 0.39.0 shipped
Expand All @@ -24,6 +35,7 @@
`JniKernelProvider`), each with parity tests against the scalar references. Unblocks the
packed Q5_1 path for `functiongemma-270m` "Q5_K_M" checkpoints (whose attention/FFN
weights are Q5_1) under `NATIVE_OPTIMIZED` — see SKaiNET-transformers#170. (#708)

### Performance

- **Primitive FP32 fast paths for the eager CPU ops** (`skainet-backend-cpu`,
Expand Down
14 changes: 12 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ Add the core dependencies (Gradle Kotlin DSL):
```kotlin
dependencies {
// Recommended: import the umbrella BOM and drop versions on the engine modules.
implementation(platform("sk.ainet:skainet-bom:0.39.0"))
implementation(platform("sk.ainet:skainet-bom:0.39.1"))

implementation("sk.ainet.core:skainet-lang-core")
implementation("sk.ainet.core:skainet-backend-cpu")
Expand Down Expand Up @@ -297,7 +297,13 @@ val withoutLabel = dataPipeline<RawDataset>()

---

## What's New in 0.39.0
## What's New in 0.39.1

- **Native Q5_0/Q5_1 packed matmul kernels — FFM, Kotlin/Native, JNI.** 0.39.0 could *load* Q5_0/Q5_1 packed but had no native-tier kernels for them: the JVM fell back to Panama and Kotlin/Native/Android to scalar. New plain-NEON C kernels (no dotprod/i8mm requirement, so they run on every AArch64 core) are wired into all three native consumers, each with parity tests against the scalar references. This unblocks the packed `NATIVE_OPTIMIZED` path for Q5_1-quantized checkpoints such as `functiongemma-270m`.
- **Eager CPU ops run primitive FP32 fast paths.** The generic per-element paths (index-array allocations, boxed accessors, dtype dispatch) dominated on-device LLM decode — 83% of end-to-end SmolLM2-135M decode time on a Pixel 8a was non-matmul overhead even with the NEON backend. Hot ops (arithmetic, activations, unary math, softmax/logSoftmax, reductions, concat, reshape) now run flat primitive loops over the dense `FloatArray` buffer, benefiting every non-JVM target — Android, Kotlin/Native, JS/Wasm. `DirectCpuExecutionContext.ops` is also cached instead of rebuilt per access.
- **README points LLM users to SKaiNET-transformers.** A callout under "Start in 5 minutes" makes clear that LLM inference lives in the SKaiNET-transformers repository — this repo is the engine underneath.

### Previously, in 0.39.0

- **On-device AI on Android — a NEON kernel backend.** New `skainet-backend-jni-cpu` module: the hand-tuned ARM matmul kernels reach Android through a JNI bridge (ART has no `java.lang.foreign`, so the FFM provider can never run there). Two `.so` tiers are built from the same sources and selected at load time from `/proc/cpuinfo` — a baseline `armv8-a` build that runs on every 64-bit core, and an `armv8.2-a+dotprod` build for the `vdotq_s32` Q4_K/Q6_K paths — so a single artifact is safe from Cortex-A53 up. Measured on a Pixel 8a: **~24 tok/s** SmolLM2-135M Q8_0 decode versus ~3.8 scalar (6.4x), clearing the on-device usability bar. The provider auto-registers via `ServiceLoader`; an app just adds the AAR.
- **Android GGUF loading no longer OOMs.** `createRandomAccessSource` returned `null` on Android, forcing every model load through a full-file heap read that exhausted the ART heap on real devices. It now streams via positional `FileChannel` reads across `skainet-io-gguf` / `-safetensors` / `-onnx`.
Expand Down Expand Up @@ -342,6 +348,10 @@ We love contributions! Whether it's a new operator, documentation, or a bug fix:

Browse the full codebase documentation on [DeepWiki](https://deepwiki.com/SKaiNET-developers/SKaiNET).

### Contributors (0.39.1)

- **Michal Harakal** ([@michalharakal](https://github.com/michalharakal)) — native Q5_0/Q5_1 packed matmul kernels across FFM, Kotlin/Native, and JNI (#708), primitive FP32 fast paths for the eager CPU ops (#949), README pointer to SKaiNET-transformers (#923)

### Contributors (0.39.0)

- **Michal Harakal** ([@michalharakal](https://github.com/michalharakal)) — Android JNI NEON kernel backend with runtime dotprod dispatch (#943, #945), Android `createRandomAccessSource` streaming loads (#922), cinterop klib archive embedding (#942), Q4_0 NEON kernel (#939), GGUF loader fail-fast (#919), tensor-storage correctness fixes (#927, #928, #929, #930, #931), AAR release publishing (#947)
Expand Down
2 changes: 1 addition & 1 deletion docs/antora.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ asciidoc:
framework_name: SKaiNET
# Current SKaiNET release — bump once per release; referenced as
# {skainet_version} in dependency snippets (blocks need subs="attributes+").
skainet_version: 0.39.0
skainet_version: 0.39.1
ksp_version: 2.2.21-2.0.5
dokka_version: 2.1.0
asciidoctorj_version: 3.0.0
6 changes: 3 additions & 3 deletions docs/modules/ROOT/pages/reference/kernel-support-matrix.adoc
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
= Kernel × platform support matrix
:description: Which compute-kernel provider serves each weight format on each KMP target.

Generated from `kernel-support.json` (version `0.39.0`) by `KernelSupportMatrixTest` — registry introspection of the registered `KernelProvider` implementations. Do not edit by hand; run `./gradlew generateKernelMatrix` to refresh.
Generated from `kernel-support.json` (version `0.39.1`) by `KernelSupportMatrixTest` — registry introspection of the registered `KernelProvider` implementations. Do not edit by hand; run `./gradlew generateKernelMatrix` to refresh.

Each cell is the best (highest-priority) provider that serves `Float32 × format` `matmul` on that platform: *native-ffm* (100) → *panama-vector* (50) → *scalar* (0). An empty cell (`—`) means no provider carries a kernel there (the format is dequant-to-FP32 only).

Expand All @@ -16,8 +16,8 @@ Each cell is the best (highest-priority) provider that serves `Float32 × format
| `Q4_K` | native-ffm | native-jni | scalar | scalar | scalar
| `Q6_K` | panama-vector | native-jni | scalar | scalar | scalar
| `Q5_K` | native-ffm | native-jni | scalar | scalar | scalar
| `Q5_1` | panama-vector | panama-vector | scalar | scalar | scalar
| `Q5_0` | panama-vector | panama-vector | scalar | scalar | scalar
| `Q5_1` | native-ffm | native-jni | scalar | scalar | scalar
| `Q5_0` | native-ffm | native-jni | scalar | scalar | scalar
|===

See also the eager backends & kernels mindmap (`docs/eager-execution-backends-and-kernels.md`) for the narrative overview and gaps.
2 changes: 1 addition & 1 deletion gradle.properties
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
GROUP=sk.ainet.core
VERSION_NAME=0.39.0
VERSION_NAME=0.39.1
POM_DESCRIPTION=SKaiNET

POM_URL=https://github.com/SKaiNET-developers/skainet/
Expand Down
Loading