From 0a0cbbc27980167ae307556e5ce4dc4928e2f9c7 Mon Sep 17 00:00:00 2001 From: Michal Harakal Date: Wed, 26 Aug 2026 21:48:15 +0200 Subject: [PATCH 1/2] =?UTF-8?q?release:=200.49.0=20=E2=80=94=20the=20SKEEP?= =?UTF-8?q?-003=20architecture,=20complete?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The deliberate 0.40.1 → 0.49.0 jump (~100 merged PRs): one storage model, resolver-owned decisions, scope-recycled eager execution, and a compile lane that carries what the runtime decides. This is the release SKaiNET-transformers and the IREE conformity pipeline build on. - CHANGELOG: full 0.49.0 notes with an explicit Breaking-changes migration map (legacy loader axes → WeightForm; dead placement machinery → AllocationResolver; skainet.tensor_encodings → skainet.tensor_layouts). - README: What's New in 0.49.0, BOM coordinate, contributors incl. @AjithGoveas (Iris provider). - Versions: gradle.properties, docs/antora.yml skainet_version, tutorial coordinates; kernel-support matrix and operator reference regenerated at 0.49.0. - Docs audit (62 pages): every reference presenting removed API as current is fixed (WeightOrientation/staging/quantPolicy prose, SKEEP-002 snippet, tensor_encodings seam); overtaken claims updated (op outputs + carriage landed, CI api-compatibility leg + branch protection, native FFM shipped, device placement resolved, ternary 'in progress' → landed, eager-execution kernel matrix gains the exact-FP32 ternary and BITNET_PLANES rows); SKEEP-003 status → Implemented and SKEEP-003a added to the skeep index; the dangling docs/design/memory prose repaired; a leaking literal block and a wrong cross-module xref fixed; the fake StableHloOptimizer(debugMode) snippet replaced with the real HloGenerator target API. Known non-blockers, left deliberately: how-to/java-cli-app.adoc and tutorials/kllama-getting-started.adoc are orphaned from nav (content review needed before linking). Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 133 ++++++++++++++++-- README.md | 48 +++++-- docs/antora.yml | 2 +- .../pages/contributing/build-from-source.adoc | 6 +- .../ROOT/pages/contributing/index.adoc | 2 +- .../pages/contributing/matmul-kernels.adoc | 2 +- .../pages/explanation/eager-execution.adoc | 2 + .../ROOT/pages/explanation/memory-model.adoc | 8 +- .../pages/explanation/operator-design.adoc | 6 +- .../explanation/packed-weight-layout.adoc | 4 +- .../pages/explanation/perf/simd-kernels.adoc | 2 +- .../pages/explanation/virtual-tensors.adoc | 10 +- .../ROOT/pages/how-to/plan-model-memory.adoc | 10 +- .../ROOT/pages/reference/architecture.adoc | 5 +- .../reference/kernel-support-matrix.adoc | 2 +- .../reference/operators/generated/index.adoc | 2 +- .../operators/generated/tensorops.adoc | 77 ++++++++++ .../pages/reference/ops-status-matrix.adoc | 7 +- .../android-classifier-getting-started.adoc | 12 +- .../pages/tutorials/hlo-getting-started.adoc | 14 +- .../tutorials/ternary-getting-started.adoc | 2 +- .../002-android-offheap-tensor-storage.adoc | 10 +- .../pages/003-unified-tensor-storage.adoc | 14 +- ...03a-placement-and-planning-resolution.adoc | 23 +-- docs/modules/skeep/pages/index.adoc | 6 +- gradle.properties | 2 +- 26 files changed, 327 insertions(+), 84 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fabb7b0fc..78fdfca62 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,18 +2,131 @@ ## [Unreleased] -### Docs +## [0.49.0] - 2026-08-26 + +Headline: **the SKEEP-003 memory & storage architecture, complete — from accepted proposal to shipped system.** +One storage model (`Storage` / `Scope` / `Format` / `Layout` / `TensorView`), resolver-owned decisions +(what form a weight takes, where its bytes live — never decided by the model author), scope-recycled eager +execution (flat-memory decode), and a compile lane that carries what the runtime decides into the exported +MLIR and `.irpa`. The version jump (0.40.1 → 0.49.0, ~100 merged PRs) is deliberate: this is the release +downstream repositories (SKaiNET-transformers, the IREE conformity pipeline) should build on, and it +removes every façade the architecture replaced. See **Breaking changes** below for the migration map. + +### Breaking changes + +- **The three legacy loader axes are gone** ([#1159](https://github.com/SKaiNET-developers/SKaiNET/issues/1159)): + `QuantPolicy`, `StagingPolicy` and `WeightOrientation` are deleted. The loader takes one + `WeightForm(encoding, order, shape, residency)` (uniform `weightForm` or per-tensor `weightFormFor`); + migration: `DEQUANTIZE_TO_FP32` → `EncodingRequest.DequantizeTo(FP32)`, `StagingPolicy.MAPPED` → + `WeightForm(residency = WeightResidency.MAPPED)`, `WeightOrientation.OUT_IN` → `WeightShapeOrientation.OUT_IN`. + `AndroidGguf.loader` takes a `WeightForm` (default mapped residency). +- **The dead placement machinery is gone** ([#1142](https://github.com/SKaiNET-developers/SKaiNET/issues/1142)): + `@Place`/`@Weights` (declared, retained, read by nothing), `sk.ainet.lang.tensor.storage.MemoryPlanner`, + `StorageSpec`, and `Placement.residency`/`Residency`. Lifetime is `ScopeKind`; weight staging is + `WeightForm.WeightResidency`; placement is decided by `AllocationResolver` (see Added). `Placement` + itself stays (KV-cache stores carry it); `@KvCache`/`@KvCacheBypass` moved to `KvCacheAnnotations.kt`. +- **The `skainet.tensor_encodings` module attribute is gone** + ([#1179](https://github.com/SKaiNET-developers/SKaiNET/issues/1179)): replaced by the machine-readable + `skainet.tensor_layouts` (`{kind, block_elems, block_bytes, bits, block_order}`); nothing outside this + repository read the old names dictionary. +- `LogicalDType` is deprecated end to end in favour of `DType` + ([#1014](https://github.com/SKaiNET-developers/SKaiNET/issues/1014)); removal at the next major. + +### Added -- **SKEEP-003 accepted — memory & storage architecture design record and roadmap** - ([#932](https://github.com/SKaiNET-developers/SKaiNET/issues/932)). - `docs/modules/skeep/pages/003-unified-tensor-storage.adoc` moves to *Accepted* and records the - thirteen design decisions (storage-first end-state delivered incrementally: `Storage` / `TensorView` / - `Tensor`, scopes, `Format(dtype, encoding)`, `TensorId`, kernel dispatch on declared formats, 2 GB - planner profile, `LogicalDType` → `DType` merge). The full proposal and the M0/M1/M2 milestone PRD are - committed under `docs/design/memory/`; the work is tracked as milestone issues - [#1001](https://github.com/SKaiNET-developers/SKaiNET/issues/1001), +- **The memory model (SKEEP-003 M0 "know before you load" / M1 "flat decode" / M2 "1.58-bit on a 2 GB board")** + ([#1001](https://github.com/SKaiNET-developers/SKaiNET/issues/1001), [#1002](https://github.com/SKaiNET-developers/SKaiNET/issues/1002), - [#1003](https://github.com/SKaiNET-developers/SKaiNET/issues/1003) with one sub-issue per feature branch. + [#1003](https://github.com/SKaiNET-developers/SKaiNET/issues/1003); slices #1004–#1042): + `Storage` (Heap/OffHeap/Mapped, ownership + liveness — a use-after-free is a loud + `StorageClosedException`), `Scope` (`ModelScope`/`ForwardScope` slab with `reset()`), + `Format(dtype, encoding)`, `Layout` (strides/offset/block geometry), `TensorView` with `prepack()` as + the visible relayout and `materialize()` as the single copy point; `TraceSink` events (allocations, + scope resets, adapter insertions); `MemoryPlan`/`MemoryPlans` header-only planning with budgets, + suggestions and a plan-vs-actual check; the `skainet-plan` CLI; `PlannerProfile` (`MOBILE_2GB` with + automatic KV quantization — and `strict`, so a missing kernel refuses instead of silently costing + several times the weight, [#1128](https://github.com/SKaiNET-developers/SKaiNET/pull/1128)); + Android mmap loading + device fit checks; KV cache preallocated in model scope with declared formats; + kernel dispatch on declared formats (`KernelKey`, registry-backed capabilities); the decode harness + and M2 acceptance runs. +- **Weight forms — one resolved decision instead of three caller flags** + ([#1109](https://github.com/SKaiNET-developers/SKaiNET/issues/1109) arc, #1114–#1120): + `WeightForm` (encoding × byte order × shape × residency) resolved by `WeightFormResolver` from + *what the file holds × the profile × what the backend's kernels can feed*; the loader honours it, + the plan prices it (a resolved dequantization shows in the table, not at the OOM), conversions are + traced, and packed weights can load directly in kernel-feed order + ([#1120](https://github.com/SKaiNET-developers/SKaiNET/issues/1120)). +- **Placement is resolver-owned** ([#1133](https://github.com/SKaiNET-developers/SKaiNET/issues/1133) → + #1142–#1144): `AllocationResolver.resolve(weight, profile, platform)` decides memory domain and scope + (mapping requires: the form asks, the platform can, the bytes are the file's bytes); + `AllocationResolver.explain()` renders every decision with its reason pre-load; `ResolvedGguf` wires + plan → load with the documented user-wins precedence (per-tensor `weightFormFor` > uniform + `weightForm` > resolver). +- **Scope-recycled eager execution** ([#1135](https://github.com/SKaiNET-developers/SKaiNET/issues/1135) → + #1145/#1146/#1173): `ExecutionContext.memoryScope` is consulted by tensor creation *and* op outputs + (`TensorDataFactory.adoptFloatArray`, `ScopedTensorDataFactory`), so + `ctx.forwardScope(slabFloats) { … }` gives steady-state decode that allocates zero new slab bytes per + step; the FP32 fast paths and the JVM Panama vector kernels are offset-aware, so slab-backed tensors + keep SIMD speed. +- **Model-footprint analysis for GGUF, safetensors and ONNX** + ([#1169](https://github.com/SKaiNET-developers/SKaiNET/issues/1169)): header-only `planInput` for all + three formats (ONNX `external_data` sidecars priced correctly — multi-GB models no longer report ~0 + bytes; sizes `Long`-safe), and `PlannerProfile.EDGE` for embedded devices where the budget *is* the + usable RAM. "Will it fit in ~2.1 GB?" is answered in seconds, without reading a tensor payload. +- **The compile lane carries what the runtime decides** + ([#1147](https://github.com/SKaiNET-developers/SKaiNET/issues/1147) → #1178/#1179/#1180): + `TensorRef` carries tensor identity (`TraceSession.identify`, registered from + `trainableParameters()`), the encoding *object* (block size intact) and packed block order across the + trace→graph boundary; the emitted module header declares structural facts per tensor + (`skainet.tensor_layouts`); `ExternalParameterRef` declares block order to the `.irpa` consumer; + `HloGenerator.generate(target = …)` runs the optimizer pipeline on the production path with + `LayoutAssignmentPass` (rank-2 packed weights get kernel-feed order; the tape's carried facts are + never overridden), and the `ResolvedComputeGraph` seams surface exactly the decisions made. +- **BitNet / ternary compute track** (#1033, #1040/#1041, #1136–#1141, #1150): + ternary encodings (`TQ1_0`/`TQ2_0`, `BITNET_B1_58`, `BITNET_PLANES` multi-plane packing), i2s GGUF + import, the vendored NeoGPU ternary f32 NEON kernel (MIT, verbatim) exposed through FFM, JNI and + Kotlin/Native including a fused lm_head kernel, requant adapters, NEON BitNet packing, and ternary + benchmarks + getting-started docs. +- **Iris dataset provider** ([#1044](https://github.com/SKaiNET-developers/SKaiNET/issues/1044), + [#1101](https://github.com/SKaiNET-developers/SKaiNET/pull/1101), contributed by @AjithGoveas): + the embedded 150-row dataset used by the new Android classifier tutorial. +- Sliding-window KV SDPA ([#1036](https://github.com/SKaiNET-developers/SKaiNET/issues/1036)) and + KV formats declared by the store ([#1077](https://github.com/SKaiNET-developers/SKaiNET/issues/1077)). + +### Fixed + +- Packed block order end to end: feed-order bytes in a type claiming canonical order decoded to + plausible garbage ([#1124](https://github.com/SKaiNET-developers/SKaiNET/issues/1124), + [#1126](https://github.com/SKaiNET-developers/SKaiNET/pull/1126)); `TensorData` now declares its + `BlockOrder` and every reader agrees on the same bytes. +- Packed ternary weights reach dispatch through the Wᵀ marker — ANY packed block storage routes through + the marker path ([#1136](https://github.com/SKaiNET-developers/SKaiNET/issues/1136), + [#1181](https://github.com/SKaiNET-developers/SKaiNET/pull/1181)). +- M2 acceptance page-fault flake ([#1107](https://github.com/SKaiNET-developers/SKaiNET/pull/1107)); + safetensors dtype mapper no longer prints a WARNING into stdout mid-parse (#1169). + +### CI & process + +- `apiCheck` has its own named PR leg (`test (api-compatibility)`) instead of hiding inside + golden-parity ([#1176](https://github.com/SKaiNET-developers/SKaiNET/pull/1176)), after a stale dump + reached `develop` unnoticed ([#1174](https://github.com/SKaiNET-developers/SKaiNET/pull/1174)); + branch protection on `develop` now requires the full `build-job` aggregator, admins included. +- Android per-target API dumps dropped (jvm + klib only, + [#1111](https://github.com/SKaiNET-developers/SKaiNET/pull/1111)). + +### Docs + +- The memory model as built: `explanation/memory-model.adoc`, `explanation/packed-weight-layout.adoc` + ([#1106](https://github.com/SKaiNET-developers/SKaiNET/pull/1106) — design drafts under + `docs/design/` retired in favour of Antora pages), `explanation/virtual-tensors.adoc` (the ML + Drift-style split, with diagrams), and SKEEP-003a (`skeep/003a-placement-and-planning-resolution.adoc`) + recording the P7/P8 resolutions. +- Tutorials whose code cannot rot: the Android classifier getting-started and the ternary + getting-started, with snippets compiled *and executed* in CI by `skainet-docs-samples` (the Iris + training loop asserts held-out accuracy ≥ 0.80). +- SKEEP-003 accepted ([#932](https://github.com/SKaiNET-developers/SKaiNET/issues/932)) with its + thirteen design decisions; how-to: plan a model's memory before loading it, including the + embedded-device (`edge`) verdict. ## [0.40.1] - 2026-08-12 diff --git a/README.md b/README.md index f636d3212..18219456f 100644 --- a/README.md +++ b/README.md @@ -51,7 +51,7 @@ Add the core dependencies (Gradle Kotlin DSL): ```kotlin dependencies { // Recommended: import the umbrella BOM and drop versions on the engine modules. - implementation(platform("sk.ainet:skainet-bom:0.40.1")) + implementation(platform("sk.ainet:skainet-bom:0.49.0")) implementation("sk.ainet.core:skainet-lang-core") implementation("sk.ainet.core:skainet-backend-cpu") @@ -296,7 +296,39 @@ val withoutLabel = dataPipeline() --- -## What's New in 0.40.1 +## What's New in 0.49.0 + +The **SKEEP-jump release**: 0.40.1 → 0.49.0, ~100 merged PRs — the SKEEP-003 memory & storage +architecture complete, from accepted proposal to shipped system. This is the release downstream +repositories (SKaiNET-transformers, the IREE conformity pipeline) should build on. It **removes +every façade the architecture replaced** — see the Breaking-changes section of +[CHANGELOG.md](CHANGELOG.md) for the migration map (`QuantPolicy`/`StagingPolicy`/`WeightOrientation` +→ `WeightForm`; `@Place`/`@Weights`/`StorageSpec`/old `MemoryPlanner` → `AllocationResolver`). + +- **One storage model** — `Storage` / `Scope` / `Format` / `Layout` / `TensorView`: enforced ownership + (use-after-free throws, loudly), scoped lifetimes, `prepack()` as the visible relayout, + `materialize()` as the single copy point. +- **Decisions are resolved, not declared** — `WeightFormResolver` picks a weight's in-memory form from + *file × profile × kernels*; `AllocationResolver` picks domain and scope, and `explain()` says why, + per tensor, before a byte of payload is read. You always outrank the resolver + (per-tensor `weightFormFor` > uniform `weightForm` > resolver). +- **Flat-memory decode** — `ctx.forwardScope(slabFloats) { … }` recycles one slab per step; creation + *and op outputs* draw from it, and the FP32 fast paths + JVM Panama vector kernels are offset-aware, + so scoped tensors keep SIMD speed. Steady-state decode allocates zero new slab bytes per step. +- **"Will it fit?" in seconds, any format** — header-only footprint plans for GGUF, **safetensors and + ONNX** (external-data sidecars priced correctly), with `PlannerProfile.EDGE` for embedded devices: + `skainet-plan model.onnx --profile edge --budget 2.1G`, exit code 0/1. +- **The compile lane carries what the runtime decides** — tensor identity, structural encodings + (`skainet.tensor_layouts`: block sizes and bit widths as integers, not names) and block order flow + into the exported MLIR and `.irpa`; `HloGenerator.generate(target = …)` runs the first layout pass + on the production path. +- **BitNet / ternary compute** — `BITNET_PLANES` multi-plane packing, i2s GGUF import, and the vendored + NeoGPU ternary f32 NEON kernel through FFM, JNI and Kotlin/Native, including a fused lm_head kernel. +- **Docs that cannot rot** — the Android classifier and ternary getting-started tutorials are compiled + *and executed* in CI (the Iris training loop asserts held-out accuracy ≥ 0.80), plus the + virtual-tensors explanation and SKEEP-003a resolution record. + +### Previously, in 0.40.1 - **Correctness hotfix: packed-quant `transpose()` was silently wrong, not crashing.** `ops.matmul(x, ops.transpose(W))` on a packed-quantized weight (Q4_0/Q5_0/Q5_1/Q8_0/Q4_K/Q5_K/Q6_K) with more than one quant block per row produced silently incorrect output — sometimes all-zero — across the scalar, Panama-vector, *and* native (FFM/JNI) kernel tiers, with no exception raised. `transpose()` now performs a real block-grid byte permutation instead of a shape-only relabel; a misaligned packed tensor now throws instead of silently truncating. Closes [#968](https://github.com/SKaiNET-developers/SKaiNET/issues/968). **Upgrading is strongly recommended** for anyone calling `ops.transpose()` on packed-quantized weights. @@ -326,13 +358,6 @@ val withoutLabel = dataPipeline() - **Both narrow formats now beat the FP32 SGEMM** — BF16 by 1.8–1.9x, FP16 by 1.5–1.7x on a 4096x11008 projection. Getting there took a zero-copy transpose for input-major weights (the per-token transpose previously widened the tensor elementwise, 4.4 s per projection), a native FFM FP16 kernel to match the existing BF16 one, and tiling both kernels so the weight is read once per matmul rather than once per input row. - **Allocation-free shape-only tracing** — `VoidTensorOps` propagates shapes through a `ShapeOnlyTensorData` that allocates no backing buffer, so a dynamic extent flows through a whole decode trace instead of throwing on a negative-size allocation. -### Previously, in 0.37.0 - -- **`Lstm` layer** — single-layer, batch-first LSTM built from existing primitives only, with `torch.nn.LSTM`-compatible gate order and a caller-owned `LstmState` + `step()` API. -- **Training essentials** — real inverted `Dropout`, mutable optimizer `lr` plus `linearWarmupCosineDecay`, bias-less and `open` `Linear`. -- **Attention scale fix** — `scaledDotProductAttention` at its default scale multiplied every score by zero on the CPU backend; it now resolves to `1/sqrt(headDim)` as documented. -- **Autograd correctness** — `CrossEntropyLoss` no longer detaches the tape, and `softmax`/`logSoftmax`/`variance` backward now work for rank ≥ 3. - See [CHANGELOG.md](CHANGELOG.md) for details and the full release history. --- @@ -357,6 +382,11 @@ We love contributions! Whether it's a new operator, documentation, or a bug fix: Browse the full codebase documentation on [DeepWiki](https://deepwiki.com/SKaiNET-developers/SKaiNET). +### Contributors (0.49.0) + +- **Michal Harakal** ([@michalharakal](https://github.com/michalharakal)) — the SKEEP-003 memory & storage architecture end to end: M0/M1/M2 milestones, the weight-form and placement-resolution arcs, scope-recycled execution, multi-format footprint analysis, the compile-lane carriage arc, the BitNet/ternary kernel track, and the release docs +- **Ajith Goveas** ([@AjithGoveas](https://github.com/AjithGoveas)) — Iris dataset provider (#1044, #1101), now powering the Android classifier tutorial + ### Contributors (0.40.1) - **Michal Harakal** ([@michalharakal](https://github.com/michalharakal)) — packed-quant `transpose()` block-grid correctness fix, all three kernel tiers (#968, #969) diff --git a/docs/antora.yml b/docs/antora.yml index a03272633..9fbd0f10a 100644 --- a/docs/antora.yml +++ b/docs/antora.yml @@ -15,7 +15,7 @@ asciidoc: framework_name: SKaiNET # Current SKaiNET release — bump once per release; referenced as # {skainet_version} in dependency snippets (blocks need subs="attributes+"). - skainet_version: 0.40.1 + skainet_version: 0.49.0 ksp_version: 2.2.21-2.0.5 dokka_version: 2.1.0 asciidoctorj_version: 3.0.0 diff --git a/docs/modules/ROOT/pages/contributing/build-from-source.adoc b/docs/modules/ROOT/pages/contributing/build-from-source.adoc index 3709d83d1..e9b2780fc 100644 --- a/docs/modules/ROOT/pages/contributing/build-from-source.adoc +++ b/docs/modules/ROOT/pages/contributing/build-from-source.adoc @@ -167,8 +167,10 @@ Applying `sk.ainet.multiplatform` to the *root* project is not supported and fai === Pre-PR Gate and the Packed-Encoding Golden Parity Tests -CI runs the test legs per target (`jvmTest`, `jsTest wasmJsTest wasmWasiTest`, `linuxX64Test`, -`assemble`) and, since the SKEEP-003 roadmap, a `golden-parity` leg. Before opening a PR, run the +CI runs the test legs per target (`jvm`, `js-wasm`, `native`, `android`, plus `assemble`), the +`golden-parity` packed-encoding gate, and — since 0.49.0 — a dedicated `api-compatibility` leg +running `apiCheck`, so a stale API dump fails under its own name. Branch protection on `develop` +requires the aggregated `build-job` to be green, admins included. Before opening a PR, run the same set locally: [source,bash] diff --git a/docs/modules/ROOT/pages/contributing/index.adoc b/docs/modules/ROOT/pages/contributing/index.adoc index 13bf39fe1..ac34c81df 100644 --- a/docs/modules/ROOT/pages/contributing/index.adoc +++ b/docs/modules/ROOT/pages/contributing/index.adoc @@ -20,7 +20,7 @@ The Contributing section is for the engineer who: - Maintains the CI workflows (smoke runs on `ubuntu-latest`, full publishable runs on the self-hosted lane). - Adds or replaces kernels in the CPU backend (scalar, Panama Vector, - the planned native FFM provider). + the native FFM provider). - Operates the self-hosted runner that publishes benchmark results. - Drafts or reviews durable API and architecture proposals in the xref:skeep:index.adoc[SKEEP proposal track]. diff --git a/docs/modules/ROOT/pages/contributing/matmul-kernels.adoc b/docs/modules/ROOT/pages/contributing/matmul-kernels.adoc index 66d6c2d41..dd31a1526 100644 --- a/docs/modules/ROOT/pages/contributing/matmul-kernels.adoc +++ b/docs/modules/ROOT/pages/contributing/matmul-kernels.adoc @@ -144,7 +144,7 @@ flowchart LR subgraph Selection["At JVM start: ServiceLoader scan"] P1["ScalarProvider
priority 0, always"] P2["PanamaVectorProvider
priority 50, JDK 21+"] - P3["NativeProvider
priority 100, planned"] + P3["NativeProvider
priority 100 (FFM)"] P1 --> Registry P2 --> Registry P3 --> Registry diff --git a/docs/modules/ROOT/pages/explanation/eager-execution.adoc b/docs/modules/ROOT/pages/explanation/eager-execution.adoc index 3b7997877..7cd46060a 100644 --- a/docs/modules/ROOT/pages/explanation/eager-execution.adoc +++ b/docs/modules/ROOT/pages/explanation/eager-execution.adoc @@ -60,6 +60,8 @@ mindmap | Q5_1 | ✅ | ✅ | ✅ | ✅ | ✅ | Q5_0 | ✅ | ✅ | ✅ | ✅ | ✅ | TQ2_0 / BitNet b1.58 (int8 activations) | ✅ | — | — | ✅ NEON | — +| BitNet b1.58 (exact FP32 activations, vendored NeoGPU kernel) | ✅ | ✅ FFM | — | ✅ JNI + fused lm_head | ✅ K/N cinterop +| BITNET_PLANES (multi-plane ternary packing) | ✅ | via kernel pack | — | via kernel pack | via kernel pack | Q2_K / Q3_K / Q8_K / IQ4 | ❌ (dequant to FP32 only) | ❌ | ❌ | ❌ | ❌ |=== diff --git a/docs/modules/ROOT/pages/explanation/memory-model.adoc b/docs/modules/ROOT/pages/explanation/memory-model.adoc index 685bc4f31..c520a878e 100644 --- a/docs/modules/ROOT/pages/explanation/memory-model.adoc +++ b/docs/modules/ROOT/pages/explanation/memory-model.adoc @@ -206,7 +206,7 @@ ARMv8.2 Cortex-A55 reference board. | Ternary encodings and `bitnet_gemv` | `TQ1_0`, `TQ2_0`, BitNet b1.58; reference kernel plus a NEON pack on Android | Memory plan, device profiles, fit check | `skainet-plan`, mobile/desktop/native profiles, two-pool device check | Tracing, metrics, process probe | Perfetto/JFR/Android exporters; TTFT, tok/s, effective bandwidth, RSS and page faults -| Mapped staging | `quantPolicy × staging` on one GGUF loader; dense FP32 tensors served from file-backed pages +| Mapped staging | one `WeightForm` per weight (`residency = MAPPED`) on one GGUF loader; dense FP32 tensors served from file-backed pages |=== === Measured @@ -248,5 +248,7 @@ Known limits, stated rather than implied: `ByteArray`s, so a buffer-aware kernel is future work. Mapping therefore lifts the Android heap ceiling for dense checkpoints, not yet for a Q4_K_M one. * *Whole-file mapping only.* Files larger than 2 GB are refused rather than mapped in windows. -* *Device placement and graph-level planning* (allocating a whole forward pass at once) were -always out of scope for this round. +* *Graph-level planning* (allocating a whole forward pass at once) stays in the downstream +compiler by decision (#1134). Device placement itself was resolved in 0.49.0: `AllocationResolver` +decides memory domain and scope per weight (#1133/#1143) — see +xref:skeep:003a-placement-and-planning-resolution.adoc[SKEEP-003a]. diff --git a/docs/modules/ROOT/pages/explanation/operator-design.adoc b/docs/modules/ROOT/pages/explanation/operator-design.adoc index 6ce1d8d4f..ff8a56514 100644 --- a/docs/modules/ROOT/pages/explanation/operator-design.adoc +++ b/docs/modules/ROOT/pages/explanation/operator-design.adoc @@ -104,8 +104,8 @@ Your article must be written in AsciiDoc and include the following sections (use • Pointers (`xref:`) to human-written math/semantics sections - Provide example AsciiDoc fragment: - [source,adoc] - ---- + [listing] + ...... // generated: do not edit == TensorOps.matmul @@ -124,7 +124,7 @@ Your article must be written in AsciiDoc and include the following sections (use |=== See xref:theory/matmul.adoc#definition[MatMul semantics] and xref:examples/matmul.adoc#examples[Examples]. - ---- + ...... - Demonstrate combining generated and human-written docs via `include::` and `xref:`, with a small folder layout: [source,text] diff --git a/docs/modules/ROOT/pages/explanation/packed-weight-layout.adoc b/docs/modules/ROOT/pages/explanation/packed-weight-layout.adoc index 8945ccdae..6f46fc035 100644 --- a/docs/modules/ROOT/pages/explanation/packed-weight-layout.adoc +++ b/docs/modules/ROOT/pages/explanation/packed-weight-layout.adoc @@ -73,7 +73,7 @@ GGUF writes dimensions in `ne` order — fastest-varying first — so a weight t Only the label is wrong, and the label is what the relayout reads: driven by `[in, out]` it permutes the wrong grid, or refuses because `out` is not a multiple of the block size. -* `StreamingGgufParametersLoader(weightOrientation = WeightOrientation.OUT_IN)` fixes the label at +* `StreamingGgufParametersLoader(weightForm = WeightForm(shape = WeightShapeOrientation.OUT_IN))` fixes the label at the boundary, reversing 2-D weights only. Nothing about the bytes changes. It defaults to `AS_STORED` — the historical behaviour — because reversing shapes changes what every consumer sees; new code should ask for `OUT_IN`. @@ -83,7 +83,7 @@ when the *first* dimension is block-aligned and the second is not — exactly wh produces — and stays quiet when both are aligned and it cannot tell. The two GGUF readers still disagree about this: the legacy `GGUFReader` reverses dimensions, the -streaming one does not. `WeightOrientation` is how a caller states which it wants rather than +streaming one does not. `WeightForm.shape` (`WeightShapeOrientation`) is how a caller states which it wants rather than discovering it. == For a downstream repository diff --git a/docs/modules/ROOT/pages/explanation/perf/simd-kernels.adoc b/docs/modules/ROOT/pages/explanation/perf/simd-kernels.adoc index ea2c3081f..d839a8d3e 100644 --- a/docs/modules/ROOT/pages/explanation/perf/simd-kernels.adoc +++ b/docs/modules/ROOT/pages/explanation/perf/simd-kernels.adoc @@ -31,7 +31,7 @@ don't carry that"). Three providers ship with the CPU backend today: | Provider | Priority | When available | Notes | `ScalarKernelProvider` | 0 | always | Three-loop reference; the parity baseline. | `PanamaVectorKernelProvider` | 50 | JDK 21+ with `--add-modules jdk.incubator.vector` and `skainet.cpu.vector.enabled != false` | Tile-blocked FMA; the production winner on every supported JVM. -| (future) `NativeKernelProvider` | 100 | JDK 22+ with the native lib loaded | Designed but not yet shipped. +| `NativeKernelProvider` (FFM) | 100 | JDK 22+ with the native lib loaded | Shipped since 0.22.0; the priority-100 winner where available. |=== == Why the SPI exists diff --git a/docs/modules/ROOT/pages/explanation/virtual-tensors.adoc b/docs/modules/ROOT/pages/explanation/virtual-tensors.adoc index 40b934bdd..e831d64b5 100644 --- a/docs/modules/ROOT/pages/explanation/virtual-tensors.adoc +++ b/docs/modules/ROOT/pages/explanation/virtual-tensors.adoc @@ -113,10 +113,12 @@ counterpart; `Scope.Ambient` (plain GC) remains the default everywhere. == Known limits, stated rather than implied -* The virtualization described here is the *eager and IO* side. The compile pipeline - (tape → StableHLO) does not carry `Layout` yet; by decision (#1134), graph-level memory - planning stays in the downstream compiler, and core's future obligation there is metadata - carriage only (#1147), gated on an in-repo consumer (#1148). +* By decision (#1134), graph-level memory *planning* stays in the downstream compiler; core's + obligation on the compile lane is metadata carriage, and that landed (#1147): tensor identity, + structural encodings and block order ride from the tape into the exported MLIR + (`skainet.tensor_layouts`) and `.irpa` references, with `LayoutAssignmentPass` making the first + layout decision on the production path. Validation against IREE waits for the conformity + pipeline (#1148). * `Storage.OffHeap` exists on every platform that can honour it, but no tensor factory allocates through it yet — off the managed heap today means *mapped, read-only* weights (dense F32; packed formats still land on the heap until the buffer-aware kernel SPI of diff --git a/docs/modules/ROOT/pages/how-to/plan-model-memory.adoc b/docs/modules/ROOT/pages/how-to/plan-model-memory.adoc index 1caaa93f9..82b5d2c1a 100644 --- a/docs/modules/ROOT/pages/how-to/plan-model-memory.adoc +++ b/docs/modules/ROOT/pages/how-to/plan-model-memory.adoc @@ -119,14 +119,14 @@ can be impossible on the heap and unremarkable when mapped. [source,kotlin] ---- -val loader = AndroidGguf.loader(path) // staging = MAPPED, quantPolicy = NATIVE_OPTIMIZED +val loader = AndroidGguf.loader(path) // weightForm = WeightForm(residency = MAPPED) loader.load(ctx, FP32::class) { name, tensor -> model.put(name, tensor) } ---- -`staging` and `quantPolicy` are independent axes of one loader: `quantPolicy` decides *what the -values are*, `staging` decides *where the bytes live*. `MAPPED` serves dense FP32 tensors as -zero-heap views over file-backed pages, and falls back to `HEAP` when the platform cannot map or -the source has no path. +The loader takes one `WeightForm` — encoding × byte order × shape × residency — resolved +for you by `WeightFormResolver` (or passed explicitly; your form always wins). `MAPPED` +residency serves dense FP32 tensors as zero-heap views over file-backed pages, and falls +back to the heap when the platform cannot map or the source has no path. NOTE: Packed (quantized) tensors still arrive as heap arrays under `MAPPED`, because the packed matmul kernels take `ByteArray`s. Mapping lifts the heap ceiling for dense checkpoints today, not diff --git a/docs/modules/ROOT/pages/reference/architecture.adoc b/docs/modules/ROOT/pages/reference/architecture.adoc index 1e722144c..42e4d0972 100644 --- a/docs/modules/ROOT/pages/reference/architecture.adoc +++ b/docs/modules/ROOT/pages/reference/architecture.adoc @@ -87,7 +87,7 @@ between modes is part of the test contract. | `skainet-backends/skainet-backend-xnnpack` | Optional XNNPACK CPU backend (FP32 matmul / conv2d / pooling) on linuxX64 / linuxArm64 / Android. | `skainet-backends/benchmarks/jvm-cpu-jmh` | JMH harness — `MatmulBench`, `KernelMatmulBench`, `QuantizedMatmulBench`, `ElementwiseAdd1MBench`, `Reductions1MBench`. | `skainet-compile/*` | Tape recording, StableHLO emission, IREE export. -| `skainet-io/*` | Model loaders (GGUF, SafeTensors, ONNX), tokenizers, IRPA writer. One GGUF loader with two independent axes: `quantPolicy` (what the values are) × `staging` (where the bytes live). +| `skainet-io/*` | Model loaders (GGUF, SafeTensors, ONNX), tokenizers, IRPA writer. One GGUF loader configured by a single `WeightForm` (encoding × byte order × shape × residency), resolved per tensor or passed explicitly. | `skainet-apps/skainet-plan` | `skainet plan ` — prints a model's memory plan from its header alone. See xref:how-to/plan-model-memory.adoc[]. | `skainet-backends/benchmarks/jvm-cpu-publish` | Publishes benchmark records as JSON (schema-checked in CI), including generation metrics when a scenario runs a decode loop. |=== @@ -353,7 +353,8 @@ consumers of it. | Kernels select on a declared `KernelKey`, not an `is`-ladder | 2026-08 | Every quantization bug was arriving as a dispatch bug. A kernel now declares the formats, layouts and capabilities it accepts, and the dispatcher inserts a visible adapter when an operand does not match. | Block order is part of the layout | 2026-08 | Two block orders for packed weights had been an unwritten contract, contradicting itself in seven places and shipping wrong numbers twice. It is now carried on `Layout`, declared in the key, and converted only by one engine-owned, idempotent function. | `matmulWeightTransposed` instead of transposing a packed weight | 2026-08 | Transposing block-quantized data is not representable — blocks quantize runs along the input dimension. What the engine called a packed transpose was a per-call layout copy that was not its own inverse; the primitive ggml and BLAS have takes the weight as `[out, in]` and converts once. -| `quantPolicy × staging` as two axes of one loader | 2026-08 | "Streaming loader" and "mapped weights helper" were separate code paths that could disagree. What the values are and where the bytes live are independent questions. +| One `WeightForm` (encoding × byte order × shape × residency) replaces `quantPolicy`/`staging`/`weightOrientation`; the three axes were removed in #1159 (0.49.0) | 2026-08 | The three flags asked the caller to resolve, per device, what the resolver can decide from file × profile × kernels; a single resolved value is priceable, traceable, and overridable. +| `quantPolicy × staging` as two axes of one loader (superseded above) | 2026-08 | "Streaming loader" and "mapped weights helper" were separate code paths that could disagree. What the values are and where the bytes live are independent questions. |=== == 10. Quality requirements diff --git a/docs/modules/ROOT/pages/reference/kernel-support-matrix.adoc b/docs/modules/ROOT/pages/reference/kernel-support-matrix.adoc index c6bcc685f..70400c938 100644 --- a/docs/modules/ROOT/pages/reference/kernel-support-matrix.adoc +++ b/docs/modules/ROOT/pages/reference/kernel-support-matrix.adoc @@ -1,7 +1,7 @@ = Kernel × platform support matrix :description: Which compute-kernel provider serves each weight format on each KMP target. -Generated from `kernel-support.json` (version `0.40.1`) by `KernelSupportMatrixTest` — registry introspection of the registered `KernelProvider` implementations. Do not edit by hand; run `./gradlew generateKernelMatrix` to refresh. +Generated from `kernel-support.json` (version `0.49.0`) by `KernelSupportMatrixTest` — registry introspection of the registered `KernelProvider` implementations. Do not edit by hand; run `./gradlew generateKernelMatrix` to refresh. Each cell is the best (highest-priority) provider that serves `Float32 × format` `matmul` on that platform: *native-ffm* (100) → *panama-vector* (50) → *scalar* (0). An empty cell (`—`) means no provider carries a kernel there (the format is dequant-to-FP32 only). diff --git a/docs/modules/ROOT/pages/reference/operators/generated/index.adoc b/docs/modules/ROOT/pages/reference/operators/generated/index.adoc index 111112b87..763ab5497 100644 --- a/docs/modules/ROOT/pages/reference/operators/generated/index.adoc +++ b/docs/modules/ROOT/pages/reference/operators/generated/index.adoc @@ -1,6 +1,6 @@ = AI-NET Operators Reference -Generated from version `0.27.0` on 2026-06-04 +Generated from version `0.49.0` on 2026-08-26 == Operators by Modality diff --git a/docs/modules/ROOT/pages/reference/operators/generated/tensorops.adoc b/docs/modules/ROOT/pages/reference/operators/generated/tensorops.adoc index f27eb1351..a9e088dce 100644 --- a/docs/modules/ROOT/pages/reference/operators/generated/tensorops.adoc +++ b/docs/modules/ROOT/pages/reference/operators/generated/tensorops.adoc @@ -242,6 +242,47 @@ include::partial$ops/tensorops/matmul.adoc[tag=examples,optional] include::partial$ops/tensorops/matmul.adoc[tag=references,optional] +== matmulWeightTransposed + +[.darc-none]#✖ Generated facts only (no human prose)# + +=== Signature + +[source,kotlin] +---- +fun matmulWeightTransposed(x:Tensor, weight:Tensor): Tensor +---- + +=== Parameters + +* `x: Tensor` + activations, `[batch, in]` or `[in]` +* `weight: Tensor` + `[out, in]` — **not** pre-transposed + +=== Return Type + +`Tensor` + +== relayoutPackedWeightForKernels + +[.darc-none]#✖ Generated facts only (no human prose)# + +=== Signature + +[source,kotlin] +---- +fun relayoutPackedWeightForKernels(weight:Tensor): Tensor +---- + +=== Parameters + +* `weight: Tensor` + +=== Return Type + +`Tensor` + == transpose [.darc-none]#✖ Generated facts only (no human prose)# @@ -808,6 +849,42 @@ fun variance(tensor:Tensor, dim:Int): Tensor `Tensor` +== argMax + +[.darc-stub]#⚠ Prose present but not DARC-validated# + +=== Signature + +[source,kotlin] +---- +fun argMax(tensor:Tensor, dim:Int): Tensor +---- + +=== Parameters + +* `tensor: Tensor` +* `dim: Int` + +=== Return Type + +`Tensor` + +=== Definition + +include::partial$ops/tensorops/argmax.adoc[tag=math,optional] + +=== Intuition + +include::partial$ops/tensorops/argmax.adoc[tag=intuition,optional] + +=== Examples + +include::partial$ops/tensorops/argmax.adoc[tag=examples,optional] + +=== References + +include::partial$ops/tensorops/argmax.adoc[tag=references,optional] + == sqrt [.darc-none]#✖ Generated facts only (no human prose)# diff --git a/docs/modules/ROOT/pages/reference/ops-status-matrix.adoc b/docs/modules/ROOT/pages/reference/ops-status-matrix.adoc index e12c55326..cff069a6c 100644 --- a/docs/modules/ROOT/pages/reference/ops-status-matrix.adoc +++ b/docs/modules/ROOT/pages/reference/ops-status-matrix.adoc @@ -1,7 +1,7 @@ = Operator Coverage Matrix :description: Cross-backend status for every operator function in SKaiNET. -Generated from `operators.json` version `0.27.0` on 2026-06-04. +Generated from `operators.json` version `0.49.0` on 2026-08-26. Rows are `Operator.function` pairs. The `Validated` column shows whether the function's documentation has been DARC-validated by a reviewer (see xref:contributing/darc-workflow.adoc[DARC workflow]). Remaining columns are backends that appear in any function's `statusByBackend` map — a missing entry means the backend makes no claim about the function (treat it as "unknown", not "not supported"). @@ -20,6 +20,8 @@ Rows are `Operator.function` pairs. The `Validated` column shows whether the fun | `TensorOps.rsubScalar` | ✖ | — | — | — | `TensorOps.rdivScalar` | ✖ | — | — | — | `TensorOps.matmul` | ✅ | — | — | — +| `TensorOps.matmulWeightTransposed` | ✖ | — | — | — +| `TensorOps.relayoutPackedWeightForKernels` | ✖ | — | — | — | `TensorOps.transpose` | ✖ | — | — | — | `TensorOps.permute` | ✖ | — | — | — | `TensorOps.conv1d` | ✖ | — | — | — @@ -47,6 +49,7 @@ Rows are `Operator.function` pairs. The `Validated` column shows whether the fun | `TensorOps.sum` | ✖ | — | — | — | `TensorOps.mean` | ✖ | — | — | — | `TensorOps.variance` | ✖ | — | — | — +| `TensorOps.argMax` | ⚠ | — | — | — | `TensorOps.sqrt` | ✖ | — | — | — | `TensorOps.pow` | ✖ | — | — | — | `TensorOps.powScalar` | ✖ | — | — | — @@ -72,7 +75,7 @@ Rows are `Operator.function` pairs. The `Validated` column shows whether the fun | `TensorOps.scaledDotProductAttention` | ✖ | — | — | — | `Similarity.cosineDistance` | ✖ | ✅ | ✅ | ✅ -| *Done* | *1 / 62* | *1 / 62* | *1 / 62* | *1 / 62* +| *Done* | *1 / 65* | *1 / 65* | *1 / 65* | *1 / 65* |=== Per-function detail including notes lives in xref:reference/operators/generated/index.adoc[Operator reference]. diff --git a/docs/modules/ROOT/pages/tutorials/android-classifier-getting-started.adoc b/docs/modules/ROOT/pages/tutorials/android-classifier-getting-started.adoc index ac9045d9f..e809719d8 100644 --- a/docs/modules/ROOT/pages/tutorials/android-classifier-getting-started.adoc +++ b/docs/modules/ROOT/pages/tutorials/android-classifier-getting-started.adoc @@ -19,12 +19,12 @@ An Android project with Kotlin. Add the SKaiNET modules: [source,kotlin] ---- dependencies { - implementation("sk.ainet.core:skainet-lang-core:0.40.1") // tensors, DSL, training - implementation("sk.ainet.core:skainet-backend-cpu:0.40.1") // CPU ops - implementation("sk.ainet.core:skainet-compile-dag:0.40.1") // autograd (training context) - implementation("sk.ainet.core:skainet-data-api:0.40.1") // Dataset / DataBatch - implementation("sk.ainet.core:skainet-data-simple:0.40.1") // embedded Iris - runtimeOnly("sk.ainet.core:skainet-backend-jni-cpu:0.40.1") // NEON kernels (see below) + implementation("sk.ainet.core:skainet-lang-core:0.49.0") // tensors, DSL, training + implementation("sk.ainet.core:skainet-backend-cpu:0.49.0") // CPU ops + implementation("sk.ainet.core:skainet-compile-dag:0.49.0") // autograd (training context) + implementation("sk.ainet.core:skainet-data-api:0.49.0") // Dataset / DataBatch + implementation("sk.ainet.core:skainet-data-simple:0.49.0") // embedded Iris + runtimeOnly("sk.ainet.core:skainet-backend-jni-cpu:0.49.0") // NEON kernels (see below) } ---- diff --git a/docs/modules/ROOT/pages/tutorials/hlo-getting-started.adoc b/docs/modules/ROOT/pages/tutorials/hlo-getting-started.adoc index 846db38fb..d73e3ee2c 100644 --- a/docs/modules/ROOT/pages/tutorials/hlo-getting-started.adoc +++ b/docs/modules/ROOT/pages/tutorials/hlo-getting-started.adoc @@ -331,12 +331,14 @@ Use SKaiNET's built-in debugging tools: [source,kotlin] ---- -// Enable HLO debugging -val optimizer = StableHloOptimizer(debugMode = true) -val optimizedHlo = optimizer.optimize(hloModule) - -// Visualize computation graph -optimizer.dumpGraphviz("rgb2gray.dot") +// Name a compile target to run the optimizer pipeline on the production path (0.49.0): +// LayoutAssignmentPass assigns kernel-feed block order to packed weights, and any passes the +// target registered via TargetOptimizers run after it. With no target, no pipeline runs and +// the emitted module is unchanged. +val module = HloGenerator.generate(model, sampleInput, target = "llvm-cpu") + +// The emitted MLIR header carries machine-readable storage facts per tensor: +// module attributes {skainet.tensor_layouts = {w = {kind = "Q4_K", block_elems = 256, ...}}} ---- === Resources and References diff --git a/docs/modules/ROOT/pages/tutorials/ternary-getting-started.adoc b/docs/modules/ROOT/pages/tutorials/ternary-getting-started.adoc index b3de58a7a..51c00b081 100644 --- a/docs/modules/ROOT/pages/tutorials/ternary-getting-started.adoc +++ b/docs/modules/ROOT/pages/tutorials/ternary-getting-started.adoc @@ -183,7 +183,7 @@ to make. The kernel, its dispatch pack, and all three bridges are merged (https://github.com/SKaiNET-developers/SKaiNET/issues/1136[#1136] tracks the effort; the C kernel is vendored verbatim from NeoGPU under MIT, agreed in -https://github.com/anjaustin/neogpu/issues/1[neogpu#1]). In progress: +https://github.com/anjaustin/neogpu/issues/1[neogpu#1]). Landed in 0.49.0: * GGUF I2_S import with keep-packed loading (https://github.com/SKaiNET-developers/SKaiNET/issues/1140[#1140]) — load a diff --git a/docs/modules/skeep/pages/002-android-offheap-tensor-storage.adoc b/docs/modules/skeep/pages/002-android-offheap-tensor-storage.adoc index 8f0364c4c..40f447733 100644 --- a/docs/modules/skeep/pages/002-android-offheap-tensor-storage.adoc +++ b/docs/modules/skeep/pages/002-android-offheap-tensor-storage.adoc @@ -232,12 +232,14 @@ keeps working unchanged and the overload is purely opt-in. Expected consumers: (packed quant blocks) and a synthesized SafeTensors file (dense F32/F16/BF16) produce bit-identical tensors under either placement. -== Implementation status (2026-08-24) +== Implementation status (updated 2026-08-26, for 0.49.0) Tracked by SKEEP-003's M2 slice https://github.com/SKaiNET-developers/SKaiNET/issues/1038[#1038]. This SKEEP stays *Draft*: the -mechanism is in `develop`, the acceptance criteria that need a physical device are not met yet, and -one of them cannot be met until an unrelated contract is fixed. +mechanism is in `develop` (and since 0.49.0 the loader speaks one `WeightForm` — the legacy +`StagingPolicy`/`QuantPolicy`/`WeightOrientation` axes were removed in #1159), the acceptance +criteria that need a physical device are not met yet, and one of them cannot be met until an +unrelated contract is fixed. Landed: @@ -247,7 +249,7 @@ Landed: (https://github.com/SKaiNET-developers/SKaiNET/issues/921[#921], https://github.com/SKaiNET-developers/SKaiNET/issues/922[#922]). * *Phase 3* — mapped loading is a *configuration of the ordinary loader* rather than a separate - helper: `StreamingGgufParametersLoader(staging = StagingPolicy.MAPPED)` + helper: `StreamingGgufParametersLoader(weightForm = WeightForm(residency = WeightResidency.MAPPED))` (https://github.com/SKaiNET-developers/SKaiNET/issues/1037[#1037]), with `AndroidGguf.loader()` making it the Android default. * *The fit check this SKEEP did not have* — `MemoryPlan.fitOn(DeviceMemory, weightsMapped)` treats diff --git a/docs/modules/skeep/pages/003-unified-tensor-storage.adoc b/docs/modules/skeep/pages/003-unified-tensor-storage.adoc index 354c0765a..86a75a0c9 100644 --- a/docs/modules/skeep/pages/003-unified-tensor-storage.adoc +++ b/docs/modules/skeep/pages/003-unified-tensor-storage.adoc @@ -1,7 +1,7 @@ = SKEEP-003: Unifying the tensor storage model — one byte-owner, enforced ownership, coherent dtype/encoding :description: SKaiNET proposal to converge the TensorData and TensorStorage layers into one storage model with real ownership, a single view mechanism, and dtype/encoding coherence. -Status: Accepted (2026-08-22) + +Status: Implemented (accepted 2026-08-22; M0/M1/M2 + the P7/P8 resolution shipped in 0.49.0) + Audience: SKaiNET maintainers and contributors + Created: 2026-08-10 + Tracking issue: https://github.com/SKaiNET-developers/SKaiNET/issues/932[#932] (umbrella) → https://github.com/SKaiNET-developers/SKaiNET/issues/1001[#1001 M0] · https://github.com/SKaiNET-developers/SKaiNET/issues/1002[#1002 M1] · https://github.com/SKaiNET-developers/SKaiNET/issues/1003[#1003 M2] + @@ -17,10 +17,9 @@ and the façades are deleted at the next major release. Pure B would leave the erased packed dtype and the `is`-ladder in place; pure A as a big-bang rewrite is the "third layer" risk named below. -The full analysis, terminology and diagrams are in the repository file -the proposal (revision 2, tracked in #932); the -milestone scoping, sample apps and acceptance numbers are in -the milestone PRD. The thirteen +The full analysis, terminology and diagrams are in the proposal (revision 2) and the milestone +PRD, both preserved in the pull requests and issues linked from #932 (the design drafts under +`docs/design/` were retired in #1106 in favour of the Antora pages linked above). The thirteen decisions recorded there: [cols="1,3",options="header"] @@ -187,8 +186,9 @@ What *is* real and worth keeping from each side: the data side's zero-copy packed transposes, `LazyZeroFloatArrayTensorData` placeholders, `RowDequantSource` for embedding-scale tensors, and the genuinely zero-copy `TensorView` family; the storage side's `TensorEncoding` → -`TensorSpec.metadata` → StableHLO `skainet.tensor_encodings` export seam, -which works end-to-end today. +`TensorSpec.metadata` → StableHLO module-attribute export seam, which works +end-to-end today (since #1179 the attribute is the structural +`skainet.tensor_layouts`). == Proposed Design diff --git a/docs/modules/skeep/pages/003a-placement-and-planning-resolution.adoc b/docs/modules/skeep/pages/003a-placement-and-planning-resolution.adoc index eabe93f44..082c289de 100644 --- a/docs/modules/skeep/pages/003a-placement-and-planning-resolution.adoc +++ b/docs/modules/skeep/pages/003a-placement-and-planning-resolution.adoc @@ -34,7 +34,7 @@ activations are a lifetime question (§2), not a device question. Every backend device arbitration policy would be untestable speculation, deferred until a second device exists. *The user always wins.* The precedence order, explicit and documented on the loader: -per-tensor `weightFormFor` > uniform `weightForm` > the three legacy parameters > the resolver. +per-tensor `weightFormFor` > uniform `weightForm` > the resolver (the three legacy parameters were removed outright in #1159). `WeightForm(DequantizeTo(FP32), residency = HEAP)` — everything dense, on the managed heap — is a supported one-liner, not a fight with the planner. @@ -68,8 +68,9 @@ Two boundaries held on purpose: `arrayOffset`; the ops fast paths that unwrap `buffer` assume offset 0. Kernels that want zero-copy take the view, which carries the offset. -Op _outputs_ still allocate raw arrays; routing `DefaultCpuOps` through the active scope and -adopting the loop in a real decode is #1146. +Op _outputs_ draw from the active scope too since #1146 (`TensorDataFactory.adoptFloatArray`, +`ScopedTensorDataFactory`), with offset-aware FP32 fast paths and Panama vector kernels (#1173) +so slab-backed tensors keep SIMD speed. == 3. Would graph-level memory planning duplicate IREE? Yes. (#1134) @@ -93,11 +94,13 @@ extension, emitted as a module attribute. There is no such import today; the fir couple the compile pipeline to the storage model for no consumer's benefit — reject it in review. ==== -The carry work itself (extend `TensorRef`, a `TensorSpecLayout` accessor, populating the -pre-built `ResolvedComputeGraph.resolvedLayout`/`backendAssignment` seams, a structural -`skainet.tensor_layouts` module attribute, `ExternalParameterRef.layout`) is #1147, deliberately -gated on an in-repo IREE consumer — the `.vmfb` parity harness of #1148 — because metadata with no -reader is write-only speculation. +The carry work landed as the #1178/#1179/#1180 arc (0.49.0): `TensorRef` carries identity, +encoding and block order; `TensorSpec` metadata accessors follow the untyped precedent; the +structural `skainet.tensor_layouts` module attribute and `ExternalParameterRef.blockOrder` declare +the facts to the consumer; and `LayoutAssignmentPass` runs on the production path via +`HloGenerator.generate(target = …)`, populating the pre-built `ResolvedComputeGraph` seams. Until +the conformity pipeline exists (#1148, the future IREE-validation SKEEP), the carriage tests are +the metadata's reader. == Where everything landed @@ -108,8 +111,8 @@ reader is write-only speculation. | `AllocationResolver` + `explain()` | #1143 | PR #1153 (re-landed #1155) | Loader consults the resolution (`ResolvedGguf`, `weightFormFor`) | #1144 | PR #1154 (re-landed #1155) | `memoryScope` wired into creation | #1145 | PR #1156 -| Op outputs through the scope + decode adoption | #1146 | open (follow-up) -| Layout/placement carriage to StableHLO | #1147 | open (gated on #1148) +| Op outputs through the scope + offset-aware SIMD | #1146, #1173 | landed (PRs #1168/#1175) +| Layout/placement carriage to StableHLO | #1147 (#1178/#1179/#1180) | landed (PRs #1182/#1183/#1184) | `.vmfb` parity acceptance run | #1148 | open (follow-up) |=== diff --git a/docs/modules/skeep/pages/index.adoc b/docs/modules/skeep/pages/index.adoc index 53a23b589..365348663 100644 --- a/docs/modules/skeep/pages/index.adoc +++ b/docs/modules/skeep/pages/index.adoc @@ -67,6 +67,10 @@ Every proposal should include: | Off-heap tensor storage on Android | xref:skeep:003-unified-tensor-storage.adoc[SKEEP-003] -| Accepted +| Implemented | Unifying the tensor storage model + +| xref:skeep:003a-placement-and-planning-resolution.adoc[SKEEP-003a] +| Resolved +| Placement, eager lifetime and graph-level planning — the P7/P8 answers |=== diff --git a/gradle.properties b/gradle.properties index 05b1dc8f0..8d312a3fd 100644 --- a/gradle.properties +++ b/gradle.properties @@ -1,5 +1,5 @@ GROUP=sk.ainet.core -VERSION_NAME=0.40.1 +VERSION_NAME=0.49.0 POM_DESCRIPTION=SKaiNET POM_URL=https://github.com/SKaiNET-developers/skainet/ From 6c0f48b4cd6ff0dd29c8120a73566023e4ea92c4 Mon Sep 17 00:00:00 2001 From: Michal Harakal Date: Thu, 27 Aug 2026 12:03:11 +0200 Subject: [PATCH 2/2] fix(cpu): bridge off-heap activations to the common ternary dispatch path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On the JVM, SKEEP-004 activations live in MemorySegments, and the common views->KernelDispatch path a packed ternary weight takes (#1136) cannot read them: element access over SegmentStorage is forbidden by design, so the first linearProject over a BitNetB158TensorData weight in a real decode loop threw the SKEEP-004 guard. matmulWeightTransposed on the JVM tier now bulk-copies such an activation to the heap once per call before handing the common path its views — decode-step activations are k floats, not weights, so the copy is noise next to the matmul. Two producers are handled: TensorData that is itself segment-backed (MemorySegmentBackedData), and TensorData whose view presents a SegmentStorage. The copy deliberately constructs DenseFloatArrayTensorData directly — this tier's dataFactory adopts float arrays back INTO segments, which would round-trip the problem. Found and verified running the real microsoft/bitnet-b1.58-2B-4T I2_S GGUF end-to-end in SKaiNET-transformers: with this fix the model loads packed and generates coherent text ("The capital of France is Paris. Paris is known for its beautiful architecture, art, and cuisine...") through the vendored ternary kernels — which also confirms the GROUP_128 default of the I2_S import (#1140's open caveat) against the official BitNet.cpp file. Refs #1136 Co-Authored-By: Claude Fable 5 --- .../ainet/exec/tensor/ops/DefaultCpuOpsJvm.kt | 46 ++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/skainet-backends/skainet-backend-cpu/src/jvmMain/kotlin/sk/ainet/exec/tensor/ops/DefaultCpuOpsJvm.kt b/skainet-backends/skainet-backend-cpu/src/jvmMain/kotlin/sk/ainet/exec/tensor/ops/DefaultCpuOpsJvm.kt index 9ec156e7c..76b583e86 100644 --- a/skainet-backends/skainet-backend-cpu/src/jvmMain/kotlin/sk/ainet/exec/tensor/ops/DefaultCpuOpsJvm.kt +++ b/skainet-backends/skainet-backend-cpu/src/jvmMain/kotlin/sk/ainet/exec/tensor/ops/DefaultCpuOpsJvm.kt @@ -191,7 +191,18 @@ internal class DefaultCpuOpsJvm( */ @Suppress("UNCHECKED_CAST") override fun matmulWeightTransposed(x: Tensor, weight: Tensor): Tensor { - if (weight.shape.rank != 2 || !isHeapPackedWeightForJvm(weight.data)) return super.matmulWeightTransposed(x, weight) + if (weight.shape.rank != 2 || !isHeapPackedWeightForJvm(weight.data)) { + // Packed weights OUTSIDE the JVM relayout list (the ternary formats, #1136) go to the + // common views→KernelDispatch path — whose kernels and reference are commonMain and + // cannot read a MemorySegment-backed activation (SKEEP-004 forbids element access over + // SegmentStorage by design). On this tier the bridge is trivial: bulk-copy the + // activation to the heap once per call — decode-step activations are k floats, not + // weights — and hand the common path heap views. + if (weight.data is sk.ainet.lang.tensor.storage.PackedBlockStorage) { + segmentActivationToHeap(x)?.let { return super.matmulWeightTransposed(it, weight) } + } + return super.matmulWeightTransposed(x, weight) + } // Already in feed order — the loader produced it that way (#1120). Nothing to permute and // nothing to cache: the kernels want the other shape label over the same bytes, which costs // an object rather than a copy of the weight. @@ -206,6 +217,39 @@ internal class DefaultCpuOpsJvm( return matmul(x, kernelOrder as Tensor) } + /** + * A heap FP32 copy of an activation whose bytes live off-heap, or null when [x] is not that + * shape. Two producers exist on this tier: TensorData that is itself segment-backed + * ([MemorySegmentBackedData]), and TensorData whose VIEW presents a + * [sk.ainet.lang.memory.SegmentStorage] (SKEEP-004 off-heap scope allocations). + */ + @OptIn(sk.ainet.lang.memory.ExperimentalMemoryApi::class) + private fun segmentActivationToHeap(x: Tensor): Tensor? { + if (x.dtype != FP32::class) return null + val count = x.shape.volume + val arr = FloatArray(count) + val src = x.data as? MemorySegmentBackedData + if (src != null) { + java.lang.foreign.MemorySegment.copy( + src.segment, java.lang.foreign.ValueLayout.JAVA_FLOAT, src.segmentByteOffset, arr, 0, count, + ) + } else { + val view = x.data.view ?: return null + if (!view.isContiguous) return null + val seg = view.storage as? sk.ainet.lang.memory.SegmentStorage ?: return null + java.lang.foreign.MemorySegment.copy( + seg.segment(), java.lang.foreign.ValueLayout.JAVA_FLOAT, + view.layout.offsetElements * java.lang.Float.BYTES.toLong(), arr, 0, count, + ) + } + // NOT dataFactory.adoptFloatArray: this JVM factory adopts float arrays back INTO + // MemorySegments — the round-trip that defeats the whole copy. The base heap class is + // what the common dispatch path can read. + @Suppress("UNCHECKED_CAST") + val data = sk.ainet.lang.tensor.data.DenseFloatArrayTensorData(x.shape, arr) as sk.ainet.lang.tensor.data.TensorData + return newTensor(data, x.dtype, x) + } + /** Relayouted weights keyed by the identity of the bytes they came from (#1096). */ private val prepackedWeightsJvm: MutableList>> = mutableListOf() private val PREPACK_CACHE_LIMIT_JVM: Int = 64