From 20ff07628759fda6c2f27b70a9fd7143bdd27d67 Mon Sep 17 00:00:00 2001 From: Michal Harakal Date: Mon, 24 Aug 2026 18:18:01 +0200 Subject: [PATCH] =?UTF-8?q?feat(memory):=20block=20order=20belongs=20to=20?= =?UTF-8?q?the=20layout=20=E2=80=94=20BlockOrder,=20a=20visible=20prepack?= =?UTF-8?q?=20adapter,=20the=20normative=20doc?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes #1094, the foundational slice of #973. #973's census found seven mutually contradicting statements of one contract inside this repository, two of which had already shipped wrong numbers (#968, #971). The contract is: a packed weight's blocks are in one of two orders, and which one is a property of the value. - `BlockOrder { ROW_MAJOR, INPUT_BLOCK_MAJOR }` on `Layout`, defaulting to ROW_MAJOR — what a file holds. The order is expressed **in the strides** (input-block-major is `[1, out]` over the block grid, not `[blocksPerRow, 1]`), so narrow, transpose, get and toFloatArray keep working unchanged on a view in either order rather than growing a branch each. - `LayoutClass.BLOCKED` splits into `BLOCKED_ROW_MAJOR` and `BLOCKED_INPUT_MAJOR`, so a kernel *declares* the order it reads in its `KernelKey` and the dispatcher can relayout instead of the caller having to know. This is what #1029 was waiting for. - `TensorView.prepack(order, scope, sink)`: the conversion as a visible adapter — allocates in the caller's scope, emits `AdapterInserted` with the byte count, returns `this` when nothing has to move. Named as the conversion it is, not as a transpose. - `RelayoutedBlockDecoder` keeps rule 4 true across a relayout: the M1 `PackedBlockDecoder` decodes from the `TensorData` it wraps rather than from the storage it is handed, so a prepacked view maps the block index back instead of silently decoding the old bytes. - `docs/design/memory/packed-weight-layout.md`: the normative contract, with the five rules and the semver policy that `packedData` byte semantics are public API. The stale kdocs the census names are corrected to point at it — `Q5_0/Q5_1TensorData` claimed input-block-major bytes and a shape-swap transpose, neither of which has been true since 0.40.1, and two kernel SPI kdocs claimed byte-identity with a storage type that holds the other order. `BlockOrderTest` uses a three-block-wide weight throughout, because the two orders coincide at one block per row — the case that hides the bug. It pins that prepacking moves the bytes to `b * rows + o`, that the decoded matrix is unchanged, that going there and back is the identity (the double-transpose hazard, which nothing detected before), that the order survives narrow / transpose / unsqueeze, and that a same-order prepack copies nothing and reports nothing. Gate: scripts/pr-gate.sh — all legs passed; --golden passed on JVM and linuxX64 (no decoder or kernel moved a bit). Co-Authored-By: Claude Fable 5 --- docs/design/memory/packed-weight-layout.md | 77 ++++++++++ .../backend/api/kernel/BitNetGemvKernel.kt | 4 +- .../sk/ainet/backend/api/kernel/KernelKey.kt | 29 +++- .../backend/api/kernel/Q5_0MatmulKernel.kt | 5 +- .../backend/api/kernel/Q6KMatmulKernel.kt | 5 +- .../api/kernel/KernelKeyDispatchTest.kt | 8 +- .../api/jvm/skainet-lang-core.api | 28 +++- .../kotlin/sk/ainet/lang/memory/Layout.kt | 67 +++++++-- .../kotlin/sk/ainet/lang/memory/TensorView.kt | 114 +++++++++++++++ .../ainet/lang/tensor/data/Q5_0TensorData.kt | 7 +- .../ainet/lang/tensor/data/Q5_1TensorData.kt | 11 +- .../sk/ainet/lang/memory/BlockOrderTest.kt | 135 ++++++++++++++++++ 12 files changed, 460 insertions(+), 30 deletions(-) create mode 100644 docs/design/memory/packed-weight-layout.md create mode 100644 skainet-lang/skainet-lang-core/src/commonTest/kotlin/sk/ainet/lang/memory/BlockOrderTest.kt diff --git a/docs/design/memory/packed-weight-layout.md b/docs/design/memory/packed-weight-layout.md new file mode 100644 index 000000000..77e723ef2 --- /dev/null +++ b/docs/design/memory/packed-weight-layout.md @@ -0,0 +1,77 @@ +# Packed weight layout — the normative contract + +Every packed (block-quantized) weight in SKaiNET is stored in **one of exactly two block orders**, +and which one it is in is a property of the value, not of the type it happens to have or of the +module that produced it. This document is that contract; kdocs link here instead of restating it. + +Written for [#973](https://github.com/SKaiNET-developers/SKaiNET/issues/973), whose census found +seven mutually contradicting statements of it inside this repository alone — two of which had +already produced wrong numbers in production ([#968](https://github.com/SKaiNET-developers/SKaiNET/issues/968), +[#971](https://github.com/SKaiNET-developers/SKaiNET/issues/971)). + +## The two orders + +A 2-D weight is logically `[out, in]`. Its blocks tile the **input** dimension, so a row of +`in` elements is `blocksPerRow = in / blockSize` blocks, and the whole weight is an +`out × blocksPerRow` grid of blocks. Flattening that grid is where the two orders come from: + +| order | flat block index | who produces it | who reads it | +|---|---|---|---| +| `BlockOrder.ROW_MAJOR` (canonical) | `o * blocksPerRow + b` | GGUF files, `Q*Quantizer`, `TernaryCodec` | `toFloatArray()`, `get()`, the reference matmul, `bitnet_gemv` | +| `BlockOrder.INPUT_BLOCK_MAJOR` (kernel feed order) | `b * out + o` | `TensorView.prepack(INPUT_BLOCK_MAJOR)` | the packed matmul kernels — scalar, Panama, native C, JNI | + +**They coincide only when `blocksPerRow == 1`.** For any weight wider than one block — that is, +virtually every real weight — reading one order as the other produces a block-permuted matrix: +finite, plausible numbers that are simply wrong. Nothing crashes. That is the entire reason this +document exists. + +## Where the order lives + +`Layout.blockOrder`. A `TensorView` over packed bytes carries it, and the order is expressed **in +the strides**, not in a branch: input-block-major is `strides = [1, out]` over the block grid +instead of `[blocksPerRow, 1]`. Everything else — `narrow`, `transpose`, `get`, `toFloatArray` — +therefore works unchanged on a view in either order, and a prepacked view still describes the same +matrix. + +`OperandKey`'s `LayoutClass` splits the same way: `BLOCKED_ROW_MAJOR` and `BLOCKED_INPUT_MAJOR`. +A kernel **declares** which order it reads, in its `KernelKey`. That is what makes the dispatcher +able to insert a relayout instead of the caller having to know. + +## Converting between them + +`TensorView.prepack(order, scope, sink)`: + +- returns `this` when the order already matches — the only free case; +- otherwise copies the blocks into `scope` and emits `TraceEvent.AdapterInserted` with the byte + count, so the conversion shows up in the trace with its price; +- the result carries the new order on its layout. + +It is a *conversion*, and it is named as one. It is not `transpose`: a true transpose of a +block-quantized weight would need runs of quantized values along the other axis, i.e. +requantization. What the engine historically called a "packed transpose" was this conversion +wearing transpose's name and a swapped shape label that lied about the data +(see #973, "the deeper semantic problem"); replacing that with a weight-transposing matmul +primitive is [#1096](https://github.com/SKaiNET-developers/SKaiNET/issues/1096). + +## Rules + +1. **A file's bytes are `ROW_MAJOR`.** Anything loaded from GGUF, produced by a quantizer, or + written by `TernaryCodec` is canonical. A loader never silently relayouts. +2. **A kernel declares its order in its key.** No kernel may assume; no caller may guess. +3. **A conversion is visible.** It allocates in a scope and emits an adapter event. A relayout that + does not appear in the trace is a bug. +4. **`get()` and `toFloatArray()` always mean the same thing** in either order — they read through + the layout. A view whose decoded content depends on which module produced it is a bug. +5. **`packedData` byte semantics are public API.** Changing the order a type holds is a + minor/major change, never a patch — this is what let a byte-layout change ship as a green-CI + hotfix once already. + +## Status + +`Layout.blockOrder`, the `LayoutClass` split, `prepack` and this document land with +[#1094](https://github.com/SKaiNET-developers/SKaiNET/issues/1094). The remaining work is tracked +as sub-issues of #973: bridging the packed SPI kernels through the ordered key +([#1095](https://github.com/SKaiNET-developers/SKaiNET/issues/1095)), the weight-transposing matmul +primitive (#1096), engine-owned prepacking and cross-repo contract fixtures +([#1097](https://github.com/SKaiNET-developers/SKaiNET/issues/1097)), and normalizing weight +orientation at the load boundary ([#1098](https://github.com/SKaiNET-developers/SKaiNET/issues/1098)). diff --git a/skainet-backends/skainet-backend-api/src/commonMain/kotlin/sk/ainet/backend/api/kernel/BitNetGemvKernel.kt b/skainet-backends/skainet-backend-api/src/commonMain/kotlin/sk/ainet/backend/api/kernel/BitNetGemvKernel.kt index f89f6af52..7d0b51b69 100644 --- a/skainet-backends/skainet-backend-api/src/commonMain/kotlin/sk/ainet/backend/api/kernel/BitNetGemvKernel.kt +++ b/skainet-backends/skainet-backend-api/src/commonMain/kotlin/sk/ainet/backend/api/kernel/BitNetGemvKernel.kt @@ -20,7 +20,7 @@ import sk.ainet.lang.tensor.storage.TensorEncoding * exactly as the vector kernels will factor them. * * Operands: `[rows, k]` activations in [I8Absmax.FORMAT] × `[n, k]` ternary weights in canonical - * (row-major block) order — the order [TernaryCodec] produces and GGUF stores. Output `[rows, n]`. + * ([sk.ainet.lang.memory.BlockOrder.ROW_MAJOR]) block order — the order [TernaryCodec] produces and GGUF stores. Output `[rows, n]`. * * The weight's codes are read once per call, not once per row: a decode step is one row against * the whole matrix, so hoisting it is the difference between O(rows·n·k) decodes and O(n·k). @@ -108,7 +108,7 @@ public class BitNetGemvKernel(override val key: KernelKey) : ViewKernel { op = "matmul", operands = listOf( OperandKey.contiguous(I8Absmax.FORMAT), - OperandKey(weightFormat, LayoutClass.BLOCKED), + OperandKey(weightFormat, LayoutClass.BLOCKED_ROW_MAJOR), ), ) diff --git a/skainet-backends/skainet-backend-api/src/commonMain/kotlin/sk/ainet/backend/api/kernel/KernelKey.kt b/skainet-backends/skainet-backend-api/src/commonMain/kotlin/sk/ainet/backend/api/kernel/KernelKey.kt index 86da23fc2..696428bd4 100644 --- a/skainet-backends/skainet-backend-api/src/commonMain/kotlin/sk/ainet/backend/api/kernel/KernelKey.kt +++ b/skainet-backends/skainet-backend-api/src/commonMain/kotlin/sk/ainet/backend/api/kernel/KernelKey.kt @@ -51,7 +51,10 @@ public data class OperandKey(val format: Format, val layout: LayoutClass) { /** Describe [view]: dense-and-gap-free is `CONTIGUOUS`, a packed layout is `BLOCKED`, anything else `STRIDED`. */ public fun of(view: TensorView): OperandKey { val cls = when { - view.layout.blocked -> LayoutClass.BLOCKED + view.layout.blocked -> when (view.layout.blockOrder) { + sk.ainet.lang.memory.BlockOrder.ROW_MAJOR -> LayoutClass.BLOCKED_ROW_MAJOR + sk.ainet.lang.memory.BlockOrder.INPUT_BLOCK_MAJOR -> LayoutClass.BLOCKED_INPUT_MAJOR + } view.isContiguous -> LayoutClass.CONTIGUOUS else -> LayoutClass.STRIDED } @@ -65,12 +68,30 @@ public data class OperandKey(val format: Format, val layout: LayoutClass) { /** * How an operand's bytes are arranged, as far as kernel selection cares: one gap-free run - * ([CONTIGUOUS]), a strided view over a larger buffer ([STRIDED]), or block-packed ([BLOCKED]). + * ([CONTIGUOUS]), a strided view over a larger buffer ([STRIDED]), or block-packed in one of the + * two orders that exist ([BLOCKED_ROW_MAJOR], [BLOCKED_INPUT_MAJOR]). + * * A kernel that declares `CONTIGUOUS` gets a gather adapter inserted for a `STRIDED` operand - * (§5.1) — the adapter is visible in the trace, never hidden inside a kernel. + * (§5.1) — the adapter is visible in the trace, never hidden inside a kernel. The two blocked + * classes exist for the same reason: a packed kernel reads its weight in *one* of the two block + * orders, and #973 is what happens when that is left implicit. A kernel declares which one it + * takes, and the dispatcher relayouts when the operand disagrees. */ @ExperimentalMemoryApi -public enum class LayoutClass { CONTIGUOUS, STRIDED, BLOCKED } +public enum class LayoutClass { + CONTIGUOUS, + STRIDED, + + /** Blocks in file order: `o * blocksPerRow + b` ([sk.ainet.lang.memory.BlockOrder.ROW_MAJOR]). */ + BLOCKED_ROW_MAJOR, + + /** Blocks in kernel feed order: `b * outputDim + o` ([sk.ainet.lang.memory.BlockOrder.INPUT_BLOCK_MAJOR]). */ + BLOCKED_INPUT_MAJOR, + ; + + /** True for either blocked class — when the question is "packed or not". */ + public val isBlocked: Boolean get() = this == BLOCKED_ROW_MAJOR || this == BLOCKED_INPUT_MAJOR +} /** Thrown when no registered kernel and no adapter chain can serve a key; lists what is registered. */ @ExperimentalMemoryApi diff --git a/skainet-backends/skainet-backend-api/src/commonMain/kotlin/sk/ainet/backend/api/kernel/Q5_0MatmulKernel.kt b/skainet-backends/skainet-backend-api/src/commonMain/kotlin/sk/ainet/backend/api/kernel/Q5_0MatmulKernel.kt index 3d51aa435..e813b2806 100644 --- a/skainet-backends/skainet-backend-api/src/commonMain/kotlin/sk/ainet/backend/api/kernel/Q5_0MatmulKernel.kt +++ b/skainet-backends/skainet-backend-api/src/commonMain/kotlin/sk/ainet/backend/api/kernel/Q5_0MatmulKernel.kt @@ -26,7 +26,10 @@ package sk.ainet.backend.api.kernel * write the `outputDim` floats starting at `output[outputOffset]`. * * Packed-weight **block-major** row contract: `weight` holds blocks laid - * out `(blockIdx * outputDim + o) * 22`. Matches `Q5_0BlockTensorData.packedData`. + * The weight is **input-block-major** (Q5_0BlockTensorData's bytes are canonical + * row-major — a weight reaches this kernel through `TensorView.prepack`, not by + * reinterpretation). One contract, written down in + * `docs/design/memory/packed-weight-layout.md` (#973). * * `inputDim` MUST be a multiple of 32 (the Q5_0 block size). */ diff --git a/skainet-backends/skainet-backend-api/src/commonMain/kotlin/sk/ainet/backend/api/kernel/Q6KMatmulKernel.kt b/skainet-backends/skainet-backend-api/src/commonMain/kotlin/sk/ainet/backend/api/kernel/Q6KMatmulKernel.kt index 8d05d3fde..cb089cbfd 100644 --- a/skainet-backends/skainet-backend-api/src/commonMain/kotlin/sk/ainet/backend/api/kernel/Q6KMatmulKernel.kt +++ b/skainet-backends/skainet-backend-api/src/commonMain/kotlin/sk/ainet/backend/api/kernel/Q6KMatmulKernel.kt @@ -23,7 +23,10 @@ package sk.ainet.backend.api.kernel * write the `outputDim` floats starting at `output[outputOffset]`. * * Packed-weight **block-major** row contract: blocks laid out - * `(blockIdx * outputDim + o) * 210`. Matches `Q6_KBlockTensorData.packedData`. + * The weight is **input-block-major** (Q6_KBlockTensorData's bytes are canonical + * row-major — a weight reaches this kernel through `TensorView.prepack`, not by + * reinterpretation). One contract, written down in + * `docs/design/memory/packed-weight-layout.md` (#973). * * `inputDim` MUST be a multiple of 256 (the Q6_K super-block size). */ diff --git a/skainet-backends/skainet-backend-api/src/commonTest/kotlin/sk/ainet/backend/api/kernel/KernelKeyDispatchTest.kt b/skainet-backends/skainet-backend-api/src/commonTest/kotlin/sk/ainet/backend/api/kernel/KernelKeyDispatchTest.kt index b9799c89f..60cc15521 100644 --- a/skainet-backends/skainet-backend-api/src/commonTest/kotlin/sk/ainet/backend/api/kernel/KernelKeyDispatchTest.kt +++ b/skainet-backends/skainet-backend-api/src/commonTest/kotlin/sk/ainet/backend/api/kernel/KernelKeyDispatchTest.kt @@ -68,8 +68,12 @@ class KernelKeyDispatchTest { val key = KernelKey.matmul(a, w) assertEquals("matmul", key.op); assertEquals(2, key.operands.size) assertEquals(OperandKey(Format.dense(FP32), LayoutClass.CONTIGUOUS), key.operands[0]) - assertEquals(OperandKey(Format(FP32, TensorEncoding.Q8_0), LayoutClass.BLOCKED), key.operands[1]) - assertEquals("matmul(Float32/Dense(4B) contiguous × Float32/Q8_0 blocked) @host", key.toString()) + assertEquals( + OperandKey(Format(FP32, TensorEncoding.Q8_0), LayoutClass.BLOCKED_ROW_MAJOR), + key.operands[1], + "a weight loaded from a file is canonical, and the key says so (#973)", + ) + assertEquals("matmul(Float32/Dense(4B) contiguous × Float32/Q8_0 blocked_row_major) @host", key.toString()) // a strided operand is a different key — that is the point of keying on layout val strided = denseView(Shape(4, 8)) { it.toFloat() }.narrow(1, 0, 4) assertEquals(LayoutClass.STRIDED, OperandKey.of(strided).layout) diff --git a/skainet-lang/skainet-lang-core/api/jvm/skainet-lang-core.api b/skainet-lang/skainet-lang-core/api/jvm/skainet-lang-core.api index 02bd67733..ccc457990 100644 --- a/skainet-lang/skainet-lang-core/api/jvm/skainet-lang-core.api +++ b/skainet-lang/skainet-lang-core/api/jvm/skainet-lang-core.api @@ -686,6 +686,14 @@ public final class sk/ainet/lang/memory/BlockDecoder$DefaultImpls { public static fun decodeElement (Lsk/ainet/lang/memory/BlockDecoder;Lsk/ainet/lang/memory/Storage;Lsk/ainet/lang/memory/Layout;J)F } +public final class sk/ainet/lang/memory/BlockOrder : java/lang/Enum { + public static final field INPUT_BLOCK_MAJOR Lsk/ainet/lang/memory/BlockOrder; + public static final field ROW_MAJOR Lsk/ainet/lang/memory/BlockOrder; + public static fun getEntries ()Lkotlin/enums/EnumEntries; + public static fun valueOf (Ljava/lang/String;)Lsk/ainet/lang/memory/BlockOrder; + public static fun values ()[Lsk/ainet/lang/memory/BlockOrder; +} + public final class sk/ainet/lang/memory/BlockSpec { public static final field Companion Lsk/ainet/lang/memory/BlockSpec$Companion; public static final field PER_TENSOR_BLOCK I @@ -811,11 +819,12 @@ public final class sk/ainet/lang/memory/I8Absmax { public final class sk/ainet/lang/memory/Layout { public static final field Companion Lsk/ainet/lang/memory/Layout$Companion; - public fun (Lsk/ainet/lang/tensor/Shape;[IJIZI)V - public synthetic fun (Lsk/ainet/lang/tensor/Shape;[IJIZIILkotlin/jvm/internal/DefaultConstructorMarker;)V + public fun (Lsk/ainet/lang/tensor/Shape;[IJIZILsk/ainet/lang/memory/BlockOrder;)V + public synthetic fun (Lsk/ainet/lang/tensor/Shape;[IJIZILsk/ainet/lang/memory/BlockOrder;ILkotlin/jvm/internal/DefaultConstructorMarker;)V public final fun byteOffsetOf ([I)J public fun equals (Ljava/lang/Object;)Z public final fun getBlockAxis ()I + public final fun getBlockOrder ()Lsk/ainet/lang/memory/BlockOrder; public final fun getBlocked ()Z public final fun getElementBytes ()I public final fun getElementCount ()J @@ -837,8 +846,8 @@ public final class sk/ainet/lang/memory/Layout { } public final class sk/ainet/lang/memory/Layout$Companion { - public final fun blocked (Lsk/ainet/lang/tensor/Shape;IIJ)Lsk/ainet/lang/memory/Layout; - public static synthetic fun blocked$default (Lsk/ainet/lang/memory/Layout$Companion;Lsk/ainet/lang/tensor/Shape;IIJILjava/lang/Object;)Lsk/ainet/lang/memory/Layout; + public final fun blocked (Lsk/ainet/lang/tensor/Shape;IIJLsk/ainet/lang/memory/BlockOrder;)Lsk/ainet/lang/memory/Layout; + public static synthetic fun blocked$default (Lsk/ainet/lang/memory/Layout$Companion;Lsk/ainet/lang/tensor/Shape;IIJLsk/ainet/lang/memory/BlockOrder;ILjava/lang/Object;)Lsk/ainet/lang/memory/Layout; public final fun rowMajor (Lsk/ainet/lang/tensor/Shape;Lsk/ainet/lang/memory/Format;J)Lsk/ainet/lang/memory/Layout; public static synthetic fun rowMajor$default (Lsk/ainet/lang/memory/Layout$Companion;Lsk/ainet/lang/tensor/Shape;Lsk/ainet/lang/memory/Format;JILjava/lang/Object;)Lsk/ainet/lang/memory/Layout; public final fun rowMajorStrides (Lsk/ainet/lang/tensor/Shape;)[I @@ -1057,6 +1066,14 @@ public final class sk/ainet/lang/memory/ProcessMemorySample { public fun toString ()Ljava/lang/String; } +public final class sk/ainet/lang/memory/RelayoutedBlockDecoder : sk/ainet/lang/memory/BlockDecoder { + public fun (Lsk/ainet/lang/memory/BlockDecoder;IILsk/ainet/lang/memory/BlockOrder;)V + public fun decodeBlock (Lsk/ainet/lang/memory/Storage;J[FI)V + public fun decodeElement (Lsk/ainet/lang/memory/Storage;Lsk/ainet/lang/memory/Layout;J)F + public fun getBlockSize ()I + public fun getBytesPerBlock ()I +} + public final class sk/ainet/lang/memory/ScalePlacement : java/lang/Enum { public static final field BLOCK_HEAD Lsk/ainet/lang/memory/ScalePlacement; public static final field BLOCK_TAIL Lsk/ainet/lang/memory/ScalePlacement; @@ -1220,6 +1237,7 @@ public final class sk/ainet/lang/memory/TensorView { public static final field Companion Lsk/ainet/lang/memory/TensorView$Companion; public fun (Lsk/ainet/lang/tensor/Shape;Lsk/ainet/lang/memory/Format;Lsk/ainet/lang/memory/Layout;Lsk/ainet/lang/memory/Storage;Lsk/ainet/lang/tensor/TensorId;Lsk/ainet/lang/memory/BlockDecoder;)V public synthetic fun (Lsk/ainet/lang/tensor/Shape;Lsk/ainet/lang/memory/Format;Lsk/ainet/lang/memory/Layout;Lsk/ainet/lang/memory/Storage;Lsk/ainet/lang/tensor/TensorId;Lsk/ainet/lang/memory/BlockDecoder;ILkotlin/jvm/internal/DefaultConstructorMarker;)V + public final fun decoderOrNull ()Lsk/ainet/lang/memory/BlockDecoder; public final fun get ([I)F public final fun getElementCount ()J public final fun getFormat ()Lsk/ainet/lang/memory/Format; @@ -1232,6 +1250,8 @@ public final class sk/ainet/lang/memory/TensorView { public final fun materialize (Lsk/ainet/lang/memory/Format;Lsk/ainet/lang/memory/Scope;)Lsk/ainet/lang/memory/TensorView; public static synthetic fun materialize$default (Lsk/ainet/lang/memory/TensorView;Lsk/ainet/lang/memory/Format;Lsk/ainet/lang/memory/Scope;ILjava/lang/Object;)Lsk/ainet/lang/memory/TensorView; public final fun narrow (III)Lsk/ainet/lang/memory/TensorView; + public final fun prepack (Lsk/ainet/lang/memory/BlockOrder;Lsk/ainet/lang/memory/Scope;Lsk/ainet/lang/memory/trace/TraceSink;)Lsk/ainet/lang/memory/TensorView; + public static synthetic fun prepack$default (Lsk/ainet/lang/memory/TensorView;Lsk/ainet/lang/memory/BlockOrder;Lsk/ainet/lang/memory/Scope;Lsk/ainet/lang/memory/trace/TraceSink;ILjava/lang/Object;)Lsk/ainet/lang/memory/TensorView; public final fun set ([IF)V public final fun squeeze (I)Lsk/ainet/lang/memory/TensorView; public final fun step (II)Lsk/ainet/lang/memory/TensorView; diff --git a/skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/memory/Layout.kt b/skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/memory/Layout.kt index 30007a3ae..47a12a3a9 100644 --- a/skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/memory/Layout.kt +++ b/skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/memory/Layout.kt @@ -2,6 +2,30 @@ package sk.ainet.lang.memory import sk.ainet.lang.tensor.Shape +/** + * Which order a block-packed buffer's blocks are in — the contract #973 reports as unwritten and + * contradictory across the engine and its converters. + * + * Two orders exist for a 2-D `[out, in]` weight whose blocks tile the input dimension: + * + * - [ROW_MAJOR] — flat block index `o * blocksPerRow + b`. What a GGUF holds, what a quantizer + * emits, what `toFloatArray()` and every `get()` assume. + * - [INPUT_BLOCK_MAJOR] — flat block index `b * outputDim + o`: all output rows' blocks for one + * input block, contiguous. What every packed matmul kernel actually reads. + * + * The two coincide only when a row is a single block, which is why mixing them produced *plausible + * but wrong numbers* rather than a crash (#968, #971). Carrying the order on the [Layout] is what + * makes it knowable from the value instead of guessed from the type: a kernel declares the order it + * reads, and the dispatcher inserts a visible relayout when the two disagree. + */ +public enum class BlockOrder { + /** `o * blocksPerRow + b` — canonical, as stored in a file. */ + ROW_MAJOR, + + /** `b * outputDim + o` — the order the packed matmul kernels feed on. */ + INPUT_BLOCK_MAJOR, +} + /** * Strides, byte offset and contiguity of a view over a [Storage] (SKEEP-003 §0 *Layout*). Pure * metadata: it says *where* the elements of a shape live inside a byte range, never what they mean @@ -19,6 +43,9 @@ import sk.ainet.lang.tensor.Shape * @property blockAxis for a [blocked] layout, the axis whose extent is measured in blocks. It is * the last axis as loaded, and it *moves* with [transpose]/[unsqueeze]/[squeeze] — which is what * lets a packed weight be transposed as metadata and still decode correctly (#1034). + * @property blockOrder for a [blocked] layout, the order its blocks are in ([BlockOrder]). Default + * [BlockOrder.ROW_MAJOR]: what a file holds. A buffer in kernel feed order says so here instead + * of leaving the next reader to guess (#973). */ @ExperimentalMemoryApi public class Layout( @@ -28,6 +55,7 @@ public class Layout( public val elementBytes: Int = 4, public val blocked: Boolean = false, public val blockAxis: Int = shape.rank - 1, + public val blockOrder: BlockOrder = BlockOrder.ROW_MAJOR, ) { init { require(strides.size == shape.rank) { "strides (${strides.size}) must match rank (${shape.rank})" } @@ -79,7 +107,7 @@ public class Layout( require(axis in 0 until shape.rank) { "axis $axis out of range for rank ${shape.rank}" } require(from >= 0 && size >= 0 && from + size <= shape[axis]) { "narrow($axis, $from, $size) outside extent ${shape[axis]}" } val dims = shape.dimensions.copyOf(); dims[axis] = size - return Layout(Shape(dims), strides.copyOf(), offsetElements + from.toLong() * strides[axis], elementBytes, blocked, blockAxis) + return Layout(Shape(dims), strides.copyOf(), offsetElements + from.toLong() * strides[axis], elementBytes, blocked, blockAxis, blockOrder) } /** Swap two axes — metadata only; the bytes are untouched (the packed-transpose trick, rule 5). */ @@ -92,7 +120,7 @@ public class Layout( // The block axis travels with its extent: a transposed packed view still knows which index // splits into (block, offset-within-block). val movedBlockAxis = when (blockAxis) { axis0 -> axis1; axis1 -> axis0; else -> blockAxis } - return Layout(Shape(dims), st, offsetElements, elementBytes, blocked, movedBlockAxis) + return Layout(Shape(dims), st, offsetElements, elementBytes, blocked, movedBlockAxis, blockOrder) } /** @@ -108,7 +136,7 @@ public class Layout( dims[axis] = (shape[axis] + step - 1) / step val st = strides.copyOf() st[axis] = strides[axis] * step - return Layout(Shape(dims), st, offsetElements, elementBytes, blocked, blockAxis) + return Layout(Shape(dims), st, offsetElements, elementBytes, blocked, blockAxis, blockOrder) } /** Insert a unit axis at [axis] (stride 0 — it is never stepped). */ @@ -119,7 +147,7 @@ public class Layout( for (i in 0..shape.rank) { if (i == axis) { dims[i] = 1; st[i] = 0 } else { dims[i] = shape[j]; st[i] = strides[j]; j++ } } - return Layout(Shape(dims), st, offsetElements, elementBytes, blocked, if (axis <= blockAxis) blockAxis + 1 else blockAxis) + return Layout(Shape(dims), st, offsetElements, elementBytes, blocked, if (axis <= blockAxis) blockAxis + 1 else blockAxis, blockOrder) } /** Drop the unit axis at [axis]. */ @@ -129,20 +157,24 @@ public class Layout( val dims = ArrayList(shape.rank - 1); val st = ArrayList(shape.rank - 1) require(!(blocked && axis == blockAxis)) { "cannot drop the block axis of a blocked layout" } for (i in 0 until shape.rank) if (i != axis) { dims += shape[i]; st += strides[i] } - return Layout(Shape(dims.toIntArray()), st.toIntArray(), offsetElements, elementBytes, blocked, if (axis < blockAxis) blockAxis - 1 else blockAxis) + return Layout(Shape(dims.toIntArray()), st.toIntArray(), offsetElements, elementBytes, blocked, if (axis < blockAxis) blockAxis - 1 else blockAxis, blockOrder) } override fun toString(): String = - "Layout($shape, strides=${strides.joinToString(",", "[", "]")}, offset=$offsetElements${if (blocked) " blocks" else ""}, ${if (isContiguous) "contiguous" else "strided"})" + "Layout($shape, strides=${strides.joinToString(",", "[", "]")}, offset=$offsetElements" + + (if (blocked) " blocks/${blockOrder.name.lowercase()}" else "") + + ", ${if (isContiguous) "contiguous" else "strided"})" override fun equals(other: Any?): Boolean = other is Layout && other.shape == shape && other.strides.contentEquals(strides) && other.offsetElements == offsetElements && - other.elementBytes == elementBytes && other.blocked == blocked && other.blockAxis == blockAxis + other.elementBytes == elementBytes && other.blocked == blocked && other.blockAxis == blockAxis && + other.blockOrder == blockOrder override fun hashCode(): Int { var h = shape.hashCode() h = 31 * h + strides.contentHashCode(); h = 31 * h + offsetElements.hashCode() h = 31 * h + elementBytes; h = 31 * h + blocked.hashCode(); h = 31 * h + blockAxis + h = 31 * h + blockOrder.hashCode() return h } @@ -164,7 +196,13 @@ public class Layout( * "element" of the layout being one packed block of `bytesPerBlock` bytes. Used for the * GGML block formats, where a view addresses whole blocks (rule 5). */ - public fun blocked(shape: Shape, blockSize: Int, bytesPerBlock: Int, offsetBlocks: Long = 0L): Layout { + public fun blocked( + shape: Shape, + blockSize: Int, + bytesPerBlock: Int, + offsetBlocks: Long = 0L, + blockOrder: BlockOrder = BlockOrder.ROW_MAJOR, + ): Layout { require(blockSize > 0 && bytesPerBlock > 0) { "block geometry must be positive" } require(shape.rank >= 1) { "blocked layout needs rank >= 1" } val last = shape[shape.rank - 1] @@ -172,7 +210,16 @@ public class Layout( val dims = shape.dimensions.copyOf() dims[dims.size - 1] = last / blockSize val blockShape = Shape(dims) - return Layout(blockShape, rowMajorStrides(blockShape), offsetBlocks, bytesPerBlock, blocked = true) + // The order *is* a stride pattern: input-block-major means "step one row to move + // one block index, step `rows` to move to the next input block". Expressing it in + // the strides rather than in a branch keeps every other operation — narrow, + // transpose, get — working unchanged on a prepacked view (#973). + val strides = if (blockOrder == BlockOrder.INPUT_BLOCK_MAJOR && blockShape.rank == 2) { + intArrayOf(1, blockShape[0]) + } else { + rowMajorStrides(blockShape) + } + return Layout(blockShape, strides, offsetBlocks, bytesPerBlock, blocked = true, blockOrder = blockOrder) } // A block that spans rows (e.g. ternary, where the whole tensor is one block): address the // flattened element sequence instead. Such a view decodes but cannot be sliced per axis. @@ -180,7 +227,7 @@ public class Layout( "neither the last extent ($last) nor the volume (${shape.volume}) is a multiple of the block size $blockSize" } val flat = Shape(shape.volume / blockSize) - return Layout(flat, rowMajorStrides(flat), offsetBlocks, bytesPerBlock, blocked = true) + return Layout(flat, rowMajorStrides(flat), offsetBlocks, bytesPerBlock, blocked = true, blockOrder = blockOrder) } } } diff --git a/skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/memory/TensorView.kt b/skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/memory/TensorView.kt index bef37a9ef..3369963c9 100644 --- a/skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/memory/TensorView.kt +++ b/skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/memory/TensorView.kt @@ -2,6 +2,9 @@ package sk.ainet.lang.memory import sk.ainet.lang.tensor.Shape import sk.ainet.lang.tensor.TensorId +import sk.ainet.lang.memory.trace.NoopTraceSink +import sk.ainet.lang.memory.trace.TraceEvent +import sk.ainet.lang.memory.trace.TraceSink import sk.ainet.lang.tensor.storage.PackedBlockStorage import sk.ainet.lang.tensor.storage.TensorEncoding import sk.ainet.lang.types.DType @@ -108,6 +111,78 @@ public class TensorView( private fun blockSize(): Int = decoder?.blockSize ?: 1 + /** + * This view's blocks rearranged into [order] — the relayout #973 asks for, as a **visible + * adapter** rather than an operation wearing transpose's name. + * + * A packed `[out, in]` weight's blocks tile the input dimension, and two orders exist for them: + * [BlockOrder.ROW_MAJOR] (`o * blocksPerRow + b`, what a file holds) and + * [BlockOrder.INPUT_BLOCK_MAJOR] (`b * outputDim + o`, what every packed matmul kernel reads). + * They coincide only when a row is one block, which is why mixing them produced plausible wrong + * numbers instead of a crash (#968, #971). + * + * The bytes are copied — this is a real conversion, allocated in [scope] and emitted as an + * `AdapterInserted` so it appears in the trace with its price. Returning `this` when the order + * already matches is the only free case. + * + * The result carries [order] on its layout, so the next reader does not have to guess. Note the + * *decoded* content is unchanged: [get] and [toFloatArray] read through the order, so a + * prepacked view still describes the same matrix. + */ + public fun prepack( + order: BlockOrder, + scope: Scope = Scope.Ambient, + sink: TraceSink = NoopTraceSink, + ): TensorView { + if (layout.blockOrder == order) return this + require(layout.blocked) { "block order only applies to a block-packed view, not $format" } + check(layout.shape.rank == shape.rank) { "this view's blocks span rows; prepack it after materialize()" } + require(shape.rank == 2) { "block relayout is defined for 2-D weights, got $shape" } + val decoder = decoderOrNull() ?: throw IllegalStateException("a packed view needs its decoder to be relayouted") + val bytesPerBlock = decoder.bytesPerBlock + val rows = shape[0] + val blocksPerRow = layout.shape[layout.blockAxis] + val source = (storage as? Storage.Heap)?.bytes + ?: throw UnsupportedOperationException("block relayout reads heap byte storage in this milestone") + val sourceOffset = (storage as Storage.Heap).arrayOffset + (layout.offsetElements * bytesPerBlock).toInt() + + // Read through this view's own addressing and write through the target order's: the block + // holding (row o, block b) moves from wherever it is now to where `order` says it goes. + val out = ByteArray(rows * blocksPerRow * bytesPerBlock) + for (o in 0 until rows) { + for (b in 0 until blocksPerRow) { + val from = (layout.indexOf(o, b) - layout.offsetElements).toInt() + val to = if (order == BlockOrder.INPUT_BLOCK_MAJOR) b * rows + o else o * blocksPerRow + b + source.copyInto(out, to * bytesPerBlock, sourceOffset + from * bytesPerBlock, sourceOffset + (from + 1) * bytesPerBlock) + } + } + val heap = Storage.Heap.bytes(out.size, scope.kind, id, sink) + out.copyInto(heap.bytes!!, heap.arrayOffset) + if (sink.isEnabled) { + sink.emit( + TraceEvent.AdapterInserted( + kind = "prepack-${order.name.lowercase()}", + from = format, + to = format, + bytes = out.size.toLong(), + target = id, + scope = scope.kind, + ), + ) + } + return TensorView( + shape, + format, + Layout.blocked(shape, decoder.blockSize, bytesPerBlock, blockOrder = order), + heap, + id, + RelayoutedBlockDecoder(decoder, rows, blocksPerRow, order), + ) + } + + /** The decoder this view was built with, if any — needed to know its block geometry. */ + public fun decoderOrNull(): BlockDecoder? = decoder + // ---- element access (rule 4: decode, never a raw byte) ---- /** The decoded logical value at [indices]; `Float` for every float dtype and every packed encoding. */ @@ -251,6 +326,45 @@ public interface BlockDecoder { } } +/** + * Decodes a **relayouted** view by translating block indices back to the order its delegate reads. + * + * `prepack` moves bytes into the order a kernel feeds on; the decoder that came with the original + * view still addresses blocks in the original order (the M1 [PackedBlockDecoder] decodes from the + * `TensorData` it wraps, not from the storage it is handed). Mapping the index instead of the bytes + * is what keeps rule 4 true across a relayout: `get()` on a prepacked view returns the same value + * it returned before (#973). + */ +@ExperimentalMemoryApi +public class RelayoutedBlockDecoder( + private val delegate: BlockDecoder, + private val rows: Int, + private val blocksPerRow: Int, + private val order: BlockOrder, +) : BlockDecoder { + override val blockSize: Int get() = delegate.blockSize + override val bytesPerBlock: Int get() = delegate.bytesPerBlock + + override fun decodeBlock(storage: Storage, blockIndex: Long, out: FloatArray, outOffset: Int) { + delegate.decodeBlock(storage, sourceIndexOf(blockIndex), out, outOffset) + } + + /** The index this block had before the relayout. */ + private fun sourceIndexOf(blockIndex: Long): Long = when (order) { + // input-block-major index `b * rows + o` → row-major `o * blocksPerRow + b` + BlockOrder.INPUT_BLOCK_MAJOR -> { + val b = blockIndex / rows + val o = blockIndex % rows + o * blocksPerRow + b + } + BlockOrder.ROW_MAJOR -> { + val o = blockIndex / blocksPerRow + val b = blockIndex % blocksPerRow + b * rows + o + } + } +} + /** A [BlockDecoder] backed by an existing [PackedBlockStorage] implementation (the M1 bridge). */ @ExperimentalMemoryApi public class PackedBlockDecoder(private val packed: PackedBlockStorage) : BlockDecoder { diff --git a/skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/tensor/data/Q5_0TensorData.kt b/skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/tensor/data/Q5_0TensorData.kt index 06913fc31..ee9e52f88 100644 --- a/skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/tensor/data/Q5_0TensorData.kt +++ b/skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/tensor/data/Q5_0TensorData.kt @@ -18,8 +18,11 @@ import sk.ainet.lang.types.DType * element[j] = d * (lo + (bitLo shl 4) - 16) * element[j + 16] = d * (hi + (bitHi shl 4) - 16) * - * Matmul packing is **input-block-major** `(blockIdx * outputDim + o)`; see - * [Q5_1TensorData] for the layout/transpose contract. + * Block order: **canonical row-major** (`o * blocksPerRow + b`), like every other + * `Q*TensorData`; the kernels read input-block-major bytes and get there through a + * relayout, never by reinterpreting these. See + * `docs/design/memory/packed-weight-layout.md` (#973) — this kdoc used to claim the + * opposite. */ public interface Q5_0TensorData : TensorData { public val blockCount: Int diff --git a/skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/tensor/data/Q5_1TensorData.kt b/skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/tensor/data/Q5_1TensorData.kt index bda194f34..55e9e0ade 100644 --- a/skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/tensor/data/Q5_1TensorData.kt +++ b/skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/tensor/data/Q5_1TensorData.kt @@ -21,10 +21,13 @@ import sk.ainet.lang.types.DType * element[j] = d * (lo + (bitLo shl 4)) + m * element[j + 16] = d * (hi + (bitHi shl 4)) + m * - * As packed by the GGUF converter for matmul, blocks are **input-block-major** - * `(blockIdx * outputDim + o)`; `Q5_1MatmulKernel` indexes them that way and the - * CPU-ops lazy transpose is a pure shape swap. The per-block [dequantizeBlock] - * below is layout-agnostic (it dequantizes the block at a flat index). + * Block order: **canonical row-major** (`o * blocksPerRow + b`) — what a GGUF holds + * and what this type's [dequantizeBlock] and `toFloatArray` assume. `Q5_1MatmulKernel` + * reads *input-block-major* bytes instead, so a weight reaches it through a + * relayout (`TensorView.prepack`), never by reinterpreting these bytes in place. + * The contract is written down once in `docs/design/memory/packed-weight-layout.md` + * (#973); this kdoc used to claim the opposite, which is the confusion that issue + * exists to end. */ public interface Q5_1TensorData : TensorData { public val blockCount: Int diff --git a/skainet-lang/skainet-lang-core/src/commonTest/kotlin/sk/ainet/lang/memory/BlockOrderTest.kt b/skainet-lang/skainet-lang-core/src/commonTest/kotlin/sk/ainet/lang/memory/BlockOrderTest.kt new file mode 100644 index 000000000..371f502f5 --- /dev/null +++ b/skainet-lang/skainet-lang-core/src/commonTest/kotlin/sk/ainet/lang/memory/BlockOrderTest.kt @@ -0,0 +1,135 @@ +package sk.ainet.lang.memory + +import sk.ainet.lang.memory.trace.RecordingTraceSink +import sk.ainet.lang.memory.trace.TraceEvent +import sk.ainet.lang.tensor.Shape +import sk.ainet.lang.tensor.data.Q8_0BlockTensorData +import sk.ainet.lang.tensor.storage.TensorEncoding +import kotlin.test.Test +import kotlin.test.assertContentEquals +import kotlin.test.assertEquals +import kotlin.test.assertFailsWith +import kotlin.test.assertNotEquals +import kotlin.test.assertTrue + +/** + * #1094 (#973): which block order a packed weight is in is a property of the value. + * + * The two orders coincide only when a row is a single block, so every assertion here uses a weight + * that is **three** blocks wide — the case where mixing them produces a block-permuted matrix of + * plausible, finite, wrong numbers. That is the failure #968/#971 shipped, and the reason the order + * is now carried rather than guessed. + */ +@OptIn(ExperimentalMemoryApi::class) +class BlockOrderTest { + + private val rows = 4 + private val blocksPerRow = 3 + private val blockSize = 32 + private val bytesPerBlock = 34 + private val cols = blocksPerRow * blockSize + + /** A Q8_0 weight whose every block is identifiable: block (o, b) is filled with `o * 16 + b`. */ + private fun weight(): TensorView { + val bytes = ByteArray(rows * blocksPerRow * bytesPerBlock) + for (o in 0 until rows) { + for (b in 0 until blocksPerRow) { + val base = (o * blocksPerRow + b) * bytesPerBlock + bytes[base] = 0x00; bytes[base + 1] = 0x3C // fp16 scale 1.0 + for (i in 0 until blockSize) bytes[base + 2 + i] = (o * 16 + b).toByte() + } + } + val data = Q8_0BlockTensorData(Shape(rows, cols), bytes) + return TensorView.packed( + Storage.Heap.wrap(bytes), Shape(rows, cols), TensorEncoding.Q8_0, PackedBlockDecoder(data), + ) + } + + @Test + fun aViewLoadedFromAFileIsRowMajor() { + val w = weight() + assertEquals(BlockOrder.ROW_MAJOR, w.layout.blockOrder, "canonical is the default, as a file holds it") + assertTrue(w.layout.blocked) + assertTrue(w.layout.toString().contains("row_major"), w.layout.toString()) + } + + @Test + fun prepackingChangesTheBytesButNotTheMatrix() { + val canonical = weight() + val kernelOrder = canonical.prepack(BlockOrder.INPUT_BLOCK_MAJOR) + + assertEquals(BlockOrder.INPUT_BLOCK_MAJOR, kernelOrder.layout.blockOrder) + assertNotEquals(canonical.storage.id, kernelOrder.storage.id, "a relayout is a copy, not a view") + + // the decoded matrix is identical — get() and toFloatArray() read through the order + assertContentEquals(canonical.toFloatArray(), kernelOrder.toFloatArray(), "the matrix must not change") + for (r in 0 until rows) for (c in 0 until cols) { + assertEquals(canonical.get(r, c), kernelOrder.get(r, c), "element ($r,$c)") + } + } + + @Test + fun theBytesActuallyMoveToKernelFeedOrder() { + val kernelOrder = weight().prepack(BlockOrder.INPUT_BLOCK_MAJOR) + val bytes = (kernelOrder.storage as Storage.Heap).bytes!! + val offset = (kernelOrder.storage as Storage.Heap).arrayOffset + // block (o, b) must now sit at flat index b * rows + o + for (o in 0 until rows) { + for (b in 0 until blocksPerRow) { + val at = offset + (b * rows + o) * bytesPerBlock + 2 + assertEquals((o * 16 + b).toByte(), bytes[at], "block ($o,$b) should be at input-major index ${b * rows + o}") + } + } + } + + @Test + fun prepackingBackIsTheIdentity() { + // the double-transpose hazard: on 0.40.1 `transpose(transpose(W)) != W` for a non-square + // block grid and nothing detected it. A relayout that names its target order cannot have + // that problem — going there and back is exactly the original. + val canonical = weight() + val roundTrip = canonical.prepack(BlockOrder.INPUT_BLOCK_MAJOR).prepack(BlockOrder.ROW_MAJOR) + assertEquals(BlockOrder.ROW_MAJOR, roundTrip.layout.blockOrder) + assertContentEquals(canonical.toFloatArray(), roundTrip.toFloatArray()) + val original = (canonical.storage as Storage.Heap).bytes!! + val after = (roundTrip.storage as Storage.Heap).bytes!! + assertContentEquals(original, after.copyOfRange((roundTrip.storage as Storage.Heap).arrayOffset, (roundTrip.storage as Storage.Heap).arrayOffset + original.size)) + assertTrue(blocksPerRow != rows, "this test is only meaningful on a non-square block grid") + } + + @Test + fun prepackingToTheOrderItAlreadyHasIsFree() { + val w = weight() + val sink = RecordingTraceSink() + val same = w.prepack(BlockOrder.ROW_MAJOR, Scope.Ambient, sink) + assertEquals(w.storage.id, same.storage.id, "no copy when nothing has to move") + assertTrue(sink.eventsOf().isEmpty(), "and nothing to report") + } + + @Test + fun aRelayoutIsVisibleAndPriced() { + val sink = RecordingTraceSink() + val scope = ForwardScope(slabFloats = rows * blocksPerRow * bytesPerBlock, sink = sink, name = "prepack") + weight().prepack(BlockOrder.INPUT_BLOCK_MAJOR, scope, sink) + val adapter = sink.eventsOf().single() + assertEquals("prepack-input_block_major", adapter.kind) + assertEquals((rows * blocksPerRow * bytesPerBlock).toLong(), adapter.bytes, "the copy is priced in bytes") + assertEquals(ScopeKind.FORWARD, adapter.scope) + scope.close() + } + + @Test + fun theOrderSurvivesTheOperationsThatDoNotMoveBytes() { + val kernelOrder = weight().prepack(BlockOrder.INPUT_BLOCK_MAJOR) + assertEquals(BlockOrder.INPUT_BLOCK_MAJOR, kernelOrder.narrow(0, 1, 2).layout.blockOrder) + assertEquals(BlockOrder.INPUT_BLOCK_MAJOR, kernelOrder.transpose().layout.blockOrder) + assertEquals(BlockOrder.INPUT_BLOCK_MAJOR, kernelOrder.unsqueeze(0).layout.blockOrder) + } + + @Test + fun onlyAPackedViewHasABlockOrderToChange() { + val dense = TensorView.dense(Storage.Heap.floats(16), Shape(4, 4)) + assertEquals(BlockOrder.ROW_MAJOR, dense.layout.blockOrder, "the default is meaningless but harmless for dense") + assertFailsWith { dense.prepack(BlockOrder.INPUT_BLOCK_MAJOR) } + } +}