diff --git a/docs/modules/ROOT/nav.adoc b/docs/modules/ROOT/nav.adoc index e20d02741..1c8ed5409 100644 --- a/docs/modules/ROOT/nav.adoc +++ b/docs/modules/ROOT/nav.adoc @@ -29,6 +29,7 @@ ** xref:explanation/skainet-for-ai.adoc[SKaiNET for AI/ML] ** xref:explanation/minerva-secure-mcu-export.adoc[How Minerva secure MCU export fits] ** xref:explanation/operator-design.adoc[Operator documentation system] +** xref:explanation/quantization-process.adoc[The quantization process] ** xref:explanation/theory/index.adoc[Mathematical theory] *** xref:explanation/theory/matmul.adoc[Matrix multiplication] ** xref:explanation/examples/index.adoc[Worked examples] diff --git a/docs/modules/ROOT/pages/explanation/quantization-process.adoc b/docs/modules/ROOT/pages/explanation/quantization-process.adoc new file mode 100644 index 000000000..508d28b4d --- /dev/null +++ b/docs/modules/ROOT/pages/explanation/quantization-process.adoc @@ -0,0 +1,183 @@ += The Quantization Process: Weights, Activations, and Calibration +:description: How SKaiNET quantizes — weight-only block quant vs activation post-training quantization (PTQ), and how calibration captures the activation ranges an int8 accelerator needs. + +Quantization in {framework_name} comes in two flavours that are easy to conflate. This page draws +the line between them, then walks the **activation post-training quantization (PTQ)** pipeline — +calibration, scales, and the int8 matmul shape — that an int8 NPU or accelerator target needs. + +For the *weight* side, this article is an overview; the depth lives in +xref:explanation/perf/quantized-simd-kernels.adoc[How quantized SIMD kernels are built] and +xref:explanation/perf/turboquant-kv-compression.adoc[TurboQuant KV-cache compression]. + +== Two regimes, one word + +[cols="1,3,3",options="header"] +|=== +| | *Weight-only quantization* | *Activation PTQ* + +| What is quantized +| The *weights* only. Activations stay fp32. +| *Both* operands of a matmul — activations and weights — are int8. + +| Where it runs +| Eager CPU/JVM kernels (`Q8_0`, `Q4_K`, `Q6_K`…), TurboQuant for the KV cache. +| Integer accelerators (NPUs) that do `i8 × i8 → i32` MACs. + +| The matmul +| `out[fp32] = Σ act[fp32] · dequant(w_block)` — dequant fused into the inner loop. +| `out = requantize( Σ q_act[i8] · q_w[i8] )` with an i32 accumulator. + +| Needs calibration? +| No. Block scales are derived from the weights at quantize time. +| *Yes.* Activation ranges are data-dependent and must be measured. + +| In the codebase +| `QuantizedMatmul`, the quantized SIMD kernels, TurboQuant. +| Calibration in the framework; int8 emission in the target backend codegen. +|=== + +The rest of this page is about the second column. + +== The int8 matmul shape + +An integer accelerator computes a matmul as `i8 × i8 → i32`, so a single fp32 projection +`y = x · Wᵀ` becomes a three-part pattern: + +[source,text] +---- +q_x = clamp(round(x / Sa), -127, 127) # quantize activation (fp32 -> i8) +acc = matmul_i8(q_x, q_w) # i8 x i8 -> i32 accumulator +y = acc * (Sa * Sw) # requantize (i32 -> fp32) +---- + +with *symmetric per-tensor* scales + +* `Sa = max|x| / 127` — the **activation** scale (data-dependent → calibrated), +* `Sw = max|W| / 127` — the **weight** scale (static → computed from the weights directly), +* `So = max|y| / 127` — the **output** scale, used when the result feeds another int8 matmul. + +Zero-point is `0` under symmetric quantization, which keeps the requantize a single multiply. +Softmax, LayerNorm and GELU stay in fp32 — only the matmuls are quantized. + +`Sw` is free: the weights are known ahead of time. `Sa` and `So` are the problem — they depend on +the *activations*, which depend on the input. That is what calibration measures. + +== Calibration: measuring activation ranges + +Calibration runs the model **eagerly in fp32** over a handful of representative inputs and records, +per matmul, the absolute-max of each operand and of the result. The per-tensor scale is then +`S = max|x| / 127`, accumulated (by max) across all calibration inputs. + +The clean way to tap the activation stream is a `TensorOps` decorator that intercepts `matmul` and +delegates everything else, using Kotlin interface delegation: + +[source,kotlin] +---- +class ObservingTensorOps( + private val base: TensorOps, + private val onMatmul: (a: Tensor<*, *>, b: Tensor<*, *>, out: Tensor<*, *>) -> Unit, +) : TensorOps by base { // <1> + override fun matmul(a: Tensor, b: Tensor): Tensor { + val out = base.matmul(a, b) + onMatmul(a, b, out) // <2> + return out + } +} +---- +<1> Every op except `matmul` flows straight through to the real backend. +<2> `a` is the activation, `b` the weight (or, for attention score matmuls, a second activation); + `out` is the result. Read their values with `tensor.data.copyToFloatArray()` and fold into a + running `max|·|`. + +Wrap a context so the model's `ctx.ops.matmul(...)` calls route through the tap — the model itself +is unchanged: + +[source,kotlin] +---- +class ObservingExecutionContext(base: ExecutionContext, onMatmul: MatmulTap) + : ExecutionContext by base { + override val ops: TensorOps = ObservingTensorOps(base.ops, onMatmul) +} + +// drive it: +val ranges = CalibrationRanges() +val ctx = ObservingExecutionContext(DirectCpuExecutionContext.create(), ranges::observe) +for (input in representativeInputs) model.forward(input, ctx) // fp32 eager, taps every matmul +val scales = ranges.symmetricInt8Scales() // S = max|x| / 127 per tensor +---- + +[NOTE] +==== +{framework_name} also exposes the `ExecutionObserver` hooks (`onOpStart` / `onOpEnd`) on an +`ExecutionContext`. Those are the natural home for this logic, but on the plain eager contexts the +per-op `notifyOp` dispatch is not wired, so a `TensorOps` decorator is the reliable tap today. An +observer implementation can share the same range-accumulation code. +==== + +The output is one scale triple per matmul — keyed by deterministic op-invocation order so the +emitter can line them up with the graph: + +[source,text] +---- +# op_index Sa Sb So absmax_act absmax_rhs absmax_out +op0 0.124454215 0.003971611 0.053741008 15.805685 0.504395 6.825108 +op1 0.124454215 0.004852055 0.063222170 15.805685 0.616211 8.029216 +... +---- + +`Sb` is the second-operand scale. For the q/k/v/out/MLP projections it simply matches the static +`Sw`; for the **attention score matmuls** (`Q·Kᵀ`, `attn·V`) the second operand is *also an +activation*, so `Sb` is a genuine calibrated scale — which is why the calibrator records it for +every matmul rather than assuming operand two is a weight. + +== Why `QuantizedMatmul` is not the activation-int8 op + +A natural question: can the existing `QuantizedMatmul` carry this? No — and the reason is worth +stating, because it shapes where the int8 emission has to live. + +* **It is weight-only and eager.** `QuantizedMatmul` dequant-fuses `Q8_0`/`Q4_K` *weight* blocks + against **fp32 activations** and returns a materialised fp32 tensor. It never records into the + tape, so it produces no int8 graph for a backend to lower. +* **The tape can't express the pattern either.** `TensorOps.matmul` is *dtype-homogeneous* — + `matmul(a: Tensor, b: Tensor): Tensor`. There is no `i8 × i8 → i32` + accumulation form, and there is no `round` primitive (only `clamp` / `convert` / `sign`). So the + `quantize → i8 matmul → requantize` pattern cannot be written at the op level today. + +The consequence is a clean split of responsibilities: + +* **The framework calibrates** — a pure read-side tap over the existing fp32 eager run, fully + expressible with today's API (above). +* **The target backend emits int8** — the `quantize / i8-matmul / requantize` lowering belongs in + the accelerator's codegen, parameterised by the `Sa` / `Sb` / `So` the calibrator produced. + +== The pipeline end to end + +[source,text] +---- + representative inputs + │ + ▼ + fp32 eager run ──tap every matmul──▶ per-tensor max|·| + (ObservingExecutionContext) │ + ▼ + symmetric scales Sa, Sb, So (= max|x| / 127) + │ + static weights ──▶ Sw, int8 weight constants + │ + ▼ + backend codegen: quantize → i8 matmul (i32) → requantize + │ + ▼ + int8 model on the accelerator +---- + +Calibration is the SKaiNET-side half and is target-agnostic: the same scales drive any int8 backend. + +== Follow-ups + +* A **tape-level int8 matmul** op (`i8 × i8 → i32`) plus a `round` primitive would let the int8 + pattern be expressed in the graph itself, rather than only in backend codegen. +* **Per-channel weight scales** (a `Sw` per output column) and **percentile / KL calibration** + (instead of plain `max|·|`) are the usual accuracy levers once the per-tensor path works. +* Wiring the `ExecutionObserver` `notifyOp` dispatch on the eager contexts would let calibration use + the observer hooks directly instead of a `TensorOps` decorator.