From cfa7829538e0e4648b27bdf0f182a0a2d3c47fa7 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sun, 8 Nov 2026 15:21:01 -0400 Subject: [PATCH 1/2] docs/npu-plan: the schema table learns about designs[], and the P4 row stops saying the derivation is all there is Three statements on this page were true when written and are not now, all of them about fields the loader reads: * the schema table had no row for designs[] -- the per-role kernel set the plan may name (#1330, #1333), with its both-directions agreement check, its bare-file-name artifact, the generator writing it only where it can be complete, and Plan::check_artifacts() refusing a missing artifact at load (and npu_plan.py --check asking the same question at import time); * the layers[] row said the forward 'derives designs from the container', which is now only the no-section case; * the 'Not yet' P4 entry stopped at A2b part 1 (#1380). A2b is complete: the M=128 ELF set in open_prefill (#1380) and the lm-head context in open_head (#1458, #1478), so q4nx_forward.cpp creates no I8Ctx at all. What is genuinely still open is the residency step -- load_block is a named no-op, the cache fills lazily inside gemv -- and that is an optimisation whose evidence is a tokens/s row, not a structural move. Docs only; no code change. --- docs/npu-plan.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/docs/npu-plan.md b/docs/npu-plan.md index 44c1f44a1..d299cdaa6 100644 --- a/docs/npu-plan.md +++ b/docs/npu-plan.md @@ -65,7 +65,8 @@ tools/npu_plan.py [--kernels DIR] [--out PATH] [--check] [--print] | `kv.key`, `kv.value` | what the KV cache is stored as, per side: `f16` (the meaning of an absent `kv` block, and of every plan written before P7) or `int8`. `int8` is int8 with **one fp32 scale per (head, token)**, taken *after* a per-head Hadamard rotation — the rotation is what makes one scale affordable, because it collapses the per-channel magnitude spread the scale would otherwise have to cover. `key` is the side that costs no un-rotation (`(QH)(KH)ᵀ = QKᵀ`), so storing K rotated and rotating Q needs no inverse transform; `value` is a separate choice because an int8 V needs one on the output, which is why K-only is the cheap variant to prove the path with. This is a description until a target opens it: a target without the capability refuses `int8` **by name**, never serves f16 in its place. That check is `check_kv_caps` in `npu/target/target.h`, called from every target's `open()`, and it is gated device-free (`ctest -R npu_target_caps`) | | `kv.hadamard.block`, `kv.hadamard.signs` | the rotation itself: `block` is `0` (no rotation) or a power of two in `[2, 65536]` — the head width the rotation runs over — and `signs` is an optional **bare file name** (no directory part, under the model dir) for per-channel signs. A `signs` file with `block: 0` is refused rather than ignored | | `moe.batched_expert_gu` | `{enabled, K, N}`; when enabled, K and N must be named | -| `layers[]` | one entry per layer, in order: `index`, `role` (`attn`/`gdn`/`moe`/`mla`), and projections `qkv {K, N, fused}`, `attn_gate {K, N}`, `gate_up {K, N}`, `down {K, N}`. The count must equal `model.layers`. The projection shapes are recorded for P4’s `NpuModel`; the current forward reads only the count (for its [plan] log line) and derives designs from the container | +| `designs[]` | **optional**: the per-role kernel set — `role` (`QKV`, `QKVLIN`, `O`, `GU`, `G`, `U`, `D`, the MoE `GB`/`UB`, the Gemma4 second attention type, the MLA and GDN roles), `M`, `K`, `N` and the artifact `elf` (a **bare file name**). When present, `Plan::designs_agree()` requires it to be exactly the set the container derives — same roles, same geometry, same artifact, **both directions** — and refuses by naming the role and both sides; a plan without the section keeps the engine's own derivation, which is what every plan written before the field means. Written by `tools/npu_plan.py` only for the families it can state completely (a partial section would be refused at load, so MLA, GDN mixes and Gemma4's second attention type / per-layer embeddings / double-wide MLP get none). `Plan::check_artifacts()` then refuses at load, before any context is opened, when a named artifact is not present and non-empty in the kernel directory — and `npu_plan.py --check` asks the same question with the same rule at import time | +| `layers[]` | one entry per layer, in order: `index`, `role` (`attn`/`gdn`/`moe`/`mla`), and projections `qkv {K, N, fused}`, `attn_gate {K, N}`, `gate_up {K, N}`, `down {K, N}`. The count must equal `model.layers`. The projection shapes are recorded for P4’s `NpuModel`; the forward reads the count (for its [plan] log line), and takes the design set from `designs[]` when the plan names it, deriving it from the container otherwise | Unknown layer keys are ignored by the loader (the current generator also writes `ssm_out`; the forward does not read it from the plan). @@ -176,11 +177,14 @@ layers[0] {"index": 0, "role": "gdn", "qkv": {"K": 5120, "N": 10240, "fused" ## Not yet - **P4** (#803): **A1 landed** in #808 (`0a2cbb9`) — the `NpuTarget` interface (`npu/target/target.h`), the XDNA - target behind it, and `open()` consuming the plan instead of duplicating the derivation. **A2 is in progress and is - tracked on its own page of record**, [npu-p4-target-split.md](npu-p4-target-split.md) §3 (steps A1/A2/A3), so this - list does not try to snapshot it — as of #1380 (`A2b, part 1`) the M=128 ELF set is adopted in - `XdnaTarget::open_prefill()` instead of in the forward. What is still true from *this* page: `XdnaTarget::load_block` - remains a named no-op that says why, so weights still fill lazily per projection. + target behind it, and `open()` consuming the plan instead of duplicating the derivation. The step-by-step record is + [npu-p4-target-split.md](npu-p4-target-split.md) §3, so this list does not snapshot it, but the two ends that touch + *this* page's fields are worth naming: the plan can now **name the design set** (#1330, #1333 — `designs[]` above), + and the setup side is **the target's** — the M=128 ELF set in `XdnaTarget::open_prefill()` (#1380) and the lm-head + context in `XdnaTarget::open_head()` (#1458, #1478), so `q4nx_forward.cpp` creates no `I8Ctx` at all. What is still + true from *this* page: `XdnaTarget::load_block` remains a named no-op, so weight residency and decoding stay as they + were — the cache fills lazily inside `gemv`, and `load_block()` turning that into an explicit block step is the one + A2 item left, an optimisation whose evidence is a tokens/s row (see §3's A2c). - **P5** (#804): Hexagon target (t2 gemv on HVX/HMX, QEMU v73 oracle) has no engine rows yet. - **P6** (#805): the shared drafter protocol is on main; tau=0.6 and Hexagon acceptance rows are still missing. - **P7** (#806): the `kv` contract above is in the plan and its arithmetic is gated in CI (`ctest -R npu_kv_int8`, From 611ab5a9d52e52fbf2ce77ccb747fa9696f02f8a Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sun, 8 Nov 2026 16:58:28 -0400 Subject: [PATCH 2/2] docs/npu-plan: the M=16 decode family is still the forward's, so A2 has two items left, not one Review at cfa78295 is right and this is the correction: the paragraph said the setup side is the target's and that q4nx_forward.cpp creates no I8Ctx at all. The prefill ELF set (#1380) and the lm-head context (#1458, #1478) did move - the forward calls open_head() at :1357 and then asks head_ctx() for it - but the M=16 decode family is still built here: the target allocates the memory (XdnaTarget::make_ctx16) while the forward decides which decode contexts exist and configures each one field by field (MD, KD, ND, blocked_b, pb16i, t2s, planes) at q4nx_forward.cpp:1306-1330, and docs/npu-p4-target-split.md section 3 says exactly that ('still in the forward, and therefore still owed: the M=16 decode family'). So A2 has two items left: A2c moves that decode-family setup behind the target, and load_block stays a named no-op whose explicit block step is an optimisation with a tokens/s row as its evidence, not a structural move. The paragraph now says that, cites the lines, and no longer claims the forward creates no I8Ctx. Docs only; copyright --check rc 0. --- docs/npu-plan.md | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/docs/npu-plan.md b/docs/npu-plan.md index d299cdaa6..18fe9b0b0 100644 --- a/docs/npu-plan.md +++ b/docs/npu-plan.md @@ -180,11 +180,19 @@ layers[0] {"index": 0, "role": "gdn", "qkv": {"K": 5120, "N": 10240, "fused" target behind it, and `open()` consuming the plan instead of duplicating the derivation. The step-by-step record is [npu-p4-target-split.md](npu-p4-target-split.md) §3, so this list does not snapshot it, but the two ends that touch *this* page's fields are worth naming: the plan can now **name the design set** (#1330, #1333 — `designs[]` above), - and the setup side is **the target's** — the M=128 ELF set in `XdnaTarget::open_prefill()` (#1380) and the lm-head - context in `XdnaTarget::open_head()` (#1458, #1478), so `q4nx_forward.cpp` creates no `I8Ctx` at all. What is still - true from *this* page: `XdnaTarget::load_block` remains a named no-op, so weight residency and decoding stay as they - were — the cache fills lazily inside `gemv`, and `load_block()` turning that into an explicit block step is the one - A2 item left, an optimisation whose evidence is a tokens/s row (see §3's A2c). + and two of the three context sets are now **the target's**: the M=128 prefill ELF set in + `XdnaTarget::open_prefill()` (#1380) and the lm-head context in `XdnaTarget::open_head()` (#1458, #1478) — the + forward calls `open_head()` (`q4nx_forward.cpp:1357`) and then asks the target for the context + (`head_ctx()`), rather than building it. **The M=16 decode family is still the forward's**, and that is the + correction to an earlier version of this paragraph: the target allocates the memory + (`XdnaTarget::make_ctx16()`), but the forward still decides which decode contexts exist and configures each one + field by field — `MD`, `KD`, `ND`, `blocked_b`, `pb16i`, `t2s`, the plane count + (`q4nx_forward.cpp:1306-1330`) — and `docs/npu-p4-target-split.md` §3 says the same in as many words ("still + in the forward, and therefore still owed: the M=16 decode family"). What is left on A2 is therefore two things, + not one: **A2c** moves that decode-family setup behind the target (it needs nothing the setup state does not + already carry), and `XdnaTarget::load_block` remains a named no-op, so weight residency and decoding stay as + they were — the cache fills lazily inside `gemv`, and turning that into an explicit block step is an optimisation + whose evidence is a tokens/s row, not a structural move. - **P5** (#804): Hexagon target (t2 gemv on HVX/HMX, QEMU v73 oracle) has no engine rows yet. - **P6** (#805): the shared drafter protocol is on main; tau=0.6 and Hexagon acceptance rows are still missing. - **P7** (#806): the `kv` contract above is in the plan and its arithmetic is gated in CI (`ctest -R npu_kv_int8`,