Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/npu-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,10 +56,14 @@ tools/npu_plan.py <model dir> [--kernels DIR] [--out PATH] [--check] [--print]
| `container.tile_order` | q4_1 tile order; only `engine-1` is implemented. Absent means `engine-1`, so containers written before the field keep loading; an unknown id is refused. The loader is its only reader in this tree |
| `fusion.gate_up` | `fused` or `split` (replaces the `full_i8_GU_*` filename probe) |
| `prefill.M` | prefill activation rows; absent/0 means "not stated". Recorded and validated, but the current forward still takes its prefill rows from `Q4NX_DESIGN_M_PREFILL` (128) |
| `prefill.sealed_elf` | optional, pb16i/t2s only: a second sealed ELF, the decode set's instances built for more rows per launch (the wide prefill family, engine#1990). A bare file name in the kernels directory. Wins over the `config.json` key; `NPU_WIDE_SEALED_ELF` still overrides both. A file that is missing or does not load is logged and prefill stays at 16 rows |
| `prefill.sealed_rows` | rows per launch of `prefill.sealed_elf`, 17..128; required when it is named |
| `decode.M` | decode activation rows; absent/0 means not stated. Recorded for P4’s `NpuModel`; the current forward selects the M=16 family through `decode.dirs`, not through this value |
| `decode.enabled` | false disables the decode twin family |
| `decode.dirs[]` | where the decode family is looked for, in order: `root` is `model` (the `-m` directory) or `kernels` (`--kernels`), `path` is relative under that root, `.` is the root itself. Empty, absolute or `..` paths are refused |
| `decode.sealed_elf` | pb16i only: the sealed ELF the decode roles run |
| `decode.narrow_elf` | optional, pb16i/t2s only: a third sealed ELF, the decode set's instances built for fewer rows per launch (the narrow decode family, engine#2073). A bare file name in the kernels directory; wins over the `config.json` key, and `NPU_NARROW_SEALED_ELF` overrides both. A file that is missing or does not load is logged and decode stays on the 16-row kernels |
| `decode.narrow_rows` | rows per launch of `decode.narrow_elf`, 1..15; required when it is named |
| `kernels.tag` | artifact-name suffix: `""`, `_b8`, `_pb16i`, `_t2s` or `_t2s2`; must agree with the container layout and `container.pb16i` |
| `attention.mode` | `causal` (default) or `noncausal_full_attention`. The latter is a prompt-encoding mode: every full-attention layer sees the whole prompt, and the forward refuses to generate with it |
| `kv.key`, `kv.value` | what the KV cache is stored as, per side: `f16` (the meaning of an absent `kv` block, and of every plan written before P7) or `int8`. `int8` is int8 with **one fp32 scale per (head, token)**, taken *after* a per-head Hadamard rotation — the rotation is what makes one scale affordable, because it collapses the per-channel magnitude spread the scale would otherwise have to cover. `key` is the side that costs no un-rotation (`(QH)(KH)ᵀ = QKᵀ`), so storing K rotated and rotating Q needs no inverse transform; `value` is a separate choice because an int8 V needs one on the output, which is why K-only is the cheap variant to prove the path with. This is a description until a target opens it: a target without the capability refuses `int8` **by name**, never serves f16 in its place. That check is `check_kv_caps` in `npu/target/target.h`, called from every target's `open()`, and it is gated device-free (`ctest -R npu_target_caps`) |
Expand Down Expand Up @@ -103,6 +107,10 @@ The loader never substitutes a default for something it cannot honour; every ref
layer count that disagrees with `model.layers`;
- `container.pb16i` with an empty `decode.sealed_elf` when `kernels.tag` is `_pb16i` and decode is enabled;
or a non-empty `sealed_elf` on a non-pb16i container.
- `prefill.sealed_elf` without `decode.sealed_elf`, with a path instead of a bare name, or with
`prefill.sealed_rows` missing or outside 17..128; or `prefill.sealed_rows` without `prefill.sealed_elf`.
- `decode.narrow_elf` without `decode.sealed_elf`, with a path instead of a bare name, or with
`decode.narrow_rows` missing or outside 1..15; or `decode.narrow_rows` without `decode.narrow_elf`.

The forward adds one more check on load: `kernels.tag` must equal the tag the container itself implies
(same block, `Q4nxNpuForward::init_impl` in `npu/forward/src/q4nx_forward.cpp`), so a plan and a container cannot
Expand Down
7 changes: 5 additions & 2 deletions docs/npu.md
Original file line number Diff line number Diff line change
Expand Up @@ -550,7 +550,9 @@ may name a second sealed ELF with the same instances built for more rows: `q4nx_
`q4nx_pb16i_sealed_wide_elf`) in `config.json`, a file in the kernels directory beside the 16-row one, and
`q4nx_sealed_wide_rows` for its row count (default 32, allowed 17..128). No pack ships one yet, so nothing
changes for an existing pack. `NPU_WIDE_SEALED_ELF=<file>` overrides the pack's file, `NPU_WIDE_SEALED_ELF=0`
switches a pack's wide family off, and `NPU_WIDE_ROWS` overrides the row count.
switches a pack's wide family off, and `NPU_WIDE_ROWS` overrides the row count. `tools/npu_plan.py` copies
the two keys into `plan.json` as `prefill.sealed_elf` and `prefill.sealed_rows`, and a loaded plan wins over
`config.json` (`docs/npu-plan.md`).

When the wide ELF loads for every role that has a 16-row kernel, the chunk width becomes its row count and a
launch of more than 16 rows takes the wide kernel; decode steps and chunks of up to 16 rows stay on the
Expand Down Expand Up @@ -583,7 +585,8 @@ takes it (a decode step, or a verify pass of a few rows); anything wider keeps t
are the wide family's: all roles or none, one hw_context, shared weight images, and a file that is missing
or does not load is logged (`[forward] narrow family NOT adopted (...)`) while decode stays on the 16-row
kernels. `narrow_family_complete` and `narrow_takes` in `batch_width.h` are pinned by
`tests/npu_pack_test.cpp`. No pack ships a narrow ELF yet.
`tests/npu_pack_test.cpp`. No pack ships a narrow ELF yet. `plan.json` carries it as `decode.narrow_elf` and
`decode.narrow_rows` (`docs/npu-plan.md`).

Measured with a 4-row set, one binary, arms 16-row / 4-row / 16-row / 4-row, three prompts generating 49
tokens each (`(tokens - 1) / (wall - wall of a 1-token request)`), two `npu-hold` tickets:
Expand Down
9 changes: 9 additions & 0 deletions npu/forward/src/q4nx_forward.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -847,6 +847,15 @@ bool Q4nxNpuForward::init_impl(xrt::device* dev, const char* model_path,
// The plan is the runtime's copy of the sealed-ELF decision; when it names one it wins, so a plan
// and a container cannot disagree about which file the decode roles run.
if (plan_.loaded() && !plan_.sealed_elf.empty()) pb16i_sealed_ = plan_.sealed_elf;
// The plan is the pack's description when there is one, so its wide family wins over config.json's keys.
if (plan_.loaded() && !plan_.sealed_wide_elf.empty()) {
wide_sealed_ = plan_.sealed_wide_elf;
wide_sealed_rows_ = plan_.sealed_wide_rows;
}
if (plan_.loaded() && !plan_.sealed_narrow_elf.empty()) {
narrow_sealed_ = plan_.sealed_narrow_elf;
narrow_sealed_rows_ = plan_.sealed_narrow_rows;
}
// Hand the layout family to the launch layer now that the plan (or the probes) has settled it.
xd_->set_pb16i(pb16i_, pb16i_dir_, pb16i_sealed_);
xd_->set_hadamard_block(had_block);
Expand Down
50 changes: 48 additions & 2 deletions npu/plan/plan.h
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
// "model": {"hidden": 1024, "layers": 28, "arch": "qwen3"},
// "container": {"layout": "rm"|"b8"|"pb16i"|"t2s", "chunk_kind": "mode0|mode1|mode2|pb16i", "pb16i": false},
// "fusion": {"gate_up": "fused"|"split"},
// "prefill": {"M": 128},
// "prefill": {"M": 128}, (+ optional "sealed_elf", "sealed_rows": a wide family)
// "decode": {"M": 16, "enabled": true, "sealed_elf": "",
// "dirs": [{"root": "model", "path": "kernels-m16"},
// {"root": "kernels", "path": "m16"},
Expand Down Expand Up @@ -206,6 +206,14 @@ struct Plan {
struct DecodeDir { bool under_model = false; std::string path; };
std::vector<DecodeDir> decode_dirs;
std::string sealed_elf; // pb16i: one sealed ELF, per-role instance names
// engine#1990: an optional wide prefill family -- a second sealed ELF with the same instances built for
// `sealed_wide_rows` rows per launch. Empty = none; the decode family then also runs prefill chunks.
std::string sealed_wide_elf;
int sealed_wide_rows = 0;
// engine#2073: an optional narrow decode family -- the same instances built for FEWER rows, taken by launches
// of 1..`sealed_narrow_rows` rows (a decode step carries one). Empty = none.
std::string sealed_narrow_elf;
int sealed_narrow_rows = 0;
struct { bool enabled = false; int K = 0, N = 0; } moe_batched_gu;
std::vector<LayerPlan> layer_plans;

Expand Down Expand Up @@ -306,7 +314,13 @@ struct Plan {
throw PlanError("plan: " + path + " fusion.gate_up \"" + gu +
"\" is not fused or split");
}
if (j.contains("prefill")) p.prefill_m = positive_or_zero(obj(j, "prefill", path), "M", path);
if (j.contains("prefill")) {
const auto& pf = obj(j, "prefill", path);
p.prefill_m = positive_or_zero(pf, "M", path);
// Optional: plans written before the wide family existed load and mean what they meant.
p.sealed_wide_elf = pf.value("sealed_elf", "");
if (pf.contains("sealed_rows")) p.sealed_wide_rows = positive_or_zero(pf, "sealed_rows", path);
}
// Optional and defaulted on purpose: a plan written before this field existed must load and
// mean exactly what it meant then (causal), so this is not a format bump.
if (j.contains("attention")) {
Expand Down Expand Up @@ -518,6 +532,8 @@ struct Plan {
p.decode_m = positive_or_zero(d, "M", path);
p.decode_enabled = d.value("enabled", true);
p.sealed_elf = d.value("sealed_elf", "");
p.sealed_narrow_elf = d.value("narrow_elf", "");
if (d.contains("narrow_rows")) p.sealed_narrow_rows = positive_or_zero(d, "narrow_rows", path);
if (d.contains("dirs")) {
if (!d.at("dirs").is_array()) throw PlanError("plan: " + path + " decode.dirs must be an array");
for (const auto& one : d.at("dirs")) {
Expand Down Expand Up @@ -587,6 +603,36 @@ struct Plan {
"the decode roles run)");
if (!p.pb16i && !p.sealed_elf.empty())
throw PlanError("plan: " + path + " names decode.sealed_elf but container.pb16i is false");
// The wide family is the decode family's instances at another width, so it cannot exist without it, and
// its row count must be one a chunk can have: more than the 16 rows of the decode family, at most 128.
if (!p.sealed_wide_elf.empty()) {
if (p.sealed_elf.empty())
throw PlanError("plan: " + path + " names prefill.sealed_elf but decode.sealed_elf is empty (the "
"wide family is the sealed decode set built for more rows)");
if (p.sealed_wide_elf.find('/') != std::string::npos || p.sealed_wide_elf.find("..") != std::string::npos)
throw PlanError("plan: " + path + " prefill.sealed_elf \"" + p.sealed_wide_elf +
"\" must be a bare file name in the kernels directory");
if (p.sealed_wide_rows < 17 || p.sealed_wide_rows > 128)
throw PlanError("plan: " + path + " prefill.sealed_rows is " + std::to_string(p.sealed_wide_rows) +
" but a wide family holds 17..128 rows");
} else if (p.sealed_wide_rows != 0) {
throw PlanError("plan: " + path + " states prefill.sealed_rows without prefill.sealed_elf");
}
// The narrow family: the same three conditions, with the row bound on the other side of 16.
if (!p.sealed_narrow_elf.empty()) {
if (p.sealed_elf.empty())
throw PlanError("plan: " + path + " names decode.narrow_elf but decode.sealed_elf is empty (the "
"narrow family is the sealed decode set built for fewer rows)");
if (p.sealed_narrow_elf.find('/') != std::string::npos ||
p.sealed_narrow_elf.find("..") != std::string::npos)
throw PlanError("plan: " + path + " decode.narrow_elf \"" + p.sealed_narrow_elf +
"\" must be a bare file name in the kernels directory");
if (p.sealed_narrow_rows < 1 || p.sealed_narrow_rows > 15)
throw PlanError("plan: " + path + " decode.narrow_rows is " + std::to_string(p.sealed_narrow_rows) +
" but a narrow family holds 1..15 rows");
} else if (p.sealed_narrow_rows != 0) {
throw PlanError("plan: " + path + " states decode.narrow_rows without decode.narrow_elf");
}
if (p.tag == "_b8" && p.layout != "b8")
throw PlanError("plan: " + path + " kernels.tag is _b8 but container.layout is " + p.layout);
if ((p.tag == "_t2s" || p.tag == "_t2s2") && p.layout != "t2s")
Expand Down
40 changes: 40 additions & 0 deletions tests/npu_plan_generate_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,46 @@ def touch(path):
check(json.load(open(os.path.join(d2c, "plan.json")))["decode"]["sealed_elf"] == "",
"a t2s container does not inherit q4nx_pb16i_sealed_elf")

# 4d. engine#1990: a wide prefill family is carried into prefill{}, with its row count; a pack without
# one keeps the two-key-free block it always had; and a layout reads only its own key.
check(p2b["prefill"] == {"M": 128}, "no wide key in config -> prefill block unchanged")
d2d = model_dir(os.path.join(root, "b4"), layout="t2s", q4nx_weight_layout_version=6,
q4nx_t2s_sealed_elf="SEALED-T2S.elf", q4nx_t2s_sealed_wide_elf="SEALED-T2S-m32.elf",
layer_types=["full_attention", "full_attention"])
run(d2d)
p2d = json.load(open(os.path.join(d2d, "plan.json")))
check(p2d["prefill"] == {"M": 128, "sealed_elf": "SEALED-T2S-m32.elf", "sealed_rows": 32},
"t2s wide key -> prefill.sealed_elf and the default 32 rows")
d2e = model_dir(os.path.join(root, "b5"), layout="t2s", q4nx_weight_layout_version=6,
q4nx_t2s_sealed_elf="SEALED-T2S.elf", q4nx_t2s_sealed_wide_elf="W64.elf",
q4nx_sealed_wide_rows=64, layer_types=["full_attention", "full_attention"])
run(d2e)
check(json.load(open(os.path.join(d2e, "plan.json")))["prefill"]["sealed_rows"] == 64,
"q4nx_sealed_wide_rows is carried")
d2f = model_dir(os.path.join(root, "b6"), layout="t2s", q4nx_weight_layout_version=6,
q4nx_t2s_sealed_elf="SEALED-T2S.elf", q4nx_pb16i_sealed_wide_elf="PB-WIDE.elf",
layer_types=["full_attention", "full_attention"])
run(d2f)
check(json.load(open(os.path.join(d2f, "plan.json")))["prefill"] == {"M": 128},
"a t2s container does not inherit q4nx_pb16i_sealed_wide_elf")

# 4e. engine#2073: the narrow decode family is carried into decode{} the same way.
check("narrow_elf" not in p2b["decode"] and "narrow_rows" not in p2b["decode"],
"no narrow key in config -> decode block unchanged")
d2g = model_dir(os.path.join(root, "b7"), layout="t2s", q4nx_weight_layout_version=6,
q4nx_t2s_sealed_elf="SEALED-T2S.elf", q4nx_t2s_sealed_narrow_elf="SEALED-T2S-m4.elf",
layer_types=["full_attention", "full_attention"])
run(d2g)
dg = json.load(open(os.path.join(d2g, "plan.json")))["decode"]
check(dg.get("narrow_elf") == "SEALED-T2S-m4.elf" and dg.get("narrow_rows") == 4,
"t2s narrow key -> decode.narrow_elf and the default 4 rows")
d2h = model_dir(os.path.join(root, "b8"), layout="t2s", q4nx_weight_layout_version=6,
q4nx_t2s_sealed_elf="SEALED-T2S.elf", q4nx_pb16i_sealed_narrow_elf="PB-NARROW.elf",
layer_types=["full_attention", "full_attention"])
run(d2h)
check("narrow_elf" not in json.load(open(os.path.join(d2h, "plan.json")))["decode"],
"a t2s container does not inherit q4nx_pb16i_sealed_narrow_elf")

# 5. The plan printer writes nothing.
r = run(d2, "--print")
check(r.returncode == 0 and '"format": "1bit-npu-plan/1"' in r.stdout, "--print emits the plan on stdout")
Expand Down
Loading
Loading