You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
npu-run cannot drive any current pack: it requires the per-role layer_ctx*.elf set, every pack ships the sealed full-ELF set #1862
It is not a missing file, it is a kernel-set generation. --help states the contract:
--kernels directory with layer_ctx{1,2,17}.elf, lmhead.elf and layer.pdi
npu/lane.cpp:69 reads layer_ctx1.elf and :71 derives the context map from layer_ctx2.elf / layer_ctx17.elf; app/unified.cpp:349 checks for npu/layer_ctx1.elf and npu/lmhead.elf. Every pack since the sealed route ships the other set:
bonsai2-27b-t2s2/kernels-m16 → SEALED-T2S.elf, full_i8_D_*, full_i8_G_* (no layer_ctx*);
the sealed/full-ELF route is driven by serve --device npu from plan.json, which npu-run does not read.
So the lane-based kernel set that npu-run requires is not present anywhere on this host — not in ~/models/npu (12 packs), not in ~/scratch-packs.
Why it matters
serve --parallel N batches requests (continuous batching: N sequences each advancing one token per pass). What npu-run --ids gave was the other thing: k rows of one prompt through one block in one launch, which is what the k-row bit-identity check needs to be meaningful — rows 2..k have to be independent of row 1 and of each other within the same pass, so a comparison against k sequential passes actually tests something.
With the per-role layout gone, that check has no CLI route on any current pack. It blocks:
any "M=k vs k × M=1" measurement on a sealed-route pack.
What would fix it
Either teach npu-run the sealed route it already has available (plan.json + SEALED-*.elf / full_i8_*, i.e. what serve --device npu uses), or keep the lane path and ship one pack with the layer_ctx{1,2,17} + lmhead set so the existing code has something to drive. The first is the one that matches every pack on disk.
Measurements above are from this host, 2026-11-10, 1bit built from npu/batchcheck-per-row (fb3d0f8) with -DONEBIT_NPU=ON -DONEBIT_NPU_RT_DIR=~/wt/npu-rt-753-pr/engine/npu/npu_rt.
The seam for the real fix is already there — this is plumbing, not new numerics
Scoped tonight while fixing the message. The sealed route already has both k-row primitives npu-run needs:
// npu/forward/include/q4nx_forward.hboolstep_batch_all(const std::vector<int>& tokens, int start_pos, std::vector<float>& logits_all);
voidbegin_verify(int rows, int start_pos);
boolcommit_verify_rows(int accepted);
step_batch_all is documented as exactly this: "the verifier has to see each row's logits, not just the tail of the batch … the layer maths is the batched body step_batch already runs — only the final norm and the LM head are applied per row, and the head is launched once for all M rows."
And it already refuses a noncausal pack by name, because in such a batch every row attends to rows after it, so per-row logits are not prefix-conditioned.
Setup is two calls plus the lock, copied from app/forward_serve.cpp:916-943:
e.fwd = std::make_unique<Q4nxNpuForward>();
if (npu_lock_taken(!host_gemm, std::getenv("NPU_NO_DEVICE_LOCK") != nullptr)) e.lock_fd = take_npu_device_lock(streams);
std::unique_ptr<xrt::device> dev; if (!host_gemm) dev = std::make_unique<xrt::device>(0);
constbool ok = dev ? e.fwd->init(*dev, model.c_str(), kernels) : e.fwd->init_host(model.c_str(), kernels);
So a --batch mode on npu-run (ids in, one step_batch_all, per-row argmax out) is ~50 lines in app/main.cpp's run_npu, no kernel work. The device lock is not optional — npu_device_lock.h is what keeps a second client from wedging the array (#1084), and #1353's lesson is the mirror image: a host-GEMM run must not take it.
What I changed tonight instead, and why
Only the failure's clarity: run_npu now inspects the kernel directory before constructing the Lane and refuses by name —
sealed set present (SEALED-* / full_i8_* ELFs) → says how many, that npu-run needs layer_ctx{1,2,17}.elf + lmhead.elf + layer.pdi, and that the sealed route is driven by 1bit serve --device npu from plan.json;
neither set → says neither is there.
Verifiable without a device or a ticket (it is a directory check), which is the only kind of change to this route I can verify on this box tonight:
$ find ~/models ~/scratch-packs -maxdepth 4 -name layer_ctx1.elf | wc -l
0
$ ls ~/models/npu/Qwen3-0.6B/kernels; python3 -c "import json;print(json.load(open('…/Qwen3-0.6B/plan.json'))['kernels'])"
(no such dir) {'tag': ''}
That is the constraint on verifying the full fix: the only pack on this host that is both sealed-route and runnable is the 27B (model.q4nx 16.8 GB, plan referencing kernels-m16); the small 1.9 packs here carry plan.json with kernels: {"tag": ""} and no kernel directory, so they exercise layout arithmetic only. A --batch implementation therefore has to be verified either against the 27B (device, GTT-marginal) or with NPU_INFER_HOST_GEMM=1 (same forward, host GEMMs, no device, no lock) — slow on 27B but exact.
If a lane that owns the serve path would rather expose it there (forward-serve --batch-ids), say so and I will drop the npu-run idea: the important part is that one CLI can hand k ids to one block on a sealed pack, not which one.
Cross-reference from a reviewer's side: #1872 is the same fact restated as a refusal, and it landed the part this issue needs first.
I reviewed #1872 tonight (771c12c, "npu-run: refuse a sealed kernel directory by name"). It adds exactly the diagnosis this issue makes — that a directory from the sealed/full-ELF generation has no layer_ctx{1,2,17}.elf, that the bare cannot read …/layer_ctx1.elf reads as a missing file rather than a different route, and that the honest failure is to name the generation that is there (SEALED-* / full_i8_*, counted) plus the command that drives it. So the refusal half of this issue is in flight; what remains is the thing the issue is actually for — either teaching npu-run to drive a sealed set, or saying in --help+docs that it is a one-route CLI (batched ids → one forward) that only the layer_ctx generation serves.
Two facts from my review that whoever claims this can use instead of re-deriving:
Also worth stating since the issue reports it as "cannot run ANY pack on this box": the boxes here are running the sealed generation, so this is not one bad pack directory — every kernel set in ~/models/npu/*/kernels-* is that generation, which is why the error reads as if npu-run is broken rather than unsupported.
Activity
bong-water-water-bong commentedon Nov 10, 2026
The seam for the real fix is already there — this is plumbing, not new numerics
Scoped tonight while fixing the message. The sealed route already has both k-row primitives
npu-runneeds:step_batch_allis documented as exactly this: "the verifier has to see each row's logits, not just the tail of the batch … the layer maths is the batched body step_batch already runs — only the final norm and the LM head are applied per row, and the head is launched once for all M rows."And it already refuses a noncausal pack by name, because in such a batch every row attends to rows after it, so per-row logits are not prefix-conditioned.
Setup is two calls plus the lock, copied from
app/forward_serve.cpp:916-943:So a
--batchmode onnpu-run(ids in, onestep_batch_all, per-row argmax out) is ~50 lines inapp/main.cpp'srun_npu, no kernel work. The device lock is not optional —npu_device_lock.his what keeps a second client from wedging the array (#1084), and #1353's lesson is the mirror image: a host-GEMM run must not take it.What I changed tonight instead, and why
Only the failure's clarity:
run_npunow inspects the kernel directory before constructing theLaneand refuses by name —SEALED-*/full_i8_*ELFs) → says how many, thatnpu-runneedslayer_ctx{1,2,17}.elf+lmhead.elf+layer.pdi, and that the sealed route is driven by1bit serve --device npufrom plan.json;Verifiable without a device or a ticket (it is a directory check), which is the only kind of change to this route I can verify on this box tonight:
That is the constraint on verifying the full fix: the only pack on this host that is both sealed-route and runnable is the 27B (
model.q4nx16.8 GB, plan referencingkernels-m16); the small 1.9 packs here carryplan.jsonwithkernels: {"tag": ""}and no kernel directory, so they exercise layout arithmetic only. A--batchimplementation therefore has to be verified either against the 27B (device, GTT-marginal) or withNPU_INFER_HOST_GEMM=1(same forward, host GEMMs, no device, no lock) — slow on 27B but exact.If a lane that owns the serve path would rather expose it there (
forward-serve --batch-ids), say so and I will drop thenpu-runidea: the important part is that one CLI can hand k ids to one block on a sealed pack, not which one.bong-water-water-bong commentedon Nov 10, 2026
Cross-reference from a reviewer's side: #1872 is the same fact restated as a refusal, and it landed the part this issue needs first.
I reviewed #1872 tonight (
771c12c, "npu-run: refuse a sealed kernel directory by name"). It adds exactly the diagnosis this issue makes — that a directory from the sealed/full-ELF generation has nolayer_ctx{1,2,17}.elf, that the barecannot read …/layer_ctx1.elfreads as a missing file rather than a different route, and that the honest failure is to name the generation that is there (SEALED-*/full_i8_*, counted) plus the command that drives it. So the refusal half of this issue is in flight; what remains is the thing the issue is actually for — either teachingnpu-runto drive a sealed set, or saying in--help+docs that it is a one-route CLI (batched ids → one forward) that only thelayer_ctxgeneration serves.Two facts from my review that whoever claims this can use instead of re-deriving:
layer_ctx1.elfis absent, so it cannot shadow the ordinary missing-file error for a partial layer set — meaning a fix here can keep it and add the sealed route underneath without the two messages fighting;SEALED-1.elfand nolayer_ctx1.elf→ new text; dir with neither → old error survives). If someone takes this issue, the sensible split is: npu-run: refuse a sealed kernel directory by name instead of "cannot read layer_ctx1.elf" #1872 lands the refusal + its arm, and this issue lands either the sealed route or the documented "not supported, here is what is" — with the arm extended to whichever way it goes.Also worth stating since the issue reports it as "cannot run ANY pack on this box": the boxes here are running the sealed generation, so this is not one bad pack directory — every kernel set in
~/models/npu/*/kernels-*is that generation, which is why the error reads as if npu-run is broken rather than unsupported.npu: the batch width falls back to the plan's prefill M, so a sealed …