Skip to content

npu-run cannot drive any current pack: it requires the per-role layer_ctx*.elf set, every pack ships the sealed full-ELF set #1862

Closed

Description

@bong-water-water-bong

What happens

npu-run — the batched token-ids entry point, the only CLI route that hands N ids to one forward pass — cannot run any pack on this box:

$ NPU_BATCH_CHECK=1 1bit npu-run --model ~/models/npu/bonsai2-27b-t2s2 \
      --kernels ~/models/npu/bonsai2-27b-t2s2/kernels-m16 --ids "1 2 3 4 5 6 7"
1bit npu-run: cannot read /home/bcloud/models/npu/bonsai2-27b-t2s2/kernels-m16/layer_ctx1.elf

Why

It is not a missing file, it is a kernel-set generation. --help states the contract:

--kernels  directory with layer_ctx{1,2,17}.elf, lmhead.elf and layer.pdi

npu/lane.cpp:69 reads layer_ctx1.elf and :71 derives the context map from layer_ctx2.elf / layer_ctx17.elf; app/unified.cpp:349 checks for npu/layer_ctx1.elf and npu/lmhead.elf. Every pack since the sealed route ships the other set:

  • bonsai2-27b-t2s2/kernels-m16 → SEALED-T2S.elf, full_i8_D_*, full_i8_G_* (no layer_ctx*);
  • the sealed/full-ELF route is driven by serve --device npu from plan.json, which npu-run does not read.
$ find ~/models ~/scratch-packs -maxdepth 4 -name layer_ctx1.elf | wc -l
0
$ find ~/models ~/scratch-packs -maxdepth 4 -path '*/npu/lmhead.elf' | wc -l
0

So the lane-based kernel set that npu-run requires is not present anywhere on this host — not in ~/models/npu (12 packs), not in ~/scratch-packs.

Why it matters

serve --parallel N batches requests (continuous batching: N sequences each advancing one token per pass). What npu-run --ids gave was the other thing: k rows of one prompt through one block in one launch, which is what the k-row bit-identity check needs to be meaningful — rows 2..k have to be independent of row 1 and of each other within the same pass, so a comparison against k sequential passes actually tests something.

With the per-role layout gone, that check has no CLI route on any current pack. It blocks:

What would fix it

Either teach npu-run the sealed route it already has available (plan.json + SEALED-*.elf / full_i8_*, i.e. what serve --device npu uses), or keep the lane path and ship one pack with the layer_ctx{1,2,17} + lmhead set so the existing code has something to drive. The first is the one that matches every pack on disk.

Measurements above are from this host, 2026-11-10, 1bit built from npu/batchcheck-per-row (fb3d0f8) with -DONEBIT_NPU=ON -DONEBIT_NPU_RT_DIR=~/wt/npu-rt-753-pr/engine/npu/npu_rt.

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The seam for the real fix is already there — this is plumbing, not new numerics

    Scoped tonight while fixing the message. The sealed route already has both k-row primitives npu-run needs:

    // npu/forward/include/q4nx_forward.h
    bool step_batch_all(const std::vector<int>& tokens, int start_pos, std::vector<float>& logits_all);
    void begin_verify(int rows, int start_pos);
    bool commit_verify_rows(int accepted);

    step_batch_all is documented as exactly this: "the verifier has to see each row's logits, not just the tail of the batch … the layer maths is the batched body step_batch already runs — only the final norm and the LM head are applied per row, and the head is launched once for all M rows."
    And it already refuses a noncausal pack by name, because in such a batch every row attends to rows after it, so per-row logits are not prefix-conditioned.

    Setup is two calls plus the lock, copied from app/forward_serve.cpp:916-943:

    e.fwd = std::make_unique<Q4nxNpuForward>();
    if (npu_lock_taken(!host_gemm, std::getenv("NPU_NO_DEVICE_LOCK") != nullptr)) e.lock_fd = take_npu_device_lock(streams);
    std::unique_ptr<xrt::device> dev;  if (!host_gemm) dev = std::make_unique<xrt::device>(0);
    const bool ok = dev ? e.fwd->init(*dev, model.c_str(), kernels) : e.fwd->init_host(model.c_str(), kernels);

    So a --batch mode on npu-run (ids in, one step_batch_all, per-row argmax out) is ~50 lines in app/main.cpp's run_npu, no kernel work. The device lock is not optional — npu_device_lock.h is what keeps a second client from wedging the array (#1084), and #1353's lesson is the mirror image: a host-GEMM run must not take it.

    What I changed tonight instead, and why

    Only the failure's clarity: run_npu now inspects the kernel directory before constructing the Lane and refuses by name —

    • sealed set present (SEALED-* / full_i8_* ELFs) → says how many, that npu-run needs layer_ctx{1,2,17}.elf + lmhead.elf + layer.pdi, and that the sealed route is driven by 1bit serve --device npu from plan.json;
    • neither set → says neither is there.

    Verifiable without a device or a ticket (it is a directory check), which is the only kind of change to this route I can verify on this box tonight:

    $ find ~/models ~/scratch-packs -maxdepth 4 -name layer_ctx1.elf | wc -l
    0
    $ ls ~/models/npu/Qwen3-0.6B/kernels; python3 -c "import json;print(json.load(open('…/Qwen3-0.6B/plan.json'))['kernels'])"
    (no such dir)   {'tag': ''}
    

    That is the constraint on verifying the full fix: the only pack on this host that is both sealed-route and runnable is the 27B (model.q4nx 16.8 GB, plan referencing kernels-m16); the small 1.9 packs here carry plan.json with kernels: {"tag": ""} and no kernel directory, so they exercise layout arithmetic only. A --batch implementation therefore has to be verified either against the 27B (device, GTT-marginal) or with NPU_INFER_HOST_GEMM=1 (same forward, host GEMMs, no device, no lock) — slow on 27B but exact.

    If a lane that owns the serve path would rather expose it there (forward-serve --batch-ids), say so and I will drop the npu-run idea: the important part is that one CLI can hand k ids to one block on a sealed pack, not which one.

  2. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Cross-reference from a reviewer's side: #1872 is the same fact restated as a refusal, and it landed the part this issue needs first.

    I reviewed #1872 tonight (771c12c, "npu-run: refuse a sealed kernel directory by name"). It adds exactly the diagnosis this issue makes — that a directory from the sealed/full-ELF generation has no layer_ctx{1,2,17}.elf, that the bare cannot read …/layer_ctx1.elf reads as a missing file rather than a different route, and that the honest failure is to name the generation that is there (SEALED-* / full_i8_*, counted) plus the command that drives it. So the refusal half of this issue is in flight; what remains is the thing the issue is actually for — either teaching npu-run to drive a sealed set, or saying in --help+docs that it is a one-route CLI (batched ids → one forward) that only the layer_ctx generation serves.

    Two facts from my review that whoever claims this can use instead of re-deriving:

    Also worth stating since the issue reports it as "cannot run ANY pack on this box": the boxes here are running the sealed generation, so this is not one bad pack directory — every kernel set in ~/models/npu/*/kernels-* is that generation, which is why the error reads as if npu-run is broken rather than unsupported.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions