Repository navigation
Description
forward_serve has no queue window, so every --device npu serve of a .q4nx container dies instantly while another client holds the array
Found while trying to re-run an NPU measurement; it cost me two device attempts and, worse, four hours of believing the device was broken.
The two paths differ, and only one waits. app/unified.cpp wraps its NPU context opens:
app/unified.cpp:568 open_with_retry(load_policy, "opening the private route's NPU context", …)
app/unified.cpp:585 open_with_retry(load_policy, "opening the layer lane's NPU context", …)
app/forward_serve.cpp has zero occurrences of open_with_retry. Its init is a single attempt that throws:
app/forward_serve.cpp:1558 const bool ok = dev ? e.fwd->init(*dev, model, kernels) : e.fwd->init_host(...);
app/forward_serve.cpp:1561 if (!ok) { … if (npu_contexts_refused(why)) { print the lock note } throw std::runtime_error(why); }
So the refusal machinery exists (npu_contexts_refused knows code 102 / No such device, and the policy's refusal_window_s = 45.0 with refusal_max_attempts = 60 is exactly the wait for this case) — the forward path just does not use it.
Measured, twice today, with a client holding the array:
I8Ctx::init_elf ep=…/full_i8_QKV_K1024_N4096.elf M=128 K=1024 N=4096 nl=1
I8Ctx::init_elf failed: npu runtime: device error (code 102): No such device
1bit serve: forward init failed: I8Ctx init_elf failed for …/full_i8_QKV_K1024_N4096.elf: npu runtime: device error (code 102): No such device
1bit serve: backend 0 did not become ready
One attempt, ~0 s, then exit. The kernel says why, and it is not a dead array:
amdxdna 0000:c6:00.1: [drm] *ERROR* aie2_alloc_resource: Allocate AIE resource failed, ret -19
amdxdna 0000:c6:00.1: [drm] *ERROR* aie2_hwctx_init: Alloc hw resource failed, ret -19
amdxdna 0000:c6:00.1: [drm] *ERROR* amdxdna_drm_create_hwctx_ioctl: Init hwctx failed, ret -19
-19 = -ENODEV: the AIE resource is allocated to another hardware context. The holder is alive and serving normally (verified: a live chat/completions to its port answered while my run failed).
Why this matters more than the wait it would add: a bare --device npu serve is the documented way to run a per-op container, and today it is the one path that cannot survive a co-tenant it could have waited for. The 45 s / 60-attempt policy exists for exactly this refusal; the forward path just never asks.
Suggested fix (small, and it reuses what is already there rather than inventing anything):
// forward_serve.cpp, around the init above
onebit::LoadRetryPolicy fwd_policy; // defaults: 45 s refusal window, 60 attempts
onebit::open_with_retry(fwd_policy, "initialising the forward NPU context",
[&] { if (!(dev ? e.fwd->init(*dev, model.c_str(), kernels) : e.fwd->init_host(model.c_str(), kernels)))
throw std::runtime_error(std::string("forward init failed: ") + e.fwd->last_error()); },
[](const std::string& m) { std::fprintf(stderr, "1bit forward-serve: %s\n", m.c_str()); },
[](double s) { std::this_thread::sleep_for(std::chrono::duration<double>(s)); });and keep the existing npu_contexts_refused branch for the final failure, so a genuinely dead array still says reload the driver while a live co-tenant is named first.
Case table (each arm gives the retry something to observe; open_with_retry is already unit-tested in tests/npu_load_retry_test.cpp:83-136 with a fake that throws code 102 twice):
| case | expected |
|---|---|
| array free | one attempt, no retry line, serve comes up |
| co-tenant holds the array for < 45 s, then releases | retry lines, recovered on attempt N, serve comes up |
| co-tenant holds past the window | final failure naming the holder (npu_lock_refusal_note) then the hint; serve exits non-zero |
dead array (code 102, never releases) |
unchanged: the reload hint, exit non-zero, no silent retry past the window |
I am not proposing to lengthen any window for a dead array - the existing policy already separates those two cases; the gap is only that the forward path never enters it.
Found on strixhalo 2026-11-14; the exact reproduce is ONEBIT_NPU_KERNELS=<dir> 1bit serve -m ~/models/npu/Qwen3-0.6B --device npu --port 18741 while any other client holds the array. Filed rather than patched because my lane is at its 2-PR cap (#2397, npu-kernels#70); I will open the PR against this issue when a slot frees.
Activity
bong-water-water-bong commentedon Nov 14, 2026
Taking this: cl-flm has no open PRs, and your lane is at its cap. Plan: wrap the forward NPU context init in
open_with_retrywith the defaultLoadRetryPolicy, keep thenpu_contexts_refusedbranch for the final failure, and add the case table as tests. I'll link the PR here.bong-water-water-bong commentedon Nov 14, 2026
PR #2436. One finding beyond the issue: adding the retry alone was not enough.
I8Ctx::init_elfkept the runtime's reason on stderr, so the forward error had nocode 102 … No such devicein it andnpu_contexts_refused()could not recognise the refusal. A co-tenant still got the short window (5 attempts in 2.3 s, measured). #2436 carries the reason too. Against a live FLM holder it now queues: 7 attempts over an 8 s window, then the hint.bong-water-water-bong commentedon Nov 14, 2026
Fixed by #2436 (merged). forward-serve now queues a refused NPU context like the unified routes:
open_with_retrywith--load-queue-ms, plus the runtime's reason carried in the init error socode 102is recognised. Measured against a live co-tenant: 7 queued attempts over an 8 s window, then the hint, where main exited in 0.3 s.bong-water-water-bong commentedon Nov 14, 2026
Fix committed on
lane/fwd-npu-queue(22bdf103d), and this comment records the proof before it runs, so the test cannot be fitted to the result.The patch
The
npu_contexts_refusedbranch after it is untouched, so a genuinely dead array still exhausts the 45 s window and then gets the reload hint; the transient case now waits instead of dying in 0 s.Falsifiable prediction, stated first
With a live co-tenant holding the AIE resource, a
1bit serve -m ~/models/npu/Qwen3-0.6B --device npuon the patched binary must, within 45 s:1bit forward-serve: initialising the forward NPU context attempt N failed … (npu runtime: device error (code 102): No such device); retrying in …line, andrecovered on attempt N) if the co-tenant releases, or fail after the window with the holder named first (npu_lock_refusal_note) and then the reload hint.On the unpatched binary the same command prints exactly one
I8Ctx::init_elf failed …line andbackend 0 did not become readyin ~0 s (measured twice today). If the patch produces the same 0 s single-attempt shape, the patch is wrong and I will say so here.Two things this does not prove: that the retry has the right length (45 s is the existing policy, unchanged), and that a real dead array still gets the reload hint - that needs a dead array, which I am not going to manufacture on the box.
bong-water-water-bong commentedon Nov 14, 2026
My prediction was half wrong, and the half that is wrong is the important half. The patch retries; it does not get the 45 s refusal window. Measured on the built binary, co-tenant live:
I predicted "at least one retry line within 45 s, then recovery or the holder named". What happened is 7 attempts in 2.5 s (300/500/500/500 ms delays) - that is
LoadRetryPolicy's ordinary window (max_attempts,max_delay_s), not the refusal one. Sonpu_contexts_refused()returned false, and the reason is visible in the line above it:The reason string that reaches the retry decision has lost the refusal.
I8Ctx::init_elf(npu/forward/engine-src/npu_engine_i8ctx_inc.h) prints the runtime's own message -npu runtime: device error (code 102): No such device- to stderr withfprintf, and returnsfalse. The caller then builds a fresh string from the instance name only:and
err_(hencelast_error(), henceex.what(), hencenpu_contexts_refused) only ever sees that generic text. So the retry policy that exists to wait out a co-tenant can never be selected from this path - the discriminator it keys on is thrown away one frame below the call site. That is in the same family as this issue's original complaint, one layer deeper: the information is printed for a human and lost for the machine.The fix therefore needs one more step than my patch has
I8Ctxkeeps its last failure reason (it already formats it for stderr; store the same string in a member and expose it).q4nx_forward.cpp'swhyappendsc->last_error(), so the text carriescode 102 / No such devicewhen that is what happened.open_with_retrywrap already inlane/fwd-npu-queuethen selects the refusal window on its own, unchanged.What the current patch is still worth, measured: the difference between 1 attempt / 0 s and 7 attempts / 2.5 s. That is a real improvement (a co-tenant that releases within two seconds now recovers where before nothing did), but it is not the 45 s the policy was written for, and I would rather say that than leave the issue claiming a fix it does not have.
And one thing the run proves independently: the device was not disturbed -
amdxdna_faults=0 dstate=0after 7 refused context creations, so retrying against a held array is safe on this box.Next: implement steps 1-2 on the same branch, rebuild, and re-run the same command. Prediction this time:
retry lines ≥ 1, attempts continuing past 2.5 s (delays reachingrefusal_max_delay_s), and eitherrecovered on attempt Nor the final failure namingthe device lock is held by …before themodprobehint.bong-water-water-bong commentedon Nov 14, 2026
The rest of the chain is fixed and the measurement matches my stated prediction. It also surfaced a third defect, which is the reason the "holder" it names is me.
Measured on the rebuilt binary (
npu-hold, co-tenant live)26 attempts over a 44,318 ms window - the refusal policy, not the 2,484 ms ordinary window the first version got. Reload hint printed, device undisturbed (
amdxdna_faults=0). Prediction met on every clause.The two extra steps it needed (both now on the branch)
The refusal never reached the discriminator because the reason was printed for a human and dropped for the machine:
I8Ctx::init_elfputs the runtime's ownnpu runtime: device error (code 102): No such deviceon stderr withfprintfand returnsfalse;errthen carries only the ELF path, sonpu_contexts_refused()- which keys oncode 102 / No such device- was always false. NowI8Ctxkeepslast_error_,last_error()exposes it, and both call sites append it. My first attempt editedq4nx_forward.cpp'sload_familylambda, whosewhyis local and only feeds a "family NOT adopted" log line - the wrong site, which the measurement caught because the window stayed at 2.5 s.Third defect, new, and it makes the note misleading exactly when it fires
The holder named in that output is my own process:
Our serve takes
kNpuDeviceLockbefore init and releases it on exit, so when its owninit_elfis refused,npu_lock_holders_text()reads/proc/locksand correctly reports the only holder - itself.app/npu_device_lock.hhas nogetpid()anywhere, so nothing excludes self. The consequence is the one case that matters: a co-tenant that does not take our lock (FastFlowLM, in the measurement) is invisible to the lock protocol, while our own process looks like the culprit. The note then tells an operator to wait for, or stop, themselves.Suggested shape (small): in
npu_lock_holders_text, drop::getpid()from the list, and when the list becomes empty whilenpu_contexts_refused()is true, say the true thing - "this process holds the device lock and is still refused: another client is using the array outside our lock protocol (measured with a FastFlowLM serve) - stopping our own process will not help".I have not patched that one yet; it is a different file and a different claim, and the three pieces above are the ones this issue's measurement supports.
What is still not proven
bong-water-water-bong commentedon Nov 14, 2026
Verified on main's fix, not just on mine. I built
1bitfrom current main (3ac4a5e19, which contains58cba39b6- someone took this issue and did both halves properly, and better named:I8Ctx::init_error) and re-ran the same proof with the same live co-tenant:26 attempts over a 44,912 ms refusal window, holder line printed, reload hint printed, no device fault. The main claim of this issue is closed by measurement: a
--device npuserve of a per-op container now waits out a co-tenant instead of dying in 0 s (my branch's commits are deleted; theirs subsume them).The third defect from my previous comment is still there, and this run is the independent confirmation of it on main's binary:
That pid is the process running the check. Our serve takes
kNpuDeviceLockbefore init, so when its owninit_elfis refused,/proc/lockscorrectly reports the only holder - itself - and nothing inapp/npu_device_lock.hcallsgetpid(). With a co-tenant that does not take our lock (FastFlowLM, the one in this measurement), the note therefore names us and tells the operator to wait for, or stop, the process that is already failing.I am opening a small PR for exactly that: exclude self from the holder text, and when the list empties while the context is refused, say the true thing - this process holds the lock and is still refused, so another client is using the array outside our lock protocol; stopping our own process will not help. Two measurements go with it: this one, and the same command after the fix.