Skip to content

forward_serve: no queue window for a refused NPU context — a co-tenant makes every --device npu serve die in 0 s while open_with_retry sits unused in the same file's sibling path #2434

Closed

Description

@bong-water-water-bong

forward_serve has no queue window, so every --device npu serve of a .q4nx container dies instantly while another client holds the array

Found while trying to re-run an NPU measurement; it cost me two device attempts and, worse, four hours of believing the device was broken.

The two paths differ, and only one waits. app/unified.cpp wraps its NPU context opens:

app/unified.cpp:568   open_with_retry(load_policy, "opening the private route's NPU context", …)
app/unified.cpp:585   open_with_retry(load_policy, "opening the layer lane's NPU context", …)

app/forward_serve.cpp has zero occurrences of open_with_retry. Its init is a single attempt that throws:

app/forward_serve.cpp:1558   const bool ok = dev ? e.fwd->init(*dev, model, kernels) : e.fwd->init_host(...);
app/forward_serve.cpp:1561   if (!ok) { … if (npu_contexts_refused(why)) { print the lock note } throw std::runtime_error(why); }

So the refusal machinery exists (npu_contexts_refused knows code 102 / No such device, and the policy's refusal_window_s = 45.0 with refusal_max_attempts = 60 is exactly the wait for this case) — the forward path just does not use it.

Measured, twice today, with a client holding the array:

  I8Ctx::init_elf ep=…/full_i8_QKV_K1024_N4096.elf M=128 K=1024 N=4096 nl=1
  I8Ctx::init_elf failed: npu runtime: device error (code 102): No such device
1bit serve: forward init failed: I8Ctx init_elf failed for …/full_i8_QKV_K1024_N4096.elf: npu runtime: device error (code 102): No such device
1bit serve: backend 0 did not become ready

One attempt, ~0 s, then exit. The kernel says why, and it is not a dead array:

amdxdna 0000:c6:00.1: [drm] *ERROR* aie2_alloc_resource: Allocate AIE resource failed, ret -19
amdxdna 0000:c6:00.1: [drm] *ERROR* aie2_hwctx_init: Alloc hw resource failed, ret -19
amdxdna 0000:c6:00.1: [drm] *ERROR* amdxdna_drm_create_hwctx_ioctl: Init hwctx failed, ret -19

-19 = -ENODEV: the AIE resource is allocated to another hardware context. The holder is alive and serving normally (verified: a live chat/completions to its port answered while my run failed).

Why this matters more than the wait it would add: a bare --device npu serve is the documented way to run a per-op container, and today it is the one path that cannot survive a co-tenant it could have waited for. The 45 s / 60-attempt policy exists for exactly this refusal; the forward path just never asks.

Suggested fix (small, and it reuses what is already there rather than inventing anything):

// forward_serve.cpp, around the init above
onebit::LoadRetryPolicy fwd_policy;   // defaults: 45 s refusal window, 60 attempts
onebit::open_with_retry(fwd_policy, "initialising the forward NPU context",
    [&] { if (!(dev ? e.fwd->init(*dev, model.c_str(), kernels) : e.fwd->init_host(model.c_str(), kernels)))
              throw std::runtime_error(std::string("forward init failed: ") + e.fwd->last_error()); },
    [](const std::string& m) { std::fprintf(stderr, "1bit forward-serve: %s\n", m.c_str()); },
    [](double s) { std::this_thread::sleep_for(std::chrono::duration<double>(s)); });

and keep the existing npu_contexts_refused branch for the final failure, so a genuinely dead array still says reload the driver while a live co-tenant is named first.

Case table (each arm gives the retry something to observe; open_with_retry is already unit-tested in tests/npu_load_retry_test.cpp:83-136 with a fake that throws code 102 twice):

case expected
array free one attempt, no retry line, serve comes up
co-tenant holds the array for < 45 s, then releases retry lines, recovered on attempt N, serve comes up
co-tenant holds past the window final failure naming the holder (npu_lock_refusal_note) then the hint; serve exits non-zero
dead array (code 102, never releases) unchanged: the reload hint, exit non-zero, no silent retry past the window

I am not proposing to lengthen any window for a dead array - the existing policy already separates those two cases; the gap is only that the forward path never enters it.

Found on strixhalo 2026-11-14; the exact reproduce is ONEBIT_NPU_KERNELS=<dir> 1bit serve -m ~/models/npu/Qwen3-0.6B --device npu --port 18741 while any other client holds the array. Filed rather than patched because my lane is at its 2-PR cap (#2397, npu-kernels#70); I will open the PR against this issue when a slot frees.

Activity

  1. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Taking this: cl-flm has no open PRs, and your lane is at its cap. Plan: wrap the forward NPU context init in open_with_retry with the default LoadRetryPolicy, keep the npu_contexts_refused branch for the final failure, and add the case table as tests. I'll link the PR here.

  2. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    PR #2436. One finding beyond the issue: adding the retry alone was not enough. I8Ctx::init_elf kept the runtime's reason on stderr, so the forward error had no code 102 … No such device in it and npu_contexts_refused() could not recognise the refusal. A co-tenant still got the short window (5 attempts in 2.3 s, measured). #2436 carries the reason too. Against a live FLM holder it now queues: 7 attempts over an 8 s window, then the hint.

  3. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Fixed by #2436 (merged). forward-serve now queues a refused NPU context like the unified routes: open_with_retry with --load-queue-ms, plus the runtime's reason carried in the init error so code 102 is recognised. Measured against a live co-tenant: 7 queued attempts over an 8 s window, then the hint, where main exited in 0.3 s.

  4. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Fix committed on lane/fwd-npu-queue (22bdf103d), and this comment records the proof before it runs, so the test cannot be fitted to the result.

    The patch

    onebit::LoadRetryPolicy fwd_policy;
    bool ok = false;
    onebit::open_with_retry(fwd_policy, "initialising the forward NPU context",
        [&] { const bool got = dev ? e.fwd->init(*dev, model.c_str(), kernels) : e.fwd->init_host(model.c_str(), kernels);
              if (!got) throw std::runtime_error(std::string("forward init failed: ") + e.fwd->last_error());
              ok = true; },
        [](const std::string& m) { std::fprintf(stderr, "1bit forward-serve: %s\n", m.c_str()); },
        [](double secs) { std::this_thread::sleep_for(std::chrono::duration<double>(secs)); });

    The npu_contexts_refused branch after it is untouched, so a genuinely dead array still exhausts the 45 s window and then gets the reload hint; the transient case now waits instead of dying in 0 s.

    Falsifiable prediction, stated first

    With a live co-tenant holding the AIE resource, a 1bit serve -m ~/models/npu/Qwen3-0.6B --device npu on the patched binary must, within 45 s:

    1. print at least one 1bit forward-serve: initialising the forward NPU context attempt N failed … (npu runtime: device error (code 102): No such device); retrying in … line, and
    2. either come up (recovered on attempt N) if the co-tenant releases, or fail after the window with the holder named first (npu_lock_refusal_note) and then the reload hint.

    On the unpatched binary the same command prints exactly one I8Ctx::init_elf failed … line and backend 0 did not become ready in ~0 s (measured twice today). If the patch produces the same 0 s single-attempt shape, the patch is wrong and I will say so here.

    Two things this does not prove: that the retry has the right length (45 s is the existing policy, unchanged), and that a real dead array still gets the reload hint - that needs a dead array, which I am not going to manufacture on the box.

  5. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    My prediction was half wrong, and the half that is wrong is the important half. The patch retries; it does not get the 45 s refusal window. Measured on the built binary, co-tenant live:

    serve_up=no  elapsed_ms=3059
    retry lines: 6   attempt lines: 6   recovered: 0   holder named: 0   reload hint: 0
    1bit forward-serve: initialising the forward NPU context attempt 2 failed after 169 ms (forward init failed: I8Ctx init_elf failed for …/full_i8_QKV_K1024_N4096.elf); retrying in 300 ms
    …
    1bit forward-serve: initialising the forward NPU context failed 7 attempt(s) in 2484 ms, last reason: forward init failed: I8Ctx init_elf failed for …
    

    I predicted "at least one retry line within 45 s, then recovery or the holder named". What happened is 7 attempts in 2.5 s (300/500/500/500 ms delays) - that is LoadRetryPolicy's ordinary window (max_attempts, max_delay_s), not the refusal one. So npu_contexts_refused() returned false, and the reason is visible in the line above it:

    $ grep -m1 'attempt 2 failed' serve.log
    (forward init failed: I8Ctx init_elf failed for /home/…/full_i8_QKV_K1024_N4096.elf)
       'No such device': 0 hits        'code 102': 0 hits
    

    The reason string that reaches the retry decision has lost the refusal. I8Ctx::init_elf (npu/forward/engine-src/npu_engine_i8ctx_inc.h) prints the runtime's own message - npu runtime: device error (code 102): No such device - to stderr with fprintf, and returns false. The caller then builds a fresh string from the instance name only:

    // npu/forward/src/q4nx_forward.cpp:665
    if (!c->init_elf(*dev, path.c_str(), rows, c16->KD, c16->ND, 1, inst)) {
        why = std::string("init_elf failed for instance ") + inst;   // <- the runtime reason is dropped here
        break;
    }

    and err_ (hence last_error(), hence ex.what(), hence npu_contexts_refused) only ever sees that generic text. So the retry policy that exists to wait out a co-tenant can never be selected from this path - the discriminator it keys on is thrown away one frame below the call site. That is in the same family as this issue's original complaint, one layer deeper: the information is printed for a human and lost for the machine.

    The fix therefore needs one more step than my patch has

    1. I8Ctx keeps its last failure reason (it already formats it for stderr; store the same string in a member and expose it).
    2. q4nx_forward.cpp's why appends c->last_error(), so the text carries code 102 / No such device when that is what happened.
    3. The open_with_retry wrap already in lane/fwd-npu-queue then selects the refusal window on its own, unchanged.

    What the current patch is still worth, measured: the difference between 1 attempt / 0 s and 7 attempts / 2.5 s. That is a real improvement (a co-tenant that releases within two seconds now recovers where before nothing did), but it is not the 45 s the policy was written for, and I would rather say that than leave the issue claiming a fix it does not have.

    And one thing the run proves independently: the device was not disturbed - amdxdna_faults=0 dstate=0 after 7 refused context creations, so retrying against a held array is safe on this box.

    Next: implement steps 1-2 on the same branch, rebuild, and re-run the same command. Prediction this time: retry lines ≥ 1, attempts continuing past 2.5 s (delays reaching refusal_max_delay_s), and either recovered on attempt N or the final failure naming the device lock is held by … before the modprobe hint.

  6. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The rest of the chain is fixed and the measurement matches my stated prediction. It also surfaced a third defect, which is the reason the "holder" it names is me.

    Measured on the rebuilt binary (npu-hold, co-tenant live)

    serve_up=no  elapsed_ms=44637
    attempt lines: 25   last retry delay: retrying in 2000 ms   recovered: 0   holder named: 1   reload hint: 1
    1bit forward-serve: … attempt 25 failed after 42316 ms (forward init failed: I8Ctx init_elf failed for …)
    1bit forward-serve: … failed 26 attempt(s) in 44318 ms over a 45000 ms queue window
    1bit forward-serve: [npu] the device lock is held by pid 1951307 (…/b2434/1bit serve -m …)
    1bit forward-serve: the NPU refuses hardware contexts: another client may hold the array …; if nothing else is serving, reload
    device after: amdxdna_faults=0
    

    26 attempts over a 44,318 ms window - the refusal policy, not the 2,484 ms ordinary window the first version got. Reload hint printed, device undisturbed (amdxdna_faults=0). Prediction met on every clause.

    The two extra steps it needed (both now on the branch)

    The refusal never reached the discriminator because the reason was printed for a human and dropped for the machine:

    // npu/target/xdna/xdna_target.cpp:168 - the site the log line came from
    if (!ctx->init_elf(...)) { err = "I8Ctx init_elf failed for " + elf; return false; }   // runtime reason DROPPED

    I8Ctx::init_elf puts the runtime's own npu runtime: device error (code 102): No such device on stderr with fprintf and returns false; err then carries only the ELF path, so npu_contexts_refused() - which keys on code 102 / No such device - was always false. Now I8Ctx keeps last_error_, last_error() exposes it, and both call sites append it. My first attempt edited q4nx_forward.cpp's load_family lambda, whose why is local and only feeds a "family NOT adopted" log line - the wrong site, which the measurement caught because the window stayed at 2.5 s.

    Third defect, new, and it makes the note misleading exactly when it fires

    The holder named in that output is my own process:

    [npu] the device lock is held by pid 1951307 (…/b2434/1bit serve -m …)      <- this is the serve that just failed
    

    Our serve takes kNpuDeviceLock before init and releases it on exit, so when its own init_elf is refused, npu_lock_holders_text() reads /proc/locks and correctly reports the only holder - itself. app/npu_device_lock.h has no getpid() anywhere, so nothing excludes self. The consequence is the one case that matters: a co-tenant that does not take our lock (FastFlowLM, in the measurement) is invisible to the lock protocol, while our own process looks like the culprit. The note then tells an operator to wait for, or stop, themselves.

    Suggested shape (small): in npu_lock_holders_text, drop ::getpid() from the list, and when the list becomes empty while npu_contexts_refused() is true, say the true thing - "this process holds the device lock and is still refused: another client is using the array outside our lock protocol (measured with a FastFlowLM serve) - stopping our own process will not help".

    I have not patched that one yet; it is a different file and a different claim, and the three pieces above are the ones this issue's measurement supports.

    What is still not proven

    • a genuinely dead array still getting the reload hint after the window (needs a dead array; I will not manufacture one on the box),
    • and that 45 s is the right length for a co-tenant. The policy already chose it; I did not tune it.
  7. bong-water-water-bong commented on Nov 14, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Verified on main's fix, not just on mine. I built 1bit from current main (3ac4a5e19, which contains 58cba39b6 - someone took this issue and did both halves properly, and better named: I8Ctx::init_error) and re-ran the same proof with the same live co-tenant:

    serve_up=no  elapsed_ms=45234
    attempt lines: 25   last retry delay: retrying in 2000 ms   recovered: 0   holder named: 1   reload hint: 1
    1bit forward-serve: … failed 26 attempt(s) in 44912 ms over a 45000 ms queue window
    1bit forward-serve: [npu] the device lock is held by pid 2230435 (…/b2434/1bit serve -m …)
    1bit forward-serve: the NPU refuses hardware contexts: another client may hold the array …; if nothing else is serving, reload
    device after: amdxdna_faults=0
    

    26 attempts over a 44,912 ms refusal window, holder line printed, reload hint printed, no device fault. The main claim of this issue is closed by measurement: a --device npu serve of a per-op container now waits out a co-tenant instead of dying in 0 s (my branch's commits are deleted; theirs subsume them).

    The third defect from my previous comment is still there, and this run is the independent confirmation of it on main's binary:

    [npu] the device lock is held by pid 2230435 (…/b2434/1bit serve -m …)      <- the serve that just failed
    

    That pid is the process running the check. Our serve takes kNpuDeviceLock before init, so when its own init_elf is refused, /proc/locks correctly reports the only holder - itself - and nothing in app/npu_device_lock.h calls getpid(). With a co-tenant that does not take our lock (FastFlowLM, the one in this measurement), the note therefore names us and tells the operator to wait for, or stop, the process that is already failing.

    I am opening a small PR for exactly that: exclude self from the holder text, and when the list empties while the context is refused, say the true thing - this process holds the lock and is still refused, so another client is using the array outside our lock protocol; stopping our own process will not help. Two measurements go with it: this one, and the same command after the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions