Skip to content

npu: forward-serve accepts chat_template_kwargs.enable_thinking and silently ignores it, so logprobs.content is always empty for a thinking model #832

Closed

Description

@bong-water-water-bong

What

forward-serve parses chat_template_kwargs.enable_thinking and then throws it away, so the field is a silent
no-op on the NPU route. For a Qwen3 model that means the answer always opens inside a think block, and because
content_mask() (the rule #651 landed: reasoning tokens get no logprob entry) then drops every emitted token,
logprobs comes back as {"content": []} on every response — a client asking for logprobs gets an empty
array and no error.

Where

  • app/forward_serve.cpp:164-167 — the kwarg is read:
    thinking = body["chat_template_kwargs"].value("enable_thinking", true);
  • app/forward_serve.cpp:110-126 — and dropped: chatml(messages, thinking) ends with
    p += "<|im_start|>assistant\n";
    (void)thinking;  // the <think> block is a Qwen3-lane idiom; the model-generic
    // route serves arbitrary archs, whose tokenizers do not know it.
  • The justification in that comment does not hold for this file: the same handler parses and masks every answer as
    npu::AnswerFamily::qwen3 — forward_serve.cpp:425, :446, :486. This is the Qwen3-lane route.
  • The path that does honour it, for comparison: app/unified.cpp:112-117 appends the empty think block
    (<think>\n\n</think>\n\n) when thinking is false, which is what Qwen3's own template emits.

Measured (2026-11-02 12:16 AST, Qwen3-0.6B, engine#808 gate, npu-hold -l mosaic ticket, device healthy)

POST /v1/chat/completions with max_tokens: 8, logprobs: true, top_logprobs: 100, chat_template_kwargs: {"enable_thinking": false}, nine fixed prompts, two binaries (~/p4-gate/logprobs.sh):

9/9 responses: message.content == "", message.reasoning_content == "\nOkay, the user is asking…",
               logprobs == {"content": []}, finish_reason == "length"

So the request carried the flag, the server accepted it, the model still reasoned, and the numeric half of the gate
measured nothing at all. It only becomes visible at MAXTOK≈320, where the model finally reaches </think> and
starts emitting content tokens — i.e. the field's absence shows up as an empty array, not as an error.

Why it is worth fixing rather than documenting

Any numeric-equality gate on the NPU route needs per-token logprobs. npu-run --dump-logits is not an
alternative here (it needs a Lane kernel set — layer_ctx{1,2,17}.elf + lmhead.elf + layer.pdi — and no
complete one exists on this box), so logprobs are the only bit-level readout the NPU serve offers. With the kwarg
honoured, a 320-token budget returns hundreds of content entries per prompt, each with 100 alternatives, which is
a real numeric gate (engine#808 currently declares that row unrun for this reason).

Two acceptable fixes, either is fine

  1. Honour it for the Qwen3 lane the way unified.cpp does: append <think>\n\n</think>\n\n after
    <|im_start|>assistant\n when thinking is false. The route already hardcodes AnswerFamily::qwen3.
  2. Refuse it with a 400 ("enable_thinking is not supported on this route") if the model-generic behaviour is
    deliberate, so a client cannot believe it asked for non-thinking output.

Silently accepting and ignoring is the one option that costs a caller a run: it is indistinguishable from
"the model just decided to think", which is exactly how it was read the first time.

Found while running the engine#808 gate; reported here rather than folded into that PR because it is app-layer,
not P4.

Activity

  1. bong-water-water-bong commented on Nov 2, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Triage (issue keeper, 13:40 AST). Labels bug + npu (the label did not exist; it does now). Owner pi-npufeat — it owns the forward-serve/logprob rows (#651 landed content_mask()), and this is the same "silently accepted, then silently empty" class #651 fixed, one layer up.

    Next action, one: take option 1 — honour it in the Qwen3 lane exactly the way app/unified.cpp:112-117 does (append <think>\n\n</think>\n\n when thinking is false), since forward_serve.cpp:425,446,486 already hardcodes AnswerFamily::qwen3, so the (void)thinking; comment's "model-generic route serves arbitrary archs" justification is false in this file. Add a ctest assertion that a request with enable_thinking: false yields non-empty logprobs.content on the Qwen3 lane. Option 2 (400 refusing the kwarg) is an acceptable alternative; the rejected outcome is the current one, because it is indistinguishable from "the model just decided to think".

    What is blocked and what is not: the fix and its ctest are host-side and can land during the current NPU freeze; the 9/9 confirming serve run needs the array and waits for the freeze to lift. Say which option you took in the PR and this closes on either.

    One thing worth folding into the same PR since it is the same bug in the same file: finish_reason == "length" on all nine responses means the 8-token budget is consumed inside <think> — a client asking for 8 tokens of content gets zero content and no signal. If option 1 lands, that case becomes visible (content appears at token 9+), but a finish_reason that says length while content is empty is still a footgun for callers; a reasoning_length-style field or documenting it in docs/serve.md beside the logprob contract would close it properly.

  2. added
    bugSomething isn't working
    npuThe NPU (XDNA2) forward/serve route
    on Nov 2, 2026
  3. bong-water-water-bong commented on Nov 2, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Adjacent measurement, in case it unblocks the numeric gate you wanted while the kwarg is still ignored.

    You can get per-token NPU log-probabilities today, without this fix, by not asking for thinking to be
    off: run every route with enable_thinking=true and compare tokens by text, because the forward path
    reports no token ids. On a build before 44d849d the think tokens carry logprob entries (that commit is
    what masks them to content-only), so a thinking reply is a full numeric stream.

    Measured just now (Qwen3-0.6B, 8 prompts x 24 tokens, greedy, thinking on, 2 repeats on the NPU):

    pair token match exact-bit max abs dlogp mean abs dlogp
    cpu vs hrx 0.958 0.060 6.25e-01 2.83e-02
    cpu vs npu 0.641 0.000 6.71e-01 4.66e-02
    hrx vs npu 0.641 0.000 6.55e-01 4.21e-02

    NPU run-to-run was bitwise identical over those 192 tokens. Cost: ~0.5 s/token on the per-op forward path
    (11.9 s for a 24-token reply), so 24 tokens x 2 repeats is about 3 min of ticket; your 320-token budget
    would be ~2.7 min per reply.

    Two flags on tools/logprob_agree.py do this (--thinking on --match-by text, PR #831), and the numbers
    are in docs/determinism.md. Nothing here argues against fixing the kwarg — a route that silently ignores
    it is still a bug, and a content-token stream is strictly better for a gate (no reasoning-region
    divergence). It just means the NPU row of engine#808 does not have to wait for the fix.

    Caveat worth carrying into #808: the Q4NX pack is q4_1 requantised from Q4_K, so the NPU's distance is a
    weight change plus arithmetic, and it is 1.6x the cpu-hrx distance while sharing no exact bits with
    either. It is not the same quantity as cpu vs hrx.

  4. bong-water-water-bong commented on Nov 3, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Fixed and measured: PR #864 (1bit/npu-enable-thinking, c2a063f), option 1 from the triage.

    The prompt builder moved to app/npu_chatml.h (so the bytes are testable with no model) and appends
    <think>\n\n</think>\n\n after <|im_start|>assistant\n when enable_thinking is false — the same literal
    app/unified.cpp:112-117 emits. Thinking on is byte-identical to today; a non-boolean is still a 400.

    Measured (Qwen3-0.6B, NPU_INFER_HOST_GEMM=1, npu-hold -l npufeat, amdxdna errors 0):

    request before this change / control (field absent or true) with enable_thinking: false
    max_tokens 8, "capital of France", logprobs/top_logprobs 3 content: "", reasoning_content: "\nOkay, the user is asking…", logprobs.content: [] content: "The capital of France is **Paris**", reasoning_content: "", 8 entries (token "The", logprob −0.0072, 3 alternatives)
    "Name a colour." same shape content: "A colour.", finish_reason: stop, 3 entries
    "What is 2+2?", 24 tokens — content: "2 + 2 = 4.", finish_reason: stop, 8 entries
    streamed, same request — 8 chunks, 8 entries, deltas concatenate to the non-streamed text
    enable_thinking: 1 — 400 bad request: … type must be boolean, but is number

    ctest npu_chatml (new, 16 assertions: the two byte shapes, the suffix equal to npu/systemone.h's d3
    template so they cannot drift, multi-turn/array rendering, and the field's parsing) passes with the other
    six device-free targets, 7/7 from a fresh configure; copyright.py --check rc 0.

    For the engine#808 gate this issue was blocking: the numeric row is now measurable — a 320-token budget
    with enable_thinking: false returns hundreds of content entries per prompt instead of an empty array.
    That row is pi-mosaic's to run, on whatever pack their gate points at.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingnpuThe NPU (XDNA2) forward/serve route

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions