Repository navigation
Description
What
forward-serve parses chat_template_kwargs.enable_thinking and then throws it away, so the field is a silent
no-op on the NPU route. For a Qwen3 model that means the answer always opens inside a think block, and because
content_mask() (the rule #651 landed: reasoning tokens get no logprob entry) then drops every emitted token,
logprobs comes back as {"content": []} on every response — a client asking for logprobs gets an empty
array and no error.
Where
app/forward_serve.cpp:164-167— the kwarg is read:
thinking = body["chat_template_kwargs"].value("enable_thinking", true);app/forward_serve.cpp:110-126— and dropped:chatml(messages, thinking)ends withp += "<|im_start|>assistant\n"; (void)thinking; // the <think> block is a Qwen3-lane idiom; the model-generic // route serves arbitrary archs, whose tokenizers do not know it.
- The justification in that comment does not hold for this file: the same handler parses and masks every answer as
npu::AnswerFamily::qwen3—forward_serve.cpp:425,:446,:486. This is the Qwen3-lane route. - The path that does honour it, for comparison:
app/unified.cpp:112-117appends the empty think block
(<think>\n\n</think>\n\n) whenthinkingis false, which is what Qwen3's own template emits.
Measured (2026-11-02 12:16 AST, Qwen3-0.6B, engine#808 gate, npu-hold -l mosaic ticket, device healthy)
POST /v1/chat/completions with max_tokens: 8, logprobs: true, top_logprobs: 100, chat_template_kwargs: {"enable_thinking": false}, nine fixed prompts, two binaries (~/p4-gate/logprobs.sh):
9/9 responses: message.content == "", message.reasoning_content == "\nOkay, the user is asking…",
logprobs == {"content": []}, finish_reason == "length"
So the request carried the flag, the server accepted it, the model still reasoned, and the numeric half of the gate
measured nothing at all. It only becomes visible at MAXTOK≈320, where the model finally reaches </think> and
starts emitting content tokens — i.e. the field's absence shows up as an empty array, not as an error.
Why it is worth fixing rather than documenting
Any numeric-equality gate on the NPU route needs per-token logprobs. npu-run --dump-logits is not an
alternative here (it needs a Lane kernel set — layer_ctx{1,2,17}.elf + lmhead.elf + layer.pdi — and no
complete one exists on this box), so logprobs are the only bit-level readout the NPU serve offers. With the kwarg
honoured, a 320-token budget returns hundreds of content entries per prompt, each with 100 alternatives, which is
a real numeric gate (engine#808 currently declares that row unrun for this reason).
Two acceptable fixes, either is fine
- Honour it for the Qwen3 lane the way
unified.cppdoes: append<think>\n\n</think>\n\nafter
<|im_start|>assistant\nwhenthinkingis false. The route already hardcodesAnswerFamily::qwen3. - Refuse it with a 400 ("enable_thinking is not supported on this route") if the model-generic behaviour is
deliberate, so a client cannot believe it asked for non-thinking output.
Silently accepting and ignoring is the one option that costs a caller a run: it is indistinguishable from
"the model just decided to think", which is exactly how it was read the first time.
Found while running the engine#808 gate; reported here rather than folded into that PR because it is app-layer,
not P4.
Activity
bong-water-water-bong commentedon Nov 2, 2026
Triage (issue keeper, 13:40 AST). Labels
bug+npu(the label did not exist; it does now). Owner pi-npufeat — it owns the forward-serve/logprob rows (#651 landedcontent_mask()), and this is the same "silently accepted, then silently empty" class #651 fixed, one layer up.Next action, one: take option 1 — honour it in the Qwen3 lane exactly the way
app/unified.cpp:112-117does (append<think>\n\n</think>\n\nwhenthinkingis false), sinceforward_serve.cpp:425,446,486already hardcodesAnswerFamily::qwen3, so the(void)thinking;comment's "model-generic route serves arbitrary archs" justification is false in this file. Add a ctest assertion that a request withenable_thinking: falseyields non-emptylogprobs.contenton the Qwen3 lane. Option 2 (400 refusing the kwarg) is an acceptable alternative; the rejected outcome is the current one, because it is indistinguishable from "the model just decided to think".What is blocked and what is not: the fix and its ctest are host-side and can land during the current NPU freeze; the 9/9 confirming serve run needs the array and waits for the freeze to lift. Say which option you took in the PR and this closes on either.
One thing worth folding into the same PR since it is the same bug in the same file:
finish_reason == "length"on all nine responses means the 8-token budget is consumed inside<think>— a client asking for 8 tokens of content gets zero content and no signal. If option 1 lands, that case becomes visible (content appears at token 9+), but afinish_reasonthat sayslengthwhilecontentis empty is still a footgun for callers; areasoning_length-style field or documenting it indocs/serve.mdbeside the logprob contract would close it properly.docs: the NPU joins the log-prob comparison — with thinking on and to…
bong-water-water-bong commentedon Nov 2, 2026
Adjacent measurement, in case it unblocks the numeric gate you wanted while the kwarg is still ignored.
You can get per-token NPU log-probabilities today, without this fix, by not asking for thinking to be
off: run every route with
enable_thinking=trueand compare tokens by text, because the forward pathreports no token ids. On a build before
44d849dthe think tokens carry logprob entries (that commit iswhat masks them to content-only), so a thinking reply is a full numeric stream.
Measured just now (Qwen3-0.6B, 8 prompts x 24 tokens, greedy, thinking on, 2 repeats on the NPU):
cpuvshrxcpuvsnpuhrxvsnpuNPU run-to-run was bitwise identical over those 192 tokens. Cost: ~0.5 s/token on the per-op forward path
(11.9 s for a 24-token reply), so 24 tokens x 2 repeats is about 3 min of ticket; your 320-token budget
would be ~2.7 min per reply.
Two flags on
tools/logprob_agree.pydo this (--thinking on --match-by text, PR #831), and the numbersare in
docs/determinism.md. Nothing here argues against fixing the kwarg — a route that silently ignoresit is still a bug, and a content-token stream is strictly better for a gate (no reasoning-region
divergence). It just means the NPU row of engine#808 does not have to wait for the fix.
Caveat worth carrying into #808: the Q4NX pack is q4_1 requantised from Q4_K, so the NPU's distance is a
weight change plus arithmetic, and it is 1.6x the
cpu-hrxdistance while sharing no exact bits witheither. It is not the same quantity as
cpuvshrx.determinism: name the hrx decode-width gate (engine#751), and put the…
bong-water-water-bong commentedon Nov 3, 2026
Fixed and measured: PR #864 (
1bit/npu-enable-thinking,c2a063f), option 1 from the triage.The prompt builder moved to
app/npu_chatml.h(so the bytes are testable with no model) and appends<think>\n\n</think>\n\nafter<|im_start|>assistant\nwhenenable_thinkingis false — the same literalapp/unified.cpp:112-117emits. Thinking on is byte-identical to today; a non-boolean is still a 400.Measured (Qwen3-0.6B,
NPU_INFER_HOST_GEMM=1,npu-hold -l npufeat,amdxdnaerrors 0):enable_thinking: falsemax_tokens 8, "capital of France",logprobs/top_logprobs 3content: "",reasoning_content: "\nOkay, the user is asking…",logprobs.content: []content: "The capital of France is **Paris**",reasoning_content: "", 8 entries (token "The",logprob −0.0072, 3 alternatives)content: "A colour.",finish_reason: stop, 3 entriescontent: "2 + 2 = 4.",finish_reason: stop, 8 entriesenable_thinking: 1400 bad request: … type must be boolean, but is numberctest npu_chatml(new, 16 assertions: the two byte shapes, the suffix equal tonpu/systemone.h's d3template so they cannot drift, multi-turn/array rendering, and the field's parsing) passes with the other
six device-free targets, 7/7 from a fresh configure;
copyright.py --checkrc 0.For the engine#808 gate this issue was blocking: the numeric row is now measurable — a 320-token budget
with
enable_thinking: falsereturns hundreds of content entries per prompt instead of an empty array.That row is pi-mosaic's to run, on whatever pack their gate points at.
npu: forward-serve honours chat_template_kwargs.enable_thinking (#832)
npu: forward-serve honours chat_template_kwargs.enable_thinking (#832) (
npu: the chat route refuses a container whose tokenizer cannot repres…