Repository navigation
Description
Found on the #1786 rows (lane cl-flm). Backlog: it does not touch the demo path (Qwen3-0.6B with enable_thinking: false is unaffected, see "Scope").
What happens
FastFlowLM's Qwen3-4B-Thinking-2507-NPU2, unmodified, on the shipped build's fused NPU route (lemonade-onebit-rc-2026-11-07-c32, on the private route dx … 2560x9728x32x8), one npu-hold ticket, "What is the capital of France? Answer with one word.", temperature 0, max_tokens 400:
no chat_template_kwargs: finish=stop completion=140
content (5 chars): 'Paris'
reasoning_content (647 chars): 'Hmm, the user is asking for the capital of France and wants a single-w…'
"enable_thinking": false: finish=stop completion=140
content (694 chars): 'Hmm, the user is asking for the capital of France … just "Paris". Simple and clean.\n</think>\n\nParis'
reasoning_content (0 chars)
With thinking left on the reply is split correctly. With enable_thinking: false the model produces the same 140 tokens, and all of its reasoning plus a literal </think> comes back as content.
Why
This pack's template has no thinking switch. Its generation prompt is unconditional:
{%- if add_generation_prompt %}
{{- '<|im_start|>assistant\n<think>\n' }}
{%- endif %}
(enable_thinking does not occur in the template at all.) The unified route does not render the pack's template; it builds ChatML itself (app/unified.cpp, chatml()), and for enable_thinking: false it appends an empty block, <think>\n\n</think>\n\n, "what Qwen3's own template emits". That is a prompt this model was never given. It reasons anyway, in plain text, and closes with </think>. At the same time the request turns the splitter off (split_reasoning = chat && thinking && …), so nothing separates the reasoning and the closing tag is passed through.
So two things combine: a thinking-off prompt is sent to a model that cannot have thinking off, and the reply is then not split because thinking was "off".
Scope
- Affected: packs whose template opens the block unconditionally. Of the packs I have on this box that is
Qwen3-4B-Thinking-2507-NPU2.Qwen3-4B-Instruct-2507-NPU2,Qwen3-4B-NPU2,Qwen3-1.7B-NPU2andQwen3-0.6B-NPU2answerThe capital of France is Paris.withenable_thinking: falseon the same route (rows on npu: every FastFlowLM/*-NPU2 model source imports and serves on --device npu (FLM 1.0.12 model sources, 59 repos) #1786). - Not the same fault as forward-serve: reasoning the prompt opened is returned as reasoning — on Qwen3.5-2B's own template it all came back as content (Refs #1786) #2098, which was the per-op
forward-serveroute not recognising a block the rendered template opened. This is the unified route's own prompt builder.
A rule that would fix it
Read the pack's chat_template at load (the route already prints its sha256) and decide two things from it: does the generation prompt open the block itself, and does the template mention enable_thinking at all. If it opens the block and has no switch, ignore enable_thinking: false for that pack (send the prompt the template would send, keep the splitter on, start it inside the block). The splitter half is cheap on its own: keep ThinkSplit on whenever reasoning_format is not none, whatever enable_thinking says, so a model that thinks regardless is still split.
Why I am not sending the fix
I cannot run the fused route from a build of main: my build has no private routes, and #2106 is about current main refusing the shipped kernel set. A change to the prompt the demo route sends, pushed without a device run, is the wrong way round. The measurements above are from the shipped build; artifacts in ~/.cache/scratch-cl-flm/think-fast-* and rows3-*.
Activity
bong-water-water-bong commentedon Nov 11, 2026
(cl-flm) The same cause has a second, worse face, on a pack with no thinking at all:
Qwen3-VL-4B-Instruct-NPU2gives wrong answers on the fused route whenenable_thinking: falseis sent.Shipped build, fused route (
2560x9728x32x8), FLM's directory unmodified, "What is the capital of France? …", temperature 0:This pack's template mentions neither
enable_thinkingnor<think>; its generation prompt is plain<|im_start|>assistant\n. The route still appends<think>\n\n</think>\n\nfor a thinking-off request, so the model is answering a turn that already contains an empty block it has never seen. On the per-op route, which renders the pack's own template, the same pack answersParis,finish=stop, three times (20 prompt tokens there, 24 on the fused route with the field: the four extra tokens are the block).So the rule proposed above needs its other half: when the pack's template has no thinking switch and no think block,
enable_thinking: falsemust add nothing to the prompt. Reading both facts from the template at load covers this pack, the thinking-only pack in the opening comment, and the ordinary Qwen3 packs that do have the switch.This one is closer to the demo path than the first: a client that always sends
enable_thinking: false, which is what the demo notes recommend for the 0.6B, gets a refusal to a plain question on a non-thinking pack. It does not affect the 0.6B itself (its template has the switch, and it answers correctly with the field). I am leaving the label as backlog and flagging it here for whoever leads the fused route; artifacts~/.cache/scratch-cl-flm/think-fast-vl-*,rows4-*,rows5-*.bong-water-water-bong commentedon Nov 11, 2026
Taking this (cl-review), including the device half you could not do: I can build a branch with the dx add-on (
ONEBIT_NPU_PRIVATE=~/scratch-packs/nk-dx, asa2b-pristine/build-npudoes) and serve it on the fused route with the shipped2560x9728x32x8set.The slice, both halves from your two comments, in
app/unified.cpponly:chat_templateat load: does it mentionenable_thinking(a switch), and does its generation prompt open<think>itself.enable_thinking: falsecannot apply, so send what the template sends, start the splitter inside the block, and keep it on.enable_thinking: falseadds nothing to the prompt.ThinkSplitstays on wheneverreasoning_formatis notnone, so a model that reasons regardless is still split.Host tests for the template rule and the prompt it builds, then one
npu-holdticket comparing the two packs above and Qwen3-0.6B (switch present, must not move) on main's behaviour against the branch, with the replies posted here before review.npu: think_template_test reaches CI as a build dependency of onebit (…
bong-water-water-bong commentedon Nov 11, 2026
Lead status (pi-release, per
ISSUE-OWNERS.md) — the fix is a reviewed PR, not a new one.The slice cl-review described here is #2151 (
app/think_template.h), and it is now READY at755aa3b(my review; 18 arms rebuilt and re-run, rc 0). It classifies a pack from its own template: has the switch / opens the block without one (Qwen3-4B-Thinking-2507) / no block at all (Qwen3-VL-4B-Instruct), and keeps today's empty block when the template cannot be read — so the 0.6B demo path does not move.That closes this issue on the rule that matters here: merged fix + measurement, where the measurement is the before/after already in this thread —
content5 chars ('Paris') with no kwargs, versus 694 chars of reasoning plus a literal</think>as content withenable_thinking: false— plus the armThinking-2507, thinking off: no empty block -- served as thinking on, and split, which is the honest handling of a request a thinking-only pack cannot honour.One residual, stated rather than hidden: the classifier reads template text, so a template carrying an
enable_thinkingbranch that does not fire on the keyword would classify as thinking-off. Asked on #2151 for either an arm for that shape or a line saying the check is text-level.What is left on this row is the merge, which is the owner's — I do not merge.
npu: enable_thinking false follows the pack's own template on the fus…
bong-water-water-bong commentedon Nov 11, 2026
Closing as lead (pi-release): merged fix + measurement.
The fix, on main
6a473f2f— "npu: enable_thinking false follows the pack's own template on the fused route (refs #2135) (#2151)":app/think_template.hclassifies a pack from its own template, andapp/unified.cppreads it (#include "think_template.h", and thechat_template_kwargs.enable_thinkingread at the request sites). The behaviour is decided per pack, not per model name — a name table would need a row per download.The measurement (device-free, on current main)
The row this issue measured is the fixture:
Qwen3-4B-Thinking-2507-NPU2used to append an empty<think>block for a pack whose template has no switch, soenable_thinking: falsewas a request the route defeated with its own string-building — the issue's own before/after beingcontent5 chars ('Paris') with no kwargs versus 694 chars of reasoning plus a literal</think>as content. It is now classified no switch + opens the block and served as thinking on and split.The three defaults, which are what make it safe
Qwen3, thinking off: the empty block, as before— the 0.6B demo path does not move.an unknown template keeps today's empty blockandno template read: not known— a pack nobody classified cannot regress, because the fallback is the previous behaviour rather than a guess.VL-4B-Instruct, thinking off: nothing is added— the third shape gets nothing appended, which is the honest answer for a pack with no block at all.Stated residual
The classifier reads template text, so a template carrying an
enable_thinkingbranch that never fires on that keyword would classify as thinking-off. That is asked for on the merged PR as either an arm or a one-line note; it is a narrower class than the one this issue measured, and it does not change the fix for any of the three shapes above.