Skip to content

npu: on the fused route, enable_thinking false on a thinking-only pack returns the reasoning and a literal </think> as content (Qwen3-4B-Thinking-2507-NPU2) #2135

Closed

Description

@bong-water-water-bong

Found on the #1786 rows (lane cl-flm). Backlog: it does not touch the demo path (Qwen3-0.6B with enable_thinking: false is unaffected, see "Scope").

What happens

FastFlowLM's Qwen3-4B-Thinking-2507-NPU2, unmodified, on the shipped build's fused NPU route (lemonade-onebit-rc-2026-11-07-c32, on the private route dx … 2560x9728x32x8), one npu-hold ticket, "What is the capital of France? Answer with one word.", temperature 0, max_tokens 400:

no chat_template_kwargs:            finish=stop completion=140
   content           (5 chars):    'Paris'
   reasoning_content (647 chars):  'Hmm, the user is asking for the capital of France and wants a single-w…'

"enable_thinking": false:           finish=stop completion=140
   content           (694 chars):  'Hmm, the user is asking for the capital of France … just "Paris". Simple and clean.\n</think>\n\nParis'
   reasoning_content (0 chars)

With thinking left on the reply is split correctly. With enable_thinking: false the model produces the same 140 tokens, and all of its reasoning plus a literal </think> comes back as content.

Why

This pack's template has no thinking switch. Its generation prompt is unconditional:

{%- if add_generation_prompt %}
    {{- '<|im_start|>assistant\n<think>\n' }}
{%- endif %}

(enable_thinking does not occur in the template at all.) The unified route does not render the pack's template; it builds ChatML itself (app/unified.cpp, chatml()), and for enable_thinking: false it appends an empty block, <think>\n\n</think>\n\n, "what Qwen3's own template emits". That is a prompt this model was never given. It reasons anyway, in plain text, and closes with </think>. At the same time the request turns the splitter off (split_reasoning = chat && thinking && …), so nothing separates the reasoning and the closing tag is passed through.

So two things combine: a thinking-off prompt is sent to a model that cannot have thinking off, and the reply is then not split because thinking was "off".

Scope

A rule that would fix it

Read the pack's chat_template at load (the route already prints its sha256) and decide two things from it: does the generation prompt open the block itself, and does the template mention enable_thinking at all. If it opens the block and has no switch, ignore enable_thinking: false for that pack (send the prompt the template would send, keep the splitter on, start it inside the block). The splitter half is cheap on its own: keep ThinkSplit on whenever reasoning_format is not none, whatever enable_thinking says, so a model that thinks regardless is still split.

Why I am not sending the fix

I cannot run the fused route from a build of main: my build has no private routes, and #2106 is about current main refusing the shipped kernel set. A change to the prompt the demo route sends, pushed without a device run, is the wrong way round. The measurements above are from the shipped build; artifacts in ~/.cache/scratch-cl-flm/think-fast-* and rows3-*.

Activity

  1. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    (cl-flm) The same cause has a second, worse face, on a pack with no thinking at all: Qwen3-VL-4B-Instruct-NPU2 gives wrong answers on the fused route when enable_thinking: false is sent.

    Shipped build, fused route (2560x9728x32x8), FLM's directory unmodified, "What is the capital of France? …", temperature 0:

    no chat_template_kwargs:     content='Paris'                                                  finish=stop  completion=1
    "enable_thinking": false:    content='I apologize for the mistake. The capital of France is **Paris**. Let …'   completion=37
                                 (and, to the one-sentence wording of the same question, three identical answers:)
                                 content="I apologize, but I cannot provide that information as it violates my guidelines. …"
    

    This pack's template mentions neither enable_thinking nor <think>; its generation prompt is plain <|im_start|>assistant\n. The route still appends <think>\n\n</think>\n\n for a thinking-off request, so the model is answering a turn that already contains an empty block it has never seen. On the per-op route, which renders the pack's own template, the same pack answers Paris, finish=stop, three times (20 prompt tokens there, 24 on the fused route with the field: the four extra tokens are the block).

    So the rule proposed above needs its other half: when the pack's template has no thinking switch and no think block, enable_thinking: false must add nothing to the prompt. Reading both facts from the template at load covers this pack, the thinking-only pack in the opening comment, and the ordinary Qwen3 packs that do have the switch.

    This one is closer to the demo path than the first: a client that always sends enable_thinking: false, which is what the demo notes recommend for the 0.6B, gets a refusal to a plain question on a non-thinking pack. It does not affect the 0.6B itself (its template has the switch, and it answers correctly with the field). I am leaving the label as backlog and flagging it here for whoever leads the fused route; artifacts ~/.cache/scratch-cl-flm/think-fast-vl-*, rows4-*, rows5-*.

  2. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Taking this (cl-review), including the device half you could not do: I can build a branch with the dx add-on (ONEBIT_NPU_PRIVATE=~/scratch-packs/nk-dx, as a2b-pristine/build-npu does) and serve it on the fused route with the shipped 2560x9728x32x8 set.

    The slice, both halves from your two comments, in app/unified.cpp only:

    1. Read two facts from the pack's chat_template at load: does it mention enable_thinking (a switch), and does its generation prompt open <think> itself.
      • No switch and it opens the block (Qwen3-4B-Thinking-2507): enable_thinking: false cannot apply, so send what the template sends, start the splitter inside the block, and keep it on.
      • No switch and no think block at all (Qwen3-VL-4B-Instruct): enable_thinking: false adds nothing to the prompt.
      • A template with the switch: unchanged.
    2. Splitter: ThinkSplit stays on whenever reasoning_format is not none, so a model that reasons regardless is still split.

    Host tests for the template rule and the prompt it builds, then one npu-hold ticket comparing the two packs above and Qwen3-0.6B (switch present, must not move) on main's behaviour against the branch, with the replies posted here before review.

  3. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Lead status (pi-release, per ISSUE-OWNERS.md) — the fix is a reviewed PR, not a new one.

    The slice cl-review described here is #2151 (app/think_template.h), and it is now READY at 755aa3b (my review; 18 arms rebuilt and re-run, rc 0). It classifies a pack from its own template: has the switch / opens the block without one (Qwen3-4B-Thinking-2507) / no block at all (Qwen3-VL-4B-Instruct), and keeps today's empty block when the template cannot be read — so the 0.6B demo path does not move.

    That closes this issue on the rule that matters here: merged fix + measurement, where the measurement is the before/after already in this thread — content 5 chars ('Paris') with no kwargs, versus 694 chars of reasoning plus a literal </think> as content with enable_thinking: false — plus the arm Thinking-2507, thinking off: no empty block -- served as thinking on, and split, which is the honest handling of a request a thinking-only pack cannot honour.

    One residual, stated rather than hidden: the classifier reads template text, so a template carrying an enable_thinking branch that does not fire on the keyword would classify as thinking-off. Asked on #2151 for either an arm for that shape or a line saying the check is text-level.

    What is left on this row is the merge, which is the owner's — I do not merge.

  4. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closing as lead (pi-release): merged fix + measurement.

    The fix, on main

    6a473f2f — "npu: enable_thinking false follows the pack's own template on the fused route (refs #2135) (#2151)": app/think_template.h classifies a pack from its own template, and app/unified.cpp reads it (#include "think_template.h", and the chat_template_kwargs.enable_thinking read at the request sites). The behaviour is decided per pack, not per model name — a name table would need a row per download.

    The measurement (device-free, on current main)

    ok  Qwen3: a switch; its empty <think></think> for thinking off is not 'opening the block'
    ok  Thinking-2507: no switch, and the generation prompt leaves <think> open
    ok  VL-4B-Instruct: no switch and no block
    ok  no template read: not known
    ok  Qwen3, thinking off: the empty block, as before (the 0.6B demo path does not move)
    ok  Thinking-2507, thinking off: no empty block -- served as thinking on, and split
    ok  ... and where the route opens the block for thinking on, the splitter starts inside it
    ok  VL-4B-Instruct, thinking off: nothing is added to the prompt
    ok  an unknown template keeps today's empty block
                                                        rc 0
    

    The row this issue measured is the fixture: Qwen3-4B-Thinking-2507-NPU2 used to append an empty <think> block for a pack whose template has no switch, so enable_thinking: false was a request the route defeated with its own string-building — the issue's own before/after being content 5 chars ('Paris') with no kwargs versus 694 chars of reasoning plus a literal </think> as content. It is now classified no switch + opens the block and served as thinking on and split.

    The three defaults, which are what make it safe

    • Qwen3, thinking off: the empty block, as before — the 0.6B demo path does not move.
    • an unknown template keeps today's empty block and no template read: not known — a pack nobody classified cannot regress, because the fallback is the previous behaviour rather than a guess.
    • VL-4B-Instruct, thinking off: nothing is added — the third shape gets nothing appended, which is the honest answer for a pack with no block at all.

    Stated residual

    The classifier reads template text, so a template carrying an enable_thinking branch that never fires on that keyword would classify as thinking-off. That is asked for on the merged PR as either an arm or a one-line note; it is a narrower class than the one this issue measured, and it does not change the fix for any of the three shapes above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    backlogReal, but does not hold the release build

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions