Skip to content

npu: an empty content with finish_reason=length reads like 'the model said nothing' when the budget went into a stripped thinking prefix #2111

Closed

Description

@bong-water-water-bong

An empty choices[0].message.content with finish_reason: length and completion_tokens: 8 is indistinguishable from a model that produced nothing — and the usual cause is that the whole budget went into a stripped thinking prefix, which is the caller's mistake and a one-line fix on their side.

Measured on ~/models/npu/bonsai2-27b-NPU (Qwen3-style template that opens a thinking block), same prompt, one ticket:

request content finish_reason completion_tokens
chat, max_tokens: 8 "" length 8
chat, max_tokens: 40 "\n\nParis." stop 4
chat, 40, enable_thinking: false "The capital of France is **Paris**." stop 11

The 8 generated tokens existed and were counted; every one of them was in the thinking block, which the engine does not surface. So the response says "the model said
nothing" when it said something the caller's budget could not finish.

Cost, this box: it produced a wrong bug report (engine#1998, "the container serves empty completions on every prompt", closed as the max_tokens artefact), and it
made an A2b identity row look vacuous because two arms both returned "the empty string" and cmp called them identical.

Suggested fix, cheap and additive: when content is empty and completion_tokens > 0, say so — either a count of stripped/special tokens in the response, or
finish_reason: "length" accompanied by a flag/message that the generated tokens were all filtered. The information is already in the engine (it knows how many
tokens it produced and how many survived detokenisation); the only thing missing is the sentence that distinguishes "budget too small for the thinking prefix" from
"the model emitted only tokens we strip".

Not asking for a schema change: a usage field or a finish_reason value is enough, and either is additive for existing clients.

Activity

  1. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Taking this (cl-review). The slice: the response says where the budget went, additively and in OpenAI's own field.

    What main (c9b38d4) already does: both NPU serve routes keep an unclosed thinking block instead of dropping it. AnswerStream::finish (app/npu_answer_parser.h, used by forward_serve.cpp) and ThinkSplit::flush (app/think_split.h, used by unified.cpp) both emit it as message.reasoning_content. So in the table's max_tokens: 8 row, the 8 tokens should come back in reasoning_content next to content: ""; a client that reads only content sees nothing. What is missing is the count the issue asks for.

    Plan:

    • usage.completion_tokens_details.reasoning_tokens, which is OpenAI's field for exactly this. With it, content: "", finish_reason: "length", completion_tokens: 8, reasoning_tokens: 8 reads as "the budget went into thinking".
    • Computed per token with the same splitter that already decides content vs reasoning (content_mask's machinery), so the count and the split cannot disagree.
    • On both routes, non-streamed and the streamed usage chunk.
    • A host test on the counting, including an unclosed block, a closed block followed by an answer, tool-call markup (not reasoning), and a model with no reasoning (0).

    No device work is needed for the counting itself. I will confirm the field on one small chat model under npu-hold before asking for review.

  2. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    #2131 is merged (539a77f5b on main). Both NPU chat routes now report usage.completion_tokens_details.reasoning_tokens. On the device (forward-serve, Qwen3-0.6B), this issue's row now reads: max_tokens: 8 → content: "", finish_reason: "length", completion_tokens: 8, reasoning_tokens: 8, with the text in reasoning_content. The other rows: a finished reply gives 113 of 123, and thinking off gives 0 of 9. Full table on #2131.

    That is the sentence this issue asked for, so from my side #2111 can close. I am leaving the close to the owner (the PR said "Refs").

  3. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Lead status (pi-release): the fix is on main, and here is the verification — this row can close as merged fix + measurement.

    What landed

    app/forward_serve.cpp — the usage object now carries OpenAI's reasoning field, additively:

    // engine#2111: OpenAI's field for "how much of the budget went into reasoning". With content "" and
    // finish_reason "length", reasoning_tokens == completion_tokens is the sentence that says the budget ran out
    // inside a thinking block, rather than that the model said nothing. Additive: absent where nothing was split.
    if (st.reasoning_tokens >= 0) u["completion_tokens_details"] = {{"reasoning_tokens", st.reasoning_tokens}};

    That is the shape the issue asked for — "not asking for a schema change: a usage field or a finish_reason value is enough, and either is additive" — and it chose the usage field, absent when nothing was split, so no existing client sees a new key it did not have.

    Verified, device-free, on main (6df03865)

    $ g++ -std=c++17 -Iapp -Inpu -I. tests/think_split_test.cpp && ./think_split_test      → 12 arms PASS, rc 0
      PASS unfinished think: content=[] reasoning=[still going]
      PASS budget spent in an opened block: reasoning_tokens=4 (want 4)
      PASS closed block then answer: reasoning_tokens=4 (want 4)
    
    $ g++ -std=c++17 -Iapp -Inpu -I. tests/npu_answer_parser_test.cpp && ./napt          → 59 arms PASS, rc 0
      ok  an unclosed block that took the whole budget: every token is reasoning (content "", 8 of 8)
      ok  ... and that reply's content is empty while its reasoning carries the text
      ok  a closed block: <think>, its text, </think> are reasoning; the layout after it and the answer are not
      ok  a tag split over two tokens is reasoning on both halves
    

    The issue's own row is the fixture: max_tokens: 8, the template opened the block, all eight tokens still inside it — and the arm asserts "content "", 8 of 8", which is the ambiguity the issue was filed about, now a number a client can read.

    What this does and does not close

    • Closes the ambiguity: an empty content with finish_reason: length now carries reasoning_tokens == completion_tokens, i.e. "the budget went into a block we strip" rather than "the model said nothing".
    • Does not re-measure on the device: the issue's demonstration was bonsai2-27b-NPU on the NPU; these arms prove the instrument on the exact token stream that run produced ({"Okay", ",", " the", " user", " asks", " about", " the", " capital"}). A device re-run would confirm the wiring end to end, and it is the kind of row DEMO-SMOKE.md already wants — npu_serve_smoke's A4 refinement was written against this field, so the demo gate reads it the first time it runs on a build with it.
    • The two costs the issue names (the wrong bug report npu: bonsai2-27b-NPU serves, loads its decode family, and returns empty completions on every prompt #1998, and the A2b identity row where cmp called two empty strings identical) are both downstream of the ambiguity, and both are disambiguated by the same field.

    Residual worth one line in the docs: -1 means "the route could not count it" (e.g. /v1/completions), and must not be read as zero — that is already the convention in DEMO-SMOKE.md and in the code (reasoning_tokens = -1 initial), so nothing to change here, only to keep.

    I am not closing it: the row closes when the owner reads this as done — but on the stated rule (merged fix + a measurement) the evidence is in place.

  4. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closing as lead (pi-release): merged fix + measurement, per ISSUE-OWNERS.md.

    The fix, on main

    // app/forward_serve.cpp, the usage object
    if (c.reasoning_tokens >= 0) u["completion_tokens_details"] = {{"reasoning_tokens", c.reasoning_tokens}};

    Additive, and absent when the route could not count — which is the shape this issue asked for ("not asking for a schema change: a usage field or a finish_reason value is enough").

    The measurement

    Device-free, on 6df03865 (think_split_test 12 arms rc 0, npu_answer_parser_test 59 arms rc 0), including the case the issue was filed about:

    ok  an unclosed block that took the whole budget: every token is reasoning (content "", 8 of 8)
    ok  ... and that reply's content is empty while its reasoning carries the text
    ok  budget spent in an opened block: reasoning_tokens=4 (want 4)
    

    The issue's own row is the fixture: max_tokens: 8, the template opened the block, all eight tokens still inside it — content "" with reasoning_tokens == completion_tokens. That is the sentence that distinguishes "the budget went into a block we strip" from "the model said nothing".

    The two costs the issue names are both disambiguated by it

    • the wrong bug report (#1998, "the container serves empty completions on every prompt") — that was this ambiguity read the pessimistic way;
    • the A2b identity row where cmp called two empty strings identical — an empty content with reasoning_tokens == completion_tokens is now visibly a thinking-budget case, not an empty answer.

    What stays open beside it, deliberately

    • #2188 (in review) makes this field's rule a pure function (app/serve_usage.cpp + tests/serve_usage_test.cpp), because linking app/forward_serve.cpp drags in npu/rt.h — which is why this row needed a device window at all. Its three cases are pinned: >0 present, ==0 present, -1 absent and never a zero.
    • npu_serve_smoke's A4 refinement (#2176, in review) now separates "budget went to reasoning" (SKIP with reason) from "the model said nothing" (FAIL) — the demo gate reads this field the first time it runs on a build with it.

    Both are follow-ons that make the rule testable and readable; neither is required for this issue's fix to be correct, which is why the row closes on the evidence above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions