You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
npu: an empty content with finish_reason=length reads like 'the model said nothing' when the budget went into a stripped thinking prefix #2111
An empty choices[0].message.content with finish_reason: length and completion_tokens: 8 is indistinguishable from a model that produced nothing — and the usual cause is that the whole budget went into a stripped thinking prefix, which is the caller's mistake and a one-line fix on their side.
Measured on ~/models/npu/bonsai2-27b-NPU (Qwen3-style template that opens a thinking block), same prompt, one ticket:
request
content
finish_reason
completion_tokens
chat, max_tokens: 8
""
length
8
chat, max_tokens: 40
"\n\nParis."
stop
4
chat, 40, enable_thinking: false
"The capital of France is **Paris**."
stop
11
The 8 generated tokens existed and were counted; every one of them was in the thinking block, which the engine does not surface. So the response says "the model said
nothing" when it said something the caller's budget could not finish.
Cost, this box: it produced a wrong bug report (engine#1998, "the container serves empty completions on every prompt", closed as the max_tokens artefact), and it
made an A2b identity row look vacuous because two arms both returned "the empty string" and cmp called them identical.
Suggested fix, cheap and additive: when content is empty and completion_tokens > 0, say so — either a count of stripped/special tokens in the response, or finish_reason: "length" accompanied by a flag/message that the generated tokens were all filtered. The information is already in the engine (it knows how many
tokens it produced and how many survived detokenisation); the only thing missing is the sentence that distinguishes "budget too small for the thinking prefix" from
"the model emitted only tokens we strip".
Not asking for a schema change: a usage field or a finish_reason value is enough, and either is additive for existing clients.
Taking this (cl-review). The slice: the response says where the budget went, additively and in OpenAI's own field.
What main (c9b38d4) already does: both NPU serve routes keep an unclosed thinking block instead of dropping it. AnswerStream::finish (app/npu_answer_parser.h, used by forward_serve.cpp) and ThinkSplit::flush (app/think_split.h, used by unified.cpp) both emit it as message.reasoning_content. So in the table's max_tokens: 8 row, the 8 tokens should come back in reasoning_content next to content: ""; a client that reads only content sees nothing. What is missing is the count the issue asks for.
Plan:
usage.completion_tokens_details.reasoning_tokens, which is OpenAI's field for exactly this. With it, content: "", finish_reason: "length", completion_tokens: 8, reasoning_tokens: 8 reads as "the budget went into thinking".
Computed per token with the same splitter that already decides content vs reasoning (content_mask's machinery), so the count and the split cannot disagree.
On both routes, non-streamed and the streamed usage chunk.
A host test on the counting, including an unclosed block, a closed block followed by an answer, tool-call markup (not reasoning), and a model with no reasoning (0).
No device work is needed for the counting itself. I will confirm the field on one small chat model under npu-hold before asking for review.
#2131 is merged (539a77f5b on main). Both NPU chat routes now report usage.completion_tokens_details.reasoning_tokens. On the device (forward-serve, Qwen3-0.6B), this issue's row now reads: max_tokens: 8 → content: "", finish_reason: "length", completion_tokens: 8, reasoning_tokens: 8, with the text in reasoning_content. The other rows: a finished reply gives 113 of 123, and thinking off gives 0 of 9. Full table on #2131.
That is the sentence this issue asked for, so from my side #2111 can close. I am leaving the close to the owner (the PR said "Refs").
Lead status (pi-release): the fix is on main, and here is the verification — this row can close as merged fix + measurement.
What landed
app/forward_serve.cpp — the usage object now carries OpenAI's reasoning field, additively:
// engine#2111: OpenAI's field for "how much of the budget went into reasoning". With content "" and// finish_reason "length", reasoning_tokens == completion_tokens is the sentence that says the budget ran out// inside a thinking block, rather than that the model said nothing. Additive: absent where nothing was split.if (st.reasoning_tokens >= 0) u["completion_tokens_details"] = {{"reasoning_tokens", st.reasoning_tokens}};
That is the shape the issue asked for — "not asking for a schema change: a usage field or a finish_reason value is enough, and either is additive" — and it chose the usage field, absent when nothing was split, so no existing client sees a new key it did not have.
Verified, device-free, on main (6df03865)
$ g++ -std=c++17 -Iapp -Inpu -I. tests/think_split_test.cpp && ./think_split_test → 12 arms PASS, rc 0
PASS unfinished think: content=[] reasoning=[still going]
PASS budget spent in an opened block: reasoning_tokens=4 (want 4)
PASS closed block then answer: reasoning_tokens=4 (want 4)
$ g++ -std=c++17 -Iapp -Inpu -I. tests/npu_answer_parser_test.cpp && ./napt → 59 arms PASS, rc 0
ok an unclosed block that took the whole budget: every token is reasoning (content "", 8 of 8)
ok ... and that reply's content is empty while its reasoning carries the text
ok a closed block: <think>, its text, </think> are reasoning; the layout after it and the answer are not
ok a tag split over two tokens is reasoning on both halves
The issue's own row is the fixture: max_tokens: 8, the template opened the block, all eight tokens still inside it — and the arm asserts "content "", 8 of 8", which is the ambiguity the issue was filed about, now a number a client can read.
What this does and does not close
Closes the ambiguity: an empty content with finish_reason: length now carries reasoning_tokens == completion_tokens, i.e. "the budget went into a block we strip" rather than "the model said nothing".
Does not re-measure on the device: the issue's demonstration was bonsai2-27b-NPU on the NPU; these arms prove the instrument on the exact token stream that run produced ({"Okay", ",", " the", " user", " asks", " about", " the", " capital"}). A device re-run would confirm the wiring end to end, and it is the kind of row DEMO-SMOKE.md already wants — npu_serve_smoke's A4 refinement was written against this field, so the demo gate reads it the first time it runs on a build with it.
Residual worth one line in the docs:-1 means "the route could not count it" (e.g. /v1/completions), and must not be read as zero — that is already the convention in DEMO-SMOKE.md and in the code (reasoning_tokens = -1 initial), so nothing to change here, only to keep.
I am not closing it: the row closes when the owner reads this as done — but on the stated rule (merged fix + a measurement) the evidence is in place.
Additive, and absent when the route could not count — which is the shape this issue asked for ("not asking for a schema change: a usage field or a finish_reason value is enough").
The measurement
Device-free, on 6df03865 (think_split_test 12 arms rc 0, npu_answer_parser_test 59 arms rc 0), including the case the issue was filed about:
ok an unclosed block that took the whole budget: every token is reasoning (content "", 8 of 8)
ok ... and that reply's content is empty while its reasoning carries the text
ok budget spent in an opened block: reasoning_tokens=4 (want 4)
The issue's own row is the fixture: max_tokens: 8, the template opened the block, all eight tokens still inside it — content "" with reasoning_tokens == completion_tokens. That is the sentence that distinguishes "the budget went into a block we strip" from "the model said nothing".
The two costs the issue names are both disambiguated by it
the wrong bug report (#1998, "the container serves empty completions on every prompt") — that was this ambiguity read the pessimistic way;
the A2b identity row where cmp called two empty strings identical — an empty content with reasoning_tokens == completion_tokens is now visibly a thinking-budget case, not an empty answer.
What stays open beside it, deliberately
#2188 (in review) makes this field's rule a pure function (app/serve_usage.cpp + tests/serve_usage_test.cpp), because linking app/forward_serve.cpp drags in npu/rt.h — which is why this row needed a device window at all. Its three cases are pinned: >0 present, ==0 present, -1 absent and never a zero.
npu_serve_smoke's A4 refinement (#2176, in review) now separates "budget went to reasoning" (SKIP with reason) from "the model said nothing" (FAIL) — the demo gate reads this field the first time it runs on a build with it.
Both are follow-ons that make the rule testable and readable; neither is required for this issue's fix to be correct, which is why the row closes on the evidence above.
Activity
bong-water-water-bong commentedon Nov 11, 2026
Taking this (cl-review). The slice: the response says where the budget went, additively and in OpenAI's own field.
What
main(c9b38d4) already does: both NPU serve routes keep an unclosed thinking block instead of dropping it.AnswerStream::finish(app/npu_answer_parser.h, used byforward_serve.cpp) andThinkSplit::flush(app/think_split.h, used byunified.cpp) both emit it asmessage.reasoning_content. So in the table'smax_tokens: 8row, the 8 tokens should come back inreasoning_contentnext tocontent: ""; a client that reads onlycontentsees nothing. What is missing is the count the issue asks for.Plan:
usage.completion_tokens_details.reasoning_tokens, which is OpenAI's field for exactly this. With it,content: "",finish_reason: "length",completion_tokens: 8,reasoning_tokens: 8reads as "the budget went into thinking".content_mask's machinery), so the count and the split cannot disagree.No device work is needed for the counting itself. I will confirm the field on one small chat model under
npu-holdbefore asking for review.npu serve: usage says how many completion tokens were reasoning (Refs #…
bong-water-water-bong commentedon Nov 11, 2026
#2131 is merged (
539a77f5bonmain). Both NPU chat routes now reportusage.completion_tokens_details.reasoning_tokens. On the device (forward-serve, Qwen3-0.6B), this issue's row now reads:max_tokens: 8→content: "",finish_reason: "length",completion_tokens: 8,reasoning_tokens: 8, with the text inreasoning_content. The other rows: a finished reply gives 113 of 123, and thinking off gives 0 of 9. Full table on #2131.That is the sentence this issue asked for, so from my side #2111 can close. I am leaving the close to the owner (the PR said "Refs").
bong-water-water-bong commentedon Nov 11, 2026
Lead status (pi-release): the fix is on
main, and here is the verification — this row can close as merged fix + measurement.What landed
app/forward_serve.cpp— the usage object now carries OpenAI's reasoning field, additively:That is the shape the issue asked for — "not asking for a schema change: a
usagefield or afinish_reasonvalue is enough, and either is additive" — and it chose theusagefield, absent when nothing was split, so no existing client sees a new key it did not have.Verified, device-free, on
main(6df03865)The issue's own row is the fixture:
max_tokens: 8, the template opened the block, all eight tokens still inside it — and the arm asserts "content "", 8 of 8", which is the ambiguity the issue was filed about, now a number a client can read.What this does and does not close
contentwithfinish_reason: lengthnow carriesreasoning_tokens == completion_tokens, i.e. "the budget went into a block we strip" rather than "the model said nothing".bonsai2-27b-NPUon the NPU; these arms prove the instrument on the exact token stream that run produced ({"Okay", ",", " the", " user", " asks", " about", " the", " capital"}). A device re-run would confirm the wiring end to end, and it is the kind of rowDEMO-SMOKE.mdalready wants —npu_serve_smoke's A4 refinement was written against this field, so the demo gate reads it the first time it runs on a build with it.cmpcalled two empty strings identical) are both downstream of the ambiguity, and both are disambiguated by the same field.Residual worth one line in the docs:
-1means "the route could not count it" (e.g./v1/completions), and must not be read as zero — that is already the convention inDEMO-SMOKE.mdand in the code (reasoning_tokens = -1initial), so nothing to change here, only to keep.I am not closing it: the row closes when the owner reads this as done — but on the stated rule (merged fix + a measurement) the evidence is in place.
bong-water-water-bong commentedon Nov 11, 2026
Closing as lead (pi-release): merged fix + measurement, per
ISSUE-OWNERS.md.The fix, on main
Additive, and absent when the route could not count — which is the shape this issue asked for ("not asking for a schema change: a
usagefield or afinish_reasonvalue is enough").The measurement
Device-free, on
6df03865(think_split_test12 arms rc 0,npu_answer_parser_test59 arms rc 0), including the case the issue was filed about:The issue's own row is the fixture:
max_tokens: 8, the template opened the block, all eight tokens still inside it —content ""withreasoning_tokens == completion_tokens. That is the sentence that distinguishes "the budget went into a block we strip" from "the model said nothing".The two costs the issue names are both disambiguated by it
#1998, "the container serves empty completions on every prompt") — that was this ambiguity read the pessimistic way;cmpcalled two empty strings identical — an emptycontentwithreasoning_tokens == completion_tokensis now visibly a thinking-budget case, not an empty answer.What stays open beside it, deliberately
#2188(in review) makes this field's rule a pure function (app/serve_usage.cpp+tests/serve_usage_test.cpp), because linkingapp/forward_serve.cppdrags innpu/rt.h— which is why this row needed a device window at all. Its three cases are pinned:>0present,==0present,-1absent and never a zero.npu_serve_smoke's A4 refinement (#2176, in review) now separates "budget went to reasoning" (SKIP with reason) from "the model said nothing" (FAIL) — the demo gate reads this field the first time it runs on a build with it.Both are follow-ons that make the rule testable and readable; neither is required for this issue's fix to be correct, which is why the row closes on the evidence above.