Repository navigation
Description
What happens
A grammar-constrained reply on /v1/chat/completions can never stop by itself: the grammar mask keeps every token
whose text the grammar accepts and drops everything else, and the model's end token (<|im_end|>, id 151645) has the
literal text <|im_end|>, which is not in the constrained language. Same for <|endoftext|>. So EOS is masked out at
every position, sample_token() never returns it, and the decode loop can only end at max_tokens.
Measured on this box (Qwen3-0.6B, NPU_INFER_HOST_GEMM=1, max_tokens: 32, temperature: 0, one npu-hold -l npufeat
ticket, engine origin/main 4ea2577, streamed row equal to non-streamed):
| request | content | generated | finish_reason |
|---|---|---|---|
response_format: json_schema (object, name+age) |
{⏎ "age": 25,⏎ "name": "John Doe"⏎} then ⏎ \t\t\t\t \t\t\t\t\t\t |
32 | length |
response_format: json_object |
{} then the same whitespace tail |
32 | length |
grammar: root ::= "Hello there." |
Hello there. (nothing else visible) |
32 | length |
| no grammar (control) | reasoning only, 32 tokens | 32 | length (expected: budget is 32) |
The JSON is complete at step 19 of 32; the remaining 13 tokens are whitespace and vocabulary-gap ids that the grammar
still accepts and that <|im_end|> cannot compete with, because it is not in the candidate set at all.
The mask trace from the same run (NPU_MASK_DEBUG, temporary instrumentation, not part of any change):
step=18 next=532 text=} allowed=422 accepting=0
step=19 next=151672 text= allowed=418 accepting=1 <- the object is complete here
step=20 next=220 text= allowed=418 accepting=1
step=21 next=220 text= allowed=415 accepting=1
step=23 next=50233 text=<tabs> allowed=409 accepting=1
step=30 next=151680 text= allowed=267 accepting=1 <- id > tokenizer size
Why this matters beyond the wasted tokens
finish_reasonis wrong for a complete answer. Everyjson_schema/json_object/ tool-grammar reply reports
length, i.e. "truncated", when the constrained answer is whole. A client that retries or rejects onlength, or an
automation that streams until it sees a stop, is told the wrong thing. It is also why thejson_schemarows on npu: response_format json_schema compiled to GBNF (#652 item 1) #735
readlength— I recorded the symptom in that PR body and did not chase it.- It costs the whole budget on every constrained request: 13 of 32 tokens here are padding, i.e. ~40 % of decode
steps, and on the device that is real NPU time per request. contentcarries trailing whitespace ({"a":1}\n \t\t…), which for a strict JSON consumer is at best noise.- A second, independent hole visible in the same trace: ids above the tokenizer's vocabulary (151672, 151680 against
size() == 151669) decode to the empty string, and the mask's "an empty piece cannot leave the grammar" rule admits
them unconditionally (app/forward_serve.cpp:360-361). The model drifts into them once the grammar leaves it nothing
to say — they are padding positions, not tokens.
Where
app/forward_serve.cpp:355-367 (refresh_allowed) and :405-411 (the EOS check that can never fire under a grammar).
The comment at :406-408 states the intent — "EOS is not masked by the grammar on purpose: a constrained answer is
still allowed to stop early (the client validates it)" — but nothing implements it: the loop that builds allowed
never sets an end token's bit, so the intent and the code disagree.
npu::sample_token() (app/npu_sampling.h:88-96) drops every id whose bit is 0, so a masked EOS is unreachable rather
than merely unlikely.
Fix (smallest complete change)
- Allow the request's end tokens (config
eos_token_id, plus<|endoftext|>) only where the grammar can end
(npu::Grammar::can_stop(state), i.e. the parse is at an accepting state). Mid-grammar they stay masked, so a
constrained answer can never be cut into something the grammar would not accept; at an accepting state the reply
stops withfinish_reason: "stop"and no padding. - Do not treat ids
>= tokenizer.size()as "empty pieces": they are not byte fragments, they are vocabulary padding.
Both are inside refresh_allowed, they touch no container and no kernel, and they are verifiable device-free in
host-GEMM mode (a npu-hold ticket is still taken for every run).
Not claimed: this does not make the mask prefer stopping — if the grammar still accepts text and the model keeps
emitting it, the budget is still a budget. What changes is that stopping is now possible, and complete answers report
stop instead of length.
Found while re-verifying #1152 on current main; unrelated to that fix, so filed rather than folded into it.
Activity
bong-water-water-bong commentedon Nov 5, 2026
PR #1188 at
cdf4bcb(1bit/npu-grammar-eos), basemain0f7c9a2— the fix as described in the issue:npu::allow_end_tokens()(app/npu_grammar.h) sets the request's end tokens' bits only whereGrammar::can_stop()says the parse may end, so mid-grammar EOS stays masked and stopping becomes possible at an accepting state.refresh_allowed()(app/forward_serve.cpp) calls it with the container'seos_token_ids plus<|endoftext|>, and no longer admits ids at or beyond the tokenizer's vocabulary (they are padding positions,decode()returns""for them, and the "an empty piece cannot leave the grammar" rule was letting the model drift into them).test_grammar_end_tokens(8 assertions) + thefinish_reasonsentence indocs/npu.md.Measured on Qwen3-0.6B in host-GEMM mode, before → after:
json_schema32 tokens /length→ 14 /stop,json_object32 → 1 /stop, client GBNF 32 → 4 /stop, and a genuinely too-small budget (max_tokens: 8) still reportslength. The new test was falsified against the pre-fix policy before being trusted.CI: build and CodeQL pass;
pr_agentis red only because the local review model on:8090went down with the c27 reboot (##[error]the local model (coder-llm on :8090) does not answer), which is the same non-code failure that hit my other PRs tonight — a bounded watcher re-runs it when:8090answers.npu: a grammar-constrained reply can stop, so json_schema/json_object…
bong-water-water-bong commentedon Nov 6, 2026
Second, independent row for this fix — and a stronger one than #1182's own: on current
mainthe samejson_schemarequest does not pad after a complete object, it produces nothing at all.main29ec80c6vs this branch, Qwen3-0.6B,NPU_INFER_HOST_GEMM=1(array never opened), onenpu-hold -l npufeatticket per arm, fresh process per arm,/no_think hi,max_tokens: 48, temperature 0, schema{name: string, age: integer}both required (~/lb/postmerge/s5.py):contentfinish_reasonmain29ec80c6""— no object at alllength33320af6){"age": 18, "name": "Lily"}stopSo the failure mode is worse than #1182 recorded: it is not "object, then whitespace padding to the cap", it is "the decode drifts into vocabulary-padding ids that decode to empty strings and never emits
{", because EOS is masked and an empty piece is admitted as leaving the grammar unchanged. Withallow_end_tokens()gated oncan_stop()and no ids above the vocabulary, the same request finishes the object in 15 tokens and stops.Two things this row also says, for whoever reviews:
finish_reasonone: the reply was schema-valid and complete, and the branch reportsstop— so this fix is what makes thejson_schemarow in npu: response_format json_schema compiled to GBNF (#652 item 1) #735 mean what its body claims;age, name(the schema declaresname, age), which is legal JSON and accepted by any schema validator; the converter's ordering claim is about the grammar's size, not about what a client can rely on.npu_sampling: ok,tools/copyright.py --checkrc 0 on this branch at33320af6.npu: a grammar-constrained reply can stop, so json_schema/json_object…
npu: bound every token pick by the tokenizer, not only the grammar's (F…