Skip to content

npu: a grammar-constrained reply can never stop - the mask drops EOS, so every json_schema/json_object/tool reply pads to max_tokens and reports finish_reason 'length' #1182

Closed

Description

@bong-water-water-bong

What happens

A grammar-constrained reply on /v1/chat/completions can never stop by itself: the grammar mask keeps every token
whose text the grammar accepts and drops everything else, and the model's end token (<|im_end|>, id 151645) has the
literal text <|im_end|>, which is not in the constrained language. Same for <|endoftext|>. So EOS is masked out at
every position, sample_token() never returns it, and the decode loop can only end at max_tokens.

Measured on this box (Qwen3-0.6B, NPU_INFER_HOST_GEMM=1, max_tokens: 32, temperature: 0, one npu-hold -l npufeat
ticket, engine origin/main 4ea2577, streamed row equal to non-streamed):

request content generated finish_reason
response_format: json_schema (object, name+age) {⏎ "age": 25,⏎ "name": "John Doe"⏎} then ⏎ \t\t\t\t \t\t\t\t\t\t 32 length
response_format: json_object {} then the same whitespace tail 32 length
grammar: root ::= "Hello there." Hello there. (nothing else visible) 32 length
no grammar (control) reasoning only, 32 tokens 32 length (expected: budget is 32)

The JSON is complete at step 19 of 32; the remaining 13 tokens are whitespace and vocabulary-gap ids that the grammar
still accepts and that <|im_end|> cannot compete with, because it is not in the candidate set at all.

The mask trace from the same run (NPU_MASK_DEBUG, temporary instrumentation, not part of any change):

step=18 next=532      text=}          allowed=422  accepting=0
step=19 next=151672   text=           allowed=418  accepting=1   <- the object is complete here
step=20 next=220      text=           allowed=418  accepting=1
step=21 next=220      text=           allowed=415  accepting=1
step=23 next=50233    text=<tabs>     allowed=409  accepting=1
step=30 next=151680   text=           allowed=267  accepting=1   <- id > tokenizer size

Why this matters beyond the wasted tokens

  1. finish_reason is wrong for a complete answer. Every json_schema / json_object / tool-grammar reply reports
    length, i.e. "truncated", when the constrained answer is whole. A client that retries or rejects on length, or an
    automation that streams until it sees a stop, is told the wrong thing. It is also why the json_schema rows on npu: response_format json_schema compiled to GBNF (#652 item 1) #735
    read length — I recorded the symptom in that PR body and did not chase it.
  2. It costs the whole budget on every constrained request: 13 of 32 tokens here are padding, i.e. ~40 % of decode
    steps, and on the device that is real NPU time per request.
  3. content carries trailing whitespace ({"a":1}\n \t\t…), which for a strict JSON consumer is at best noise.
  4. A second, independent hole visible in the same trace: ids above the tokenizer's vocabulary (151672, 151680 against
    size() == 151669) decode to the empty string, and the mask's "an empty piece cannot leave the grammar" rule admits
    them unconditionally (app/forward_serve.cpp:360-361). The model drifts into them once the grammar leaves it nothing
    to say — they are padding positions, not tokens.

Where

app/forward_serve.cpp:355-367 (refresh_allowed) and :405-411 (the EOS check that can never fire under a grammar).
The comment at :406-408 states the intent — "EOS is not masked by the grammar on purpose: a constrained answer is
still allowed to stop early (the client validates it)" — but nothing implements it: the loop that builds allowed
never sets an end token's bit, so the intent and the code disagree.

npu::sample_token() (app/npu_sampling.h:88-96) drops every id whose bit is 0, so a masked EOS is unreachable rather
than merely unlikely.

Fix (smallest complete change)

  • Allow the request's end tokens (config eos_token_id, plus <|endoftext|>) only where the grammar can end
    (npu::Grammar::can_stop(state), i.e. the parse is at an accepting state). Mid-grammar they stay masked, so a
    constrained answer can never be cut into something the grammar would not accept; at an accepting state the reply
    stops with finish_reason: "stop" and no padding.
  • Do not treat ids >= tokenizer.size() as "empty pieces": they are not byte fragments, they are vocabulary padding.

Both are inside refresh_allowed, they touch no container and no kernel, and they are verifiable device-free in
host-GEMM mode (a npu-hold ticket is still taken for every run).

Not claimed: this does not make the mask prefer stopping — if the grammar still accepts text and the model keeps
emitting it, the budget is still a budget. What changes is that stopping is now possible, and complete answers report
stop instead of length.

Found while re-verifying #1152 on current main; unrelated to that fix, so filed rather than folded into it.

Activity

  1. bong-water-water-bong commented on Nov 5, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    PR #1188 at cdf4bcb (1bit/npu-grammar-eos), base main 0f7c9a2 — the fix as described in the issue:

    • npu::allow_end_tokens() (app/npu_grammar.h) sets the request's end tokens' bits only where Grammar::can_stop() says the parse may end, so mid-grammar EOS stays masked and stopping becomes possible at an accepting state.
    • refresh_allowed() (app/forward_serve.cpp) calls it with the container's eos_token_ids plus <|endoftext|>, and no longer admits ids at or beyond the tokenizer's vocabulary (they are padding positions, decode() returns "" for them, and the "an empty piece cannot leave the grammar" rule was letting the model drift into them).
    • test_grammar_end_tokens (8 assertions) + the finish_reason sentence in docs/npu.md.

    Measured on Qwen3-0.6B in host-GEMM mode, before → after: json_schema 32 tokens / length → 14 / stop, json_object 32 → 1 / stop, client GBNF 32 → 4 / stop, and a genuinely too-small budget (max_tokens: 8) still reports length. The new test was falsified against the pre-fix policy before being trusted.

    CI: build and CodeQL pass; pr_agent is red only because the local review model on :8090 went down with the c27 reboot (##[error]the local model (coder-llm on :8090) does not answer), which is the same non-code failure that hit my other PRs tonight — a bounded watcher re-runs it when :8090 answers.

  2. bong-water-water-bong commented on Nov 6, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Second, independent row for this fix — and a stronger one than #1182's own: on current main the same json_schema request does not pad after a complete object, it produces nothing at all.

    main 29ec80c6 vs this branch, Qwen3-0.6B, NPU_INFER_HOST_GEMM=1 (array never opened), one npu-hold -l npufeat ticket per arm, fresh process per arm, /no_think hi, max_tokens: 48, temperature 0, schema {name: string, age: integer} both required (~/lb/postmerge/s5.py):

    build completion tokens content finish_reason
    main 29ec80c6 48 (16.5 s of host GEMM) "" — no object at all length
    this branch (33320af6) 15 {"age": 18, "name": "Lily"} stop

    So the failure mode is worse than #1182 recorded: it is not "object, then whitespace padding to the cap", it is "the decode drifts into vocabulary-padding ids that decode to empty strings and never emits {", because EOS is masked and an empty piece is admitted as leaving the grammar unchanged. With allow_end_tokens() gated on can_stop() and no ids above the vocabulary, the same request finishes the object in 15 tokens and stops.

    Two things this row also says, for whoever reviews:

    • it is a content finding, not a finish_reason one: the reply was schema-valid and complete, and the branch reports stop — so this fix is what makes the json_schema row in npu: response_format json_schema compiled to GBNF (#652 item 1) #735 mean what its body claims;
    • the emitted key order is age, name (the schema declares name, age), which is legal JSON and accepted by any schema validator; the converter's ordering claim is about the grammar's size, not about what a client can rely on.

    npu_sampling: ok, tools/copyright.py --check rc 0 on this branch at 33320af6.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions