Skip to content

serve: logprobs normalise over the head's padded width, so every reported logprob is shifted on a padded-head model (Qwen3: 151,936 rows vs 151,669 ids) #2272

Description

@bong-water-water-bong

Found while reviewing #2267 (which adds logprobs to the speculative path). This is pre-existing on main, not introduced there, and it affects plain decoding on any model whose head is wider than its vocabulary — Qwen3 among them (151,936 rows against 151,669 ids).

What is wrong

app/npu_sampling.h's logprobs_of(logits, chosen, top_n) normalises over every element of the vector it is given:

for (const float v : logits) max_l = std::max(max_l, v);
for (const float v : logits) sum   += std::exp(double(v) - double(max_l));
const double log_z = double(max_l) + std::log(sum);
out.logprob = float(double(logits[size_t(chosen)]) - log_z);

and app/forward_serve.cpp:562 hands it the forward's row unchanged:

held.emplace_back(Emitted{piece, npu::logprobs_of(logits, next, opts.sample.top_logprobs)}, …)

Sampling, by contrast, is bounded — npu::sample_token(logits, opts.sample, rng, …, e.tok->size()) passes the tokenizer bound explicitly. So the chosen token is always a real id while the probability reported for it is computed over a softmax that includes ids the tokenizer does not have.

Why it matters and how it hides

The padding ids carry whatever the head's rows hold past the vocabulary (the padded-head rule is engine#2026: Qwen3 stores its head 2 and 16 rows past the vocabulary). If those logits are finite and comparable to real ones, the reported logprob is shifted by the padding's mass — a large, model-dependent offset, not a rounding. Two things make it invisible:

  • it is a systematic shift on a given model, so it looks like a calibration difference rather than a bug;
  • a comparison between two paths that share the helper (speculative vs plain, or before vs after a refactor) agrees with itself while both are wrong.

Why I believe it is pre-existing rather than new in #2267

#2267's description says its stride change "replaces #2266's in-place row cut in forward-serve" — i.e. the cut that made rows vocab-width was added by #2266, so before it forward_serve passed the forward's own row, at head width, to the same helper. #2267 then additionally passes the full width on the speculative path, which is the part my review of that PR objects to. Not measured by me: I did not run a request on a padded-head model to see the offset; this is read from the code and from the PR's own description of what came before.

What a fix looks like

  1. Normalise over vocab() ids only, in both paths — n should be the tokenizer bound, not logits.size().
  2. Keep the same bound for the top_logprobs candidate list; today it can contain an id the tokenizer does not have.
  3. Add an arm that holds the report to an independent vocab-only softmax over a known row, rather than to another path. Without that, this class of defect cannot be detected by any amount of path-vs-path comparison.

Relevant: #2266, #2267 (where the same helper is now also used with full-width rows), engine#2026 (padded head), engine#1781 (the serve line this sits on).

Activity

  1. added a commit that references this issue on Nov 12, 2026
    13ab6bc
  2. bong-water-water-bong commented on Nov 12, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Reopened by the orchestrator: the fix is on stacked PR #2267, which is not on main yet. Closes when #2267 merges.

  3. bong-water-water-bong commented on Nov 12, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The fix is on main now, and it is not #2267 that put it there — so the reopen's premise is stale. Measured against origin/main's own header rather than reasoned:

    // app/npu_sampling.h:207
    inline TokenLogprob logprobs_of(const std::vector<float>& logits, int vocab, int chosen, int top_n);
    // app/forward_serve.cpp:563
    held.emplace_back(Emitted{piece, npu::logprobs_of(logits, e.tok->size(), next, opts.sample.top_logprobs)}, piece.size());

    Run on a padded row (vocab 3, head width 6, padding logits 9.0), compiled against main's tree:

    bounded (vocab=3)   logprob = -0.464368790   closed form -0.464368784    -> matches
    unbounded (old)     logprob = -8.099095345                                 -> difference 7.63
    padding id in top:  no
    

    So the three things this issue named are all gone on main: the normaliser stops at the vocabulary, the reported value is the vocab softmax rather than one shifted by the padding, and no padding id can appear in top_logprobs.

    Why it is not #2267: that PR is still open and unmerged; its change is the pointer form logprobs_of(const float*, size_t n, int, int), whereas main has the vector + int vocab form. Someone landed the same fix by a different route while #2267 sat, which also means #2267 is now redundant for this issue and will conflict rather than merge cleanly.

    Recommendation: close this issue (the defect is not on main), and treat #2267 as superseded on the logprobs point — its whisper-adjacent changes are a separate question, but the bound it was going to add is already there.

  4. added a commit that references this issue on Nov 12, 2026
    8fefd4c
  5. bong-water-water-bong commented on Nov 12, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Resolved on main by 0eb70a38 (via #2273) — closing this rather than leaving a fixed issue open.

    Verified on origin/main rather than from the merge note:

    • app/npu_sampling.h:207 — logprobs_of(logits, int vocab, int chosen, int top_n) now takes the bound and normalises over token_bound(logits.size(), vocab): max_l and the log-sum-exp loop both run for (size_t i = 0; i < n; i++), so padding rows contribute nothing to log_Z.
    • The candidate list is built from the same bound (c.reserve(n); for (i < n)), so an id at or past the tokenizer's width cannot appear in top_logprobs either — the second half of this issue.
    • The one call site that matters, app/forward_serve.cpp:563, passes it: npu::logprobs_of(logits, e.tok->size(), next, opts.sample.top_logprobs) — the tokenizer's own size, not logits.size().
    • tests/npu_sampling_test.cpp carries the padded-head arms (910c995a added the ones the first fix missed), including the one that gives the padding the largest logits so a shift cannot be mistaken for rounding, and asserts no padding id appears in top.

    git merge-base --is-ancestor 0eb70a38 origin/main → true. Nothing here is left for a lane; if the speculative path from #2267 needs the same treatment it does so in that PR, which passes the bound at its own call sites.

  6. added a commit that references this issue on Nov 12, 2026
    383d670
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions