You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
serve: logprobs normalise over the head's padded width, so every reported logprob is shifted on a padded-head model (Qwen3: 151,936 rows vs 151,669 ids) #2272
Found while reviewing #2267 (which adds logprobs to the speculative path). This is pre-existing on main, not introduced there, and it affects plain decoding on any model whose head is wider than its vocabulary — Qwen3 among them (151,936 rows against 151,669 ids).
What is wrong
app/npu_sampling.h's logprobs_of(logits, chosen, top_n) normalises over every element of the vector it is given:
for (constfloat v : logits) max_l = std::max(max_l, v);
for (constfloat v : logits) sum += std::exp(double(v) - double(max_l));
constdouble log_z = double(max_l) + std::log(sum);
out.logprob = float(double(logits[size_t(chosen)]) - log_z);
and app/forward_serve.cpp:562 hands it the forward's row unchanged:
Sampling, by contrast, is bounded — npu::sample_token(logits, opts.sample, rng, …, e.tok->size()) passes the tokenizer bound explicitly. So the chosen token is always a real id while the probability reported for it is computed over a softmax that includes ids the tokenizer does not have.
Why it matters and how it hides
The padding ids carry whatever the head's rows hold past the vocabulary (the padded-head rule is engine#2026: Qwen3 stores its head 2 and 16 rows past the vocabulary). If those logits are finite and comparable to real ones, the reported logprob is shifted by the padding's mass — a large, model-dependent offset, not a rounding. Two things make it invisible:
it is a systematic shift on a given model, so it looks like a calibration difference rather than a bug;
a comparison between two paths that share the helper (speculative vs plain, or before vs after a refactor) agrees with itself while both are wrong.
Why I believe it is pre-existing rather than new in #2267
#2267's description says its stride change "replaces #2266's in-place row cut in forward-serve" — i.e. the cut that made rows vocab-width was added by #2266, so before it forward_serve passed the forward's own row, at head width, to the same helper. #2267 then additionally passes the full width on the speculative path, which is the part my review of that PR objects to. Not measured by me: I did not run a request on a padded-head model to see the offset; this is read from the code and from the PR's own description of what came before.
What a fix looks like
Normalise over vocab() ids only, in both paths — n should be the tokenizer bound, not logits.size().
Keep the same bound for the top_logprobs candidate list; today it can contain an id the tokenizer does not have.
Add an arm that holds the report to an independent vocab-only softmax over a known row, rather than to another path. Without that, this class of defect cannot be detected by any amount of path-vs-path comparison.
Relevant: #2266, #2267 (where the same helper is now also used with full-width rows), engine#2026 (padded head), engine#1781 (the serve line this sits on).
The fix is on main now, and it is not #2267 that put it there — so the reopen's premise is stale. Measured against origin/main's own header rather than reasoned:
// app/npu_sampling.h:207inline TokenLogprob logprobs_of(const std::vector<float>& logits, int vocab, int chosen, int top_n);
// app/forward_serve.cpp:563
held.emplace_back(Emitted{piece, npu::logprobs_of(logits, e.tok->size(), next, opts.sample.top_logprobs)}, piece.size());
Run on a padded row (vocab 3, head width 6, padding logits 9.0), compiled against main's tree:
bounded (vocab=3) logprob = -0.464368790 closed form -0.464368784 -> matches
unbounded (old) logprob = -8.099095345 -> difference 7.63
padding id in top: no
So the three things this issue named are all gone on main: the normaliser stops at the vocabulary, the reported value is the vocab softmax rather than one shifted by the padding, and no padding id can appear in top_logprobs.
Why it is not #2267: that PR is still open and unmerged; its change is the pointer form logprobs_of(const float*, size_t n, int, int), whereas main has the vector + int vocab form. Someone landed the same fix by a different route while #2267 sat, which also means #2267 is now redundant for this issue and will conflict rather than merge cleanly.
Recommendation: close this issue (the defect is not on main), and treat #2267 as superseded on the logprobs point — its whisper-adjacent changes are a separate question, but the bound it was going to add is already there.
Resolved on main by 0eb70a38 (via #2273) — closing this rather than leaving a fixed issue open.
Verified on origin/main rather than from the merge note:
app/npu_sampling.h:207 — logprobs_of(logits, int vocab, int chosen, int top_n) now takes the bound and normalises over token_bound(logits.size(), vocab): max_l and the log-sum-exp loop both run for (size_t i = 0; i < n; i++), so padding rows contribute nothing to log_Z.
The candidate list is built from the same bound (c.reserve(n); for (i < n)), so an id at or past the tokenizer's width cannot appear in top_logprobs either — the second half of this issue.
The one call site that matters, app/forward_serve.cpp:563, passes it: npu::logprobs_of(logits, e.tok->size(), next, opts.sample.top_logprobs) — the tokenizer's own size, not logits.size().
tests/npu_sampling_test.cpp carries the padded-head arms (910c995a added the ones the first fix missed), including the one that gives the padding the largest logits so a shift cannot be mistaken for rounding, and asserts no padding id appears in top.
git merge-base --is-ancestor 0eb70a38 origin/main → true. Nothing here is left for a lane; if the speculative path from #2267 needs the same treatment it does so in that PR, which passes the bound at its own call sites.
Activity
serve: logprobs normalise over the tokenizer's ids, not the head's pa…
test: the padded-head arms for #2272, which the fix did not ship
sampling: the vector logprobs_of takes the vocabulary as a required a…
bong-water-water-bong commentedon Nov 12, 2026
Reopened by the orchestrator: the fix is on stacked PR #2267, which is not on main yet. Closes when #2267 merges.
bong-water-water-bong commentedon Nov 12, 2026
The fix is on
mainnow, and it is not #2267 that put it there — so the reopen's premise is stale. Measured againstorigin/main's own header rather than reasoned:Run on a padded row (vocab 3, head width 6, padding logits 9.0), compiled against main's tree:
So the three things this issue named are all gone on
main: the normaliser stops at the vocabulary, the reported value is the vocab softmax rather than one shifted by the padding, and no padding id can appear intop_logprobs.Why it is not #2267: that PR is still open and unmerged; its change is the pointer form
logprobs_of(const float*, size_t n, int, int), whereasmainhas the vector +int vocabform. Someone landed the same fix by a different route while #2267 sat, which also means #2267 is now redundant for this issue and will conflict rather than merge cleanly.Recommendation: close this issue (the defect is not on
main), and treat #2267 as superseded on the logprobs point — its whisper-adjacent changes are a separate question, but the bound it was going to add is already there.Merge origin/main into #2267: resolve against #2273's logprobs bound
bong-water-water-bong commentedon Nov 12, 2026
Resolved on
mainby0eb70a38(via #2273) — closing this rather than leaving a fixed issue open.Verified on
origin/mainrather than from the merge note:app/npu_sampling.h:207—logprobs_of(logits, int vocab, int chosen, int top_n)now takes the bound and normalises overtoken_bound(logits.size(), vocab):max_land the log-sum-exp loop both runfor (size_t i = 0; i < n; i++), so padding rows contribute nothing tolog_Z.c.reserve(n); for (i < n)), so an id at or past the tokenizer's width cannot appear intop_logprobseither — the second half of this issue.app/forward_serve.cpp:563, passes it:npu::logprobs_of(logits, e.tok->size(), next, opts.sample.top_logprobs)— the tokenizer's own size, notlogits.size().tests/npu_sampling_test.cppcarries the padded-head arms (910c995aadded the ones the first fix missed), including the one that gives the padding the largest logits so a shift cannot be mistaken for rounding, and asserts no padding id appears intop.git merge-base --is-ancestor 0eb70a38 origin/main→ true. Nothing here is left for a lane; if the speculative path from #2267 needs the same treatment it does so in that PR, which passes the bound at its own call sites.serve: speculative decode serves logprobs from each token's own row; …