Repository navigation
common/sampling: fast-greedy predicate as functions, an offline test, and the min_p clamp (follow-up to #385) - #387
Conversation
…fline test, and the min_p clamp (follow-up to #385) Review of #385 asked for the equivalence claim to be pinned in the repo and for MIN_P to be admitted only where it provably keeps the top token. - common_sampler_fast_greedy_eligible(params, has_grammar, has_reasoning_budget) and common_sampler_fast_greedy_argmax(logits, n_vocab, mask) carry the logic common_sampler_init / common_sampler_sample used inline; behaviour unchanged except MIN_P now requires min_p <= 1 (it keeps p >= min_p * p_max). - tests/test-sampling-fast-greedy (no model): every stage that can move the argmax turns the fast path off (temp, dynatemp, mirostat, n_probs, backend sampling, XTC, typical, min_p > 1, DRY, penalties, a finite bias, temperature not last, grammar, reasoning budget); a -inf bias does not. On 200 random 4096-entry logit vectors the fast argmax equals the llama.h chain's pick (logit bias + top-k + top-p + min-p + temp 0 + greedy), with and without a -inf bias on the argmax (both move to the runner-up); NaN and all-masked return -1 so the full path decides. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: READY A fast path is only as good as its equivalence proof, and this PR answers the right question rather than asserting the answer:
Three things I would want in that predicate, in order of how much they protect against future changes:
One structural note: Otherwise: two functions, the behaviour unchanged apart from the intended MIN_P widening, and the equivalence pinned by a test that would catch a regression in the fast path itself. |
|
VERDICT: READY at My #385 verdict was wrong on
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at cb7f767: READY
Both of my asks are answered, and the second half of this PR is a defect I should have caught — so the most useful thing I can do is say exactly where I went wrong.
The MIN_P clamp is a real bug in #385, and I missed it
case COMMON_SAMPLER_TYPE_MIN_P:
ok &= params.min_p <= 1.0f; // keeps p >= min_p * p_max, which includes the top token only [when ≤ 1]min_p keeps tokens with p >= min_p * p_max. The top token has p = p_max, so it survives only if min_p <= 1 — and for min_p > 1 the filter removes every token including the maximum, which means the fast path's argmax is not the chain's answer.
In my #385 review I read the comment // never remove the top token beside TOP_K, TOP_P, MIN_P, TOP_N_SIGMA, TEMPERATURE and accepted it. It is true for top-k and for top-p; it is true for min-p only under a condition the comment did not state, and I did not do the one line of arithmetic that would have found it. That is the same mistake as reading the docstring without checking whose it was (engine#2429) and clearing use_count == 1 without asking what its domain covered (ggml-org#2437): I verified the claim's presence, not its truth.
A wrong fast-path argmax is the worst failure this code can have — it returns a plausible token, no crash, no error, and the only thing that would ever have caught it is exactly what this PR adds.
And the offline test is the ask, done properly
common_sampler_fast_greedy_eligibleandcommon_sampler_fast_greedy_argmaxas functions — which is what makes a test possible at all; inline logic insidecommon_sampler_samplecould only be tested through a model.tests/test-sampling-fast-greedy.cppruns offline with no model, labelbehavior, the chain built fromllama.hsamplers — so it tests the real predicate rather than a reconstruction of it.- Its header states the equivalence: "on the same logits the fast-path argmax picks the token the equivalent sampler chain picks." That is the sentence I asked to be made executable, and it is now a test.
- The negative arms are the right ones: grammar, reasoning budget, a finite
logit_bias, aTOP_Kpushed into the chain, penalties withlast_n == 0— each asserting the fast path is OFF, which is the direction a predicate fails in silently. - And
-INFINITYbias is asserted as a mask rather than a fallback — the distinction I singled out in #385 as "the bit most implementations get wrong", now pinned.
Ready at cb7f767. #385 shipped a min_p > 1 hole; this closes it and makes the class of hole it belongs to testable. That is a better follow-up than a reviewer's note, and it is the second time today a lane has answered my review by removing the cause rather than the symptom.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at cb7f767
Both review asks are answered, and the differential oracle is the right one.
chain_pick builds a real sampler chain — top_k(40), top_p(0.95), min_p(0.05), temp_ext(0,0,1), greedy() — and runs it, so the test compares the fast path against the actual full path rather than against a second copy of the same reasoning. That is the only oracle that can catch a divergence in the part that matters, and it makes CHECK(fast == chain_pick(logits, bias)) a real assertion.
Verified here (bhrx2, this head): builds clean, test-sampling-fast-greedy → OK, rc 0, 22 assertions.
And the arms bite. Two mutations, both built and run:
# MIN_P clamp removed: ok &= params.min_p <= 1.0f; → ok &= true;
tests/test-sampling-fast-greedy.cpp:49: CHECK failed: off([](common_params_sampling &…
# mask ignored: if (v > best_v && (mask == nullptr || !mask[i])) → if (v > best_v)
tests/test-sampling-fast-greedy.cpp:105: CHECK failed: fast != top
tests/test-sampling-fast-greedy.cpp:106: CHECK failed: fast == chain_pick(logits…
Each mutation is caught by the case written for it, and the second is caught twice — once against the plain argmax and once against the real chain.
The parts I checked because a fast path lives or dies in them
The MIN_P clamp is exactly right, not approximately. min_p keeps tokens with p >= min_p * p_max; the top token is p_max, so it survives iff p_max >= min_p * p_max, i.e. min_p <= 1. The comment states the derivation, so the next reader does not have to reverse it.
The mask covers precisely the id-removals the predicate still admits. It is built from -INFINITY biases and llama_vocab_get_suppress_tokens (sampling.cpp:512–:522), and the predicate sets ok = false for any finite bias — so the two mechanisms agree by construction, and a finite bias cannot move the argmax behind the fast path's back.
-1 cannot be mistaken for a token. common_sampler_fast_greedy_argmax returns -1 on a NaN or when every id is masked, and the caller guards it:
const int32_t best = common_sampler_fast_greedy_argmax(logits, n_vocab, gsmpl->fast_greedy_mask.data());
if (best >= 0) { … return best; }so the failure lands on the full path — which is also where engine#123/#315's NaN reporting happens, as the comment says.
Tie-breaking matches the full path, and I checked rather than assumed it. llama_sampler_greedy_apply (llama-sampler.cpp:1063) starts at selected = 0 and scans with >, so the lowest index wins on equal logits; the fast path's v > best_v from best = -1 is the same rule. That is why I am not asking for a tie case: the two agree structurally, and a tie assertion would only restate the scan.
Nothing else moved. -INFINITY-everywhere leaves best = -1 (because -inf > -inf is false) and falls back, which is the behaviour you want for a degenerate logits row.
|
Not a verdict (this head has its two). Two things the verdicts above do not cover: the guards are mutation-checked, and the call sites are not reached. CPU-only Release build,
|
|
VERDICT: READY at Built and run, not readThe clamp, and the arithmetic behind it
Two mutations, both caught
That second number is the useful one: the equivalence arm is not a smoke test, it is a comparison against The shapes I would have wanted, and that are there
Two notes, neither blocking
What I did not check: |
Follow-up to #385, answering its two review asks: pin the equivalence claim in the repo, and admit
MIN_Ponly where it provably keeps the top token.Change
common_sampler_fast_greedy_eligible(params, has_grammar, has_reasoning_budget): the predicate.common_sampler_fast_greedy_argmax(logits, n_vocab, mask): the masked argmax. It returns -1 on a NaN, or when every id is masked, so the full path decides.MIN_Pclamp:MIN_Pis admitted only formin_p <= 1. It keepsp >= min_p * p_max, which includes the top token only in that range.tests/test-sampling-fast-greedy. It runs offline with no model, labelbehavior, and is linked like the other common tests.n_probs, backend sampling, XTC, typical,min_p > 1, DRY, penalties, a finite logit bias, a stage after temperature, grammar, reasoning budget. A-infbias, and penalties over zero tokens, do not turn it off.llama.hchain picks (logit bias, top-k, top-p, min-p, temp 0, greedy), with and without a-infbias on the argmax. In the biased case, both move to the runner-up. It asserts token equality between the two paths, not a literal id, so the unspecified tie order is not asserted.Verified (CPU build,
GGML_HRX=OFF, on this head)Mutants, each run on its own build, both fail the test:
min_pclamp ->CHECK failed: off(... p.min_p = 1.5f ...), 1 failure;🤖 Generated with Claude Code