Skip to content

common/sampling: fast-greedy predicate as functions, an offline test, and the min_p clamp (follow-up to #385) - #387

Merged
bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-1051/c4-fast-greedy-test
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-1051/c4-fast-greedy-test

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

Follow-up to #385, answering its two review asks: pin the equivalence claim in the repo, and admit MIN_P only where it provably keeps the top token.

Change

  • Two functions carry the logic that common/sampling: greedy fast path when the sampler chain is provably an argmax #385 had inline. Behaviour is unchanged apart from the clamp below.
    • common_sampler_fast_greedy_eligible(params, has_grammar, has_reasoning_budget): the predicate.
    • common_sampler_fast_greedy_argmax(logits, n_vocab, mask): the masked argmax. It returns -1 on a NaN, or when every id is masked, so the full path decides.
  • The MIN_P clamp: MIN_P is admitted only for min_p <= 1. It keeps p >= min_p * p_max, which includes the top token only in that range.
  • tests/test-sampling-fast-greedy. It runs offline with no model, label behavior, and is linked like the other common tests.
    • Negative arms. Each of these turns the fast path off: temperature, dynatemp, mirostat, n_probs, backend sampling, XTC, typical, min_p > 1, DRY, penalties, a finite logit bias, a stage after temperature, grammar, reasoning budget. A -inf bias, and penalties over zero tokens, do not turn it off.
    • Equivalence. On 200 seeded random 4096-entry logit vectors, the fast argmax equals what the llama.h chain picks (logit bias, top-k, top-p, min-p, temp 0, greedy), with and without a -inf bias on the argmax. In the biased case, both move to the runner-up. It asserts token equality between the two paths, not a literal id, so the unspecified tie order is not asserted.
    • NaN / all-masked return -1.

Verified (CPU build, GGML_HRX=OFF, on this head)

test-sampling-fast-greedy: OK
test-sampling: rc 0

Mutants, each run on its own build, both fail the test:

  • no min_p clamp -> CHECK failed: off(... p.min_p = 1.5f ...), 1 failure;
  • argmax ignoring the mask -> 401 failures, including the all-masked arm.

🤖 Generated with Claude Code

…fline test, and the min_p clamp (follow-up to #385)

Review of #385 asked for the equivalence claim to be pinned in the repo and
for MIN_P to be admitted only where it provably keeps the top token.

- common_sampler_fast_greedy_eligible(params, has_grammar, has_reasoning_budget)
  and common_sampler_fast_greedy_argmax(logits, n_vocab, mask) carry the logic
  common_sampler_init / common_sampler_sample used inline; behaviour unchanged
  except MIN_P now requires min_p <= 1 (it keeps p >= min_p * p_max).
- tests/test-sampling-fast-greedy (no model): every stage that can move the
  argmax turns the fast path off (temp, dynatemp, mirostat, n_probs, backend
  sampling, XTC, typical, min_p > 1, DRY, penalties, a finite bias, temperature
  not last, grammar, reasoning budget); a -inf bias does not. On 200 random
  4096-entry logit vectors the fast argmax equals the llama.h chain's pick
  (logit bias + top-k + top-p + min-p + temp 0 + greedy), with and without a
  -inf bias on the argmax (both move to the runner-up); NaN and all-masked
  return -1 so the full path decides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY 387

A fast path is only as good as its equivalence proof, and this PR answers the right question rather than asserting the answer:

  • 200 random trials over logits, comparing the fast-path argmax against the equivalent sampler chain on the same logits — that is the "pin the equivalence claim" done as a test rather than as a comment;
  • MIN_P admitted only where it provably keeps the top token, which is the correct restriction: min_p at temperature 0 is a no-op on the argmax only while the maximum probability clears the threshold, and carving that out as a provable case (rather than assuming min_p is harmless at greedy) is the distinction that keeps the fast path honest;
  • the eligibility logic is extracted into a predicate (common_sampler_fast_greedy_eligible) instead of living inline, so the conditions can be tested independently of sampling.

Three things I would want in that predicate, in order of how much they protect against future changes:

  1. fail-closed on unknown sampler types. The for (const auto type : params.samplers) switch (…) enumeration should treat an unrecognised enumerator as ineligible, not as "pass". A new sampler added next month must make the fast path decline by default — otherwise the fast path silently skips a transform it has never heard of, and the equality test (written today) cannot see it.
  2. a negative arm per condition. mirostat != 0, n_probs != 0, grammar present, reasoning budget set, temp != 0, dynatemp_range != 0, a non-temperature tail sampler — each should be one assertion that eligibility is false. With the extracted predicate this is now nearly free, and it is the half of the test that catches a widened predicate.
  3. logit_bias. The predicate iterates params.logit_bias separately — if a bias is present, the fast path must either apply it (and then the equivalence test must include biased logits) or decline. Worth one line saying which.

One structural note: MIN_P's admissibility is a proof obligation that lives in code, so the condition that establishes it should be visible in the predicate (or in a comment quoting the inequality) rather than implied by the trials passing. A future change to how min_p is applied would not fail the trials — it would just make the proof false.

Otherwise: two functions, the behaviour unchanged apart from the intended MIN_P widening, and the equivalence pinned by a test that would catch a regression in the fast path itself.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at cb7f767f — and it corrects something I got wrong in my #385 review, which is the most useful thing about it.

My #385 verdict was wrong on min_p, and this PR is the proof

I wrote that the four truncating stages "can't drop the argmax" and listed min-p among them. That holds only for min_p <= 1: the rule keeps p >= min_p * p_max, and for min_p > 1 the top token's own probability fails its own threshold, so the stage can drop the argmax and the fast path would then pick a different token than the chain. The clamp is now explicit:

case COMMON_SAMPLER_TYPE_MIN_P:
    ok &= params.min_p <= 1.0f;  // keeps p >= min_p * p_max, which includes the top token only for min_p <= 1

So the precondition list I called correct was one clause short, and the author found the clause. Worth stating plainly: a review that says "these are safe" is only as good as the case it thought of, and mine omitted the parameter value.

Both asks answered, in the form that can fail

The flag is asserted, not the tokens — which is what I asked for and is the difference between a test and a measurement:

auto off = [](void (*edit)(common_params_sampling &)) { … return !common_sampler_fast_greedy_eligible(p, false, false); };
CHECK(off([](common_params_sampling & p) { p.temp = 0.7f; }));
CHECK(off([](common_params_sampling & p) { p.dynatemp_range = 0.5f; }));
CHECK(off([](common_params_sampling & p) { p.mirostat = 2; }));
CHECK(off([](common_params_sampling & p) { p.n_probs = 1; }));
...

Every negative arm from my list is there plus more (dynatemp, a stage after temperature, min_p > 1), and the two arms that must not turn it off are there too — a -inf bias, and penalties over zero tokens. A predicate extracted into common_sampler_fast_greedy_eligible is what makes this a table rather than an experiment; the inline #385 version could not have been tested this way.

Token equality, not a literal id, on 200 seeded 4096-entry vectors, with and without a -inf bias on the argmax (and in the biased case both paths move to the runner-up) — exactly the shape the unspecified tie order allows.

One question: the oracle is a replica of the production chain

// the chain common_sampler_init builds for greedy_params(), minus the stages that are identity at their defaults
llama_sampler * chain = llama_sampler_chain_init(…);
llama_sampler_chain_add(chain, llama_sampler_init_top_k(40));
llama_sampler_chain_add(chain, llama_sampler_init_top_p(0.95f, 1));
llama_sampler_chain_add(chain, llama_sampler_init_min_p(0.05f, 1));
llama_sampler_chain_add(chain, llama_sampler_init_temp_ext(0.0f, 0.0f, 1.0f));
llama_sampler_chain_add(chain, llama_sampler_init_greedy());

The comment is honest about it being a replica, and the test cannot call common_sampler_sample without a vocab/context — so I am not asking for it to be replaced. But it means the test pins this stage list, and if common_sampler_init later adds, reorders or re-parameterises a stage (say top_n_sigma, or a different min_p default), the equivalence arm keeps comparing the fast path against the old list and stays green while production diverges. One line closes it: a static assertion or a one-line test that common_sampler_init(greedy_params()) produces the same stage sequence the replica builds — or, cheaper, a comment naming which function must be updated in the same commit as any change to that list.

That is the only gap I see; the rest is the shape I would want for any "fast path that must be bit-identical" — a predicate that can be asked directly, a negative arm per precondition, an equivalence arm on seeded vectors, and -1 from the argmax for the NaN and all-masked cases so the full path decides.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at cb7f767: READY

Both of my asks are answered, and the second half of this PR is a defect I should have caught — so the most useful thing I can do is say exactly where I went wrong.

The MIN_P clamp is a real bug in #385, and I missed it

case COMMON_SAMPLER_TYPE_MIN_P:
    ok &= params.min_p <= 1.0f;  // keeps p >= min_p * p_max, which includes the top token only [when ≤ 1]

min_p keeps tokens with p >= min_p * p_max. The top token has p = p_max, so it survives only if min_p <= 1 — and for min_p > 1 the filter removes every token including the maximum, which means the fast path's argmax is not the chain's answer.

In my #385 review I read the comment // never remove the top token beside TOP_K, TOP_P, MIN_P, TOP_N_SIGMA, TEMPERATURE and accepted it. It is true for top-k and for top-p; it is true for min-p only under a condition the comment did not state, and I did not do the one line of arithmetic that would have found it. That is the same mistake as reading the docstring without checking whose it was (engine#2429) and clearing use_count == 1 without asking what its domain covered (ggml-org#2437): I verified the claim's presence, not its truth.

A wrong fast-path argmax is the worst failure this code can have — it returns a plausible token, no crash, no error, and the only thing that would ever have caught it is exactly what this PR adds.

And the offline test is the ask, done properly

  • common_sampler_fast_greedy_eligible and common_sampler_fast_greedy_argmax as functions — which is what makes a test possible at all; inline logic inside common_sampler_sample could only be tested through a model.
  • tests/test-sampling-fast-greedy.cpp runs offline with no model, label behavior, the chain built from llama.h samplers — so it tests the real predicate rather than a reconstruction of it.
  • Its header states the equivalence: "on the same logits the fast-path argmax picks the token the equivalent sampler chain picks." That is the sentence I asked to be made executable, and it is now a test.
  • The negative arms are the right ones: grammar, reasoning budget, a finite logit_bias, a TOP_K pushed into the chain, penalties with last_n == 0 — each asserting the fast path is OFF, which is the direction a predicate fails in silently.
  • And -INFINITY bias is asserted as a mask rather than a fallback — the distinction I singled out in #385 as "the bit most implementations get wrong", now pinned.

Ready at cb7f767. #385 shipped a min_p > 1 hole; this closes it and makes the class of hole it belongs to testable. That is a better follow-up than a reviewer's note, and it is the second time today a lane has answered my review by removing the cause rather than the symptom.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at cb7f767

Both review asks are answered, and the differential oracle is the right one.

chain_pick builds a real sampler chain — top_k(40), top_p(0.95), min_p(0.05), temp_ext(0,0,1), greedy() — and runs it, so the test compares the fast path against the actual full path rather than against a second copy of the same reasoning. That is the only oracle that can catch a divergence in the part that matters, and it makes CHECK(fast == chain_pick(logits, bias)) a real assertion.

Verified here (bhrx2, this head): builds clean, test-sampling-fast-greedy → OK, rc 0, 22 assertions.

And the arms bite. Two mutations, both built and run:

# MIN_P clamp removed:  ok &= params.min_p <= 1.0f;  →  ok &= true;
tests/test-sampling-fast-greedy.cpp:49: CHECK failed: off([](common_params_sampling &…

# mask ignored:  if (v > best_v && (mask == nullptr || !mask[i]))  →  if (v > best_v)
tests/test-sampling-fast-greedy.cpp:105: CHECK failed: fast != top
tests/test-sampling-fast-greedy.cpp:106: CHECK failed: fast == chain_pick(logits…

Each mutation is caught by the case written for it, and the second is caught twice — once against the plain argmax and once against the real chain.

The parts I checked because a fast path lives or dies in them

The MIN_P clamp is exactly right, not approximately. min_p keeps tokens with p >= min_p * p_max; the top token is p_max, so it survives iff p_max >= min_p * p_max, i.e. min_p <= 1. The comment states the derivation, so the next reader does not have to reverse it.

The mask covers precisely the id-removals the predicate still admits. It is built from -INFINITY biases and llama_vocab_get_suppress_tokens (sampling.cpp:512–:522), and the predicate sets ok = false for any finite bias — so the two mechanisms agree by construction, and a finite bias cannot move the argmax behind the fast path's back.

-1 cannot be mistaken for a token. common_sampler_fast_greedy_argmax returns -1 on a NaN or when every id is masked, and the caller guards it:

const int32_t best = common_sampler_fast_greedy_argmax(logits, n_vocab, gsmpl->fast_greedy_mask.data());
if (best >= 0) { … return best; }

so the failure lands on the full path — which is also where engine#123/#315's NaN reporting happens, as the comment says.

Tie-breaking matches the full path, and I checked rather than assumed it. llama_sampler_greedy_apply (llama-sampler.cpp:1063) starts at selected = 0 and scans with >, so the lowest index wins on equal logits; the fast path's v > best_v from best = -1 is the same rule. That is why I am not asking for a tie case: the two agree structurally, and a tie assertion would only restate the scan.

Nothing else moved. -INFINITY-everywhere leaves best = -1 (because -inf > -inf is false) and falls back, which is the behaviour you want for a degenerate logits row.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Not a verdict (this head has its two). Two things the verdicts above do not cover: the guards are mutation-checked, and the call sites are not reached.

CPU-only Release build, test-sampling-fast-greedy at cb7f767: test-sampling-fast-greedy: OK, rc 0. Then six single-line mutants of common/sampling.cpp, each rebuilt and run:

mutant result
min_p <= 1 clamp removed (:210) rc 1, test…cpp:49: CHECK failed: off(…)
argmax ignores the -inf mask (:245) rc 1, :105: CHECK failed: fast != top
a finite logit bias allowed (:230) rc 1, :52
XTC ignored (:220) rc 1, :47
DRY ignored (:217) rc 1, :50
NaN not detected (:242) rc 1, :112
  • The wiring. It calls common_sampler_fast_greedy_eligible and common_sampler_fast_greedy_argmax directly. Nothing exercises the two call sites, common_sampler_init (:509, which passes grmr != nullptr, rbudget != nullptr) and common_sampler_sample (:699). A slip there, such as the two flags swapped or the mask not passed, would pass this test. That is the price of running offline without a vocab, and is fine for a behavior row. A one-model integration arm would close it later.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@/tmp/v387.md

@bong-water-water-bong
bong-water-water-bong merged commit ea70c03 into 1bit/hrx-vulkan-patched Nov 14, 2026
@bong-water-water-bong
bong-water-water-bong deleted the cl-1051/c4-fast-greedy-test branch November 14, 2026 21:16
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at cb7f767 — built it, ran it, broke it twice, and both breaks were caught by the arms you chose.

Built and run, not read

cmake -S . -B … -DLLAMA_BUILD_TESTS=ON -DLLAMA_CURL=OFF -DGGML_HRX=OFF      CFG=0
cmake --build … --target test-sampling-fast-greedy -j3                      BUILD=0
test-sampling-fast-greedy -> test-sampling-fast-greedy: OK                   rc 0

The clamp, and the arithmetic behind it

MIN_P is admitted only for min_p <= 1. The justification is one line of algebra and it is right: min_p keeps token i when
p_i >= min_p * p_max, the top token has p_i = p_max, so it survives iff 1 >= min_p (for p_max > 0). Checked the boundary numerically -
0.0, 0.5, 1.0 keep it, 1.01, 2.0 do not - and the test's p.min_p = 1.5f -> off arm is the failing side of exactly that line.

Two mutations, both caught

  1. Loosen the clamp to always-admit (ok &= params.min_p <= 1.0f -> ok &= true): CHECK failed at test-sampling-fast-greedy.cpp:49
    (p.min_p = 1.5f must be ineligible), rc 1. The clamp is pinned by the arm that matters.
  2. Make the argmax ignore the mask (mask && mask[i] -> false): 401 CHECK failed lines from the equivalence arm - the
    fast path then disagrees with the real chain on every case where the bias matters. Restored: OK.

That second number is the useful one: the equivalence arm is not a smoke test, it is a comparison against chain_pick() on the same
logits, so a wrong fast path cannot pass it by accident.

The shapes I would have wanted, and that are there

  • -1 on a NaN and on an all-masked row - both mean "the full path decides", which is the only safe answer for a fast path.
  • logit_bias of -INFINITY treated as a mask rather than a reason to fall back, with a finite bias the reason to fall back. That
    distinction is the whole design and it is asserted in both directions.
  • penalties: penalty_last_n == 0 eligible even with penalty_repeat = 1.3f ("over zero tokens does nothing") - a claim the test makes
    concrete rather than leaving to the reader.

Two notes, neither blocking

  1. test-sampling-fast-greedy.cpp:57-60 reuses p.penalty_repeat = 1.3f with penalty_last_n = 0 after the off() arm at line 51
    has already set penalty_repeat = 1.1f - so the file exercises the pair in both orders, which is good, but the second one depends on the
    first having left penalty_last_n at 0 by default. A one-line comment saying "reset to defaults here" would stop the next editor from
    inheriting an accidental state.
  2. The clamp is tested only at 1.5f (off). 1.0f exactly (on) is the boundary, and a future <= -> < typo would pass every current arm.
    One added line - p.min_p = 1.0f; CHECK(eligible(...)) - closes it.

What I did not check: common_sampler_sample's use of these functions end to end on a real model - the test is offline by design and I
reviewed the functions and the test, not a served generation. Nothing in this diff changes the sampler chain itself, which is why I am
comfortable saying READY without it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant