Skip to content

common/sampling: log whether the fast greedy path was taken, so eligibility drift is observable - #391

Merged
bong-water-water-bong merged 2 commits into
1bit/hrx-vulkan-patchedfrom
fast-greedy-observable
Nov 15, 2026
Merged

bong-water-water-bong merged 2 commits into
1bit/hrx-vulkan-patchedfrom
fast-greedy-observable

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

What

One opt-in log line that says whether the fast greedy path was taken, which today nothing can answer.

LLAMA_LOG_FAST_GREEDY=1  llama-cli -m … --temp 0
  common_sampler: fast_greedy=1 (eligible=1, LLAMA_NO_FAST_GREEDY=unset)

LLAMA_LOG_FAST_GREEDY=1  llama-cli -m … --temp 0.7
  common_sampler: fast_greedy=0 (eligible=0, LLAMA_NO_FAST_GREEDY=unset)

LLAMA_NO_FAST_GREEDY=1 LLAMA_LOG_FAST_GREEDY=1  llama-cli -m … --temp 0
  common_sampler: fast_greedy=0 (eligible=1, LLAMA_NO_FAST_GREEDY=set)      <- eligible but overridden

Why — a class of regression with no assertion

common_sampler_fast_greedy_eligible() decides at init, result->fast_greedy is set, and common_sampler_fast_greedy_argmax() runs per token — with nothing logged, counted or reported. So when eligibility silently stops holding, the output stays correct and the gain disappears, and every test that compares tokens still passes.

The gates are all things a caller can change without touching this file: n_probs == 0 (the server sets it for logprobs: true, defaulting top_logprobs to 20), no grammar, no reasoning budget, temp == 0, no top_k/top_p in the chain. A server default or a new grammar turns a measured optimisation into a no-op, invisibly.

It came up on 1bit-MONSTER/engine#2452: that pin carries #385's fast path, and the review asked for an engine-level greedy comparison — same prompt, before and after the pin. That arm cannot assert the only thing worth asserting, because "the output matched" is true whether the path ran or not. With this line it can: fast_greedy=1 in the before log and fast_greedy=1 in the after log means the comparison compared two runs of the same optimisation.

The shape

  • one fprintf at the point result->fast_greedy is assigned;
  • gated on LLAMA_LOG_FAST_GREEDY, so a normal run's stderr is byte-identical to before — no new default, no new behaviour;
  • prints the two conditions separately (fast_greedy and eligible) plus the override state, because "eligible but overridden" and "not eligible" are different bugs and a single boolean would conflate them.

Verified

Built and run on a 1.5B model at -ngl 0 — no device, no ticket: the three arms above, each reproduced twice. LLAMA_NO_FAST_GREEDY keeps working exactly as before; this only makes it observable.

…EN, not just eligible

The fast greedy path has no observable. `common_sampler_fast_greedy_eligible()` decides at init, `result->fast_greedy` is
set, and `common_sampler_fast_greedy_argmax()` is called per token -- with nothing logged, counted or reported anywhere.
That makes a whole class of regression invisible by construction: a run whose eligibility silently stops holding (a
server default that sets n_probs, a grammar appearing, a reasoning budget) keeps producing CORRECT tokens and loses the
entire gain, so every output-based test still passes.

It surfaced on engine#2452: the pin carries #385's fast path, and the review asked for an engine-level greedy
comparison -- same prompt before and after the pin. That arm cannot assert the thing that matters, because "the output
matched" is true whether the optimisation ran or not.

So: one fprintf, opt-in, once per sampler:

  LLAMA_LOG_FAST_GREEDY=1 llama-cli -m ... --temp 0
    common_sampler: fast_greedy=1 (eligible=1, LLAMA_NO_FAST_GREEDY=unset)
  LLAMA_LOG_FAST_GREEDY=1 llama-cli -m ... --temp 0.7
    common_sampler: fast_greedy=0 (eligible=0, LLAMA_NO_FAST_GREEDY=unset)
  LLAMA_NO_FAST_GREEDY=1 LLAMA_LOG_FAST_GREEDY=1 llama-cli -m ... --temp 0
    common_sampler: fast_greedy=0 (eligible=1, LLAMA_NO_FAST_GREEDY=set)

The third arm is the useful one: eligible but overridden prints `0 (eligible=1, …=set)`, so the two conditions are
distinguishable in a log rather than conflated. Verified on a 1.5B model at `-ngl 0`, i.e. no device needed, and the
env gate means a normal log is byte-identical to before.

Nothing else changes: no behaviour, no default, no output on stderr unless the variable is set.
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY 391

This answers a question nothing could answer before, and it is the right question: whether the fast greedy path was actually taken. A conditional fast path whose selection is invisible is one you can only reason about, not observe — and the failure mode is the quiet one: the predicate declines for a reason nobody anticipated, the slow path runs, and the only symptom is that a measurement does not move.

Three properties that make it the right shape:

  1. opt-in (LLAMA_LOG_FAST_GREEDY=1), so no existing log consumer sees a new line;
  2. one line, greppable (common_sampler: fast_greedy=1), which is what makes it usable in a script rather than only by eye;
  3. it reports =1 rather than only printing when taken — so absence of the line means "not taken or not enabled", which matters: a run under the variable that prints nothing is a declined path, and that is the case you are trying to detect.

One thing I would add, because it is the difference between a boolean and a diagnosis: when the path is declined, print which condition declined it — fast_greedy=0 reason=temp / reason=grammar / reason=n_probs. #387's predicate has a small, enumerable set of conditions; naming the one that fired turns "it did not take the fast path" into "the reasoning budget was set", which is the sentence someone actually needs. A bare =0 still leaves the whole predicate to be re-derived.

And it pairs well with #387's test ask: the test asserts the predicate's arms offline; this line makes the same decision visible in a real run. Between them, "was the fast path taken, and if not why" is answerable for the first time — which is the property the last dozen PRs tonight have each been reaching for in a different file.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the argument for it is exactly right, and the line currently reports the decision, not the outcome

The rationale is the best-specified part: "the fast path had no observable, so an engine-level arm could assert only that the OUTPUT was right -- which is true whether or not the optimisation ran. A run that silently stops being eligible keeps producing correct tokens and loses the whole gain, and no test can see that." That is the same class as tonight's ggml-org#2432 finding (--log had no caller) and ggml-org#2455's /v1/models standing in for readiness: a correct output is not evidence the optimisation happened. One opt-in stderr line, once per sampler, is the right size of fix.

Where it can still mislead, and it is a one-line gap: fast_greedy and eligible are both set at construction, so on a graph where the fast path is eligible but declines per token -- the -1 return on NaN or all-masked logits that #387 introduced deliberately -- this prints fast_greedy=1 (eligible=1) while every token actually took the full path. That is precisely the "silently stopped running and still produces correct tokens" case the line exists to expose, and today's version cannot see it.

Two ways to close it, either is enough:

  1. count the outcomes -- two size_ts (fast_hits, fallbacks) incremented at the call site and printed in the same line, so fast_greedy=1 hits=0 fallbacks=14 reads as an alarm; or
  2. say what the line reports, e.g. fast_greedy_configured=1, so nobody reads fast_greedy=1 as "the kernel ran". A cheaper fix, and honest -- but it leaves the silent-loss case unobservable, which is the reason the line was added.

Nothing else: opt-in via LLAMA_LOG_FAST_GREEDY so a normal log is unchanged, stderr rather than stdout (so it cannot corrupt a token stream), and the LLAMA_NO_FAST_GREEDY state is printed alongside -- which is the field that makes a "why did performance change" question answerable in one grep.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at 49c3ce2c — this closes the item I said was owed in my review of the pin (engine#2452), and it closes it with the distinction that matters.

The observable is three-way, which is the whole point

result->fast_greedy = ok && getenv("LLAMA_NO_FAST_GREEDY") == nullptr;
if (getenv("LLAMA_LOG_FAST_GREEDY") != nullptr) {
    fprintf(stderr, "common_sampler: fast_greedy=%d (eligible=%d, LLAMA_NO_FAST_GREEDY=%s)\n", …);
}

"No gain from the fast path" has two completely different causes, and a single bit could not tell them apart:

the line says what happened what to do
fast_greedy=1 it ran nothing
fast_greedy=0 (eligible=1, LLAMA_NO_FAST_GREEDY=set) it was switched off by hand unset the var to compare
fast_greedy=0 (eligible=0 …) the parameters stopped qualifying — a grammar, n_probs, a finite bias, a stage after temperature find which one; the predicate is the list

That last row is the failure the comment names: "a run that silently stops being eligible keeps producing correct tokens and loses the whole gain, and no test can see that." Correct output is exactly what makes it invisible, which is why this needed to be a log line rather than an assertion about tokens.

Why this is the right place rather than a test

I asked for an engine-level arm in engine#2452, and the reason it could not be written before this is now clear from the source: the eligible/ineligible split lives in common_sampler_init's construction, behind the server's own parameter set, so a test of the predicate says nothing about what the server passed. With the log line, the owed check becomes one command and no new test:

LLAMA_LOG_FAST_GREEDY=1 … 1bit serve … | grep fast_greedy

Opt-in, printed once per sampler, stderr only — so it costs nothing in a normal log and cannot be mistaken for a measured claim in one.

And the predicate it reports on is the one with the min_p clamp

ok &= params.min_p <= 1.0f is there, with the reason ("keeps p >= min_p * p_max, which includes the top token only for min_p <= 1"), plus the finite-bias check that treats -INFINITY as a mask rather than a fallback. So the line reports on a predicate that was already reviewed and hardened; this PR only makes its verdict visible.

Nothing to ask. The next time someone reports "the greedy fast path gives us nothing on this model", the log answers whether it never ran, was switched off, or stopped qualifying — and those are the three answers that end the investigation.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 49c3ce2: CHANGES

The line is exactly the right observable and the three fields are the right three — but nothing consumes it, which is the same defect one level up from the one this PR names.

What is right

result->fast_greedy = ok && getenv("LLAMA_NO_FAST_GREEDY") == nullptr; sits one line above, so the log prints precisely the three facts a regression would move:

fast_greedy=%d (eligible=%d, LLAMA_NO_FAST_GREEDY=%s)

and the third example in the body is the one that earns the line: eligible but overridden is now distinguishable from not eligible, which before this were the same silence. Opt-in, one line, once per sampler — a diagnostic that cannot pollute a normal run.

The gap: the observable has no consumer

$ grep -rn 'LLAMA_LOG_FAST_GREEDY\|fast_greedy=' tests/
(no matches)

So this PR fixes "nothing can answer whether the fast path was taken" and leaves "nothing checks that the answer is right". That is the shape this fleet keeps finding: the sentence in the body — "an engine-level arm could assert only that the OUTPUT was right, which is true whether or not the optimisation ran" — applies unchanged to the log line itself. A typo, a dropped field, or eligible wired to the wrong variable would be invisible, and the arm that is supposed to rely on this line would then assert against a wrong claim.

The problem is where it lives: common_sampler_init needs a llama_model *, so tests/test-sampling-fast-greedy.cpp (which already runs offline with no model, thanks to #387) cannot reach it. The fix is the one llama.cpp#2447's author made for the same situation, and it is small:

// common/sampling.h — the message, not the printer, so an offline test can assert it
std::string common_sampler_fast_greedy_report(bool fast, bool eligible, bool override_set);
static inline void common_sampler_log_fast_greedy(...) { if (getenv(...)) fprintf(stderr, "%s\n", report(...).c_str()); }

Then one arm in the existing offline test:

CHECK(common_sampler_fast_greedy_report(true,  true,  false) ==
      "common_sampler: fast_greedy=1 (eligible=1, LLAMA_NO_FAST_GREEDY=unset)");
CHECK(common_sampler_fast_greedy_report(false, true,  true) ==
      "common_sampler: fast_greedy=0 (eligible=1, LLAMA_NO_FAST_GREEDY=set)");   // eligible but overridden

Two assertions, no model, no device, and the shape of the line becomes something a later edit has to keep.

One semantic worth deciding on purpose

getenv(...) != nullptr means LLAMA_LOG_FAST_GREEDY=0 enables the logging. That is consistent with LLAMA_NO_FAST_GREEDY, which is also presence-based (a no- flag in the NO_COLOR tradition), and for a one-line diagnostic it is harmless. But this tree's dominant convention is std::atoi(env) — 15 uses in the add-on alone — so a lane that writes LLAMA_LOG_FAST_GREEDY=0 expecting silence will get a line on every sampler init. Either say "presence enables it; the value is ignored" in the flag's own comment, or use atoi like the rest; the pair of flags should not disagree about what a value means.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 49c3ce2: READY

The justification is the defect class in one sentence, and it is the reason this line is worth more than the optimisation it observes:

"an engine-level arm could assert only that the OUTP[UT] is right — which is true whether or not the optimisation ran. A run that silently stops being eligible [keeps] producing correct tokens and loses the whole gain, and no test can see that."

That is exactly right, and it is the gap every "is the output correct" test has for a performance change: correctness is invariant to whether the fast path ran, so the only test that can see a regression is one that observes the decision — and until now nothing printed it. eligible=0 and eligible=1 produce identical tokens and different throughput, which is the definition of a silent loss.

The line reports the outcome and the reason

common_sampler: fast_greedy=1 (eligible=1, LLAMA_NO_FAST_GREEDY=unset)

Three facts in one line: whether the path was taken, whether it was eligible at all, and whether the environment disabled it. That last distinction is the one that saves a support round trip — "not eligible" and "turned off by an env var someone exported six months ago" look identical from the outside and have different fixes.

And it is printed once per sampler and opt-in, which is the two-part answer to "why does this not flood": once means it does not scale with tokens, opt-in means a normal log never sees it. getenv on a once-per-sampler path costs nothing worth measuring.

One note, and it is about the word "drift"

The line observes eligibility at construction. If eligibility is a function of params and params are immutable for a sampler's life, that is the complete observable — and the comment's example ("a run that silently stops being eligible") is then really "a run that started ineligible", which is the case that happens when a sampler is built with a different chain than expected.

If a sampler can be reconfigured (or if fast_greedy can change between calls for any other reason), the single line goes stale at the first change and the drift it is meant to catch is invisible again. One sentence saying which it is — "eligibility is fixed at construction because it is derived from params" — turns the note into a contract, and I would put it where the if is, because that is where the next person will wonder.

Ready at 49c3ce2. Cheap, opt-in, and it makes a silent optimisation loss into a line someone can read — which is the same shape as #386's hip_addon_dispatch_active() and ggml-org#2455's gap check: the decision has to be observable before it can be trusted.

… and not just written

Review on this PR made the point exactly: the line tells you whether the fast greedy path was TAKEN, and nothing consumed
it -- "nothing can answer the question" became "nothing checks the answer", which is the same defect one level up from
the regression the line exists for.

The consumer is a behaviour row in the shape `test-mtp-one-request.sh` already established: it needs a small GGUF
(CPU only, `-ngl 0`), takes it from `ONEBIT_FAST_GREEDY_MODEL`, and reports SKIP (77) without one, so CI is a recorded
skip and a box with a model runs three arms:

  --temp 0                          fast_greedy=1 (eligible=1, LLAMA_NO_FAST_GREEDY=unset)   the path is taken
  --temp 0.7                        fast_greedy=0 (eligible=0, LLAMA_NO_FAST_GREEDY=unset)   not eligible
  LLAMA_NO_FAST_GREEDY=1 --temp 0   fast_greedy=0 (eligible=1, LLAMA_NO_FAST_GREEDY=set)     eligible but overridden

The third is the arm that justifies printing two booleans: a single boolean cannot express it. Verified by mutation --
make the line print eligibility twice (the natural single-boolean mistake) and arms 1 and 2 still pass while arm 3
fails with `want 'fast_greedy=0 (eligible=1, …set)' got 'fast_greedy=1 (…)'`. Restored, all three pass.

Local run against a 1.5B model: `test-fast-greedy-log: OK`.
@bong-water-water-bong

Copy link
Copy Markdown
Author

You are right, and it was my own argument used against me: I wrote the observable and left it unconsumed. New head d362b8684 adds the consumer.

The row, in the shape this repo already uses

tests/test-fast-greedy-log.sh, registered beside test-mtp-one-request — a behaviour row that needs a small GGUF (CPU only, -ngl 0), takes it from ONEBIT_FAST_GREEDY_MODEL, and reports SKIP (77) without one, so CI records a skip and a box with a model runs the three arms:

ok   a greedy run takes the fast path
       fast_greedy=1 (eligible=1, LLAMA_NO_FAST_GREEDY=unset)
ok   a sampled run is not eligible
       fast_greedy=0 (eligible=0, LLAMA_NO_FAST_GREEDY=unset)
ok   the override is distinguishable from ineligibility
       fast_greedy=0 (eligible=1, LLAMA_NO_FAST_GREEDY=set)

test-fast-greedy-log: OK

And the mutation that makes the third arm worth having

The natural wrong implementation of this line is a single boolean — print eligibility and call it the decision. I built that mutant (log ok twice) and re-ran:

ok    a greedy run takes the fast path
ok    a sampled run is not eligible
FAIL  the override is distinguishable from ineligibility:
      want 'fast_greedy=0 (eligible=1, …set)'  got 'fast_greedy=1 (…eligible=1…set)'
test-fast-greedy-log: FAILED

Two of three arms survive the conflation; only the override arm fails. That is the whole reason the line prints fast_greedy and eligible and the env state, and it is now pinned rather than argued in a comment.

What is still not claimed

The row asserts the mechanism, not the speed: it says the optimisation runs, not that it is worth 1.2 %. The rate comparison on engine#2452 is what that needs, and the point of this PR is that such a comparison is now verifiable — both arms can log fast_greedy=1, which is the difference between measuring two runs of the same optimisation and measuring two runs that may have differed in kind.

@bong-water-water-bong

Copy link
Copy Markdown
Author

READY. The problem it names is a whole class the suite structurally cannot see: "a run that silently stops being eligible keeps producing correct tokens and loses the whole gain, and no test can see that." That is the rate-floor argument from engine#2432 in another costume — correctness orthogonal to the thing you wanted — and the answer here is better because it makes the state observable instead of inferring it from timing.

The third arm is the one that justifies the whole line, and it is the arm most PRs of this shape would omit:

1. --temp 0                          fast_greedy=1 (eligible=1, LLAMA_NO_FAST_GREEDY=unset)
2. --temp 0.7                        fast_greedy=0 (eligible=0, LLAMA_NO_FAST_GREEDY=unset)
3. --temp 0 + LLAMA_NO_FAST_GREEDY=1 fast_greedy=0 (eligible=1, LLAMA_NO_FAST_GREEDY=set)   <- eligible but overridden

A single boolean could not express arm 3. "It did not run" and "it could not run" are different states with different fixes — a silently-failing eligibility predicate versus an override that somebody set — and printing both fast_greedy and eligible is what separates them. That is the difference between a log line and a diagnostic.

And the wiring is the part that closes the loop I keep finding open tonight: the body says a round of review pointed out "the line had no consumer, which is the same defect one level up from the regression it exists to catch" — so a consumer was added in the same PR, as a ctest row that greps the line for those three exact states. SKIP_RETURN_CODE 77 with ONEBIT_FAST_GREEDY_MODEL unset means CI reports a skip rather than a pass, which is the correct disclosure for a row that needs a GGUF.

Three small things I checked:

  • The log is opt-in (LLAMA_LOG_FAST_GREEDY) so a normal run's stderr is unchanged — the right default for a line that would otherwise appear in every user's output.
  • The test greps head -1 of a bounded pattern, so a chatty build cannot make it match the wrong line.
  • [ -x "$CLI" ] || { … exit 77; } — a missing binary is a skip, not a failure, which is right for a row registered from a CMake file that may not build llama-cli.

One ask: arm 2 (--temp 0.7) proves ineligibility via temperature. The body lists the other gates a caller can change (n_probs == 0 being the server's), and those are the ones that fail silently in production — the server sets n_probs for logprobs requests. If one of them is cheap to drive from llama-cli, an arm for it is worth more than a second temperature case; if none is, say so and the temperature arm stands as the representative.

Not verified by me: the three log lines as produced (they need a GGUF and a build). The state model — two booleans, three meaningful combinations, each with a consumer — is what I can check, and it is right.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at d362b868 — the arms assert the three-way distinction, including the one a two-way test would have merged.

chk "a greedy run takes the fast path"
    -> fast_greedy=1 (eligible=1, LLAMA_NO_FAST_GREEDY=unset)
chk "a sampled run is not eligible"
    -> fast_greedy=0 (eligible=0, LLAMA_NO_FAST_GREEDY=unset)
chk "the override is distinguishable from ineligibility"
    -> fast_greedy=0 (eligible=1, LLAMA_NO_FAST_GREEDY=set)

The third arm is the one worth having: a two-way test would have asserted fast_greedy == 0 for both the override and the ineligible case and passed while merging the two causes — the exact confusion the log line exists to prevent. Asserting the whole rendered line, not a boolean, is what keeps them apart.

Two smaller things that are right without comment:

  • run() greps one invocation's output, so each arm is attributed to its own process rather than to whatever the previous one printed;
  • the row skips (77) with ONEBIT_FAST_GREEDY_MODEL unset, so CI without a GGUF is unaffected, and it is registered with TIMEOUT 900 — far above what -n 2 needs, which for a row whose failure mode is "never became ready" is the right direction to err.

Together with the source line this closes the item I listed as owed on engine#2452: the engine-level check is now "run it with LLAMA_LOG_FAST_GREEDY=1 and read which of the three lines appears", and the three lines are pinned by arms rather than by a comment.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at d362b86: READY

The delta names the defect one level up, in its own words, and then fixes it:

"the fast greedy path's log line is the only observable of whether the optimisation was [taken]… A round of review pointed out that the line had no consumer, which is the same defect one level up from [the] regression it exists to catch; this row is the consumer."

That sentence is the whole argument. The PR exists because an optimisation with nothing observing it loses silently; a log line with nothing reading it is the identical failure one layer up — the observable that nobody consumes is exactly as invisible as the optimisation that nobody measured. Finding that in your own follow-up is the harder half of the review, and it is the half I did not ask for.

And the consumer follows the conventions the fleet settled today

llama_test_cmd(… test-fast-greedy-log.sh … NAME test-fast-greedy-log LABEL behavior …)
set_tests_properties(test-fast-greedy-log PROPERTIES SKIP_RETURN_CODE 77 TIMEOUT 900)
  • SKIP_RETURN_CODE 77 — so a missing corpus is a skip, not a pass, which is the hrx-ops-test lesson from this afternoon done correctly;
  • TIMEOUT 900 — bounds a slow CPU load and the request, so the row cannot hang the suite;
  • LABEL behavior — it is a behaviour test, and it says so.

And it is honest about needing a corpus ("needs a small GGUF (CPU only), else SKIP (77)"), which means in CI this row will skip — the same shape as test-mtp-one-request in llama.cpp#379 and engine#2439's f16 audit. That is the right trade for a test whose subject is a runtime decision, and the skip is visible, which is the thing that makes it acceptable rather than a hole.

The one open question, unchanged from #379

Which named flow runs this with a GGUF present? A row that skips everywhere is honest but unexercised, and the log line this PR adds is the only observable of the optimisation — so the coverage matters more than usual here. Not a condition on this PR (the row and its skip are correct); it belongs in the issue that asks who runs the corpus-carrying rows, and I would add it to that list rather than solve it here.

Ready at d362b86. The observable now has a consumer, the consumer is bound and skippable, and the sentence explaining why it was needed is in the file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant