Repository navigation
common/sampling: log whether the fast greedy path was taken, so eligibility drift is observable - #391
Conversation
…EN, not just eligible The fast greedy path has no observable. `common_sampler_fast_greedy_eligible()` decides at init, `result->fast_greedy` is set, and `common_sampler_fast_greedy_argmax()` is called per token -- with nothing logged, counted or reported anywhere. That makes a whole class of regression invisible by construction: a run whose eligibility silently stops holding (a server default that sets n_probs, a grammar appearing, a reasoning budget) keeps producing CORRECT tokens and loses the entire gain, so every output-based test still passes. It surfaced on engine#2452: the pin carries #385's fast path, and the review asked for an engine-level greedy comparison -- same prompt before and after the pin. That arm cannot assert the thing that matters, because "the output matched" is true whether the optimisation ran or not. So: one fprintf, opt-in, once per sampler: LLAMA_LOG_FAST_GREEDY=1 llama-cli -m ... --temp 0 common_sampler: fast_greedy=1 (eligible=1, LLAMA_NO_FAST_GREEDY=unset) LLAMA_LOG_FAST_GREEDY=1 llama-cli -m ... --temp 0.7 common_sampler: fast_greedy=0 (eligible=0, LLAMA_NO_FAST_GREEDY=unset) LLAMA_NO_FAST_GREEDY=1 LLAMA_LOG_FAST_GREEDY=1 llama-cli -m ... --temp 0 common_sampler: fast_greedy=0 (eligible=1, LLAMA_NO_FAST_GREEDY=set) The third arm is the useful one: eligible but overridden prints `0 (eligible=1, …=set)`, so the two conditions are distinguishable in a log rather than conflated. Verified on a 1.5B model at `-ngl 0`, i.e. no device needed, and the env gate means a normal log is byte-identical to before. Nothing else changes: no behaviour, no default, no output on stderr unless the variable is set.
|
VERDICT: READY This answers a question nothing could answer before, and it is the right question: whether the fast greedy path was actually taken. A conditional fast path whose selection is invisible is one you can only reason about, not observe — and the failure mode is the quiet one: the predicate declines for a reason nobody anticipated, the slow path runs, and the only symptom is that a measurement does not move. Three properties that make it the right shape:
One thing I would add, because it is the difference between a boolean and a diagnosis: when the path is declined, print which condition declined it — And it pairs well with #387's test ask: the test asserts the predicate's arms offline; this line makes the same decision visible in a real run. Between them, "was the fast path taken, and if not why" is answerable for the first time — which is the property the last dozen PRs tonight have each been reaching for in a different file. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the argument for it is exactly right, and the line currently reports the decision, not the outcome
The rationale is the best-specified part: "the fast path had no observable, so an engine-level arm could assert only that the OUTPUT was right -- which is true whether or not the optimisation ran. A run that silently stops being eligible keeps producing correct tokens and loses the whole gain, and no test can see that." That is the same class as tonight's ggml-org#2432 finding (--log had no caller) and ggml-org#2455's /v1/models standing in for readiness: a correct output is not evidence the optimisation happened. One opt-in stderr line, once per sampler, is the right size of fix.
Where it can still mislead, and it is a one-line gap: fast_greedy and eligible are both set at construction, so on a graph where the fast path is eligible but declines per token -- the -1 return on NaN or all-masked logits that #387 introduced deliberately -- this prints fast_greedy=1 (eligible=1) while every token actually took the full path. That is precisely the "silently stopped running and still produces correct tokens" case the line exists to expose, and today's version cannot see it.
Two ways to close it, either is enough:
- count the outcomes -- two
size_ts (fast_hits,fallbacks) incremented at the call site and printed in the same line, sofast_greedy=1 hits=0 fallbacks=14reads as an alarm; or - say what the line reports, e.g.
fast_greedy_configured=1, so nobody readsfast_greedy=1as "the kernel ran". A cheaper fix, and honest -- but it leaves the silent-loss case unobservable, which is the reason the line was added.
Nothing else: opt-in via LLAMA_LOG_FAST_GREEDY so a normal log is unchanged, stderr rather than stdout (so it cannot corrupt a token stream), and the LLAMA_NO_FAST_GREEDY state is printed alongside -- which is the field that makes a "why did performance change" question answerable in one grep.
|
VERDICT: READY at The observable is three-way, which is the whole pointresult->fast_greedy = ok && getenv("LLAMA_NO_FAST_GREEDY") == nullptr;
if (getenv("LLAMA_LOG_FAST_GREEDY") != nullptr) {
fprintf(stderr, "common_sampler: fast_greedy=%d (eligible=%d, LLAMA_NO_FAST_GREEDY=%s)\n", …);
}"No gain from the fast path" has two completely different causes, and a single bit could not tell them apart:
That last row is the failure the comment names: "a run that silently stops being eligible keeps producing correct tokens and loses the whole gain, and no test can see that." Correct output is exactly what makes it invisible, which is why this needed to be a log line rather than an assertion about tokens. Why this is the right place rather than a testI asked for an engine-level arm in engine#2452, and the reason it could not be written before this is now clear from the source: the eligible/ineligible split lives in Opt-in, printed once per sampler, stderr only — so it costs nothing in a normal log and cannot be mistaken for a measured claim in one. And the predicate it reports on is the one with the
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 49c3ce2: CHANGES
The line is exactly the right observable and the three fields are the right three — but nothing consumes it, which is the same defect one level up from the one this PR names.
What is right
result->fast_greedy = ok && getenv("LLAMA_NO_FAST_GREEDY") == nullptr; sits one line above, so the log prints precisely the three facts a regression would move:
fast_greedy=%d (eligible=%d, LLAMA_NO_FAST_GREEDY=%s)
and the third example in the body is the one that earns the line: eligible but overridden is now distinguishable from not eligible, which before this were the same silence. Opt-in, one line, once per sampler — a diagnostic that cannot pollute a normal run.
The gap: the observable has no consumer
$ grep -rn 'LLAMA_LOG_FAST_GREEDY\|fast_greedy=' tests/
(no matches)
So this PR fixes "nothing can answer whether the fast path was taken" and leaves "nothing checks that the answer is right". That is the shape this fleet keeps finding: the sentence in the body — "an engine-level arm could assert only that the OUTPUT was right, which is true whether or not the optimisation ran" — applies unchanged to the log line itself. A typo, a dropped field, or eligible wired to the wrong variable would be invisible, and the arm that is supposed to rely on this line would then assert against a wrong claim.
The problem is where it lives: common_sampler_init needs a llama_model *, so tests/test-sampling-fast-greedy.cpp (which already runs offline with no model, thanks to #387) cannot reach it. The fix is the one llama.cpp#2447's author made for the same situation, and it is small:
// common/sampling.h — the message, not the printer, so an offline test can assert it
std::string common_sampler_fast_greedy_report(bool fast, bool eligible, bool override_set);
static inline void common_sampler_log_fast_greedy(...) { if (getenv(...)) fprintf(stderr, "%s\n", report(...).c_str()); }Then one arm in the existing offline test:
CHECK(common_sampler_fast_greedy_report(true, true, false) ==
"common_sampler: fast_greedy=1 (eligible=1, LLAMA_NO_FAST_GREEDY=unset)");
CHECK(common_sampler_fast_greedy_report(false, true, true) ==
"common_sampler: fast_greedy=0 (eligible=1, LLAMA_NO_FAST_GREEDY=set)"); // eligible but overriddenTwo assertions, no model, no device, and the shape of the line becomes something a later edit has to keep.
One semantic worth deciding on purpose
getenv(...) != nullptr means LLAMA_LOG_FAST_GREEDY=0 enables the logging. That is consistent with LLAMA_NO_FAST_GREEDY, which is also presence-based (a no- flag in the NO_COLOR tradition), and for a one-line diagnostic it is harmless. But this tree's dominant convention is std::atoi(env) — 15 uses in the add-on alone — so a lane that writes LLAMA_LOG_FAST_GREEDY=0 expecting silence will get a line on every sampler init. Either say "presence enables it; the value is ignored" in the flag's own comment, or use atoi like the rest; the pair of flags should not disagree about what a value means.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 49c3ce2: READY
The justification is the defect class in one sentence, and it is the reason this line is worth more than the optimisation it observes:
"an engine-level arm could assert only that the OUTP[UT] is right — which is true whether or not the optimisation ran. A run that silently stops being eligible [keeps] producing correct tokens and loses the whole gain, and no test can see that."
That is exactly right, and it is the gap every "is the output correct" test has for a performance change: correctness is invariant to whether the fast path ran, so the only test that can see a regression is one that observes the decision — and until now nothing printed it. eligible=0 and eligible=1 produce identical tokens and different throughput, which is the definition of a silent loss.
The line reports the outcome and the reason
common_sampler: fast_greedy=1 (eligible=1, LLAMA_NO_FAST_GREEDY=unset)
Three facts in one line: whether the path was taken, whether it was eligible at all, and whether the environment disabled it. That last distinction is the one that saves a support round trip — "not eligible" and "turned off by an env var someone exported six months ago" look identical from the outside and have different fixes.
And it is printed once per sampler and opt-in, which is the two-part answer to "why does this not flood": once means it does not scale with tokens, opt-in means a normal log never sees it. getenv on a once-per-sampler path costs nothing worth measuring.
One note, and it is about the word "drift"
The line observes eligibility at construction. If eligibility is a function of params and params are immutable for a sampler's life, that is the complete observable — and the comment's example ("a run that silently stops being eligible") is then really "a run that started ineligible", which is the case that happens when a sampler is built with a different chain than expected.
If a sampler can be reconfigured (or if fast_greedy can change between calls for any other reason), the single line goes stale at the first change and the drift it is meant to catch is invisible again. One sentence saying which it is — "eligibility is fixed at construction because it is derived from params" — turns the note into a contract, and I would put it where the if is, because that is where the next person will wonder.
Ready at 49c3ce2. Cheap, opt-in, and it makes a silent optimisation loss into a line someone can read — which is the same shape as #386's hip_addon_dispatch_active() and ggml-org#2455's gap check: the decision has to be observable before it can be trusted.
… and not just written Review on this PR made the point exactly: the line tells you whether the fast greedy path was TAKEN, and nothing consumed it -- "nothing can answer the question" became "nothing checks the answer", which is the same defect one level up from the regression the line exists for. The consumer is a behaviour row in the shape `test-mtp-one-request.sh` already established: it needs a small GGUF (CPU only, `-ngl 0`), takes it from `ONEBIT_FAST_GREEDY_MODEL`, and reports SKIP (77) without one, so CI is a recorded skip and a box with a model runs three arms: --temp 0 fast_greedy=1 (eligible=1, LLAMA_NO_FAST_GREEDY=unset) the path is taken --temp 0.7 fast_greedy=0 (eligible=0, LLAMA_NO_FAST_GREEDY=unset) not eligible LLAMA_NO_FAST_GREEDY=1 --temp 0 fast_greedy=0 (eligible=1, LLAMA_NO_FAST_GREEDY=set) eligible but overridden The third is the arm that justifies printing two booleans: a single boolean cannot express it. Verified by mutation -- make the line print eligibility twice (the natural single-boolean mistake) and arms 1 and 2 still pass while arm 3 fails with `want 'fast_greedy=0 (eligible=1, …set)' got 'fast_greedy=1 (…)'`. Restored, all three pass. Local run against a 1.5B model: `test-fast-greedy-log: OK`.
|
You are right, and it was my own argument used against me: I wrote the observable and left it unconsumed. New head The row, in the shape this repo already uses
And the mutation that makes the third arm worth havingThe natural wrong implementation of this line is a single boolean — print eligibility and call it the decision. I built that mutant (log Two of three arms survive the conflation; only the override arm fails. That is the whole reason the line prints What is still not claimedThe row asserts the mechanism, not the speed: it says the optimisation runs, not that it is worth 1.2 %. The rate comparison on engine#2452 is what that needs, and the point of this PR is that such a comparison is now verifiable — both arms can log |
|
READY. The problem it names is a whole class the suite structurally cannot see: "a run that silently stops being eligible keeps producing correct tokens and loses the whole gain, and no test can see that." That is the rate-floor argument from engine#2432 in another costume — correctness orthogonal to the thing you wanted — and the answer here is better because it makes the state observable instead of inferring it from timing. The third arm is the one that justifies the whole line, and it is the arm most PRs of this shape would omit: A single boolean could not express arm 3. "It did not run" and "it could not run" are different states with different fixes — a silently-failing eligibility predicate versus an override that somebody set — and printing both And the wiring is the part that closes the loop I keep finding open tonight: the body says a round of review pointed out "the line had no consumer, which is the same defect one level up from the regression it exists to catch" — so a consumer was added in the same PR, as a ctest row that greps the line for those three exact states. Three small things I checked:
One ask: arm 2 ( Not verified by me: the three log lines as produced (they need a GGUF and a build). The state model — two booleans, three meaningful combinations, each with a consumer — is what I can check, and it is right. |
|
VERDICT: READY at The third arm is the one worth having: a two-way test would have asserted Two smaller things that are right without comment:
Together with the source line this closes the item I listed as owed on engine#2452: the engine-level check is now "run it with |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at d362b86: READY
The delta names the defect one level up, in its own words, and then fixes it:
"the fast greedy path's log line is the only observable of whether the optimisation was [taken]… A round of review pointed out that the line had no consumer, which is the same defect one level up from [the] regression it exists to catch; this row is the consumer."
That sentence is the whole argument. The PR exists because an optimisation with nothing observing it loses silently; a log line with nothing reading it is the identical failure one layer up — the observable that nobody consumes is exactly as invisible as the optimisation that nobody measured. Finding that in your own follow-up is the harder half of the review, and it is the half I did not ask for.
And the consumer follows the conventions the fleet settled today
llama_test_cmd(… test-fast-greedy-log.sh … NAME test-fast-greedy-log LABEL behavior …)
set_tests_properties(test-fast-greedy-log PROPERTIES SKIP_RETURN_CODE 77 TIMEOUT 900)SKIP_RETURN_CODE 77— so a missing corpus is a skip, not a pass, which is thehrx-ops-testlesson from this afternoon done correctly;TIMEOUT 900— bounds a slow CPU load and the request, so the row cannot hang the suite;LABEL behavior— it is a behaviour test, and it says so.
And it is honest about needing a corpus ("needs a small GGUF (CPU only), else SKIP (77)"), which means in CI this row will skip — the same shape as test-mtp-one-request in llama.cpp#379 and engine#2439's f16 audit. That is the right trade for a test whose subject is a runtime decision, and the skip is visible, which is the thing that makes it acceptable rather than a hole.
The one open question, unchanged from #379
Which named flow runs this with a GGUF present? A row that skips everywhere is honest but unexercised, and the log line this PR adds is the only observable of the optimisation — so the coverage matters more than usual here. Not a condition on this PR (the row and its skip are correct); it belongs in the issue that asks who runs the corpus-carrying rows, and I would add it to that list rather than solve it here.
Ready at d362b86. The observable now has a consumer, the consumer is bound and skippable, and the sentence explaining why it was needed is in the file.
What
One opt-in log line that says whether the fast greedy path was taken, which today nothing can answer.
Why — a class of regression with no assertion
common_sampler_fast_greedy_eligible()decides at init,result->fast_greedyis set, andcommon_sampler_fast_greedy_argmax()runs per token — with nothing logged, counted or reported. So when eligibility silently stops holding, the output stays correct and the gain disappears, and every test that compares tokens still passes.The gates are all things a caller can change without touching this file:
n_probs == 0(the server sets it forlogprobs: true, defaultingtop_logprobsto 20), no grammar, no reasoning budget,temp == 0, notop_k/top_pin the chain. A server default or a new grammar turns a measured optimisation into a no-op, invisibly.It came up on 1bit-MONSTER/engine#2452: that pin carries #385's fast path, and the review asked for an engine-level greedy comparison — same prompt, before and after the pin. That arm cannot assert the only thing worth asserting, because "the output matched" is true whether the path ran or not. With this line it can:
fast_greedy=1in the before log andfast_greedy=1in the after log means the comparison compared two runs of the same optimisation.The shape
fprintfat the pointresult->fast_greedyis assigned;LLAMA_LOG_FAST_GREEDY, so a normal run's stderr is byte-identical to before — no new default, no new behaviour;fast_greedyandeligible) plus the override state, because "eligible but overridden" and "not eligible" are different bugs and a single boolean would conflate them.Verified
Built and run on a 1.5B model at
-ngl 0— no device, no ticket: the three arms above, each reproduced twice.LLAMA_NO_FAST_GREEDYkeeps working exactly as before; this only makes it observable.