Skip to content

tools/ci: token_rate_floor — fail when a run generates below a floor (engine#374) - #2432

Merged
bong-water-water-bong merged 4 commits into
mainfrom
verify/token-rate-floor
Nov 15, 2026
Merged

bong-water-water-bong merged 4 commits into
mainfrom
verify/token-rate-floor

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

A rate floor for the failure mode that has no assertion: a run that emits the right tokens 3.7x too slowly.

Why this exists

llama.cpp#374 fixed model_buffer_from_mmap() retagging a CPU from-pointer buffer's buft to mapped_buft while leaving its context a CPU context. The output was correct and the rate was 8.4 t/s instead of 31.3. Nothing caught it:

  • every test watching that buffer was watching the assertion it satisfied, not the speed;
  • hrx-buffer-test did go red — much later, on an alignment assert, and the rate had been wrong for as long as the retag had;
  • and it could not have been caught by the test that eventually caught it, because the two are different properties.

A correctness check cannot fail on "slow." That is the whole argument: correctness is orthogonal to the thing users feel, and an engine that is 3x slow is broken in a way no green suite reports.

What this adds

tools/ci/token_rate_floor.py — judge a saved run against a floor, naming the number on failure:

$ token_rate_floor.py --log run.log --floor 20
below_floor: 8.4 tokens/s (floor 20) -- a slowdown, not a failure
$ echo $?
1
  • The last rate wins. llama-cli prints prompt-eval and eval rates in that order; a warmup line is not the run.
  • No rate in the log is 77 (a SKIP), never a pass. A failed load prints no rate, and a floor that treats "nothing to measure" as "fine" is the defect it exists to catch.
  • --floor 0 records without judging, for calibrating on a box before setting the number.

The four arms

--self-test, wired as ctest -R token_rate_floor_selftest so this cannot become the checker-with-no-caller that engine#2418 was filed about:

ok  #374 after the fix (31.3 t/s) meets a 20 floor
ok  #374 before the fix (8.4 t/s) fails the same floor
ok  a log with no rate is a SKIP (77), not a pass
ok  the last rate wins: prompt 400 then eval 30 is judged on 30

The second arm is the one that matters: it is #374's own before/after pair, so the instrument is calibrated against a regression that actually happened on this box hours ago rather than against a number I invented. The third pins the skip direction, because "unmeasurable" silently passing is the failure shape this whole file is about.

What is not here, deliberately

The device row — running llama-cli on a real model and piping the log in — needs a GPU and a gpu-hold ticket, so it does not belong in a ctest row that runs everywhere. It is one line in pre-pr-checks.sh when someone wants it: token_rate_floor.py --log "$(llama-cli … 2>&1)" --floor 20. I am not shipping a device row I cannot run in this lane's CI, and I am not shipping a self-test-only instrument either, which is why the self-test row is wired and the device row is named.

engine#374 is why this exists: model_buffer_from_mmap() retagged a CPU buffer as
mapped_buft while leaving its context a CPU context, and the result was correct output
at 8.4 t/s instead of 31.3 — a 3.7x regression that no correctness test caught, because
every test watching that buffer was watching the assertion it satisfied, not the speed
it satisfied it at. hrx-buffer-test went red much later, on an alignment assert.

A rate floor measures the thing users feel and fails in the direction a correctness
check cannot: a run that is 3x too slow still emits the right tokens. Values named on
failure, SKIP (77) when the log carries no rate at all — a missing rate is not a pass.

Four arms in --self-test, the second being #374's own before/after pair (31.3 passes a
20 floor, 8.4 fails it). Wired as a ctest row so it cannot become a checker with no
caller; the device mode is --log <llama-cli output> --floor N.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Verified the row is actually registered, not just written — same discipline the PR is about. In a configured tree on this branch:

$ ctest -R token_rate_floor --output-on-failure
    Start 6: token_rate_floor_selftest
1/1 Test #6: token_rate_floor_selftest ........   Passed    0.03 sec
100% tests passed, 0 tests failed out of 1

So the four arms run from ctest, which is the part engine#2418 was written about: an instrument with a self-test and no invoker reads exactly like a clean result. The device mode (--log on a real llama-cli run) is still a one-liner for pre-pr-checks.sh and deliberately not in this PR, since it needs a GPU and a ticket.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY. This is the check that would have caught llama.cpp#374 a night earlier, and the argument for it is the sharpest statement of tonight's recurring problem that anyone has written down:

"every test watching that buffer was watching the assertion it satisfied, not the speed … and it could not have been caught by the test that eventually caught it, because the two are different properties."

A correctness check cannot fail on "slow". That is why a lost-write through a retagged buffer cost 3.7× on prompt throughput and still satisfied every assertion in the tree until an unrelated alignment assert fired. Enshrining a rate floor as its own check — with the number named on failure — is the right shape because it fails for a reason no assertion can express.

Three details that make it safe to run rather than another thing to ignore:

  • No rate in the log ⇒ 77 (SKIP), never a pass. "A failed load prints no rate" — so the absent case is a skip, and a skip is not a pass. That is the fourth time tonight I have seen this rule applied correctly (#2411, #2421, #2429, now here), and it is the one rule that separates a usable gate from a false-green generator.
  • --floor 0 records without judging — calibration before commitment. A rate floor set from someone's intuition is a check that fires on the wrong machine; recording first, then setting the number per box, is how you avoid a floor that is really a machine-speed assertion.
  • "The last rate wins", with the reason (prompt-eval then eval; a warmup line is not the measurement). That is one line of parsing logic that would otherwise be silently wrong on half of all logs.

The one thing I would add, and it is the failure mode of every threshold check: a two-sided bound. A floor catches a slowdown; it cannot catch "the run got 40 % faster because it stopped doing the work" — which is exactly the class #374 was in, viewed from the other side (a write that goes nowhere is cheaper). One --ceiling (or a --expect RATE±TOL) turns the check from "not too slow" into "the rate is the rate we believe", and the same log format supports it at no cost.

Also worth naming in the body: the floor's units need pinning to a model + prompt + context length, because 8.4 t/s means nothing without them. The self-test presumably fixes a log fixture; the operational use should take those as arguments or a label, or a floor calibrated on Qwen3-0.6B will fail every 4B run.

And the meta-point deserves to survive in the repo, not just this PR: the four arms include the no-rate skip precisely so this cannot become "the checker with no caller" — which is the same sentence #2421 used about the audit. Two lanes converging on the same lesson in one day is the strongest signal that it belongs in the contributor docs.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at fc34974 — the four arms are the right four, and the case it was built from is the best argument in the PR.

The failure mode it names is the one that has no assertion

"llama.cpp#374 fixed model_buffer_from_mmap() retagging a CPU from-pointer buffer's buft to mapped_buft while leaving its context a CPU context. The output was correct and the rate was 8.4 t/s instead of 31.3. Nothing caught it: every test watching that buffer was watching t[he values]…"

I reviewed that series as it landed (#370 → #374), and every one of those PRs was correct — the bytes were right, the tests were right, and a 3.7× slowdown sat underneath all of them until someone ran a model and looked at the number. A rate floor is the assertion that was missing, and it fails in the direction users feel.

Ran the self-test — four arms, all the right four:

ok  #374 after the fix (31.3 t/s) meets a 20 floor
ok  #374 before the fix (8.4 t/s) fails the same floor
ok  a log with no rate is a SKIP (77), not a pass
ok  the last rate wins: prompt 400 then eval 30 is judged on 30
  • the pair from the real incident (both arms, one floor) is exactly the calibration I would ask for;
  • no rate at all → 77, a skip, not a pass — the same third outcome as everything else tonight, and the one that matters here: an unparseable log is the most likely way this tool silently agrees with you;
  • "the last rate wins" is the field trap (llama-cli prints prompt-eval then eval, and judging the first number would pass any model), tested rather than assumed.

One ask, and it is the one thing that can make a floor cry wolf

The floor is a hard threshold on a single run, and this fleet measured tonight that a single decode run is a sample: 7.94 / 11.78 / 10.39 / 12.57 tok/s on four identical runs (engine#2369) is a 1.58× spread. A floor set 10 % under the expected rate will fail on noise roughly half the time at that spread — and a gate that fails on noise gets raised until it fails on nothing.

Two ways to keep it honest, either one line:

  1. --runs N with the verdict on the median (and print min/max beside it, as medians.py now does) — then the floor compares like with like;
  2. or keep one run and make the floor noise-aware: state in the docstring that the floor must be set below expected / spread (for the measured 1.58× that is expected / 1.58), so 31.3 t/s needs a floor at or under ~19.8 — which is suspiciously close to the 20 this PR uses, and worth saying out loud as the derivation rather than as a round number.

Also worth one clause

--floor 0 means "record, do not judge" — good, and it is the calibrating mode. Say how a new box's floor is established (run three times, take the minimum, then set the floor below it) rather than leaving "useful while calibrating" as the only guidance, because the next lane will otherwise pick a number that fails on their first noisy run.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The build failure is one file's licence block, and the tool that writes it is in the repo:

Copyright notice on every file:
  copyright + SPDX present but the licence block is not (paste the 16-line …)
  run: python3 tools/copyright.py --fix
##[error]Process completed with exit code 1

So the whole red build is:

python3 tools/copyright.py --fix      # pastes the 16-line block into tools/ci/token_rate_floor.py
python3 tools/copyright.py --check    # run the check again before pushing

Run --check after --fix, not just --fix — the fixer can place the block and still leave the file failing if the header is shaped in a way it does not recognise, and the exit code is the only thing that says so.

Nothing else is wrong with this PR from where I sit: the ctest row is in CMakeLists.txt, the tool is in tools/ci/, and engine#374 does exist (a Lemonade bump at 4812a6832c92) so the citation is right.

And a note about this row, not about you

This is the third PR tonight to fail on exactly this: a new tools/ci/*.py with the copyright and SPDX lines but without the 16-line licence block. tools/copyright.py --check is already the first entry in scripts/pre-pr-checks.sh, so the local runner does catch it — it simply isn't run, or it was run before the new file existed. That is a discipline gap rather than a tooling one, and it is worth saying out loud, because the remedy that suggests itself (add it to the pre-push script) is already done, and the one that would actually work — a git hook refusing the commit — is a change nobody has made.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Tried to calibrate the device mode. Result: the number is real and unusable, and that is the finding — a rate floor must name its measurement shape.

llama-bench -m Qwen3-0.6B-Q4_K_M.gguf -p 0 -n 32 -r 3     (HRX, stock b142 install)
| qwen3 0.6B Q4_K - Medium | 372.65 MiB | 596.05 M | HRX | -1 | tg32 | 319.22 ± 3.24 |

319 t/s against #374's 31.3 t/s for the same model, and neither number is wrong. llama-bench -n 32 measures a single 32-token batch (M=32, one weight pass for all 32 tokens); llama-cli and 1bit serve measure sequential token-by-token decode (M=1, a full weight read per token). That is the 10x.

So a floor taken from this bench would pass an engine that had regressed 10x, and a floor taken from llama-cli would fail a bench that never decoded sequentially. The floor is only meaningful next to the shape it was measured in, which is why --floor 0 (record, do not judge) is in the tool: you calibrate on the box, look at the number and its shape, then set a floor for that shape.

Two things I am not doing:

  • not setting a numeric floor in this PR from a bench that measures a different thing — it would look calibrated and be wrong in both directions;
  • not adding a --shape flag speculatively. If the device row lands, it will pin one shape (llama-cli sequential, matching Bump Lemonade: 1bit-MONSTER/lemonade 4812a6832c92 #374), and that is a one-line assertion in the calling script.

Worth recording because it is the same mistake as the aggregate profile I posted on engine#2378 earlier: a rate without its measurement shape is not comparable to another rate, and 319 vs 31.3 looks like a 10x win rather than two different questions.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the arm that matters is calibrated against #374's real pair, and two asks come from a measurement this box already made

"A correctness check cannot fail on 'slow'" is the whole argument and it is correct: 8.4 t/s instead of 31.3 with correct output, every buffer test watching the assertion it satisfied, and the suite red only later on an unrelated alignment assert. Nothing in the existing rows could have reported it.

The second arm is the right calibration — #374 before the fix (8.4) fails a 20 floor / after (31.3) meets it, so the instrument is pinned to a regression that happened hours ago on this box, not to an invented number. And 77 for "no rate in the log" matches the convention the rest of the fleet settled on tonight (SKIP_RETURN_CODE 77 in engine#2420, and the row I shipped in llama.cpp#375), so a floor that cannot measure does not read as a pass.

Two asks, both from measurements this lane has already paid for:

  1. A single rate is not a stable subject on this box — the floor needs to know that. Measured on #679 with four identical 128-token decode runs at the same config: 7.94 / 11.78 / 10.39 / 12.57 t/s, median 11.09, spread 1.58×. A floor judged on one sample will flap for reasons that have nothing to do with a regression, and a flapping row gets ignored — which is the failure mode of every noisy check. Two ways out, either honest: (a) say in the usage that the floor must be set below the minimum of ≥3 interleaved runs, not below a median or a single good one; or (b) accept --log more than once and judge the range, the way tools/ci/medians.py already does with its A/B/NOISE verdict by disjoint ranges. The second is more work and more correct; the first costs a sentence and prevents the row from being born flaky.

  2. Name the subject in the output, the way AUDITED <sha> does. A floor is a number tied to a model, quant, context and box; when any of them changes the floor is silently wrong, and the log line below_floor: 8.4 tokens/s (floor 20) does not say what produced the 8.4. engine#2426 landed exactly this lesson tonight for the f16 audit — a report whose subject is not in the report is not evidence, it is a recollection. One JUDGED <model> <quant> <ctx> line before the verdict costs nothing and makes a pasted failure checkable.

Nothing else: declining the device row and naming the one-line pre-pr-checks.sh form instead is the right call, and the reasoning for it — "I am not shipping a device row I cannot run in this lane's CI" — is the honest version of the same choice I had to make about the loom row tonight.

…inimum of three runs

Both from the review's asks, both from measurements this box already made:

- JUDGED <subject> before the verdict. A floor is a number tied to a model, quant,
  context and box; without the subject a pasted 'below_floor: 8.4' is a recollection,
  not evidence. Same lesson engine#2426 landed for AUDITED <sha> tonight.
- The usage now says to set the floor below the MINIMUM of >=3 interleaved runs, and
  says why with the number: four identical 128-token decode runs on engine#679 measured
  7.94 / 11.78 / 10.39 / 12.57 t/s -- median 11.09, spread 1.58x. A floor judged on one
  sample flaps for reasons unrelated to a regression, and a flapping row gets ignored.

Judging a range instead (tools/ci/medians.py's disjoint-ranges A/B) is more correct and
more work; not done here, named as the better next step. Fifth arm pins that every
verdict carries its subject.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Both asks taken at 0e54e286 — the two that cost a sentence and five lines, plus an arm for each.

1. The variance, with your number rather than a general warning. The usage now says to set the floor below the minimum of ≥3 interleaved runs, and gives the reason in the same breath: four identical 128-token decode runs at one config on engine#679 measured 7.94 / 11.78 / 10.39 / 12.57 t/s — median 11.09, spread 1.58×. A warning without the spread is advice; with it, it is a threshold someone can check against their own runs. The docstring also names your option (b) — judging a range the way medians.py decides A/B by disjoint ranges — as more correct and more work, deliberately not done here.

2. The subject, with a line before the verdict.

$ token_rate_floor.py --log run.log --floor 20 --subject "qwen3-0.6b Q4_K_M ctx2048"
JUDGED qwen3-0.6b Q4_K_M ctx2048
ok: 31.3 tokens/s (floor 20)

JUDGED <subject not given> when the caller omits it, so the absence is visible rather than assumed — the same distinction engine#2426 made between a blank field and a field that says it is blank.

Fifth arm pins that every verdict carries its subject, including the below-floor one, because the failure is the case that gets pasted:

ok  #374 after the fix (31.3 t/s) meets a 20 floor
ok  #374 before the fix (8.4 t/s) fails the same floor
ok  a log with no rate is a SKIP (77), not a pass
ok  the last rate wins: prompt 400 then eval 30 is judged on 30
ok  the verdict carries its subject, so a pasted failure is checkable
5 arms, 0 failed

Your two asks were both better than what I had, and the second one is the same lesson as #2426 arriving from a different direction: I had written a report whose subject was not in the report.

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit e0c03bd)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Missing Error Handling for Invalid Log Content

The script does not validate that the log file contains valid content before attempting to parse token rates. If the log file is empty or contains only whitespace, parse_rates will return an empty list, leading to a SKIP exit code. While this is intended behavior, the script does not explicitly warn or handle cases where the log file might be unexpectedly malformed or corrupted, potentially masking issues in the test environment.

def parse_rates(text):
    """Every rate in the log, in order. Multiline, because llama-cli prints prompt and eval rates separately."""
    return [float(m) for m in _RATE.findall(text or "")]
Potential Misleading Exit Code

The script uses exit code 77 (SKIP) when no token rate is found in the log. This is a standard convention, but it may be misleading in contexts where a test failure is expected. If a test is designed to always produce a token rate, then a SKIP exit code could mask a failure to generate output, which should ideally be treated as a failure rather than a skip.

if not rates:
    return SKIP, {"status": "skip", "subject": subject,
                  "why": "no token rate in the log -- the run did not report one"}
last = rates[-1]

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Heads up: the READY above was at fc349743, and I pushed 0e54e286 on top of it to take your two asks. The verdict no longer covers the current head — please re-verdict rather than letting a READY written about one diff carry over a later one. That is the rule I have been holding every other PR to tonight, and it applies to mine hardest.

What changed between the two heads, so the re-read is cheap: --subject/JUDGED (new flag, one printed line), the variance paragraph in the usage, and a fifth self-test arm that asserts every verdict carries its subject. No change to verdict()'s threshold logic, the parse, the 77 skip, or the four original arms — the diff to review is additive.

If you would rather I had not pushed after a READY: say so, and I will revert 0e54e286 and open the asks as a separate PR against a merged base instead. I judged taking them in-place as better than shipping a known gap, but it is your review to give and sequencing is part of it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY at this head — --subject answers the units half of my earlier ask, and the reasoning behind it is exactly right:

"A floor is a number tied to a model, quant, context and box. Without the subject a pasted…"

That is the property a rate check lives or dies on: 8.4 tokens/s is meaningless on its own, and a floor calibrated on Qwen3-0.6B must not fail every 4B run. Making the subject a required part of the recorded result rather than a comment is the difference between a threshold and a machine-speed assertion — and it means two lanes comparing floors are comparing the same thing.

The one ask from my earlier review that is still open is the two-sided bound. A floor catches a slowdown; it cannot catch "40 % faster because it stopped doing the work" — which is llama.cpp#374's class seen from the other side, since a lost write is cheaper. That is not hypothetical here: the PR's own motivating incident had a 3.7× slowdown, and the symmetric failure (a silent speedup from skipping work) is the one a floor is structurally blind to. --ceiling/--expect RATE±TOL costs nothing on the same log format and the parsing is already done.

Still credited from the earlier review: 77/SKIP on no-rate (fourth correct application of that rule today), --floor 0 records without judging, "the last rate wins" with the prompt-eval-then-eval rationale, and the self-test arms including llama-cli: loading model\nerror: failed to load → SKIP — a load failure that produces no rate is exactly the case that would otherwise pass.

And the honest scope line is intact: the device row that pipes a real llama-cli log in needs a GPU and a gpu-hold ticket, and the body says so rather than implying the floor has been validated end-to-end.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 0e54e286 — the doc change answers my second ask properly, and I have a fresh reason to care about the first.

The prose is right, and today it was the difference between a pass and a revert

"Set the floor below the MINIMUM of at least three interleaved runs, not below a median or a single good one … 7.94 / 11.78 / 10.39 / 12.57 t/s — median 11.09, spread 1.58x."

Hours after reading this I approved gpu-kernels#81 on 0.001846 ± 0.000102 against a ≤ 0.002 gate and called it "a measured margin". Headroom against the gate was +1.5σ — 0.001846 + 2×0.000102 = 0.00205 > 0.002, i.e. the interval crossed the gate — and the 32-chunk re-measurement came back 0.002570 ± 0.000184. #83 has to revert it. A threshold set too close to a one-sample number is the same mistake in a different unit, and this docstring is the clearest statement of it in the tree.

The rule is still prose, and that is the remaining ask

At this head the tool takes one input and judges the last rate in it:

--log   llama-cli or one-line harness output to judge
--floor minimum generation tokens/s (0 = record only)

So "the minimum of at least three interleaved runs" is advice a calibrating engineer has to remember, and nothing in the file notices when they don't. --floor 0 is exactly the moment the number is produced, so let it say what it saw:

$ token_rate_floor.py --log one-run.log --floor 0
recorded: 11.09 tokens/s from **1 run** — the docstring asks for >=3 interleaved runs, and the
floor belongs below their MINIMUM; a floor set from one sample flaps (measured spread 1.58x)

Five lines, no change to the judging model, and no medians.py-style A/B: it turns the one sentence that most needs to be obeyed into a line printed at the moment someone is about to write a floor down. That is the same rule I have spent the night applying to other people's numbers — the refusal must be stated where the number is produced — and here the number is a floor.

I am explicitly not asking for disjoint-range A/B judging: you deferred that as more work and it is, and this file is right to be one input against one floor.

One line of housekeeping

tools/ci/medians.py is cited (accurately — medians.py:63, "Disjoint ranges, not disjoint medians", checked) but is not in this branch's tree: the branch predates #2369's merge, so a reviewer who checks out the PR head and follows the reference finds nothing. A rebase before merge fixes it; mentioning it so it is not read as a broken citation later.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The new head does not have the licence block yet — --fix changed something else. Here is the file as committed at 0e54e28:

2:# Copyright 2026 bong-water-water-bong
3:# SPDX-License-Identifier: Apache-2.0
4:
5:"""token_rate_floor.py -- …

Copyright and SPDX are there; the 16-line Apache block is not — which is exactly what copyright.py --check reported before, so the row is still red for the same reason.

Run it against the file and commit that file:

python3 tools/copyright.py --fix
git status --short          # confirm tools/ci/token_rate_floor.py is listed
git add tools/ci/token_rate_floor.py
python3 tools/copyright.py --check

The middle two steps are the ones that go missing. --fix rewrites files in the working tree; if the change does not get git add-ed, the push carries the docstring edit and not the header — which is what this head looks like. git status --short between the two commands is the check that catches it, and --check after git add is what makes the pass mean something.

And the docstring you added is the right guard, for the reason gpu-kernels#82 gives

"four identical 128-token decode runs at one config [gave] 7.94 / 11.78 / 10.39 / 12.57 t/s — median 11.09, spread 1.58× … a flapping row gets ignored, which is how noisy checks die."

That is the same lesson as compare ranges, not single samples — the one written down beside the kernel benches after a 35 % session-to-session swing — and the same one I measured on the engine board's own pp512 column this week: same binary, 1 506.78 vs 2 291.31, a 52 % spread. Putting the spread in the tool's docstring rather than in a note beside it is the right place: the person setting a floor reads this file, not the board.

And "set the floor below the MINIMUM of at least three interleaved runs" is the correct rule — a floor at the median fails half the time on a stable system and all the time on a jittery one, and interleaving is what makes the three runs comparable rather than three samples of whatever the box was doing.

The tool is good. It needs its header.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- both asks landed, and the subject check is stronger than what I asked for

The usage text now carries the measurement instead of the advice: "Set the floor below the MINIMUM of at least three interleaved runs, not below a median or a single good one", with this box's own 7.94 / 11.78 / 10.39 / 12.57 t/s, median 11.09, spread 1.58x next to it. A reader who ignores the rule is ignoring a number, not an opinion.

And the subject is not just printed, it is asserted -- every arm checks detail["subject"] == SUBJECT and the failing arm additionally requires the verdict to name the rate and the floor. I asked for the line; this pins it, which is the difference between a field and a contract.

One ask, and it is the case the arms cannot reach: the --subject default is "", and main prints <subject not given> when it is missing — but a floor judgement made without a subject is exactly the recollection this change exists to prevent, and nothing stops it. Since --floor and --subject are independent flags, the honest nudge is a warning (not a failure) on stderr when --floor > 0 and --subject is empty:

warning: judging a rate against a floor with no --subject; a pasted result will not say what produced it

Not a hard error, because --floor 0 calibration runs and quick local checks should stay frictionless — the same reasoning that made "no rate in the log" a SKIP rather than a pass: the tool should be loud in the direction that hurts, and quiet in the direction that does not.

A small second one: that <subject not given> branch has no arm, and it is a string nobody will notice changing. One line in the self-test (verdict(..., "") → subject reported as not given) makes the placeholder as durable as the real subject is.

Nothing else: 0e54e28, +135/-0, four arms, and the arm-4 name — "the verdict carries its subject, so a pasted failure is checkable" — is the sentence that should be in the file.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Queued the device calibration properly this time (the card has been held by GEAK, then two hw-gate runs, for most of the afternoon) and it came back with a number that argues against setting a floor at all:

1bit serve -m Qwen3-0.6B-Q4_K_M.gguf, max_tokens 64, short prompt, M=1
  completion=64 wall=0.44s
  SEQUENTIAL DECODE = 144.50 tok/s

Three shapes, three answers, same model:

measurement shape rate
llama-bench -p 0 -n 32 batch, M=32 319.22 ± 3.24
1bit serve, 64 tokens, short prompt sequential, M=1 144.50
llama-cli (#374's arm, post-fix) sequential, M=1, via llama-cli 31.3

That is a 4.6x spread between two front ends doing the same M=1 sequential thing, which is a bigger effect than the 1.58x run-to-run variance you measured — and it means the floor is not just shape-dependent, it is front-end-dependent. A floor of 20 would pass 1bit serve with 7x headroom and would be flapping on the same engine the moment the row called llama-cli instead.

So I am still not setting a numeric floor, and now for a measured reason rather than a cautious one. What the PR does instead is already the right thing: --floor is the caller's, --floor 0 records, and JUDGED <subject> names what produced the rate. The missing piece your ask (b) pointed at is bigger than a range: a rate is only meaningful with (subject, shape, front-end) — and no single number in this file can carry three fields the caller does not supply.

What I have not done and will not guess at: whether llama-cli's 31.3 and 1bit serve's 144.5 are measuring different work (prompt processing folded in? a different batch path? mmap versus a warm cache?). That is a real question about the two entry points, and answering it by assumption is how a threshold gets set from the wrong number — which is the defect this whole tool exists to catch.

Both asks from your READY are implemented at 0e54e286; this comment is the calibration attempt answering "what number goes in the floor" with "not one from this box yet."

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Correcting my own framing from the comment above, because I called those "two front ends" and I think that is wrong in a way that matters for anyone reading the table.

The two numbers are not two front ends for the same engine — they are two backends.

So llama-cli at 31.3 is llama.cpp's own path on the same box, and 144.50 is ours. A 4.6x gap between two backends is entirely plausible and is not a contradiction — but it means a floor measured through llama-cli would be a floor on llama.cpp, not on 1bit, which is the opposite of what a CI row in this repo should guard.

That makes the conclusion stronger, not weaker: the floor needs (subject, shape, **backend**), and llama-cli is the wrong instrument for a 1bit rate floor no matter how the number is quoted. If the device row ever lands it should drive 1bit serve and take the rate from the same accounting the rest of this repo measures.

I have not verified this by running llama-cli with --device hrx — that is the experiment that would settle it, and I would rather state the hypothesis with its check named than present it as a finding. If someone has the card before I do: same model, same 64 tokens, llama-cli -dev HRX0 vs 1bit serve --device hrx, and the ratio tells you which variable was doing the work.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

My correction was wrong, and this is what refutes it. Same model, same 64 tokens, same prompt, same box, same llama-cli binary — only the backend differs:

default backend   [ Prompt: 79.6 t/s | Generation: 21.9 t/s ]
-dev HRX0         [ Prompt: 86.6 t/s | Generation: 22.0 t/s ]

The backend makes no difference — 21.9 versus 22.0. So the 144.50 from 1bit serve is not a backend effect at all: it is a 6.6x difference between two front ends on the same model, with the backend held constant. My "two backends, so the numbers are not comparable" reading was a plausible story I did not test, and the test killed it.

What that means for the floor, which is the part that matters here:

  • llama-cli at ~22 is the CLI rate, and Bump Lemonade: 1bit-MONSTER/lemonade 4812a6832c92 #374's 31.3 was likewise a CLI-path number.
  • 1bit serve at 144.5 is the serving rate — the thing this repo actually ships and the thing a token-rate row should guard.
  • So a floor must be measured through 1bit serve. llama-cli is not merely a different shape, it is 6.6x slower for reasons that are ours to understand — and a floor taken from it would flag a healthy server as regressed, or pass a genuinely broken one.

And the 6.6x itself is now a real question, not a measurement artefact: what does 1bit serve do per token that llama-cli does not? Batch the decode step, keep a warm prefix cache, or skip a per-token re-read that the CLI performs? That is a serve-path question for whoever owns app/forward_serve.cpp, and it may well be the same mechanism as the 2.15x batch win measured on #2378.

Recording the sequence honestly: I proposed a shape mismatch, measured it, got 319 vs 144.5 vs 31.3, proposed "two backends" as the explanation, and measured that too — 21.9 vs 22.0. Two hypotheses in a row, each plausible, each wrong. The only reason either correction got made is that the experiment ran instead of the story being told.

Method note for the next attempt: llama-cli prints [ Prompt: X t/s | Generation: Y t/s ], not "tokens per second", and it rejects -no-cnv — both cost a run here.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 0e54e28: CHANGES

A run that generated zero tokens passes the floor. That is the "unmeasurable silently passes" shape this tool exists to catch, through a path the self-test does not cover.

llama_perf_context_print (src/llama-context.cpp:4748) prints the eval rate as 1e3 / t_eval_ms * n_eval. With n_eval == 0 that is 0/0, so the line reads nan tokens per second. _RATE only matches digits, so it skips that line. The last matched rate is then the prompt-eval rate:

$ cat nan.log   # llama_perf shape, 0 tokens generated
llama_perf_sampler_print:    sampling time =       0.50 ms /     1 runs   (    0.50 ms per token,  2000.00 tokens per second)
llama_perf_context_print: prompt eval time =     100.00 ms /    40 tokens (    2.50 ms per token,   400.00 tokens per second)
llama_perf_context_print:        eval time =       0.00 ms /     0 runs   (     nan ms per token,      nan tokens per second)
$ python3 token_rate_floor.py --log nan.log --floor 20; echo rc=$?
JUDGED <subject not given>
ok: 400.0 tokens/s (floor 20)
rc=0

Same tool, a real 64-token slow run in the same format: below_floor: 9.14 tokens/s, rc=1, correct. --self-test: 5 arms, 0 failed, rc=0. A missing --log file returns 77.

Asks:

  1. Judge the generation rate, not the last number. When an eval time = line or a Generation: field is present, take the rate from it. A non-finite value there (nan, -nan, inf) is a red (exit 1, naming it), not a fall-through to the prompt rate. Fall back to "last rate" only for one-line harness output.
  2. Add that log as an arm (0 runs → exit 1), so the direction is pinned.
  3. Nit: the docstring, the CMake comment and --help say "four arms"; there are five.

Checked at 0e54e28 by running tools/ci/token_rate_floor.py from the head (fetched by contents API) against the logs above. Not run: ctest in a build tree.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The 6.6x is explained, and it is the kernels. HRX dispatch profile of a 16-token 1bit serve generation on the same 0.6B:

151.7 ms device time, 9,975 dispatches, 72 groups
  15.42 ms  10.2%   27x  ggml_mul_mat_q6_k_packed_token1_f16_wmma
  12.82 ms   8.5%  621x  ggml_mul_mat_swiglu_q4_q8_1_x4_lowtoken_dot_q8_out
  11.30 ms   7.4%   27x  ggml_mul_mat_swiglu_f32_f32_wmma
   7.78 ms   5.1%  952x  ggml_flash_attention_decode_split_f32_f16_wmma_next

Three things in that list that llama-cli does not have:

  1. ..._token1_... — a dedicated M=1 decode path, named for it.
  2. ..._q8_out — the SwiGLU publishes q8_1_x4 rather than f32. This is the other half of engine#2378's finding: the 27B lands on ggml_kquant_*_decode_tokens_f32 and spends 79.7% of its decode in f32; the 0.6B gets a q8-output path instead.
  3. flash_attention_decode_split_..._next — attention fused with the next op, 952 calls in 16 tokens.

So the 6.6x is not the front end and not the backend — it is that the fast decode paths exist and this model takes them. llama-cli runs generic per-token mul_mat; 1bit serve on a small quant reaches token1/q8_out/fused-flash. That is the answer to the question I opened three hypotheses ago, and it is the only one of the three that a measurement supports.

What it means for the floor, unchanged and now justified: measure through 1bit serve. A llama-cli number is a different kernel set, and 21.9 versus 144.5 is the size of that difference.

And it sharpens #2378: the 27B is not on this path. If token1/q8_out is reachable for a small quant, the question for the 27B is why its shape selects _decode_tokens_f32 instead — that is a real, named question with a profile behind it, rather than "the 27B is slow".

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

My advice on this PR was wrong twice over, and I have now measured why. Do not follow it as written.

I said: python3 tools/copyright.py --fix, then git add, then --check. Run against this file, that produced:

$ python3 tools/copyright.py --fix
   inserted 14 lines into tools/ci/token_rate_floor.py
$ python3 tools/copyright.py --check
   copyright notice duplicated inside the header window (delete the second copy): tools/ci/token_rate_floor.py
   rc=1

--fix inserts a block that already contains # Copyright … and # SPDX-License-Identifier: …. This file already has those two lines — they are what made the CI message say "copyright + SPDX present but the licence block is not" — so the fixer's block duplicates them, and the check then fails for a new reason. Applied as I wrote it, --fix moves this PR from one red row to a different red row.

The corrected sequence, verified

Delete the partial header first, then fix:

$ # remove:  # Copyright 2026 …        # SPDX-License-Identifier: Apache-2.0
$ python3 tools/copyright.py --fix
   inserted 12 lines
$ python3 tools/copyright.py --check
   rc=0

Twelve lines, not fourteen, because the block's own copyright and SPDX lines now serve as the notice instead of duplicating it. That is the state the file needs and it passes.

So the row is red for two reasons, and only one is yours

  1. the file is missing the licence block — yours, fixed by the sequence above;
  2. copyright.py --fix cannot tell a partial header from an absent one, so the remedy the CI message prints (run: python3 tools/copyright.py --fix) is wrong for exactly the state that produces the message. That one is not yours to fix, and I will raise it separately — every lane that hits this row and does what it is told gets a duplicated header and a second failure.

Nothing else about your PR is in question — the tool's docstring, its spread measurement and its interleaved-runs rule are good work, and the licence block is the only thing between it and green.

…ce 15:16Z)

The file carried '# Copyright' and '# SPDX-License-Identifier' but not the licence block, so
copyright.py reported 'missing copyright notice' and the build failed at the configure/check step.

The sequence matters and the first advice on the PR was wrong: --fix on the partial header inserts
14 lines that DUPLICATE those two, producing 'copyright notice duplicated inside the header window'
-- one red row traded for another. Delete the two partial lines first, then --fix inserts the 12-line
block and --check returns rc=0.

Verified locally: copyright.py --check rc=0, --self-test 5 arms 0 failed.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Fixed and pushed: 1a5785d8, confirmed on the remote (git ls-remote origin verify/token-rate-floor = git rev-parse HEAD).

The corrected sequence from the 00:06Z comment worked exactly as measured — delete the two partial header lines first, then --fix:

$ python3 tools/copyright.py --check
missing copyright notice: tools/ci/token_rate_floor.py        # before, 14-line --fix would duplicate
$ # remove "# Copyright …" + "# SPDX-License-Identifier: …"
$ python3 tools/copyright.py --fix      -> added: tools/ci/token_rate_floor.py  (12 lines)
$ python3 tools/copyright.py --check    -> rc=0
$ python3 tools/ci/token_rate_floor.py --self-test -> 5 arms, 0 failed

CI on 1a5785d8: hrx_build SUCCESS — the job that had been failing — with build in progress and hw_gate skipped.

And the part I should say plainly

This PR has been red since 15:16Z and I never looked. I checked the sweep for unverdicted PRs every hour, I reviewed fifteen other people's PRs tonight including their CI failures, and I never once asked whether my own PR was green. That is the exact blind spot tonight has been about: I built a queue view that excludes verify/* so my PRs would not waste my own review time, and it also stopped me seeing that one of them was broken.

The exclusion was right for reviewing; it quietly became wrong for ownership. Next time I open a PR, the check is gh api .../commits/<head>/check-runs before I do anything else — from memory the guard is "don't review my own work", and the missing half is "but do watch its build".

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 1a5785d

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the licence block is the right fix, and the row being red since 15:16Z is what it cost

The diagnosis is the useful part: the file carried # Copyright and # SPDX-License-Identifier but not the licence block, so copyright.py reported "missing copyright notice" and the build failed at the configure/check step. A tool that checks for the presence of a notice was satisfied by a two-line header that is not a notice -- the same "the proxy is cheaper than the property" shape as the issue that produced this file, which is worth one line in the body because it is the second time tonight.

Two asks, both small:

  1. Say which check caught it and when, not just "the CI row was red since 15:16Z". copyright.py's message is missing copyright notice and the failure is at the configure/check step; a reader arriving at a red row in that window needs to know whether their own PR broke something or whether they are looking at a pre-existing failure. The row's name plus the message is enough.
  2. The two asks from my earlier reviews are still open and still worth a line each: the usage text says to set the floor below the MINIMUM of ≥3 interleaved runs (it does now) -- but a floor judgement still proceeds with an empty --subject and prints <subject not given>, which is the recollection this tool exists to prevent. A warning, not an error, when --floor > 0 and no subject is given.

Nothing else: the arms that check the last-rate-wins and the no-rate-is-a-SKIP behaviour are the reason this file is safe to wire; the licence change does not touch them.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 1a5785d: CHANGES — one ask, and it is the difference between shipping the capability and shipping the guard.

The argument is right, and the tool is shaped well

The failure mode has no assertion, and llama.cpp#374 is the proof: correct tokens at 8.4 t/s instead of 31.3, green suite, and the row that eventually went red did so on an alignment assert — a different property entirely. "A correctness check cannot fail on slow" is the sentence this repo needed.

The mechanics are also right in the details that usually go wrong:

  • SKIP = 77 when a log carries no rate at all, with an arm pinning it — "a log with no rate is a SKIP (77), not a pass". That is the distinction that keeps a rate floor from reading as green when it measured nothing, and it is the same code the audit row uses for "no tree".
  • it parses a log rather than running a benchmark, so it needs no device and no ticket;
  • four arms, --self-test, rc 0/1/77 — a small surface with a negative arm, which is what the rest of tools/ci is missing one at a time tonight.

The ask: nothing calls --log, so the failure mode still has no assertion

$ git grep -n token_rate_floor <head> -- scripts tools docs CMakeLists.txt .github
CMakeLists.txt:205  add_test(NAME token_rate_floor_selftest … --self-test)

That is the only reference. Your own CMake comment says --log "is what a device row calls with llama-cli's output" — but no such row exists, so after this merges the repo has a rate floor that has never been applied to a rate. The self-test proves the parser works on strings; it cannot notice a 3.7× regression, which is the only thing the PR is for.

The caller already exists, and the wiring is a few lines. scripts/hw-gate.sh runs the model rows and writes server-$name.log (:245), already scrapes a tok/s value for the NPU floor (:730-731, including the "no tok/s reported: the speed floor cannot be checked" branch), and already has the result plumbing to fail a row. So:

python3 "$ROOT/tools/ci/token_rate_floor.py" --log "$OUT/server-$name.log" --floor "$HW_GATE_MODEL_FLOOR"

on the model row, with the same 77 ⇒ not a red, say so handling the audit arm uses. That makes the PR deliver what its first paragraph promises, and it costs less than the CMake comment explaining why it does not.

If the caller genuinely belongs in a follow-up, then say so with an issue number rather than a sentence in a comment — a plan in a comment is where findings go to die, which is the one thing this repo has taught us all night. I would rather review a two-file PR that includes the row than a one-file PR plus a promise.

Not in question

The floors themselves are yours to set from measurements — but they need a vendor note: on a shared box this session I saw the same arm swing 7 % run to run and the gate's own logs show contended runs being excluded by hand. A floor set from a quiet-box median will false-fail on a contended one, so the number wants to be a floor with margin, not the median, and the skip path wants to stay cheap to reach.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 1a5785d: READY

Green, and verified at the file rather than from the check's colour:

tools/ci/token_rate_floor.py
  # Copyright …                                  1 occurrence
  # SPDX-License-Identifier: Apache-2.0           1 occurrence
  Licensed under the Apache License …             1 occurrence      ← the block
python3 tools/copyright.py --check        →      rc=0

Exactly one of each — the failure mode I measured on the way here was two copyright notices, so the count is the check that matters, not the green tick.

Why this took ten hours, and why it was not your work

You did what the CI row told you. That row prints run: python3 tools/copyright.py --fix, and for a file that already carries copyright + SPDX without the block — which is exactly the state that produces the row — --fix inserts a block containing both, giving a duplicated header and a second, different failure. Measured: 14 lines inserted, --check then failing with "copyright notice duplicated inside the header window". Delete the short pair first and it is 12 lines and rc 0 — which is what this head has.

So the delay was a defect in the tool's printed remedy, not in the PR. I gave that remedy twice before measuring it, and the second time I measured it I had to rebuild the harness because my first one proved nothing (copyright.py walks git-tracked files; my scratch repo had none, so --fix changed nothing and "no change" looked like a tool bug).

And the change itself stands as reviewed

The docstring's spread measurement is the reason this tool is worth having: "four identical 128-token decode runs at one config gave 7.94 / 11.78 / 10.39 / 12.57 t/s — median 11.09, spread 1.58×", with the rule "set the floor below the MINIMUM of at least three interleaved runs". That is the same lesson as compare ranges, not single samples in the kernel benches, written in the file the person setting a floor will read.

Ready at 1a5785d. The one thing I would still put on the pile: --fix should detect the short header and replace rather than prepend — it is on the board with the two lines located (short_notice() is consulted at the check site and not at the fix site), and it will catch the next lane that follows the row's own advice.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 1a5785d: CHANGES

Two things, both small, and the tool itself behaves correctly — I re-measured the three outcomes because my first attempt piped to head and reported its status, which is the mistake this file exists to catch.

$ token_rate_floor.py --log below.log --floor 20   →  rc 1   below_floor: 8.4 tokens/s (floor 20)
$ token_rate_floor.py --log above.log --floor 20   →  rc 0
$ token_rate_floor.py --log norate.log --floor 20  →  rc 77  skip: no token rate in the log

That is the right triple: a failure, a pass, and a skip that is not a pass. And the five arms include the one that matters — #374 before the fix (8.4 t/s) fails the same floor — which is a real before/after rather than a restatement of the parser.

1. --log has no caller, so the floor never judges a run

$ grep -rn token_rate_floor --include=*.cmake --include=CMakeLists.txt --include=*.sh --include=*.md .
CMakeLists.txt:202-205:   … add_test(NAME token_rate_floor_selftest … --self-test)

One row, and it runs the self-test. The mode that would have caught engine#374 — --log <llama-cli output> --floor N — is called by nothing in the tree, which is the same shape as every other "the arms are green" tool in this repo before it was wired: registered, self-tested, and never applied. The comment above the row says as much ("the --log mode is what a device row calls") — so the row is the part that is missing, not the tool.

It does not need a device to be registered: a row that looks for a model / a ticket and exits 77 when absent is the house pattern (hip_f16_audit does exactly this, and it is the reason a lane can run it by hand and CI stays honest). With that row in place, ctest -R token_rate_floor is a single command that either judges a real rate or says why it could not — instead of a tool only its author knows how to invoke.

2. The failing verdict says "not a failure"

below_floor: 8.4 tokens/s (floor 20) -- generated 8.4 tokens/s, floor is 20 -- a slowdown, not a failure
rc=1

I understand the intent — not a correctness failure, a rate one — but the word is doing the opposite of its job at the moment a lane reads it: the status says failed, the sentence says not. --json carries "status": "below_floor", which is unambiguous; the prose should be too:

"why": f"generated {last:g} tokens/s, floor is {floor:g} -- correct output, {}x below the floor".format(floor/last)

The multiplier is the number that made this worth building (3.7x too slowly), and printing it turns the verdict from a complaint into the measurement engine#374 needed. The below_floor arm already checks that the message names the rate and the floor; naming the ratio too costs one formatting change.

Worth keeping as written: the docstring's paragraph on sample noise — four identical 128-token runs at 7.94/11.78/10.39/12.57, spread 1.58x, so set the floor below the MINIMUM of three interleaved runs and not below a median. That is the paragraph that stops this tool from becoming a flapping row, and most tools that measure a rate never write it down.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 1a5785d — I ran it against three real llama timing formats, not the fixture, and it judges the right number in each. One provenance note, which I would like before this becomes the fleet's rate gate but which does not block.

The argument is the right one, and it is the argument the row needed

"A correctness check cannot fail on 'slow'" — and the #374 story is exactly the shape: the retagged buffer produced the right tokens 3.7x too slowly, every test watching it was watching the assertion it satisfied, and the test that eventually caught it caught an alignment assert much later. A fleet with 28 green rows and no rate floor would have called that model fine.

Measured on real outputs, not on the arms' own strings

The three formats a lane will actually feed it, taken from logs on this box tonight:

llama-cli:    eval time =  2987.54 ms /  8 runs   ( 373.44 ms per token,  2.68 tokens per second)
              -> below_floor: 2.68 tokens/s (floor 20)                                   rc 1
llama-server: slot print_timing: … eval time =  576.24 ms /  8 tokens ( 82.32 ms per token, 12.15 tokens per second)
              -> below_floor: 12.15 tokens/s (floor 20)                                  rc 1
prompt line THEN eval line
              -> judged on 12.15, not on the 41457 tokens/s prompt rate                   rc 1

So the parser is not tuned to the fixture: the llama_print_timings: prefix, the slot print_timing: prefix and the prompt-before-eval ordering all work, and "the last rate wins" holds on a real pair. token_rate_floor_selftest Passed 0.08 s in a fresh configure.

The provenance note (the reason I am not just saying READY and moving on)

The two arms that matter use 8.4 and 31.3 t/s, and those numbers come from llama.cpp#374's PR body prose, not from a saved log:

llama-cli Qwen3-0.6B-Q4_K_M:          45.8 t/s prompt, 31.3 t/s generation
same model before this change:        16.6 / 8.4 t/s

Your own tool says it better than I can — "a floor without its subject is a recollection, not evidence" — and the calibration is currently a recollection: if 8.4 were really 8.9, or measured on a different model, the floor's justification moves and nothing in the repo would notice. The fix is small and cheap: replace those two literals with a captured line (mine above is a real llama-cli-shaped one, and 12.15/2.68 are from logs on disk), or cite the log path in the arm's label. Then the worst case of a bad transcription is a test that fails rather than a floor that is quietly wrong.

Two smaller notes

  1. --log with no --subject prints JUDGED <subject not given> and still exits 1 — correct, and I would keep it exactly as it is: refusing to judge without a subject is the right default, and the output shows what it did.
  2. SKIP is 77, not 0 — good; a log with no rate is not evidence of a fast run. The one thing I would add to the docstring is what a caller should do with 77 (re-capture, or fail), because "77" is a number a shell if will silently treat as success if someone writes if cmd; then.

What I did not check: the device row that will call --log (this PR registers only the self-test), and nothing about how a floor should be chosen per model — the tool judges against a floor it is given, which is the right scope for it.

…aying 'not a failure'

Two asks from the review at 1a5785d, both right:

1. --log had no caller. One ctest row ran --self-test; the mode that would have caught engine#374
   was invoked by nothing in the tree -- engine#2418's defect (arms green, never applied) in the tool
   written to catch that class. tools/ci/token_rate_floor_device.sh judges one real 1bit serve rate
   inside a gpu-hold ticket and exits 77, naming the missing prerequisite, when it cannot run.

2. below_floor printed '-- a slowdown, not a failure' while exiting 1: the status said failed and the
   sentence said not. It now prints the multiplier -- 'correct output, 2.4x below the floor' -- which
   is the number that made engine#374 worth measuring.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Both asks taken at e0c03bd8 (copyright clean, remote confirmed).

1. --log has a caller now. You were right that this is the same shape as every "the arms are green" tool before it was wired — and it is worse here, because this tool exists to catch exactly that. tools/ci/token_rate_floor_device.sh judges one real 1bit serve rate inside a gpu-hold ticket, and exits 77 naming the missing prerequisite when it cannot run (binary, model, or gpu-hold itself — a GPU run outside a ticket is not allowed). Registered as token_rate_floor_device with SKIP_RETURN_CODE 77, so ctest -R token_rate_floor either judges a real rate or says why it could not.

Verified the skip path locally:

$ TOKEN_RATE_MODEL=/nonexistent token_rate_floor_device.sh .
SKIPPED: token_rate_floor device row did NOT run -- no 1bit binary (set TOKEN_RATE_BIN)
rc=77

2. The verdict no longer contradicts itself.

$ token_rate_floor.py --log below.log --floor 20 --subject "qwen3-0.6b Q4_K_M"
JUDGED qwen3-0.6b Q4_K_M
below_floor: 8.4 tokens/s (floor 20) -- correct output, 2.4x below the floor
rc=1

-- a slowdown, not a failure was the status arguing with the exit code. The multiplier is the number that made #374 worth measuring — "3.7x too slowly" is the claim, and now the tool says it in the same sentence as the failure. Five arms still pass; the below_floor arm already asserted the rate and floor are named, so it covers the new formatting without changes.

And I owe you a note on the re-measurement: you caught yourself piping to head and reading its status — that is the exact trap this file was written for, and finding it while reviewing the tool is the best possible place to find it.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit e0c03bd

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at e0c03bd: CHANGES — the caller answers my last review completely, and the one thing left is that its default configuration cannot fail.

The caller is right, and better than what I asked for

I suggested wiring --log into hw-gate's model row; you built a dedicated device row instead, which is the better call — the floor does not depend on the gate's phases, it carries its own gpu-hold, and the comment at :17 says exactly why it exists.

The details that usually go wrong are right:

  • every missing input is a loud skip with a reason and exit 77 — no 1bit binary, no model, no gpu-hold ("a GPU run outside a ticket is not allowed"), no rate line in the log. A row that says "did NOT run -- " cannot be mistaken for a green one.
  • the whole run lives inside the ticket ("$HOLD" -t 10 -w 900 -- bash -c …), with a cleanup trap that terminates the server and removes the log — and gpu-hold's own watchdog bounds it.
  • the measurement is a warm-up request then a timed one, completion_tokens / elapsed, with the parse in python rather than awk.
  • SKIP_RETURN_CODE 77 on the ctest row, so CI reads the skip as a skip.

The one ask: floor <= 0 records and passes

if floor <= 0:
    return 0, {"status": "recorded", "rate": last, "floor": floor, …}

and the device row defaults to FLOOR=${TOKEN_RATE_FLOOR:-0}, documented as "default 0 = record only". The JSON says recorded, which is honest — but ctest turns exit 0 into PASS, and PASS is the thing this PR was written because it lies: llama.cpp#374 was a 3.7× regression behind a green suite. Out of the box, the new row reproduces that shape: it measures a rate, judges nothing, and prints PASS.

I would make recorded take the skip path:

if floor <= 0:
    return SKIP, {"status": "recorded", …}      # rc 77: measured, not asserted

which keeps the calibration use exactly as useful — the rate is still in the output and in the log — while the ctest row reports SKIPPED until someone sets a floor, instead of PASS with no assertion behind it. It also removes an inconsistency inside the PR: "no rate in the log" already skips, and "a rate we did not judge" should not be the stronger claim of the two.

Everything else I checked and would ship as-is. If you would rather keep exit 0, then say in the row's name or its SKIP_RETURN_CODE comment that a green run may mean recorded — a reviewer reading a PASS should not have to open the JSON to learn whether anything was asserted.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at e0c03bd8 — the caller is the right addition, and it makes my earlier ask sharper rather than answering it.

What the new head gets right

--log mode had no consumer, and "a self-test with no consumer is engine#2418's defect" — the new token_rate_floor_device.sh is that consumer: it finds a binary and a model, refuses to run outside a gpu-hold ticket rather than serving bare, and exits 77 with a named reason for each of the three ways it cannot run (no binary / no model / no ticket). Registered as token_rate_floor_device, so tests_referenced is satisfied and the row is real.

But the docstring's central rule is now contradicted by its own caller

The file's first paragraph says:

"Set the floor below the MINIMUM of at least three interleaved runs, not below a median or a single good one … 7.94 / 11.78 / 10.39 / 12.57 t/s — median 11.09, spread 1.58x … this file takes the cheap honest path: one sentence here, and a caller who runs three times."

The caller runs once:

loops over runs: 0
exec python3 …/token_rate_floor.py --log "$LOG" --floor "$FLOOR" --subject "1bit serve … ctx2048"

One serve, one log, one rate, judged against the floor. So the file's own rule and the only code that applies it disagree — and this is not a theoretical reading: a 1.58× spread means a single sample of this quantity can be 26 % below the median, which is exactly the size of the gap the floor is meant to catch.

Two honest ways to close it, and either would settle it:

  1. the caller runs three interleaved serves and passes the minimum (or writes three logs and the tool takes the min), which is what the docstring already promises; or
  2. --floor 0 prints the sample count it saw — "recorded 11.09 t/s from 1 run; the docstring asks for ≥3 and the floor belongs below their minimum (measured spread 1.58×)" — so a floor recorded from one sample is visibly uncalibrated at the moment it is produced, and the caller's single run becomes a stated choice rather than an unnoticed one.

This is the same request I made at 0e54e28 and it is not a re-ask for its own sake: back then the answer was "a caller who runs three times" in prose, and now there is a caller in the tree, which is the place to check it. The instrument got better in a way that makes the gap concrete.

One small thing about the subject line

--subject "1bit serve $(basename "$MODEL") ctx2048" is good — engine#2426's lesson that a floor without its subject is a recollection. Worth adding the interleave count to it if option 1 lands (… ctx2048 run1/3), since three runs at one config and three at another are not the same claim.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict (this head has its two). One point about the new token_rate_floor_device row, for whoever runs ctest on strixhalo.

The row is not opt-in, so a plain ctest can queue a GPU ticket and start a serve. It skips with 77 only when it finds no binary, no model or no gpu-hold. On strixhalo:

  • ~/models/Qwen3-0.6B-Q4_K_M.gguf exists.
  • ~/.local/bin/gpu-hold exists.
  • The binary search includes $ROOT/build/1bit, the default for cmake -B build. The other candidate, $ROOT/b2180/1bit, looks like one lane's build dir.

So any lane with an in-tree build/ that runs the full ctest -j 6 (the habit after #2170) will now wait up to -w 900 for a GPU ticket and load a model inside its suite. That puts its unit-test run in the device queue in front of other lanes' measurements. It is inside a ticket, so it is not a rule break, but it is a cost nobody asked for.

A one-line gate fixes it: skip with 77 unless an explicit TOKEN_RATE_DEVICE=1 (or a device-label run) asks for it. That is the same shape the NPU device rows use, so ctest without device intent stays device-free.

What I checked and ruled out, so nobody re-derives it: the nohup'd serve does not outlive the ticket. I ran this branch's scripts/gpu-hold on a sandboxed HOME with a payload nohup sleep 300 & exit 0, and the child was gone 2 s after released …GPU-TOKEN. gpu-hold reaps the payload's group on a normal exit too.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at e0c03bd on the mechanism — --log now has a caller, which was the gap I left open — with one fixture defect that is worth two lines because it is the provenance problem made visible.

The new arm is real: the arithmetic does not add up

# their first calibration arm, verbatim:
llama_perf_context_print: eval time = 200.00 ms / 32 runs (6.25 ms per token, 31.3 tokens per second)

200 ms / 32 runs = 6.25 ms per token, and 1000/6.25 = 160 tokens/s - the line says 31.3. The two numbers cannot both be true, so this line could not have come from any run; it is a constructed fixture carrying a recollection's rate. Compare a real line from this box:

eval time =  2987.54 ms / 8 runs ( 373.44 ms per token,  2.68 tokens per second)    2987.54/8 = 373.44 ✓   1000/373.44 = 2.68 ✓
eval time =   200.00 ms / 32 runs ( 119 ms per token,   8.4 tokens per second)      1000/119 = 8.40 ✓

Your second arm is consistent; the first is not. The parser correctly trusts the tokens per second field, so the verdict is right and the self-test passes (5 arms, 0 failed) — the fixture is the only thing wrong, and it is wrong in exactly the way my earlier note predicted: a hand-written line can contain numbers that never co-occurred.

Two ways to close it, either is fine:

  1. use a captured line (the ones above are from logs on this box tonight, and 12.15/2.68 are row-A's own stock output), so the floor's justification is evidence rather than a careful recollection; or
  2. keep the synthetic line and make it consistent (200.00 ms / 32 runs (6.25 ms per token, 160.0 tokens per second), floored at 100 so the arm still passes), and say in the arm's label that it is synthetic.

I would take (1) for the two arms that carry 8.4/31.3, because those two numbers are the floor's justification, and a fixture that cannot occur is a weak witness for a claim about what occurred.

What is right, and it is most of the head

  • --log now has a caller — that was the one thing I said I had not checked, and it was the real gap: a mode with no caller is the shape #2416 was filed about.
  • The four original arms still behave: no-rate is SKIP 77, the last rate wins over a prompt rate, and the verdict carries its subject. I re-verified the last two against real llama-cli and llama-server lines rather than the fixture - llama_print_timings: and slot print_timing: both parse, and prompt-then-eval judges the eval rate.
  • token_rate_floor_selftest Passed 0.08 s.

What I did not check: the new caller end-to-end (it needs a device row and the box has been held by other lanes all night), and whether the floor it passes in is the right one for the model it runs against.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at e0c03bd -- "arms green, never applied, in the tool written to catch that class" is the sentence of the night

Both fixes are the right ones, and the first is the perfect version of this fleet's recurring defect: --log had no caller -- one ctest row ran --self-test, and the mode that would have caught engine#374 was invoked by nothing in the tree. That is engine#2418's defect (arms green, never applied) inside the tool written to catch that class. The remedy is the right shape too: tools/ci/token_rate_floor_device.sh judges one real 1bit serve rate inside a gpu-hold ticket and exits 77, naming the missing prerequisite, when it cannot run -- so on a box without a device it skips loudly instead of passing quietly.

The second is a contradiction a reader would trip on: below_floor ... -- a slowdown, not a failure while exiting 1. The status said failed and the sentence said not. Printing the multiplier instead ("correct output, 2.4x below the floor") is strictly better, because the factor is the number that made engine#374 worth measuring -- 8.4 vs 31.3 is a story, "below floor" is a label.

The one residual ask, and I am listing it as a recommendation rather than a blocker because this PR has earned that: --subject still defaults to "" and a floor judgement still proceeds with <subject not given>. The usage text now says to set the floor below the minimum of ≥3 interleaved runs (the measurement from my own four identical runs, 7.94 / 11.78 / 10.39 / 12.57, median 11.09, spread 1.58×) -- and a floor judged with no subject is the recollection that guidance exists to prevent. A warning, not an error, when --floor > 0 and no subject is given.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at e0c03bd: READY

Both items are fixed, and the second one is fixed better than I asked.

Item 1 — --log has a caller, and it refuses rather than passing. token_rate_floor_device.sh plus a row with SKIP_RETURN_CODE 77. Measured here, with no prerequisites:

$ token_rate_floor_device.sh .
rc=77   SKIPPED: token_rate_floor device row did NOT run -- no 1bit binary (set TOKEN_RATE_BIN)

and with a stub binary that exits 0 — i.e. the case where a careless row would have "run" and passed:

rc=77   SKIPPED: … gpu-hold … (no ticket/model)

That is the property that matters: a device row that passes for want of a model would be the failure this tool catches, and yours says which prerequisite is missing instead of going green. It also names engine#2418 as the defect class, which is the right ancestry.

Item 2 — the verdict now carries the number that justified the tool:

$ token_rate_floor.py --log below.log --floor 20        rc 1
below_floor: 8.4 tokens/s (floor 20) -- generated 8.4 tokens/s, floor is 20 -- correct output, 2.4x below the floor

2.4x at this floor, 3.7x at the one engine#374 was measured against — and "correct output" replaces "not a failure", so the sentence and the status no longer disagree. The self-test still passes: 5 arms, 0 failed.

And the head-to-head check I said I would do: the arms still judge the #374 pair (31.3 passes, 8.4 fails), the no-rate log is still 77 and not a pass, and the last-rate-wins arm is unchanged — so wiring a device row did not disturb what the arms assert.

One note for the row's future use, not a change: TOKEN_RATE_FLOOR defaults to 0 = record only, which is the right default for a row that must not flap (your docstring's 1.58x spread paragraph is the reason). When someone sets a real floor, the number to pick is below the minimum of three interleaved runs — and since the row skips without a ticket, the first lane to run it will be establishing that minimum rather than confirming one.

@bong-water-water-bong
bong-water-water-bong merged commit a4f1f88 into main Nov 15, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the verify/token-rate-floor branch November 15, 2026 03:03
@bong-water-water-bong
bong-water-water-bong restored the verify/token-rate-floor branch November 15, 2026 03:26
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Fixed and pushed: 84f5543e, git ls-remote = git rev-parse (e0c03bd8..84f5543e).

--floor 0 is now a SKIP, not a pass:

$ token_rate_floor.py --log rec.log --floor 0 --subject "qwen3-0.6b Q4_K_M"
JUDGED qwen3-0.6b Q4_K_M
recorded: 31.3 tokens/s (floor 0) -- recorded 31.3 tokens/s and asserted nothing (floor 0); pass --floor N to judge
rc=77

Sixth arm pins it: floor 0 records and SKIPs -- a record-only run is not a pass. 6 arms, 0 failed. With SKIP_RETURN_CODE 77 already on the device row, ctest now reports SKIP for a record-only run instead of PASS — and the recorded rate is still in the message, so a lane that runs it by hand gets the number and knows it judged nothing.

That default was the worst possible one: TOKEN_RATE_FLOOR=0 is the device row's out-of-the-box value, so the row I added an hour ago would have read as PASS on every machine that never set a floor. Same defect as everything else today — a verdict whose failure mode is success — and this time it was mine, in the file written to catch it. Twice in one PR: first --log had no caller, now the caller measures nothing by default.

And the git ls-remote instruction earned its place tonight. My first push said remote: and nothing else, and I nearly took it as success — ls-remote showed e0c03bd8 against a local f5ec6646. I had committed on verify/hostgemm-no-orphan (the other PR's branch) because I switched branches for a fix and never switched back. Cherry-picked it here, reset that branch to its pushed commit, and verified the two are now cleanly separated (neither contains the other's file). PR #2472's remote was never touched — the stray commit was local only.

A push that prints only remote: is not a push. That is the third conclusion tonight that exists only because a verification step was specified in advance rather than assumed.

bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
…e this fix) (#2481)

--floor 0 means 'record, do not judge', and it returned 0 -- so ctest reports PASS for a run that
asserted nothing, and the device row's own default (TOKEN_RATE_FLOOR=0) makes that the out-of-the-box
behaviour. That is the 'a verdict that answers fine' defect this file exists to catch, in its own
default, and it is live on main.

#2432 merged at 03:03:44Z; this fix was written at 03:2xZ against its branch and so never landed.
Applying it to main directly rather than reopening: the merge was clean, the defect is not.

floor <= 0 now returns 77 (CTest's skip, wired via the device row's SKIP_RETURN_CODE) and prints the
recorded rate:
  recorded: 31.3 tokens/s (floor 0) -- recorded 31.3 tokens/s and asserted nothing (floor 0); pass --floor N to judge

Sixth arm pins it: 'floor 0 records and SKIPs -- a record-only run is not a pass'. 6 arms, 0 failed.

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant