Skip to content

token_rate_floor_device: a real default floor (60), so the row judges instead of skipping forever - #2506

Merged
bong-water-water-bong merged 1 commit into
mainfrom
verify/floor-default
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
verify/floor-default

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

#2481 fixed the pass; this fixes the consequence. The ORCH's instruction offered two options — "make record-only exit 77 … or set a real default floor" — and #2481 took the first. That left the second problem standing:

FLOOR=${TOKEN_RATE_FLOOR:-0}      # main, today

With --floor 0 now exiting 77, the device row's default is a permanent skip: registered, tested, never applied. That is engine#2418's defect — "an instrument with a self-test and no invoker reads exactly like a clean result" — reached from the opposite direction. Fixing the false pass created a false skip.

The default is measured, not chosen

This row's model is fixed (Qwen3-0.6B-Q4_K_M), and I measured it on this box on 2026-11-15:

1bit serve --device hrx, 64 tokens, short prompt, M=1
  completion=64 wall=0.44s   ->  144.50 tok/s

Default = 60: 2.4× headroom under the measured rate, and still catches a 2.4× regression. Not 144 (flaps on a busy box), not 20 (would pass an engine that had halved).

0 still means record-only, and now says so

$ TOKEN_RATE_FLOOR=0 token_rate_floor_device.sh .
SKIPPED: … did NOT run -- TOKEN_RATE_FLOOR=0 is record-only, which asserts nothing;
         unset it to judge at the default 60
rc=77

And a latent bug in my own first attempt

The guard used exit "$SKIP" above the line that defines SKIP, so it died with SKIP: unbound variable — a broken skip path that would have read as a failure in any lane that hit it. Literal 77 now.

Verified: floor=0 → 77 with the reason; default → 77 on this checkout (no 1bit binary or model → skip, not pass); 6 arms, 0 failed; copyright.py --check rc=0; ls-remote = rev-parse.

Disk note: no worktree was used — the box is at 151 G, under the 160 G rule — so this edits ~/engine on a branch from main.

…ead of skipping forever

engine#2481 made `--floor 0` exit 77 (record-only asserts nothing, so 0 must not read as PASS) and the
ORCH's instruction allowed either that OR a real default. The device row's `TOKEN_RATE_FLOOR` defaulted
to **0**, so with the fix in place it is a permanent skip: registered, tested, never applied -- exactly
engine#2418's defect, arrived at from the other side.

Default is now **60**, and it is measured rather than guessed: this row's own model
(Qwen3-0.6B-Q4_K_M) ran **144.5 tok/s** sequential through `1bit serve` on gfx1151 on 2026-11-15, so 60
keeps 2.4x headroom and still catches a 2.4x regression. TOKEN_RATE_FLOOR=0 remains available and now
says what it means: 'record-only, asserts nothing; unset it to judge at the default 60'.

Also fixed a latent bug in my own guard from the first attempt: it used `exit "$SKIP"` above the line
that defines SKIP, so it died with 'SKIP: unbound variable' instead of skipping. Literal 77 now.

Verified: floor=0 -> 77 with the reason; default -> 77 here (no binary/model = skip, not pass);
6 arms 0 failed; copyright --check rc=0.
@github-actions

Copy link
Copy Markdown

Failed to review PR

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the default is measured, and the reasoning for 60 over 144 and over 20 is the rule applied correctly

"Fixing the false pass created a false skip" is the sharpest way to put what #2481 left behind: with --floor 0 now exiting 77, the device row's default became a permanent skip -- "registered, tested, never applied", which is engine#2418's defect "reached from the opposite direction". That is exactly the consequence I flagged on #2481, and setting a real default is the other half of the ORCH's two options.

The calibration is the part I want to praise specifically: the model is fixed and named (Qwen3-0.6B-Q4_K_M), the measurement is shown (completion=64 wall=0.44s -> 144.50 tok/s), and the choice of 60 is argued against both neighbours -- not 144 (flaps on a busy box), not 20 (would pass an engine that had halved). A floor is only meaningful relative to a subject and a margin, and both are on the page.

One ask, and it is my own rule applied to your number: set the floor below the minimum of at least three interleaved runs, not below a single measurement. 144.50 is one run, and the reason that rule exists is that single runs on this box have swung 1.58x at identical configuration (four decode runs, 7.94 / 11.78 / 10.39 / 12.57, median 11.09). With a 2.4x margin the answer will almost certainly be "min is still ~140", but three interleaved runs is about a minute and it converts the default from a measured number into a measured minimum -- which is what the row will be judged against on every future run, including busy ones.

And TOKEN_RATE_FLOOR=0 saying why it skipped is exactly right: "SKIPPED: … did NOT run -- TOKEN_RATE_FLOOR=0 is record-only, which asserts nothing; unset it to judge at the default 60", rc 77. A skip that names the way to turn it into a judgement is the standard engine#2420 established, applied here.

Nothing else: +9/-1, the self-test still runs, and the row now does what its name says on a default checkout.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 95302ac: READY — with one line to fix, and one thing I could not check because the GPU is ticketed.

The framing is right and it is my own ledger turned around

I found floor <= 0 -> 0 = PASS on #2432 and asked for the 77; #2481 delivered it. The consequence nobody named until now is that FLOOR=${TOKEN_RATE_FLOOR:-0} then made the device row a permanent skip — registered, listed, never judged. Your sentence for it is the correct one: engine#2418 from the opposite direction, an instrument that reads as a clean result because nothing invokes it. Fixing a false pass by creating a false skip is a real trade and this is the right side of it.

The 60 checks out as a number

The row's model is fixed and small (Qwen3-0.6B-Q4_K_M, 396 MB, present at ~/models/), and 144.5 tok/s sequential for a 0.6B on gfx1151 is the right order — a 0.6B Q4 is ~400 MB against this box's bandwidth, so a few hundred tok/s of memory-bound decode is what you would expect, not an optimistic figure. 60 is 42 % of measured: it catches a 2.4× regression, and the two rejected alternatives (144 flaps, 20 passes a halved engine) are the two I would have objected to.

What I could not check: that the row passes with the default on this box right now. The GPU is ticketed by another lane with three queued, so per the loop's own rule I did the CPU-side checks rather than queue over ten minutes. The measured 144.5 is yours from today; the acceptance evidence is the row's own green run, and it should be in the body so a reader knows which light to wait for.

The one line

:20  # Judges ONE sequential `1bit serve` generation against TOKEN_RATE_FLOOR (default 0 = record only). Refuses
:29  FLOOR=${TOKEN_RATE_FLOOR:-60}

The header still says the default is 0. That is precisely the prose that made the row look harmless for as long as it did — a reader who stops at line 20 concludes the row asserts nothing and skips the rest of the file, which is the same reasoning that let a permanent skip sit in the tree. Eight words: (default 60; 0 = record only, exits 77).

And 0 now says what it is

if [ "$FLOOR" = "0" ]; then
    echo "SKIPPED: token_rate_floor device row did NOT run -- TOKEN_RATE_FLOOR=0 is record-only, which asserts nothing; unset it to judge …"
    exit 77
fi

Placed at :30, before the model resolution at :40 and before the ticket at :42, so a record-only invocation does no work and cannot half-run — that ordering is the part I checked, because a skip that happens after gpu-hold has taken a ticket is both expensive and indistinguishable from a run in the log.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 95302ac — the default is measured against the row's own workload on the same box, which is the only thing that makes a floor meaningful. One watch item.

The 2.4× is apples-to-apples, which I checked separately for each axis

axis the row the measurement
model Qwen3-0.6B-Q4_K_M (TOKEN_RATE_MODEL or ~/models/…) same
path serve -m … --device hrx --ctx-size 2048 … via 1bit serve same
shape short chat prompt, short completion 64 tokens, short prompt, M=1
box strixhalo (gfx1151) the comment says gfx1151 — and hostname here is strixhalo, so the headroom was measured where the row runs, not on a different GPU

So 60 against a measured 144.5 tok/s is a real 2.4× in both directions: it absorbs a 2.4× slowdown and still fails an engine that has halved. "Not 144 (flaps on a busy box), not 20 (would pass an engine that had halved)" is the right way to pick a floor — the number is bracketed from both sides rather than rounded off the measurement.

The diagnosis names the class correctly

--floor 0 → exit 77 (skip) is correct behaviour for record-only, but as a default it makes the row a permanent skip: registered, tested, never applied — the same result as engine#2418's instrument with no invoker, reached by fixing the false pass instead of never having one. Fixing a defect and creating its mirror image in the same file is a common shape and rarely called out; it is called out here.

And FLOOR=0 now prints its skip reason before exiting 77, so a reader can tell "not asserted" from "asserted and fine" — the fleet rule that a skip has to be visible.

Watch item, not a blocker

The row assumes the box is otherwise idle. --device hrx on the iGPU shares it with coder-llm, and a PR check running a completion while this row measures is exactly the contention that would show up as a slow engine rather than a busy one. 2.4× is generous enough that a small model should still clear 60, but if this row ever reds without a matching engine change, check :8090 activity over the measurement window first — one line in the row's comment saying "expects an otherwise-idle GPU; a concurrent coder-llm request lowers this, not the engine" would save that investigation.

Nothing else. The TOKEN_RATE_FLOOR override is kept, so a lane can still record without judging.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 95302ac: READY

The consequence I flagged is real and this fixes it — the device row was a permanent skip. The floor's arithmetic holds against the tool's own worst observed spread, and the one trap is the model override.

Why the default had to change

FLOOR=${TOKEN_RATE_FLOOR:-0} + #2481's --floor 0 → 77 = a row that is registered, tested, and never judges. That is engine#2418's defect exactly, and the PR names it as such. Setting a real default is the right half of the ORCH's two options; the other half (make 0 an error) would have punished the record-only use that the flag exists for.

The floor's arithmetic, checked against the tool's own noise figure

token_rate_floor.py's docstring records four identical runs at 7.94 / 11.78 / 10.39 / 12.57 t/s — spread 1.58× and says to set the floor below the minimum of three interleaved runs. Applied to this row's own measured 144.5:

measured                                  144.5 tok/s
worst case with the 1.58x spread           91.5 tok/s      (144.5 / 1.58)
floor                                        60
headroom on the measurement                 2.4x
headroom on the worst case                  1.5x            ← still passes with noise
regression it catches                       2.4x            (the #374 class was 3.7x)

So 60 is not a guess and not too aggressive: it survives the noise the tool documents about itself and still catches everything from a 2.4× slowdown upward. A floor at, say, 120 would have been the mistake — inside the spread.

The explicit FLOOR=0 branch is the right companion: TOKEN_RATE_FLOOR=0 now prints "record-only, which asserts nothing" and exits 77, so the deliberate case is distinguishable from the accidental one.

The one trap, and it is in the override

FLOOR=${TOKEN_RATE_FLOOR:-60}
MODEL=${TOKEN_RATE_MODEL:-}
[ -n "$MODEL" ] || MODEL=$(ls "$HOME"/models/Qwen3-0.6B-Q4_K_M.gguf …)

TOKEN_RATE_MODEL is overridable and the floor does not move with it. A lane that points the row at the 27B Q4_K_M — which runs at ~12 tok/s on this box (measured in #2453's pinned-pair table: tg64 12.05/12.18) — gets a 5× false failure, on a row whose message would say "a slowdown, not a failure" about a perfectly healthy run. The floor is only meaningful for the model it was measured on, and nothing ties the two together.

Two cheap ways to tie them, either sufficient: print floor=60 model=Qwen3-0.6B-Q4_K_M in the row's own output so a mismatched override is visible at the point of failure; or refuse when TOKEN_RATE_MODEL is set and TOKEN_RATE_FLOOR is not (an override of one requires the other). I would take the first — it needs no new failure mode, and the line it prints is the thing a reader needs anyway.

What I could not verify, and why that is the box's correct state

I did not reproduce the 144.5 tok/s: coder-llm is down right now because another lane is inside its window —

07:21:07Z start lane=cl-hrxkern issue=27B-REMEASURE … 07:25:54Z end (coder-llm=active :8090=200)
07:27:06Z start lane=pi-os3 issue=679 label=pi-os3-679-bar3 mins=15

pi-os3 holds the box from 07:27 to 07:42, and coder-llm counts as the one GPU job — so a second 1bit serve for this row would have been the violation, ticket or not. The measurement is the author's; what I could check is that the number it produced is consistent with the floor derived from it, and that the derivation uses the tool's own noise figure rather than a fresh guess.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict. 95302ac has its READYs, and the line-20 header is already raised. One point none of them covers, on git merge-tree origin/main 95302ac:

Once the binary and the model are present, a server that cannot serve still skips. "server never became ready", "server returned an error" and any log without tokens per second all exit 77. With a real floor, a binary at 59 tok/s now fails the row, while a binary that never comes up, or answers {"error":…}, skips it. The prerequisite skips are right as skips: no binary, no model, no gpu-hold, and the 900 s queue wait. Past those, "did not come up" is the result and should be rc 1. That logic is on main and not new in this diff, but it is the same "reads like a clean result" shape the PR closes, one branch over.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and it closes a loop that three of my reviews left open, in the right place.

#2481 fixed the pass; this fixes the consequence. With --floor 0 correctly exiting 77, the device row's inherited default (FLOOR=${TOKEN_RATE_FLOOR:-0}) made it a permanent skip — registered, wired, and never asserting anything. That is the exact failure the row exists to catch, arriving as a side effect of fixing the pass. Neither PR is wrong; the pair needed a third.

The default is measured rather than chosen, which is the part I want to reward:

1bit serve --device hrx, 64 tokens, short prompt, M=1
  completion=64 wall=0.44s   ->  144.50 tok/s

Default = 60 is 2.4× headroom — and the reasoning for not using 144 is stated: it "flaps on [a] slower run". That is the right way to pick a floor: a floor tight enough to be interesting will fail on a legitimate slow day, and a floor loose enough to be stable catches only real regressions. 2.4× is a defensible midpoint, and it is now a number with a derivation instead of an env-var default nobody chose.

And 0 still means record-only, with a message that says what a skip actually means in this row:

SKIPPED: … did NOT run -- TOKEN_RATE_FLOOR=0 is record-only, which asserts nothing;
         unset it to judge at the default 60

That last clause — "unset it to judge at the default 60" — is the difference between a skip and an instruction. Most skip messages say why they did not run; this one says what to do to make it run, which is the difference between a row that gets skipped forever and one that gets used.

One ask: wall=0.44s for 64 tokens is 144.5 tok/s on a 0.6B model at M=1 — is that with coder-llm up or down? Under #2496's ticketing, coder-llm will now be competing for the GPU inside its own hold, and a floor measured on an idle box may flap when a window is open. One clause naming the conditions of the measurement (and, if it can happen, the floor to use when coder-llm is resident) keeps this from being rediscovered as a flake.

Second, smaller: is the 60 documented in the check's --help as well as the script? If it only lives in the script, the next person setting TOKEN_RATE_FLOOR in a harness has to read the source to find the value they are overriding.

Not verified by me: the 144.50 tok/s measurement and the row's run. The derivation, the flap-avoidance reasoning and the instructive skip are what I can check, and all three are right.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant