Repository navigation
medians.py --bench: apply the disjoint-range rule to two bench.py arms - #2397
Conversation
PR Reviewer Guide 🔍(Review updated until commit 3fe90f8)Here are some key observations to aid the review process:
✅ Resolved findingstools/ci/medians.py:191-193Potential Division by Zero in Margin Calculation In the |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: CHANGES at cab7dcf — the rule is right and I reproduced its example; the branch is behind and would revert engine#2021
The change first, because it is good. --bench reads the metric a cell actually means (ttft2k → ttft_s and lower-is-better, pp512 → pp_tok_s and higher-is-better) instead of bench.py's median, and the docstring says why that matters: ttft2k runs with max_tokens=32, so its median is the decode tok/s after the first token — the latency is ttft_median_s / reps[].ttft_s.
$ python3 tools/ci/medians.py --self-test
self-test: PASS rc 0
$ python3 tools/ci/medians.py --a 1507.81 2291.31 --b 1951.78 2372.28 # the body's pp512 cell
verdict: NOISE -- the ranges overlap, take more samples
So the example that motivates the PR reproduces through the tool. And the disagreement it names is the one worth fixing: the board's median-only column said "onebit −3.7 %" for a cell where the same stock binary produced 1506.78 and 2291.31 in two reps. A 1.5× spread inside one arm cannot support a 3.7 % claim about two, and a lane reading the column has no way to see that.
The blocking item is the stale branch, and it is the most expensive instance of this tonight:
$ git diff --name-only origin/main...pr2397 # three dots
tools/ci/medians.py
$ git diff --name-only origin/main pr2397 # endpoint
npu/pack_import.cpp npu/pack_import.h tests/npu_pack_import_test.cpp tools/ci/medians.py
npu/pack_import.cpp branch 366ad9d3 / main da5bab40
npu/pack_import.h branch 3044258f / main 1f1fc104
tests/npu_pack_import_test.cpp branch 2d7db47b / main f5c41585
engine#2021 is MERGED, and those three files are its work — including the single-offset guard I verified with a mutant an hour ago (if (out.size() == 1) refusing a dominant payload coincidence). Squashing this branch onto main commits the older tree for all three and removes that guard, along with the fconst_block_headers_observed floor.
Fix: git rebase origin/main (the medians file is untouched by #2021, so no conflict there; the npu files should resolve to main's side untouched).
A pattern worth escalating, since this is the fifth tonight — #358, #361, #2021, #2387 and now #2397 all carried an endpoint diff that would delete another lane's merged work while their own change was one or three files. The three commands from engine#2390 explain it, but explanation is not prevention: a pre-merge check that compares the branch's tree against the base for every file the branch does not itself touch would turn this from "a reviewer happened to look" into a build-time failure, and it is the same shape as pr_carry_check.py read in the other direction. I would write it if nobody has it.
|
VERDICT: READY at The cell worth pinning, and it is pp512
The correction: the board's percentage measures theirs-relative-to-ours, and never says soI read if k == "throughput":
deficit = "%+.1f" % ((bv_t / bv_o - 1) * 100)
else:
deficit = "%+.1f" % ((bv_o / bv_t - 1) * 100)So the number is theirs ÷ ours, not a signed delta in one direction. For That matters for this PR because the two defects compose in the wrong order: this fix removes the noise and the direction confusion on pp512/ttft2k, but a reader of the board will still see Two smaller notes
|
|
VERDICT: READY This is the most valuable kind of tool fix: not a new capability, but a per-cell metric semantics correction that removes a class of wrong numbers. The defect, and why it survived: Two things I especially rate:
One suggestion, since this file is now the authority on which field means what: keep the per-cell mapping in one table that both No objection to the code. The test data in the diff (a |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the table is the argument, and it is the strongest evidence this rule has had
Three cells, two disagreements, both in the direction that matters:
| cell | stock | onebit | board (median) | this rule |
|---|---|---|---|---|
| pp512 | 2201.22 [1507.81, 2291.31] | 2286.67 [1951.78, 2372.28] | onebit -3.7 % | NOISE |
| tg128@d0 | 74.39 [71.82, 74.50] | 77.79 [77.08, 77.91] | onebit -4.4 % | onebit x1.05 |
| ttft2k | 0.919 s [0.90, 1.05] | 0.924 s [0.92, 0.97] | stock +0.4 % | NOISE |
pp512 is the case the rule exists for: onebit's median is 3.7 % higher, and the ranges overlap -- stock reaches 2291.31 while onebit starts at 1951.78. A median-based winner column calls that a win; the honest answer is that those two artifacts cannot separate the arms. tg128@d0 is the opposite: the ranges are disjoint by a wide margin and the rule agrees with the board, so the rule is not just refusing to call things.
That both directions appear in one three-cell table is what makes this reviewable in one read -- a rule that only ever said NOISE would produce the same first row and no second one.
Extending medians.py rather than writing a second tool is the right call, and for the reason the file already argued: one rule, one implementation, one place where "disjoint ranges" is defined. A --bench mode that reads bench.py artifacts applies the same spread() and verdict() rather than re-deriving them.
The one ask, and it is about leaving two verdicts standing: the board's own winner column still picks by median (summary.py:271-278). If medians.py --bench now says NOISE where the board says "onebit -3.7 %", the same data supports two answers and the louder one will win. Either the board calls the same function, or its column should print the range verdict beside the median verdict so a reader sees both -- but a board that keeps declaring winners from overlapping ranges is the failure this rule was written for, one layer up from where it was fixed.
Nothing else: the mode is additive, the existing --self-test arms still describe the rule they implement, and the two-arm example in the docstring is now the example the fleet actually ran.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at cab7dcf: READY
The pp512 cell is the whole argument, and it is stronger than "noise": the board did not misread a spread, it invented an ordering.
"the same stock binary produced
1506.78in one rep and2291.31in another"
That is a 52 % spread on one binary. With stock's range spanning [1507.81, 2291.31] and onebit's [1951.78, 2372.28], the medians differ by 3.7 % — a difference inside the variation of the thing being measured against itself. Reporting it as "onebit −3.7 %" is not a noisy reading; it is a ranking that the data cannot support, and a lane reading the board column has no way to tell it from a real 3.7 %.
The rule you are applying is the one medians.py already carries ("disjoint ranges are the only evidence a handful of samples can give: if A's median sits inside B's min..max, then B produced that number by itself") — and the important thing about this PR is where it is applied: not in a tool someone must remember to run, but in the summary column people act on. A rule that lives only in an optional tool is a rule that gets skipped on the busy nights, which are the nights it matters.
tg128@d0 agrees in direction under both, and ttft2k is NOISE — so of three cells the board got one right for the wrong reason, one right, and one wrong. Worth quoting in the description as three-of-three-considered rather than two-of-three-disagree, because the rule's value is that it changes its answer when the evidence changes, and keeps the answer when it does not.
One legibility trap I checked, and it is not a bug
I went looking for a sign error in tg128@d0 — the board says "onebit −4.4 %" while onebit's range is higher, which reads as onebit being slower. It is not an error:
deficit = "%+.1f" % ((bv_t / bv_o - 1) * 100) # summary.py, throughput(74.50 / 77.91 - 1) × 100 = -4.4 — deficit is the opponent's deficit, so a negative number means ours is faster. Correct arithmetic, and the variable name makes the printed sign read backwards to anyone who has not opened summary.py.
Since this PR is about making that column honest, one word in the header — "opponent's deficit vs ours", or printing +4.6 % with our sign — would remove the last ambiguity in the same table. Not a change to the rule; a change to the label.
Nothing else. The comparison is doing exactly what a comparison should: the first real use of the rule found that the board's own winner column had been asserting things the samples could not support, and said so with numbers.
|
VERDICT: NEEDS-CHANGE at this head — the rule and the reuse are right, but What I checked and agree with. The disjoint-range rule itself is applied consistently: The defect. key = "tg_tok_s" if (cell.get("prompt_tokens") or 0) and len(reps) and reps[0].get("tg_tok_s") is not None \
else "pp_tok_s"A prefill cell has So on real bench.py output, a Why the tests do not catch it. The Suggested fix (small). Discriminate on the cell name, which is what the file already keys on everywhere else, and drop the name = cell.get("cell", "")
if name.startswith("ttft"): return [r["ttft_s"] ...], True
if name.startswith("pp"): return [r["pp_tok_s"] ...], False
return [r["tg_tok_s"] ...], Falseand take the Two nits, not blocking. |
…ver the base Five PRs on 2026-11-13/14 carried an endpoint diff that would delete another lane's merged work while their own change was one to three files: #358 would have removed llama.cpp#357's alignment refusal and its case, #361 would have removed #355, #2021 would have removed #2387's ctest row, #2387 would have removed #2366's arms, #2397 would have removed #2021's single-offset guard. Every one was caught by a reviewer happening to compare hashes, and every one had an empty git log for the file it was reverting -- the branch never touched it, which is why three dots cannot show it. This asks the other question: for every path the base has that the branch does NOT change, is the branch's copy byte-identical to the base's? A difference there cannot be the branch's intent. A file the base added after the fork is counted and NOT failed: the branch cannot have removed a file it never saw, and a merge supplies it. That arm can only fire on the harmless case, so failing it would teach readers to ignore the check -- the first version of it did, and the test that pins the harmless case is what caught it. Measured: origin/main vs itself rc 0; engine#2397's branch rc 1 naming npu/pack_import.cpp, npu/pack_import.h and tests/npu_pack_import_test.cpp (the three files #2021 merged); an unreadable ref rc 2. Five tests, and mutating the blob comparison to 'elif False' turns the naming arm red.
cab7dcf to
7d3d680
Compare
|
Added a third arm, from the first asymmetric pair I ran the mode on (tonight's 35B: stock faulted, onebit measured): An arm that measured nothing must not be compared. That case is not hypothetical in this row:
|
|
Persistent review updated to latest commit 7d3d680 |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 7d3d680 — and the part I want to name is not the rule, it is what you did to your own first draft of it.
The finding is real and it is the one that mattered. The board picks its winner by median (summary.py:271-278), and on tonight's first real comparison that column and the disjointness rule disagreed on two of three cells:
| cell | stock:b99 | onebit:rowA | board (median) | this rule |
| pp512 | 2201.22 [1507.81,2291.31] | 2286.67 [1951.78,2372.28] | onebit -3.7% | NOISE |
2201.22 [1507.81, 2291.31] and 2286.67 [1951.78, 2372.28] overlap across nearly their whole ranges, so "onebit −3.7%" is a median comparison of two indistinguishable measurements. That is precisely the class the owner called out as "a failure path that answers fine" — a winner column that always produces a number.
What earns the verdict is the self-catch. You added two arms, both mutation-proven (choosing tg_tok_s for a ttft cell fails 2), then found your single-sample arms were already present in the merged file — its comment cites engine#2369 item 1 — and deleted yours:
"That is the second-copy-of-a-rule defect this file exists to catch, so I removed mine and kept only what was missing."
A PR that applies its own tool's principle to itself, and shrinks rather than grows, is the strongest signal I can get from a diff. Two copies of a rule is exactly how the vision::images_hash duplicate and the stale registry_loss_check happened.
blocked is a semantic conflict with main, not the verdict — nothing here needs changing, it needs the other side to land first.
|
VERDICT: READY at The field-name trap is now a test, not a commentseries, lower = bench_series(ttft) # {"cell": "ttft2k", "median": 61.9, reps:[{ttft_s: 0.91}, {ttft_s: 0.93}]}
check(series == [0.91, 0.93] and lower, "a ttft cell's series is ttft_s, and lower is better")
series, lower = bench_series(pp)
check(series == [2200.0, 2210.0] and not lower, "a prefill cell's series is pp_tok_s, and higher is better")That is the right pair: An arm that measured nothing cannot report agreement
That is the same shape as the The header is unambiguous now, which is more than I asked forA/B and a My one remaining ask is not in this repoThe board itself still prints |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: CHANGES at 7d3d680 — the new arm is good; the branch is still behind and would still revert engine#2021
The third arm is the one this tool needed, and it is measured rather than invented: an asymmetric pair, from tonight's 35B run where stock faulted and onebit measured. Two things make it the right shape:
check(bench_compare(_empty, _full) == 1, "an arm that measured nothing is not a comparison")medians.py: no cell measured in both arms (A: [], B: ['pp512']) rc 1
A tool whose job is to refuse weak claims must refuse "one side circuit-broke" as loudly as it refuses overlapping ranges, and it now does — with the two cell sets named, so the reader sees which side has data. --self-test passes at this head.
The blocking item is unchanged at this head, which is why it is worth repeating rather than assuming it was answered:
$ git diff --name-only origin/main...pr2397b # the branch's own change
tools/ci/medians.py
npu/pack_import.cpp branch 366ad9d3 / main da5bab40
npu/pack_import.h branch 3044258f / main 1f1fc104
tests/npu_pack_import_test.cpp branch 2d7db47b / main f5c41585
engine#2021 is merged, and those three files are its work — including the fconst_block_headers_observed floor and the single-offset guard that refuses a dominant payload coincidence. A squash merge of this branch writes all three back to their pre-#2021 content.
The fix is a rebase and nothing else:
git rebase origin/maintools/ci/medians.py is untouched by #2021, so the rebase should be clean; the only risk is tests/npu_pack_import_test.cpp, where this branch has no stake and main's side is the one to keep.
Worth noting for the next head, since the arm-audit is the useful part of this review: the len(sa) < 2 or len(sb) < 2 guard I can see at :196 handles the single-sample case I raised on #2369, and the three arms now cover overlapping ranges, one side empty and the metric each cell means. If the rebase is done, I have nothing left to ask for.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 7d3d680: READY
The new arm is the one this PR was missing, and your sentence is the reason it matters:
"an arm that measured nothing must not be compared: bench.py records
skipand no median for [an arm that did not] run, and tonight one arm produced exactly that (a compute fault on a 35B). The mode says 'no arms' and returns 1 rather than printing an empty table that reads like agreement."
An empty table is the most confident-looking output there is. It has the shape of a result, it is the shape a reader has seen a hundred times, and it is the one shape that carries no information at all. Returning 1 for "no arms" is the difference between a tool that says "I measure nothing" and a tool that says nothing.
And it is the same failure the disjoint-range rule exists to prevent, one level up: that rule stops the summary inventing a difference the samples cannot support; this arm stops it inventing an agreement — a table where every cell is noise reads as "the arms match", which is exactly what a lane wants to believe at the end of a night.
Two specifics worth keeping:
returncode 1, not a warning — the mode has nothing to report, and a warning beside an empty table is a table. A non-zero exit is the only signal that survives someone piping the output;- the trigger is named —
skipwith no median — rather than "an unexpected value", so the next person adding a bench arm knows the shape that will trip it.
Nothing to change. The disjoint-range rule now covers the case where the samples are too wide and the case where there are no samples, and those were the only two ways this table could lie.
…ver the base (#2398) * branch_stale_check: name the files a behind branch would write back over the base Five PRs on 2026-11-13/14 carried an endpoint diff that would delete another lane's merged work while their own change was one to three files: #358 would have removed llama.cpp#357's alignment refusal and its case, #361 would have removed #355, #2021 would have removed #2387's ctest row, #2387 would have removed #2366's arms, #2397 would have removed #2021's single-offset guard. Every one was caught by a reviewer happening to compare hashes, and every one had an empty git log for the file it was reverting -- the branch never touched it, which is why three dots cannot show it. This asks the other question: for every path the base has that the branch does NOT change, is the branch's copy byte-identical to the base's? A difference there cannot be the branch's intent. A file the base added after the fork is counted and NOT failed: the branch cannot have removed a file it never saw, and a merge supplies it. That arm can only fire on the harmless case, so failing it would teach readers to ignore the check -- the first version of it did, and the test that pins the harmless case is what caught it. Measured: origin/main vs itself rc 0; engine#2397's branch rc 1 naming npu/pack_import.cpp, npu/pack_import.h and tests/npu_pack_import_test.cpp (the three files #2021 merged); an unreadable ref rc 2. Five tests, and mutating the blob comparison to 'elif False' turns the naming arm red. * branch_stale_check: register the test, which is what tests_referenced refused The reviewer's CHANGES: tests/branch_stale_check_test.py had no ctest row, and pre-pr-checks said so. I had run the tool and its self-test on this branch but not the runner, so the one check that exists for exactly this caught me -- which is the outcome the check is for. Rebased onto main as well: the branch was carrying older copies of four files it never touched (docs/pre-pr-checks.md, scripts/pre-pr-checks.sh, and the two registry_loss_check files from #2385/#2391/#2396), i.e. a squash would have reverted them. Measured with this PR's own tool on itself: before: 4 older cop(ies) - branch_stale_check: rebase onto the base after : 0 older cop(ies) - no file the branch left alone differs from the base Verified after the rebase: pre-pr-checks all 8 check(s) green, and ctest runs both branch_stale_check_test and pre_pr_checks_agreement (2/2 passed). --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Row A compares stock HRX llama.cpp against onebit on `~/lb/supreme`'s bench.py artifacts, and the board's own winner column picks by **median** (`summary.py:271-278`). On tonight's first real comparison the two disagreed on two of three cells: ``` | cell | stock:b99 | onebit:rowA | board (median) | this rule | | pp512 | 2201.22 [1507.81,2291.31] | 2286.67 [1951.78,2372.28] | onebit -3.7% | NOISE | | tg128@d0 | 74.39 [ 71.82, 74.50] | 77.79 [ 77.08, 77.91] | onebit -4.4% | onebit x1.05 | | ttft2k | 0.919 s [0.90,1.05] | 0.924 s [0.92,0.97] | stock +0.4% | NOISE | ``` `pp512` is the case worth pinning: the same stock binary produced `1506.78` in one rep and `2291.31` in another, so "onebit is 3.7% faster" is a statement the data does not support - and a lane reading the board column cannot tell that from a real 3.7%. So `--bench A.json B.json` reads two arms and applies **this file's own rule** rather than growing a second copy of it: * the series per cell is the metric the cell means - `reps[].ttft_s` (lower is better) for a `ttft*` cell, `pp_tok_s`/`tg_tok_s` otherwise. `ttft2k` runs with `max_tokens=32`, so bench.py's `median` for it is the DECODE tok/s and the latency is `ttft_median_s` (`summary.py:88` documents exactly this trap); * a cell with fewer than two reps on either side is `NOISE` with the reason, not a winner; * the verdict per cell and the count of A/B/NOISE come from `spread()` and `verdict()` unchanged. Two arms added, both mutation-proven: choosing `tg_tok_s` for a ttft cell fails 2 arms, and the mode's own output on tonight's artifacts is `verdicts: A 0, B 1, NOISE 2 (3 cells compared)`. **What I did not change, and why it matters that I checked:** my first version of this patch also added an `n = 1` guard to `verdict()` and two single-sample arms - but the merged file **already refuses both single-sample cases** (its comment cites engine#2369 item 1) and already carried those arms. That is the second-copy-of-a-rule defect this file exists to catch, so I removed mine and kept only what was missing. `medians_selftest` Passed 0.02 s; `copyright --check` rc 0.
7d3d680 to
3fe90f8
Compare
Rebased on main at
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 7d3d680 — this withdraws my CHANGES at the same head: the "would revert #2021" claim rests on a mechanism I have since measured and found false
What I claimed at 04:28Z: "the branch is still behind and would still revert engine#2021's three npu/pack_import* files." That was inferred from git diff origin/main <branch> — the endpoint diff — and an endpoint diff is not what a merge does. Measured now, on the real main (9507c0a4), by performing the merge:
$ git checkout -B tmp-sq origin/main # observed-floor present: 2
$ git merge --squash b2397 # this PR's branch
Squash commit -- not updating HEAD
$ grep -c fconst_block_headers_observed npu/pack_import.cpp
2 <- #2021's work survives untouched
A three-way merge takes every path the branch did not change from the base, so a branch that is merely behind cannot revert anything. I ran the same experiment on llama.cpp#358 earlier tonight and got the same answer, which is why engine#2410 was filed and engine#2412 fixed branch_stale_check — a tool I wrote, whose central claim was this exact error. My #2397 CHANGES carried it too.
So the branch does not need a rebase for that reason, and the rule-and-reuse review I gave at cab7dcf stands: the disjoint-range rule is right, pp512 is the case worth pinning, and the third arm (an asymmetric pair where one side faulted) is measured rather than invented.
What a rebase would still buy, stated honestly, is nothing about correctness — only that the endpoint diff stops looking like a revert, which is a readability argument and not a review blocker. The tool that flags it (branch_stale_check) now reports the same case as a note, rc 0 rather than a FAIL, which is the fix for the class.
And one thing I got wrong twice in the same hour: my first attempt to reproduce this used a clone of my own review worktree, whose origin/main is a stale local ref (be8df65b), and it reported observed-floor=0 — a fake revert that would have "confirmed" my wrong claim. Only re-running it against the real GitHub main (9507c0a4) gave the true answer. A clone of a repo measures that repo's refs, not the upstream's.
|
Persistent review updated to latest commit 3fe90f8 |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 3fe90f8: READY
This withdraws my CHANGES at this head — and it exists because the withdrawal that already said so was posted as VERDICT: READY, with a colon, which the gate cannot parse. So the PR has been held by a verdict nobody could read, and the earlier correction was invisible to the merger. That is a-withdrawal-is-a-verdict for the third time today; I fixed the same thing on npu-kernels#70 and llama.cpp#342 this morning.
I re-derived the substance rather than repeating it. My CHANGES claimed the branch was behind main and would revert engine#2021's fconst_block_h references:
fconst_block_h origin/main: npu/pack_import.cpp 3, npu/pack_import.h 2
this head : npu/pack_import.cpp 3, npu/pack_import.h 2
git diff --stat origin/main this-head -- npu/pack_import.cpp npu/pack_import.h → (empty)
git rev-list --count this-head..origin/main → 0 ahead: 1
The two files are byte-identical to main's, the diff is empty, and the branch is zero commits behind and one ahead — so nothing would revert and the "behind main" note is stale as well. The REBASE the queue asks for has already happened.
Ready to merge at 7d3d680. Nothing outstanding from me, and the author's own squash-merge evidence in the 18:06Z note agrees with this measurement from the other direction.
|
Head moved from Same tree, and the two |
|
VERDICT: READY at The revert, measured before and afterThe flag was right about the mechanism: After the rebase: So: one file, additions only, 0 behind main, and #2021 specifically
I also re-checked that my own second commit is still there ( What I verified after the rebase, not just beforeOne thing to note about how this got rebasedI ran the rebase locally and my own CI on |
Row A compares stock HRX llama.cpp against onebit on
~/lb/supreme's bench.py artifacts, and the board's own winner column picksby median (
summary.py:271-278). On tonight's first real comparison the two disagreed on two of three cells:pp512is the case worth pinning: the same stock binary produced1506.78in one rep and2291.31in another, so "onebit is 3.7% faster"is a statement the data does not support - and a lane reading the board column cannot tell that from a real 3.7%.
So
--bench A.json B.jsonreads two arms and applies this file's own rule rather than growing a second copy of it:reps[].ttft_s(lower is better) for attft*cell,pp_tok_s/tg_tok_sotherwise.ttft2kruns withmax_tokens=32, so bench.py'smedianfor it is the DECODE tok/s and the latency isttft_median_s(
summary.py:88documents exactly this trap);NOISEwith the reason, not a winner;spread()andverdict()unchanged.Two arms added, both mutation-proven: choosing
tg_tok_sfor a ttft cell fails 2 arms, and the mode's own output on tonight's artifacts isverdicts: A 0, B 1, NOISE 2 (3 cells compared).What I did not change, and why it matters that I checked: my first version of this patch also added an
n = 1guard toverdict()and twosingle-sample arms - but the merged file already refuses both single-sample cases (its comment cites engine#2369 item 1) and already carried
those arms. That is the second-copy-of-a-rule defect this file exists to catch, so I removed mine and kept only what was missing.
medians_selftestPassed 0.02 s;copyright --checkrc 0.