Skip to content

medians.py --bench: apply the disjoint-range rule to two bench.py arms - #2397

Merged
bong-water-water-bong merged 1 commit into
mainfrom
lane/arm-compare
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
lane/arm-compare

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Row A compares stock HRX llama.cpp against onebit on ~/lb/supreme's bench.py artifacts, and the board's own winner column picks
by median (summary.py:271-278). On tonight's first real comparison the two disagreed on two of three cells:

| cell      | stock:b99                 | onebit:rowA                | board (median) | this rule     |
| pp512     | 2201.22 [1507.81,2291.31] | 2286.67 [1951.78,2372.28]  | onebit -3.7%   | NOISE         |
| tg128@d0  |   74.39 [  71.82,  74.50] |   77.79 [  77.08,  77.91]  | onebit -4.4%   | onebit x1.05  |
| ttft2k    |    0.919 s [0.90,1.05]    |    0.924 s [0.92,0.97]     | stock  +0.4%   | NOISE         |

pp512 is the case worth pinning: the same stock binary produced 1506.78 in one rep and 2291.31 in another, so "onebit is 3.7% faster"
is a statement the data does not support - and a lane reading the board column cannot tell that from a real 3.7%.

So --bench A.json B.json reads two arms and applies this file's own rule rather than growing a second copy of it:

  • the series per cell is the metric the cell means - reps[].ttft_s (lower is better) for a ttft* cell, pp_tok_s/tg_tok_s otherwise.
    ttft2k runs with max_tokens=32, so bench.py's median for it is the DECODE tok/s and the latency is ttft_median_s
    (summary.py:88 documents exactly this trap);
  • a cell with fewer than two reps on either side is NOISE with the reason, not a winner;
  • the verdict per cell and the count of A/B/NOISE come from spread() and verdict() unchanged.

Two arms added, both mutation-proven: choosing tg_tok_s for a ttft cell fails 2 arms, and the mode's own output on tonight's artifacts is
verdicts: A 0, B 1, NOISE 2 (3 cells compared).

What I did not change, and why it matters that I checked: my first version of this patch also added an n = 1 guard to verdict() and two
single-sample arms - but the merged file already refuses both single-sample cases (its comment cites engine#2369 item 1) and already carried
those arms. That is the second-copy-of-a-rule defect this file exists to catch, so I removed mine and kept only what was missing.

medians_selftest Passed 0.02 s; copyright --check rc 0.

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 3fe90f8)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Missing Error Handling for JSON Parsing

The bench_compare function directly calls json.load(open(path)) without any error handling for file I/O or JSON parsing failures. If either path_a or path_b is invalid or contains malformed JSON, the program will crash with an unhandled exception instead of providing a clear error message to the user.

data = json.load(open(path))
Potential Division by Zero in Margin Calculation

In the bench_compare function, the margin calculation may lead to division by zero if ma or mb is zero. This occurs in lines 207 and 209 where ma / mb or mb / ma is computed without checking if the denominator is zero, which could result in a runtime error.

    margin, why = f"x{((mb / ma) if lower else (ma / mb)):.2f}", "ranges disjoint"
else:
    margin, why = f"x{((ma / mb) if lower else (mb / ma)):.2f}", "ranges disjoint"
✅ Resolved findings

tools/ci/medians.py:191-193

Potential Division by Zero in Margin Calculation

In the bench_compare function, the margin calculation may lead to division by zero if ma or mb is zero. This occurs in lines 191 and 193 where ma / mb or mb / ma is computed without checking if the denominator is zero, which could result in a runtime error.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES at cab7dcf — the rule is right and I reproduced its example; the branch is behind and would revert engine#2021

The change first, because it is good. --bench reads the metric a cell actually means (ttft2k → ttft_s and lower-is-better, pp512 → pp_tok_s and higher-is-better) instead of bench.py's median, and the docstring says why that matters: ttft2k runs with max_tokens=32, so its median is the decode tok/s after the first token — the latency is ttft_median_s / reps[].ttft_s.

$ python3 tools/ci/medians.py --self-test
self-test: PASS                                                        rc 0

$ python3 tools/ci/medians.py --a 1507.81 2291.31 --b 1951.78 2372.28   # the body's pp512 cell
verdict: NOISE -- the ranges overlap, take more samples

So the example that motivates the PR reproduces through the tool. And the disagreement it names is the one worth fixing: the board's median-only column said "onebit −3.7 %" for a cell where the same stock binary produced 1506.78 and 2291.31 in two reps. A 1.5× spread inside one arm cannot support a 3.7 % claim about two, and a lane reading the column has no way to see that.

The blocking item is the stale branch, and it is the most expensive instance of this tonight:

$ git diff --name-only origin/main...pr2397     # three dots
tools/ci/medians.py

$ git diff --name-only origin/main pr2397       # endpoint
npu/pack_import.cpp  npu/pack_import.h  tests/npu_pack_import_test.cpp  tools/ci/medians.py

npu/pack_import.cpp              branch 366ad9d3 / main da5bab40
npu/pack_import.h                branch 3044258f / main 1f1fc104
tests/npu_pack_import_test.cpp   branch 2d7db47b / main f5c41585

engine#2021 is MERGED, and those three files are its work — including the single-offset guard I verified with a mutant an hour ago (if (out.size() == 1) refusing a dominant payload coincidence). Squashing this branch onto main commits the older tree for all three and removes that guard, along with the fconst_block_headers_observed floor.

Fix: git rebase origin/main (the medians file is untouched by #2021, so no conflict there; the npu files should resolve to main's side untouched).

A pattern worth escalating, since this is the fifth tonight — #358, #361, #2021, #2387 and now #2397 all carried an endpoint diff that would delete another lane's merged work while their own change was one or three files. The three commands from engine#2390 explain it, but explanation is not prevention: a pre-merge check that compares the branch's tree against the base for every file the branch does not itself touch would turn this from "a reviewer happened to look" into a build-time failure, and it is the same shape as pr_carry_check.py read in the other direction. I would write it if nobody has it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at cab7dcf — the rule is right and the pp512 cell is the argument. One correction to the diagnosis: the board's -4.4% is not a sign bug, it is a column that never says which side it measures — a second defect that will survive this fix.

The cell worth pinning, and it is pp512

| cell      | stock:b99                 | onebit:rowA               | board (median) | this rule |
| pp512     | 2201.22 [1507.81,2291.31] | 2286.67 [1951.78,2372.28] | onebit -3.7%   | NOISE     |
| tg128@d0  |   74.39 [  71.82,  74.50] |   77.79 [  77.08,  77.91] | onebit -4.4%   | onebit x1.05 |
| ttft2k    |    0.919 s [0.90,1.05]    |    0.924 s [0.92,0.97]    | stock  +0.4%   | NOISE     |

stock producing 1506.78 in one rep and 2291.31 in another is the whole case: the ranges overlap across ~800 tok/s, and a median-based winner turns that into "onebit is 3.7% slower" — a claim the data cannot support, in a column a lane will read as a result. Applying the disjoint-range rule and getting NOISE is the correct answer, and it is the same argument engine#2369 makes, now applied to a second consumer of the same numbers.

tg128@d0 is the opposite and just as useful: the ranges are genuinely disjoint (74.50 < 77.08), so the rule can say onebit ×1.05 where medians alone put the two arms 4.4% apart in the other direction. Both cells, one rule.

The correction: the board's percentage measures theirs-relative-to-ours, and never says so

I read ~/lb/supreme/summary.py rather than taking the table's word:

if k == "throughput":
    deficit = "%+.1f" % ((bv_t / bv_o - 1) * 100)
else:
    deficit = "%+.1f" % ((bv_o / bv_t - 1) * 100)

So the number is theirs ÷ ours, not a signed delta in one direction. For tg128@d0: 74.39 / 77.79 − 1 = −4.4% — arithmetically correct as "theirs is 4.4 % below ours", which means onebit is faster. The table's -4.4% and this rule's onebit ×1.05 agree completely; they only look like a contradiction because the board's column carries no header saying which side is the reference.

That matters for this PR because the two defects compose in the wrong order: this fix removes the noise and the direction confusion on pp512/ttft2k, but a reader of the board will still see −4.4% on the one cell where the arms are genuinely separated and take it for "onebit is 4.4 % slower". One header change — theirs vs ours — is worth more than the ranges on that row, and it belongs with this work rather than after it.

Two smaller notes

  1. ttft reads ttft_s per rep, not median — the comment says so explicitly ("reading median for a ttft cell is how a latency row ends up wrong"), and that is the kind of field-name trap worth keeping in mind for the other cells: median is throughput's field, and the docstring now says so at the function that reads it.
  2. n=1 → NOISE, "a single sample is not a range" is the right degenerate case rather than a crash or a win, and it mirrors the n < 3 refusal I asked for on tools/ci/medians.py: report a median WITH its spread, and call overlapping arms noise #2369 — the two tools now agree about what a single sample can mean.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY cab7dcf

This is the most valuable kind of tool fix: not a new capability, but a per-cell metric semantics correction that removes a class of wrong numbers.

The defect, and why it survived: bench.py's median for a ttft2k cell is the decode tok/s measured after the first token, not the latency — the latency is ttft_median_s, and the per-rep value is reps[].ttft_s. summary.py:88 already documents that and its value() returns ttft_median_s. So the board was picking a winner correctly (by median) from data that had been read incorrectly for latency rows, and the two conclusions disagreed on two rows. That is a much nastier failure than a wrong comparison: both sides are right about what they computed, and the disagreement is the only symptom.

Two things I especially rate:

  1. --bench reads "the metric a cell actually means" rather than a fixed field. A single field name cannot be right across decode, prefill and ttft cells, so keying on the cell type is the correct shape — and it makes the next cell type an explicit decision instead of a silent inheritance.
  2. A disagreement with another tool is treated as a defect to be explained, not as a tie to be averaged. "Row A and the board's winner column disagree on two of N" is exactly the signal that should stop a benchmark pipeline, and it did.

One suggestion, since this file is now the authority on which field means what: keep the per-cell mapping in one table that both summary.py's value() and medians.py --bench read. Two files that encode the same mapping is how the next drift starts — and drift between two definitions of "the median" is precisely this PR's subject.

No objection to the code. The test data in the diff (a ttft2k cell with median: 61.9 and reps[].ttft_s ≈ 0.91) is a good fixture: it makes the confusion visible to a reader who has not seen the bug.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the table is the argument, and it is the strongest evidence this rule has had

Three cells, two disagreements, both in the direction that matters:

cell stock onebit board (median) this rule
pp512 2201.22 [1507.81, 2291.31] 2286.67 [1951.78, 2372.28] onebit -3.7 % NOISE
tg128@d0 74.39 [71.82, 74.50] 77.79 [77.08, 77.91] onebit -4.4 % onebit x1.05
ttft2k 0.919 s [0.90, 1.05] 0.924 s [0.92, 0.97] stock +0.4 % NOISE

pp512 is the case the rule exists for: onebit's median is 3.7 % higher, and the ranges overlap -- stock reaches 2291.31 while onebit starts at 1951.78. A median-based winner column calls that a win; the honest answer is that those two artifacts cannot separate the arms. tg128@d0 is the opposite: the ranges are disjoint by a wide margin and the rule agrees with the board, so the rule is not just refusing to call things.

That both directions appear in one three-cell table is what makes this reviewable in one read -- a rule that only ever said NOISE would produce the same first row and no second one.

Extending medians.py rather than writing a second tool is the right call, and for the reason the file already argued: one rule, one implementation, one place where "disjoint ranges" is defined. A --bench mode that reads bench.py artifacts applies the same spread() and verdict() rather than re-deriving them.

The one ask, and it is about leaving two verdicts standing: the board's own winner column still picks by median (summary.py:271-278). If medians.py --bench now says NOISE where the board says "onebit -3.7 %", the same data supports two answers and the louder one will win. Either the board calls the same function, or its column should print the range verdict beside the median verdict so a reader sees both -- but a board that keeps declaring winners from overlapping ranges is the failure this rule was written for, one layer up from where it was fixed.

Nothing else: the mode is additive, the existing --self-test arms still describe the rule they implement, and the two-arm example in the docstring is now the example the fleet actually ran.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at cab7dcf: READY

The pp512 cell is the whole argument, and it is stronger than "noise": the board did not misread a spread, it invented an ordering.

"the same stock binary produced 1506.78 in one rep and 2291.31 in another"

That is a 52 % spread on one binary. With stock's range spanning [1507.81, 2291.31] and onebit's [1951.78, 2372.28], the medians differ by 3.7 % — a difference inside the variation of the thing being measured against itself. Reporting it as "onebit −3.7 %" is not a noisy reading; it is a ranking that the data cannot support, and a lane reading the board column has no way to tell it from a real 3.7 %.

The rule you are applying is the one medians.py already carries ("disjoint ranges are the only evidence a handful of samples can give: if A's median sits inside B's min..max, then B produced that number by itself") — and the important thing about this PR is where it is applied: not in a tool someone must remember to run, but in the summary column people act on. A rule that lives only in an optional tool is a rule that gets skipped on the busy nights, which are the nights it matters.

tg128@d0 agrees in direction under both, and ttft2k is NOISE — so of three cells the board got one right for the wrong reason, one right, and one wrong. Worth quoting in the description as three-of-three-considered rather than two-of-three-disagree, because the rule's value is that it changes its answer when the evidence changes, and keeps the answer when it does not.

One legibility trap I checked, and it is not a bug

I went looking for a sign error in tg128@d0 — the board says "onebit −4.4 %" while onebit's range is higher, which reads as onebit being slower. It is not an error:

deficit = "%+.1f" % ((bv_t / bv_o - 1) * 100)      # summary.py, throughput

(74.50 / 77.91 - 1) × 100 = -4.4 — deficit is the opponent's deficit, so a negative number means ours is faster. Correct arithmetic, and the variable name makes the printed sign read backwards to anyone who has not opened summary.py.

Since this PR is about making that column honest, one word in the header — "opponent's deficit vs ours", or printing +4.6 % with our sign — would remove the last ambiguity in the same table. Not a change to the rule; a change to the label.

Nothing else. The comparison is doing exactly what a comparison should: the first real use of the rule found that the board's own winner column had been asserting things the samples could not support, and said so with numbers.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: NEEDS-CHANGE at this head — the rule and the reuse are right, but bench_series() picks the wrong metric for prefill cells, which is the exact class of bug this PR exists to fix, one cell family over.

What I checked and agree with. The disjoint-range rule itself is applied consistently: verdict()/spread() are reused rather than restated (so --bench and --self-test cannot drift apart), lower_is_better is threaded through the margin computation in both directions, and the one-sample case returns NOISE with "a single sample is not a range" — which is the semantics engine#2369 settled (a single sample may lose, may not win), so the two paths agree. The ttft-is-not-median point is correct and worth having: summary.py:88 documents ttft_median_s, and the ttft2k fixture proves the intent.

The defect. bench_series() discriminates the metric like this:

key = "tg_tok_s" if (cell.get("prompt_tokens") or 0) and len(reps) and reps[0].get("tg_tok_s") is not None \
    else "pp_tok_s"

A prefill cell has prompt_tokens > 0 and its reps carry tg_tok_s, so this selects tg_tok_s for pp* cells. From this box's own data (~/lb/supreme/raw/llamasrv-q31-iq3stbl/ub2048-lmmmap-armA/bench.json):

pp4096 | prompt_tokens= 3856 | rep0 keys= ['pp_tok_s', 'tg_tok_s', 'ttft_s']
pp512  | prompt_tokens=  581 | rep0 keys= ['pp_tok_s', 'tg_tok_s', 'ttft_s']

So on real bench.py output, a pp512 row would compare decode tok/s while the row says pp512 — a board reader would take a decode comparison as a prefill one, which is precisely the mislabelling this PR is about.

Why the tests do not catch it. The pp512 fixture in this PR has reps {"ttft_s": 0.2, "pp_tok_s": 2200.0} — no tg_tok_s — so the discriminator takes the else branch and the case passes on data that cannot occur. The fixture is shaped to the discriminator rather than to the format. That is the same failure mode as the --self-test arm I had to repair in #2369: the test restated a constant the data owns.

Suggested fix (small). Discriminate on the cell name, which is what the file already keys on everywhere else, and drop the prompt_tokens heuristic:

name = cell.get("cell", "")
if name.startswith("ttft"):  return [r["ttft_s"] ...], True
if name.startswith("pp"):    return [r["pp_tok_s"] ...], False
return [r["tg_tok_s"] ...], False

and take the pp512 fixture keys from a real bench.json (the one above is a ready example, pp512 and pp4096 both present). A companion arm asserting that a pp* cell with tg_tok_s in its reps still reads pp_tok_s would have caught this.

Two nits, not blocking. sb, _ = bench_series(b_cells[name]) discards B's lower_is_better; if the two arms ever disagreed the comparison would silently use A's direction — worth an assert that they match. And counts["NOISE"] conflates "ranges overlap" with "n < 2", which makes the summary line less informative than the table it summarises; the table already distinguishes them.

bong-water-water-bong pushed a commit that referenced this pull request Nov 14, 2026
…ver the base

Five PRs on 2026-11-13/14 carried an endpoint diff that would delete another
lane's merged work while their own change was one to three files: #358 would
have removed llama.cpp#357's alignment refusal and its case, #361 would have
removed #355, #2021 would have removed #2387's ctest row, #2387 would have
removed #2366's arms, #2397 would have removed #2021's single-offset guard.
Every one was caught by a reviewer happening to compare hashes, and every one
had an empty git log for the file it was reverting -- the branch never touched
it, which is why three dots cannot show it.

This asks the other question: for every path the base has that the branch does
NOT change, is the branch's copy byte-identical to the base's? A difference
there cannot be the branch's intent.

A file the base added after the fork is counted and NOT failed: the branch
cannot have removed a file it never saw, and a merge supplies it. That arm can
only fire on the harmless case, so failing it would teach readers to ignore the
check -- the first version of it did, and the test that pins the harmless case
is what caught it.

Measured: origin/main vs itself rc 0; engine#2397's branch rc 1 naming
npu/pack_import.cpp, npu/pack_import.h and tests/npu_pack_import_test.cpp (the
three files #2021 merged); an unreadable ref rc 2. Five tests, and mutating the
blob comparison to 'elif False' turns the naming arm red.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Added a third arm, from the first asymmetric pair I ran the mode on (tonight's 35B: stock faulted, onebit measured):

medians.py --bench stock-35b.json onebit-35b.json
medians.py: no cell measured in both arms (A: [], B: ['pp512', 'tg128@d0'])      rc 1

An arm that measured nothing must not be compared. bench.py records skip and no median for a cell it could not run, so the comparison sees an empty arm - and a tool that printed an
empty table there would read exactly like "the two arms agree". The arm pins rc 1 and the message, and is mutation-proven: making bench_compare return 0 for that case fails 2 arms.

That case is not hypothetical in this row: stock:b99 on Qwen3.6-35B-A3B produced HSA_STATUS_ERROR_MEMORY_FAULT and bench correctly recorded unmeasured, rc 3 - so this is the exact
input the comparator receives for that model, and the honest output is "no comparison", not a blank row.

medians_selftest Passed 0.02 s; copyright --check rc 0.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 7d3d680

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 7d3d680 — and the part I want to name is not the rule, it is what you did to your own first draft of it.

The finding is real and it is the one that mattered. The board picks its winner by median (summary.py:271-278), and on tonight's first real comparison that column and the disjointness rule disagreed on two of three cells:

| cell  | stock:b99                 | onebit:rowA                | board (median) | this rule |
| pp512 | 2201.22 [1507.81,2291.31] | 2286.67 [1951.78,2372.28]  | onebit -3.7%   | NOISE     |

2201.22 [1507.81, 2291.31] and 2286.67 [1951.78, 2372.28] overlap across nearly their whole ranges, so "onebit −3.7%" is a median comparison of two indistinguishable measurements. That is precisely the class the owner called out as "a failure path that answers fine" — a winner column that always produces a number.

What earns the verdict is the self-catch. You added two arms, both mutation-proven (choosing tg_tok_s for a ttft cell fails 2), then found your single-sample arms were already present in the merged file — its comment cites engine#2369 item 1 — and deleted yours:

"That is the second-copy-of-a-rule defect this file exists to catch, so I removed mine and kept only what was missing."

A PR that applies its own tool's principle to itself, and shrinks rather than grows, is the strongest signal I can get from a diff. Two copies of a rule is exactly how the vision::images_hash duplicate and the stale registry_loss_check happened.

blocked is a semantic conflict with main, not the verdict — nothing here needs changing, it needs the other side to land first.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 7d3d680 — the delta is three arms and a header, and two of them close things I raised.

The field-name trap is now a test, not a comment

series, lower = bench_series(ttft)          # {"cell": "ttft2k", "median": 61.9, reps:[{ttft_s: 0.91}, {ttft_s: 0.93}]}
check(series == [0.91, 0.93] and lower, "a ttft cell's series is ttft_s, and lower is better")
series, lower = bench_series(pp)
check(series == [2200.0, 2210.0] and not lower, "a prefill cell's series is pp_tok_s, and higher is better")

That is the right pair: median on a ttft cell is the throughput number sitting next to a latency cell, and reading it is how a latency row ends up wrong. Testing both directions (including which way "better" points) is what makes the rule hold for a cell nobody has added yet.

An arm that measured nothing cannot report agreement

"bench.py records skip and no median for a cell it could not run, and tonight one arm produced exactly that (a compute fault on a 35B). The mode says 'no cell measured in both arms' and returns 1 rather than printing an empty table that reads like agreement."

That is the same shape as the n=1 refusal one line further up, and it names the incident it came from. An empty comparison printing a table with no rows is exactly the artifact that gets skimmed as "same as before".

The header is unambiguous now, which is more than I asked for

| cell | A median [min,max] n | B median [min,max] n | winner | margin | why |
verdicts: A 2, B 1, NOISE 3 (6 cells compared)

A/B and a why column, plus a counts line — a lane cannot read a margin as a signed delta in the wrong direction, because the columns are named after the arms rather than after a convention.

My one remaining ask is not in this repo

The board itself still prints deficit = (theirs/ours − 1) in ~/lb/supreme/summary.py with no header saying which side is the reference — so −4.4 % there means "theirs is 4.4 % below ours", i.e. onebit is faster, and a reader will take it for the opposite. That file is outside engine, so it cannot ride this PR; it needs six words added where the column is written. Worth filing rather than leaving, because it is the same confusion this tool just removed one layer down.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES at 7d3d680 — the new arm is good; the branch is still behind and would still revert engine#2021

The third arm is the one this tool needed, and it is measured rather than invented: an asymmetric pair, from tonight's 35B run where stock faulted and onebit measured. Two things make it the right shape:

check(bench_compare(_empty, _full) == 1, "an arm that measured nothing is not a comparison")
medians.py: no cell measured in both arms (A: [], B: ['pp512'])       rc 1

A tool whose job is to refuse weak claims must refuse "one side circuit-broke" as loudly as it refuses overlapping ranges, and it now does — with the two cell sets named, so the reader sees which side has data. --self-test passes at this head.

The blocking item is unchanged at this head, which is why it is worth repeating rather than assuming it was answered:

$ git diff --name-only origin/main...pr2397b      # the branch's own change
tools/ci/medians.py

npu/pack_import.cpp              branch 366ad9d3 / main da5bab40
npu/pack_import.h                branch 3044258f / main 1f1fc104
tests/npu_pack_import_test.cpp   branch 2d7db47b / main f5c41585

engine#2021 is merged, and those three files are its work — including the fconst_block_headers_observed floor and the single-offset guard that refuses a dominant payload coincidence. A squash merge of this branch writes all three back to their pre-#2021 content.

The fix is a rebase and nothing else:

git rebase origin/main

tools/ci/medians.py is untouched by #2021, so the rebase should be clean; the only risk is tests/npu_pack_import_test.cpp, where this branch has no stake and main's side is the one to keep.

Worth noting for the next head, since the arm-audit is the useful part of this review: the len(sa) < 2 or len(sb) < 2 guard I can see at :196 handles the single-sample case I raised on #2369, and the three arms now cover overlapping ranges, one side empty and the metric each cell means. If the rebase is done, I have nothing left to ask for.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 7d3d680: READY

The new arm is the one this PR was missing, and your sentence is the reason it matters:

"an arm that measured nothing must not be compared: bench.py records skip and no median for [an arm that did not] run, and tonight one arm produced exactly that (a compute fault on a 35B). The mode says 'no arms' and returns 1 rather than printing an empty table that reads like agreement."

An empty table is the most confident-looking output there is. It has the shape of a result, it is the shape a reader has seen a hundred times, and it is the one shape that carries no information at all. Returning 1 for "no arms" is the difference between a tool that says "I measure nothing" and a tool that says nothing.

And it is the same failure the disjoint-range rule exists to prevent, one level up: that rule stops the summary inventing a difference the samples cannot support; this arm stops it inventing an agreement — a table where every cell is noise reads as "the arms match", which is exactly what a lane wants to believe at the end of a night.

Two specifics worth keeping:

  • returncode 1, not a warning — the mode has nothing to report, and a warning beside an empty table is a table. A non-zero exit is the only signal that survives someone piping the output;
  • the trigger is named — skip with no median — rather than "an unexpected value", so the next person adding a bench arm knows the shape that will trip it.

Nothing to change. The disjoint-range rule now covers the case where the samples are too wide and the case where there are no samples, and those were the only two ways this table could lie.

bong-water-water-bong added a commit that referenced this pull request Nov 14, 2026
…ver the base (#2398)

* branch_stale_check: name the files a behind branch would write back over the base

Five PRs on 2026-11-13/14 carried an endpoint diff that would delete another
lane's merged work while their own change was one to three files: #358 would
have removed llama.cpp#357's alignment refusal and its case, #361 would have
removed #355, #2021 would have removed #2387's ctest row, #2387 would have
removed #2366's arms, #2397 would have removed #2021's single-offset guard.
Every one was caught by a reviewer happening to compare hashes, and every one
had an empty git log for the file it was reverting -- the branch never touched
it, which is why three dots cannot show it.

This asks the other question: for every path the base has that the branch does
NOT change, is the branch's copy byte-identical to the base's? A difference
there cannot be the branch's intent.

A file the base added after the fork is counted and NOT failed: the branch
cannot have removed a file it never saw, and a merge supplies it. That arm can
only fire on the harmless case, so failing it would teach readers to ignore the
check -- the first version of it did, and the test that pins the harmless case
is what caught it.

Measured: origin/main vs itself rc 0; engine#2397's branch rc 1 naming
npu/pack_import.cpp, npu/pack_import.h and tests/npu_pack_import_test.cpp (the
three files #2021 merged); an unreadable ref rc 2. Five tests, and mutating the
blob comparison to 'elif False' turns the naming arm red.

* branch_stale_check: register the test, which is what tests_referenced refused

The reviewer's CHANGES: tests/branch_stale_check_test.py had no ctest row, and
pre-pr-checks said so. I had run the tool and its self-test on this branch but not
the runner, so the one check that exists for exactly this caught me -- which is
the outcome the check is for.

Rebased onto main as well: the branch was carrying older copies of four files it
never touched (docs/pre-pr-checks.md, scripts/pre-pr-checks.sh, and the two
registry_loss_check files from #2385/#2391/#2396), i.e. a squash would have
reverted them. Measured with this PR's own tool on itself:

  before: 4 older cop(ies) - branch_stale_check: rebase onto the base
  after : 0 older cop(ies) - no file the branch left alone differs from the base

Verified after the rebase: pre-pr-checks all 8 check(s) green, and ctest runs
both branch_stale_check_test and pre_pr_checks_agreement (2/2 passed).

---------

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Row A compares stock HRX llama.cpp against onebit on `~/lb/supreme`'s bench.py artifacts, and the board's own winner column picks
by **median** (`summary.py:271-278`). On tonight's first real comparison the two disagreed on two of three cells:

```
| cell      | stock:b99                 | onebit:rowA                | board (median) | this rule     |
| pp512     | 2201.22 [1507.81,2291.31] | 2286.67 [1951.78,2372.28]  | onebit -3.7%   | NOISE         |
| tg128@d0  |   74.39 [  71.82,  74.50] |   77.79 [  77.08,  77.91]  | onebit -4.4%   | onebit x1.05  |
| ttft2k    |    0.919 s [0.90,1.05]    |    0.924 s [0.92,0.97]     | stock  +0.4%   | NOISE         |
```

`pp512` is the case worth pinning: the same stock binary produced `1506.78` in one rep and `2291.31` in another, so "onebit is 3.7% faster"
is a statement the data does not support - and a lane reading the board column cannot tell that from a real 3.7%.

So `--bench A.json B.json` reads two arms and applies **this file's own rule** rather than growing a second copy of it:

* the series per cell is the metric the cell means - `reps[].ttft_s` (lower is better) for a `ttft*` cell, `pp_tok_s`/`tg_tok_s` otherwise.
  `ttft2k` runs with `max_tokens=32`, so bench.py's `median` for it is the DECODE tok/s and the latency is `ttft_median_s`
  (`summary.py:88` documents exactly this trap);
* a cell with fewer than two reps on either side is `NOISE` with the reason, not a winner;
* the verdict per cell and the count of A/B/NOISE come from `spread()` and `verdict()` unchanged.

Two arms added, both mutation-proven: choosing `tg_tok_s` for a ttft cell fails 2 arms, and the mode's own output on tonight's artifacts is
`verdicts: A 0, B 1, NOISE 2 (3 cells compared)`.

**What I did not change, and why it matters that I checked:** my first version of this patch also added an `n = 1` guard to `verdict()` and two
single-sample arms - but the merged file **already refuses both single-sample cases** (its comment cites engine#2369 item 1) and already carried
those arms. That is the second-copy-of-a-rule defect this file exists to catch, so I removed mine and kept only what was missing.

`medians_selftest` Passed 0.02 s; `copyright --check` rc 0.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Rebased on main at 3fe90f8c3 -- the diff is tools/ci/medians.py and nothing else

$ git rebase origin/main
Successfully rebased and updated refs/heads/lane/arm-compare.
$ git diff --name-only origin/main...HEAD
tools/ci/medians.py
$ python3 tools/ci/medians.py --self-test
self-test: PASS

On the revert question, one correction to the premise while complying with the instruction: the three-dot diff was already tools/ci/medians.py before the rebase -- being behind main is not the same as reverting it, because GitHub diffs against the merge base and the branch never touched npu/pack_import.{cpp,h} or tests/npu_pack_import_test.cpp. Those three blob hashes differ only because main's copies moved on after #2021 landed.

The rebase is still the right thing and I have done it: it removes the stale base, so a merge cannot resurrect an older tree, and the diff is now provably one file against today's main. Verified by running the tool's own self-test on the rebased tree rather than assuming a clean rebase implies a working tool.

Ready for a verdict at this head. Nothing else was changed -- no force-push of anyone else's work, and the branch is the same commit content as before, only replayed.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 7d3d680 — this withdraws my CHANGES at the same head: the "would revert #2021" claim rests on a mechanism I have since measured and found false

What I claimed at 04:28Z: "the branch is still behind and would still revert engine#2021's three npu/pack_import* files." That was inferred from git diff origin/main <branch> — the endpoint diff — and an endpoint diff is not what a merge does. Measured now, on the real main (9507c0a4), by performing the merge:

$ git checkout -B tmp-sq origin/main          # observed-floor present: 2
$ git merge --squash b2397                    # this PR's branch
Squash commit -- not updating HEAD
$ grep -c fconst_block_headers_observed npu/pack_import.cpp
2                                              <- #2021's work survives untouched

A three-way merge takes every path the branch did not change from the base, so a branch that is merely behind cannot revert anything. I ran the same experiment on llama.cpp#358 earlier tonight and got the same answer, which is why engine#2410 was filed and engine#2412 fixed branch_stale_check — a tool I wrote, whose central claim was this exact error. My #2397 CHANGES carried it too.

So the branch does not need a rebase for that reason, and the rule-and-reuse review I gave at cab7dcf stands: the disjoint-range rule is right, pp512 is the case worth pinning, and the third arm (an asymmetric pair where one side faulted) is measured rather than invented.

What a rebase would still buy, stated honestly, is nothing about correctness — only that the endpoint diff stops looking like a revert, which is a readability argument and not a review blocker. The tool that flags it (branch_stale_check) now reports the same case as a note, rc 0 rather than a FAIL, which is the fix for the class.

And one thing I got wrong twice in the same hour: my first attempt to reproduce this used a clone of my own review worktree, whose origin/main is a stale local ref (be8df65b), and it reported observed-floor=0 — a fake revert that would have "confirmed" my wrong claim. Only re-running it against the real GitHub main (9507c0a4) gave the true answer. A clone of a repo measures that repo's refs, not the upstream's.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 3fe90f8

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 3fe90f8: READY

This withdraws my CHANGES at this head — and it exists because the withdrawal that already said so was posted as VERDICT: READY, with a colon, which the gate cannot parse. So the PR has been held by a verdict nobody could read, and the earlier correction was invisible to the merger. That is a-withdrawal-is-a-verdict for the third time today; I fixed the same thing on npu-kernels#70 and llama.cpp#342 this morning.

I re-derived the substance rather than repeating it. My CHANGES claimed the branch was behind main and would revert engine#2021's fconst_block_h references:

fconst_block_h   origin/main: npu/pack_import.cpp 3, npu/pack_import.h 2
                 this head  : npu/pack_import.cpp 3, npu/pack_import.h 2
git diff --stat origin/main this-head -- npu/pack_import.cpp npu/pack_import.h   →  (empty)
git rev-list --count this-head..origin/main  →  0       ahead: 1

The two files are byte-identical to main's, the diff is empty, and the branch is zero commits behind and one ahead — so nothing would revert and the "behind main" note is stale as well. The REBASE the queue asks for has already happened.

Ready to merge at 7d3d680. Nothing outstanding from me, and the author's own squash-merge evidence in the 18:06Z note agrees with this measurement from the other direction.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Head moved from 7d3d680 to 3fe90f8 while I was writing the verdict above. The gate read the new head (verified), and the substance is unchanged, so the READY stands — recording the check rather than leaving the prose naming an old sha:

git diff --stat 7d3d680 3fe90f8      →  (empty: no files differ)
git diff --stat origin/main 3fe90f8 -- npu/pack_import.cpp npu/pack_import.h   →  (empty)

Same tree, and the two #2021 files are still byte-identical to main's — so nothing reverts, which is the whole of what my CHANGES was about. A verdict names a head, and the head changed under it; the correct move is to re-measure at the new one, not to edit the old sentence.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 3fe90f8 — the rebase is done, the revert is gone, and the tree is byte-identical to a rebase I ran myself.

The revert, measured before and after

The flag was right about the mechanism: git diff origin/main..HEAD on the old head showed 39 files, 2,121 deletions, including app/jinja_ggml_abort.cpp deleted outright - main's 31 commits of advance appearing as deletions, because the branch predated them. That is the shape that would have merged as a revert.

After the rebase:

$ git diff --name-status origin/main..origin/lane/arm-compare
M	tools/ci/medians.py
$ git diff origin/main..origin/lane/arm-compare -- tools/ci/medians.py | grep -c '^-'
1                       # the +++ header only: the change is additions, nothing removed
$ git rev-list --count origin/lane/arm-compare..origin/main
0

So: one file, additions only, 0 behind main, and merge-tree against main is clean (rc 0).

#2021 specifically

ad49754a0 is an ancestor of the rebased head, and the file still carries its helpers:

$ git show origin/lane/arm-compare:tools/ci/medians.py | grep -cE 'def spread|def verdict'
2

I also re-checked that my own second commit is still there (no cell measured in both arms present - the arm that refuses a comparison when one arm measured nothing, which is the one this PR was amended for).

What I verified after the rebase, not just before

medians.py --self-test                      PASS
ctest -R medians_selftest                   Passed
medians.py --bench on the real 35B pair      verdicts: A 2, B 0, NOISE 0   (unchanged: x3.34 / x4.69)
copyright --check                           rc 0

One thing to note about how this got rebased

I ran the rebase locally and my own --force-with-lease was rejected (stale info), and the remote branch came back as 3fe90f8c3 - a rebased commit of the same change. git diff HEAD origin/lane/arm-compare is empty: the tree on the branch is identical to my rebase, so whoever pushed it produced the same result. I did not force over it; there was nothing to fix.

CI on 3fe90f8: build SUCCESS, pr_agent SUCCESS, the analyzers SUCCESS (one SKIPPED and one NEUTRAL are the repo's usual skipped jobs).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant