Bean et al. reviewed 445 AI benchmarks for NeurIPS 2025 and found that 16% of them ran a statistical test of any kind. I was in the other 84%. I had built a tool-selection benchmark, published a seven-model leaderboard off 44 items, and written the ranking down as though it meant something.
Then I put confidence intervals on it and most of it went away.
This is the tool I wrote to do that, extracted so it can be pointed at any benchmark, including yours. No dependencies — standard library only, Python 3.10+.
python -m errorbars trials.json -o STATISTICS.mdInput is one row per attempt. That is the entire schema:
[
{ "model": "claude-opus-4-7", "item": "a01", "passed": true },
{ "model": "gpt-5", "item": "a01", "passed": false }
]Wilson intervals, not Wald. At 43 of 44 the normal-approximation interval runs past 100%, which is not a coherent claim about a proportion. Wilson stays inside [0, 1] at the boundary, which is exactly where small benchmarks live.
Exact McNemar, not chi-square. Your models all saw the same items, so the design is paired and the discordant counts are single digits. The chi-square approximation is not valid at those counts. This computes the binomial tail directly and Holm-adjusts across the family.
Which items are doing work. An item that every model passes contributes nothing to a ranking. Neither does one that every model fails. The report counts them, and then asks the narrower question that actually matters: is there any item that separates the models at the top of my table?
Whether the ordering would survive a rerun. A leaderboard is a point estimate of an ordering, and nobody reports uncertainty on an ordering. This bootstraps it.
examples/ contains the adapter for my own tool-selection benchmark — 7 models, 665 trials, the 44-item large tier. Running the audit on it produces, among other things:
| Smallest gap 44 items can detect at 80% power | 19.0 percentage points |
| Items carrying no ranking information | 14 of 44 (31.8%) |
| Items separating the top four | 4 |
| Items separating #1 from #2 | 0 |
| Opus vs Sonnet, discordant pairs | 0 |
| P(published order reproduces) | 11.5% |
Zero discordant pairs means Opus and Sonnet did not tie on the score. They returned the same answer on all 44 items. My leaderboard printed one above the other anyway.
The only comparisons that survive Holm correction are the ones separated by roughly twenty points. The interesting result — GPT-5 Mini beating GPT-5, p = 0.0018 — was real, survives correction, and I had buried it under a ranking that wasn't.
When I first ran this analysis by hand I got 24% for the rank-order stability. The library says 11.5%. Both are right; they use different conventions for what happens when two models draw equal scores in a resampled run.
rank_stability(scores, tie_break="random") # 11.5%
rank_stability(scores, tie_break="published") # 24.0%Breaking ties in favour of whoever was printed higher doubles the apparent stability. A leaderboard does that implicitly every time it puts one row above another, and no leaderboard I have read states which convention it used. The default here is random, because a tie is not evidence of an ordering and should not be counted as one.
The bootstrap treats models as independent, which they are not. Correlated models reproduce their order more often than the estimate suggests, so the stability number is a ceiling, not a point estimate. The report says so in its own output; I would rather ship a stated limitation than a quiet one.
items_needed is the two-proportion normal approximation, unpaired. A paired design needs fewer items, though not dramatically fewer unless your models are highly correlated — and if they are that correlated, the item count is not your problem.
pip install -e ".[dev]"
python -m pytest46 tests. The expected values come from published worked examples, exact combinatorial identities, or properties that must hold algebraically — not from a previous run of this code. A test that only checks the code still does what it did last week catches refactors and nothing else.
MIT.