Problem
The registry variation header card (for example, “vlt 34.25s”) is an unweighted mean of the fixture means, with DNF values imputed as the slowest successful competitor. A large fixture can dominate the number: in the investigation, Babylon represented about 48% of vlt’s registry-clean sum. This made npm 35.33s vs vlt 34.25s look effectively tied even though vlt led by 10–40% on five of six fixtures.
PR #148 fixed card ordering so it matches the displayed average, but intentionally left this aggregation unchanged. The most prominent number is therefore internally consistent but still easy to misinterpret.
Decision needed
Choose a summary that communicates cross-fixture performance fairly. Candidates from the investigation:
- median of fixture values;
- show the already-computed wins count;
- at minimum, clearly label the current aggregation and DNF treatment.
Acceptance criteria
Source
Performance investigation: item 1 — misleading leaderboard average
Problem
The registry variation header card (for example, “vlt 34.25s”) is an unweighted mean of the fixture means, with DNF values imputed as the slowest successful competitor. A large fixture can dominate the number: in the investigation, Babylon represented about 48% of vlt’s
registry-cleansum. This made npm 35.33s vs vlt 34.25s look effectively tied even though vlt led by 10–40% on five of six fixtures.PR #148 fixed card ordering so it matches the displayed average, but intentionally left this aggregation unchanged. The most prominent number is therefore internally consistent but still easy to misinterpret.
Decision needed
Choose a summary that communicates cross-fixture performance fairly. Candidates from the investigation:
Acceptance criteria
Source
Performance investigation: item 1 — misleading leaderboard average