Skip to content

[BUG] Make the registry leaderboard summary meaningful across fixtures #155

Description

@lukekarrys

Problem

The registry variation header card (for example, “vlt 34.25s”) is an unweighted mean of the fixture means, with DNF values imputed as the slowest successful competitor. A large fixture can dominate the number: in the investigation, Babylon represented about 48% of vlt’s registry-clean sum. This made npm 35.33s vs vlt 34.25s look effectively tied even though vlt led by 10–40% on five of six fixtures.

PR #148 fixed card ordering so it matches the displayed average, but intentionally left this aggregation unchanged. The most prominent number is therefore internally consistent but still easy to misinterpret.

Decision needed

Choose a summary that communicates cross-fixture performance fairly. Candidates from the investigation:

  • median of fixture values;
  • show the already-computed wins count;
  • at minimum, clearly label the current aggregation and DNF treatment.

Acceptance criteria

  • The registry leaderboard summary’s meaning is explicit in the UI.
  • A single large fixture cannot silently dominate the headline without that weighting being intentional and documented.
  • Ordering and displayed values use the same selected metric.
  • DNF handling is documented and tested.
  • Fixture filtering recomputes the summary correctly.

Source

Performance investigation: item 1 — misleading leaderboard average

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions