Dashboard: collapse per-level status badges into a single highest-level badge#347
Conversation
Previously a model with both an L3 skip and an L5 xfail rendered two inline badges (e.g. youtu showed `Skip` next to `XFail`). The badges were redundant — the highest-level failure conveys the most useful information. Collapse the four per-level badges into a single badge whose source is chosen by precedence: L5 xfail > L4 xfail > L3 xfail > L3 skip > L2 xfail / xfail_graph_only.
|
@copilot This doesn't look great. Theoretically every level can have some combination of badges (xfail, skip, etc...). It seems from here that skips at other levels (L5, L4) wouldn't be counted and that only XFails would. Theoretically a skip can occur ate every level. Can you ensure these also are reported. |
The previous single-badge logic only considered skip at L3 and xfail at L2-L5, which meant L4/L5 skip statuses (the dominant skip population — many models have YAML cases marked skip_reason) showed no badge at all. Replace the if/else cascade with a loop that walks L5 → L2 and emits the first xfail OR skip encountered. Both exception kinds are now reportable at every level where the data model permits them.
Good catch — the previous version only badged skip at L3 (which was the original behavior), so L4/L5 skips silently dropped. Reworked the selection into a single L5→L2 loop that emits the first xfail or skip encountered, so any exception status at any level is now surfaced as the one badge. Verified:
|
|
@copilot Are there any other badges we need to account for at the time being? |
Looking at the row next to
Elsewhere in the dashboard:
So with the L5→L2 walk in |
Performance Comparison
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Models with both an L3 skip and an L5 xfail (e.g.
youtu) rendered two inline badges —Skipnext toXFail— instead of a single badge reflecting the most consequential outstanding issue. The same problem applied to L4/L5 skips, which previously weren't badged at all.Changes
scripts/templates/dashboard.html.j2: replace the four independentl2Badge..l5Badgeemissions with a singlestatusBadgeselected by walking levels L5 → L2 and emitting the first non-pass status (xfail or skip) encountered. L2 also handles itsxfail_graph_onlyvariant.${statusBadge}${ratioBadge}.Both xfail and skip are now reportable at every level where the data model permits them, so the badge always reflects the highest-level outstanding issue regardless of its kind.
After the change:
youtu(L3 skip + L5 xfail) →XFail(from L5)fsmt(L3/L4/L5 skip) →Skip(from L5)apertus(L4/L5 skip) →Skip(from L5)Lower-level exceptions remain visible via the L1–L5 dot strip and the expanded detail view.