fix(reportgen): warn when a kv_cache row hides a collapsed group or a failed run

[FileSystemGuy]

Fixes #836.

A kv_cache row could be built from one of several runs in its workload group, chosen by filesystem-discovery order, with no indication on the row. A second, unrelated path let a run that lost most of its per-rank result files publish a row shaped exactly like a clean one.

Base is reportgen-column-parity (the integration branch) rather than main — these touch the same reportgen files the #826–#829 stack rewrote.

What changed

Deterministic selection. _select_kvcache_run replaces runs[0]. Discovery order was genuinely arbitrary: of the three groups in the v3.0 tree holding two or more real runs, it published the earlier run for two and the later run for the third. Earliest run_datetime has no better claim than latest, but it is a rule rather than an accident, and it matches the positional convention Rules.md §2.1.17 already uses for training.

datasize excluded from selection. The kv_cache grouping key has no command component (D-05), so a datasize leaf lands in the same group as the run it sized. It writes no summary.json. An earlier draft of this change selected those leaves and blanked all twelve KVCache metrics on rows v3.0-0019 / 0021 / 0023 / 0142; the expectation diff caught it and it is now pinned by a test.

Two warnings on Result.issues. One when a row is built from a subset of its group, naming the published run and every dropped one; one when the published run recorded partial_failure, with the missing-file and failed-trial counts. The collapsed-group warning requires two or more actual run invocations, so ANL polaris-nvme's pair of datasize leaves stays quiet.

format_issues_list(show_all=False) now shows CLOSED warnings. The filter dropped every issue whose validation was CLOSED, conflating two orthogonal fields — validation is the submission category, severity is whether a human needs to look. Without this the new warnings reach only the log stream, since the fixed webpage-parity schema has no issues column. The datagen leaf-presence warnings, written to be "worth surfacing but not invalidating", had been invisible for the same reason and now print. Warnings are badged [WARN] in yellow rather than green [CLOSED], which on "row built from 1 of 3 runs" reads as approval; the stray None: placeholder is dropped for issues carrying no parameter.

Findings behind the change

The issue's original diagnosis was wrong on two counts; the correction is posted there in full.

crux-eagle is not a novel layout the tool failed to parse. Each run's metadata.result_dir records the path the tool wrote it to — crux-eagle, crux-eagle-n8, crux-eagle-n64, each canonical <system>/kv_cache/llama3.1-8b/run/<ts>/. Three system directories were merged into one at packaging time. Accepting the layout would not fix the collapse, because systemname is path-derived and all three still produce the same grouping key.

The two dropped runs are failed runs, not lost measurements: 8nodex8ppn is missing 403 per-rank result files (~70 %) and 64nodex8ppn 4473 (~97 %), both with 9 of 9 mpirun trials exiting non-zero. partial_failure and trial_failures are written by kvcache.py:919,960 and were read by nothing, which is the more general bug — open/farmgpu/.../llama3.1-70b-instruct/run/20260709_215001 publishes today missing 45 rank files.

Verification

Five commits, TDD RED→GREEN throughout. All four CI suites green: 3080 + 911 + 238 + 228 passed, 1 skipped, 0 failures.

Against the v3.0 tree the change produces 3 collapsed-group warnings (ANL crux-eagle 3 runs, Everpure 51hosts_20 and 51hosts_30 2 runs each) and 1 partial_failure warning, adding five lines to the printed summary — the four above plus one datagen leaf warning that had never been visible.

One published row moves: v3.0-0019 (Everpure 51hosts_20) switches from its later run to its earlier one, 85736 → 66113 tok/s and 11 other cells, because discovery order had picked the later run there. Which run represents a submitter's intended submission is not something the tool can know; that question is being raised with the three affected submitters in ApparentProblems.md alongside the results refresh.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[FileSystemGuy]

Closing — these commits belong directly on reportgen-column-parity, which is serving as the working main branch until the submission round completes. Pushed there instead.