docs(manuscript): evidence matrix with a mechanically enforced anonymisation rule

[Ultimate-Storm]

Step 3 of the lessons-learned manuscript plan: the evidence matrix, which is both the analysis instrument and the paper's appendix.

23 incidents, one row each — date, phase, lesson, sites, symptom, root cause, impact, evidence pointer, issue, PR, job id, preventive change, general lesson. Seeded from F1–F10 and E1–E3, plus nine operational findings not previously in any catalogue:

O1	Model transfers DERP-relayed through Frankfurt (#553)
O2	Deploy test sharing project name and ports with production (#544)
O3	importorskip hiding the collector tests through two live 8-site runs
O4	Four handbook tests that have never run (#565)
O5	CI concurrency evicting queued validations (#554)
O6	Four of eight site join dates wrong by ~6 weeks (#560)
O7	A zero class count rendered as blank — a finding read as a gap
O8	Robust aggregation costing 0.40 relative error under 94-fold imbalance (#566)
O9	Confidence anti-correlated with accuracy; recalibration does not fix it

The anonymisation rule is enforced, not intended

The plan commits to naming sites where the fact is neutral and verifiable, and pseudonymising them wherever the variable describes participation — response latency, unavailable periods, support burden. Each row declares attribution_kind, and the validator fails if a participation row names its site anywhere in the prose, not merely in the site column.

Verified by injecting a violation:

F9: participation row names RSH in 'impact' — anonymise the prose, not just the site column
exit code with violation: 1
exit code when clean:    0

CI-usable as-is. A validator that cannot fail is worth nothing — which is the same lesson as O3 and O4 in the matrix itself.

The pseudonym mapping is deliberately not ordered by dataset size. Those sizes are published under real names in the data-by-site deck, so a size-ordered mapping would re-identify every anonymised row on sight.

Sourcing

Every row carries at least one of evidence / issue / PR / job id. Any that did not would be reported as [UNSOURCED] rather than left to look established. Currently: none.

Coverage gaps are surfaced, not hidden

 !! 1. Consortium governance is part of the technical system       0
    2. Site readiness is staged, not binary                        3
    3. Infrastructure heterogeneity dominates                      7
    4. Detect before training, contain locally                     3
    5. Observability and provenance are prerequisites              2
    6. Documentation, testing and automation encode knowledge      3
    7. Distributed computation != valid evaluation                 5
 !! 8. Prototype to consortium requires redesign                   0

Lessons 1 and 8 have no incident rows, because those arguments are carried by correspondence and architecture rather than by incidents. The tool says so explicitly so the manuscript does not imply the matrix supports them.

Contents

• docs/manuscript/evidence_matrix.csv — the data
• docs/manuscript/evidence_matrix_public.md — generated appendix, grouped by lesson
• scripts/manuscript/build_evidence_matrix.py — validate (default) or --publish

Docs and one script; no runtime code touched.

🤖 Generated with Claude Code