feat: verify ensemble forecasters

README.md

@@ -114,7 +114,8 @@ experiment:
         # Stratification dimension to use as scorecard columns (e.g. region, season).
         stratification: region
         # Variables and metrics as scorecard rows. Format: VAR:METRIC1,METRIC2,...
-        # Supported metrics: RMSE, R2, ETS, POD, FAR (categorical metrics require thresholds).
+        # Supported metrics: RMSE, R2, ETS, POD, FAR (categorical metrics require thresholds),
+        # CRPS (ensemble runs; a deterministic participant uses its MAE).
         variables:
           - "T_2M:RMSE,R2"
           - "TOT_PREC:RMSE,ETS"
@@ -382,11 +383,35 @@ runs:
 - A member that finished keeps a `.ok` file under
   `logs/inference_execute/{run_id}-{init_time}/`, so a rerun after a failure only runs
   the missing members.
-- Verification, plots and MEC use member 001 only, with a warning.
+- Verification scores the ensemble (see below); plots, SAL, score maps and MEC use
+  member 001 only, with a warning.
 - Not allowed on the inline `forecaster` of a `temporal_downscaler`.
 - Changing `members` or `seed` gives the run a new `run_id`; the inference environment
   is reused.
 
+#### Ensemble verification
+
+An ensemble run is verified from all its members, in the usual `verif_*.nc` files of
+the run:
+
+- Its deterministic scores (`BIAS`, `RMSE`, `MAE`, contingency scores, ...) are those
+  of the ensemble mean. There is no separate score for a single member: to verify one,
+  run it as a deterministic forecaster.
+- Ensemble scores are added next to them: the fair `CRPS`, the ensemble spread
+  `SPREAD` and spread-skill ratio `SSR`, the mean score of the single members
+  `MEMBER_RMSE`, Brier scores per threshold, and the counts of the rank histogram and
+  reliability diagram.
+- Categorical parameters (`PTYPE`) have no ensemble mean (rain, 1, and snow, 5,
+  would average to freezing rain, 3), so their contingency scores (`ETS`, `POD`, `FAR`,
+  ...) are NaN for an ensemble run. The Brier scores and reliability diagrams of their
+  `eq` thresholds score them instead.
+- The forecast statistics (`mean`, `var`, `min`, `max`) are averaged over the members.
+- A point where any member is missing counts as missing for the ensemble (and, with
+  `score_on_common_points`, is left out for every participant).
+- Each member is loaded like a deterministic run (derived and period parameters,
+  lapse-rate correction with the orography of member 001). Memory and runtime of the
+  verification jobs grow with the number of members.
+
 
 ## Installation
 

resources/report/dashboard/template.html.jinja2

@@ -191,6 +191,7 @@
         </div>
     </div>
 
+    {% if crps_note %}<p class="note">{{ crps_note | e }}.</p>{% endif %}
     <div id="legend-chart"></div>
 
     <div id="chart-scroll">