Extend functionality for MetDB observations and related point-based observation recipe support

[ukmo-huw-lewis]

Addresses #2084

Contribution checklist

Aim to have all relevant checks ticked off before merging. See the developer's guide for more detail.

• Documentation has been updated to reflect change.
• New code has tests, and affected old tests have been updated.
• All tests and CI checks pass.
• Ensured the pull request title is descriptive.
• Ensure rose-suite.conf.example has been updated if new diagnostic added.
• Conda lock files have been updated if dependencies have changed.
• Attributed any Generative AI, such as GitHub Copilot, used in this PR.
• Marked the PR as ready to review.

[github-actions]

[ukmo-huw-lewis]

Requesting initial review from David Flack (@daflack), but will keep as draft PR at this stage.

Please advise on PR scope, and balance of splitting into smaller chunks, vs review and apply full change-set given dependencies.

Note this PR depends on #2161 being reviewed and merged initially, as will help make additional operators added to 'plot.py' more streamlined (i.e. makes use of some changes separated to that PR), plus addition of _plot_colormaps.py reflected here, but for review along with other reductions to length of plot.py under #2161.
Will update after new changes with reduced changeset after successful completion of #2161, and at that point change status to "Ready for review".

Note coverage checks fail, but these issues are largely resolved by #2161. Any outstanding gaps in test coverage will be addressed before PR marked as "Ready for review".

Note also a dependency on changes to CSET-restricted-files fetch-obs-metdb.py - already reviewed and on main.

[ukmo-huw-lewis]

As general overview of changes, there are 3 main areas:

A) Updates to operators to efficiently process point observations.

• read.py

1. add _check_combine_point_observations function. This enables reading in different MetDB OBS files (e.g. separate files for LNDSYN, SHPSYN, WOW retrievals, and combine into single cube of successively numbered stations). This enables wildcard reading of different ob types, but treatment as 1 set of "observations" in CSET processing. If users wish to select only certain ob types, this can be achieved by only setting those obtypes in input paths.

• misc.py (changes to support comparisons across CubeList of Obs | Model1 | Model2 | Model3 | .... :

1. extended functionality of subtraction operator to work with CubeList and preserve cubelist attributes.
2. extended functionality of extract_common_points to work with CubeList.

• regrid.py

1. update interpolate_to_point_cube with more pythonic approach to regridding, and add support for regridding Cube (e.g. 1 model) or CubeList (i.e. multiple models) from same call. This change greatly speeds up processing of point observation comparisons (and now relatively quick processing).

• plot.py

1. note many changes here reflect early ideas now implemented in Refactor plot.py #2161 and therefore anticipating tidier changeset once that merged to main and this branch updated.
2. add more automated treatment of map projections in _setup_spatial_map. This should likely sit in different PR, but became useful in testing obs over N. Pole and S. Pole subarea cutouts.
3. include some aesthetic modifications to alpha of coastlines and borders. Again, this should likely sit in different PR.
4. update _plot_and_save_spatial_plot to include "scatter" as valid plottype. This enables 'scattermap' plots of point observations without need for separate spatial plotting, enabling reuse of all existing spatial map functionality for observation points rather than bespoke code (needing additional updates).
5. similarly, update _plot_and_save_spatial_plot to optionally receive "scatter_cube" as input. This enables over-plotting of point obs on top of pcolormesh of spatial model fields (utilising some of the 'multi_spatial' plotting).
6. some aesthetic changes to switch frameon=True in legends - this seems to give clearer legends in my view.
7. added new plot functionality to support _plot_and_save_scatter_series to enable multi-model/obs scatter and hexbin plot types.
8. add support for looping over realization (all station-by-station model vs obs matchups considered in same output) or over 'station' (separate output plots generated for each station) in histogram, scatter and timeseries plotting functions.

[ukmo-huw-lewis]

B) addition of new recipes

See recipes/observations for full listing:

• generic_obs_spatial_plot.yaml
• generic_model_obs_spatial_sequence,yaml
• generic_model_obs_spatial_difference.yaml
• generic_model_obs_histogram.yaml
• generic_model_obs_timeseries.yaml
• generic_model_obs_timeseries_bias.yaml
• generic_model_obs_timeseries_points.yaml
• generic_obs_scatter.yaml

[ukmo-huw-lewis]

C) Workflow updates to support new observations and introduce more consistent env var naming conventions

See updates to observations/rose-meta.conf, rose-suite.conf.example and loaders/observations.py

Example outputs for UKV vs LM vs LMV comparisons to LNDSYN and WOW observations available at https://wwwspice/~huw.lewis/CSET/LMV_May2026/

Other outputs generated via cset bake on command-line can be shared where useful to support review.

[David Flack (daflack)]

Just a few comments from me to think about, as a first go round. I really like the recipes, really simple, clear and straightforward and easy for people to adapt.

Potential ideas on splitting the PR: recipes, additional capability, and hexplot addition.

Quick question/thought: Is there a conversion file for the names to match the obs with the model as getting from surface_fields? I thought I saw this in an earlier version but it appears to have dropped off. Also wondering what would happen if the surface fields had a large range (e.g. with variables we do not have observations for) as if observation comparisons they would crash so is it worth flagging this up in the documentation/having a separate entry just for observations - although I appreciate that would be a little repetitive from the user perspective so may not be the best idea.

Overall, from a science perspective a great, and well needed, addition. I've not looked at it technically, but from what I can tell it looks technically sound, and the evidence you have provided is good.