Skip to content

Never search neighbors or triangulate in latitude/longitude degrees - #8

Merged
matschaer merged 10 commits into
mainfrom
fix/kdtree-no-latlon
Nov 11, 2026
Merged

matschaer merged 10 commits into
mainfrom
fix/kdtree-no-latlon

Conversation

@matschaer

Copy link
Copy Markdown
Collaborator

The generic path of interp2poi and interp2grid (all KDTree methods, and griddata for linear / cubic) ran in latitude/longitude degrees whenever the grid had lat/lon coordinates, even for native LV95 grids. In Switzerland a degree of longitude is about 0.68 of a degree of latitude, so east–west distances counted about 1.45 times too much. The neighbors, the IDW weights and the triangulations were distorted.

Coordinates are now always in meters:

  • Native coordinates first. Native coordinates projected in meters are used even when lat/lon coordinates are present (get_coords_name_crs(prefer_projected=True), only in the generic path).
  • CRS choice:
    • if one side (source or target) is in meters, its CRS is used;
    • if both are, the larger grid's CRS;
    • if both are lat/lon, LV95 when the target is within lv95_max_offset (700 km, env GRID_PROCESS__lv95_max_offset) of the LV95 central northing, otherwise earth-centered earth-fixed coordinates (ECEF), which supports global grids.
  • linear, linear_3D and cubic raise a clear error in the ECEF case, since a triangulation is degenerate there. For regular lat/lon grids, try_reg=True remains available.
  • Altitude column. Height methods no longer force Swiss coordinates; the altitude is always the last coordinate column.

Further fixes found along the way (separate commits):

  • Sphere datum. Lat/lon or rotated-pole CRSs on a sphere (CF earth_radius) are treated as WGS84, so the datum shift to LV95 (about 180 m) is applied. parse_grid_mapping builds such CF mappings directly (pyproj.CRS.from_cf took about 0.3 s per call) and caches the other ones.
  • Numpy values in grid mappings. set_grid_mapping and parse_grid_mapping accept numpy values (e.g. a standard_parallel array).
  • create_dst_grid_from_src now spans exactly the reprojected source domain. Before, it was e.g. 80 km too wide for ICON-CH2 in LV95 and missed 28 km of IFS.
  • Longitude convention and 360° seam. With lat/lon sources whose longitude convention (0..360 or −180..180) differs from the target's, the source was sliced to the wrong longitudes (wrong values in the generic path) and try_reg=True returned NaN. Global grids are now also interpolated across their 360° seam. The longitude buffer of the slicing is widened by 1/cos(latitude).
  • Flat neighborhoods in temp. POIs whose neighbors all have the same altitude (pole rows of lat/lon grids, flat orography) now use the fixed lapse rate fallback instead of raising an error.
  • Unstructured destinations. interp2grid raises a clear error for an unstructured destination grid (not supported).
  • Two global example notebooks:
    • GFS 2 m temperature to 33 stations worldwide, compared with GHCN-hourly observations;
    • ERA5 2 m temperature to the IFS 0.1° grid.

Behaviour changes

  • Values change slightly for grids previously searched in lat/lon: neighbors at about one grid spacing, the IDW weights, and linear/cubic results. This includes native LV95 grids with lat/lon coordinates.
  • linear, linear_3D and cubic raise an error for geographic domains beyond lv95_max_offset.
  • get_grid_crs / parse_grid_mapping return lat/lon and rotated-pole CF mappings on a sphere based on WGS84.
  • create_dst_grid_from_src produces different (correct) bounds.
  • set_grid_mapping writes the grid mapping with sorted keys.
  • temp no longer raises for flat neighborhoods.

Benchmark (124 cases in fresh processes: the operational data4web POIs and seamlessweatherprovider grid cases, the notebook cases, the MIR comparison cases and others; main and the branch run back to back):

  • Totals: on par with main, within run-to-run noise. In the last run, first calls took 154 → 164 s and next calls 132 → 134 s (operational next calls 108.2 → 109.7 s). An earlier run gave the branch about 10 % faster.
  • Per case: the median first-call difference is +14 ms. Only the memory-bound large-k cases vary by more, in both directions between runs.
  • Longitude fix, measured on the 47 operational cases: identical runtime. Values are identical, except for tie-breaking between exactly equidistant neighbors (ICON-CH2 LV95 to the 2 km grid, which is offset by half a cell).
  • Regular grid path (topographic descriptors of data4web from the topo catalog v2, dask arrays, to the 5622 production POIs; warm-up, then main and the branch alternated over two rounds): the horizon with nearest (13.5 s) and 3 mosmix descriptors with linear (1.4 s) are on par with main, with identical values and peak memory. grid-process itself takes 0.03–0.06 s of each call in both versions; the rest is dask reading the zarr chunks.

The benchmark scripts are kept outside the repository.


Related issue

None


Checklist

  • Code is well documented and follows project conventions (linted, formatted)
  • CHANGELOG updated (if applicable)
  • License header is added to all new files (the only new files are the two notebooks, which, like the existing ones, have no header)
  • For a new contributors, their name has been added to the AUTHORS file
  • No sensitive data or secrets are exposed
  • Unit tests added covering the new code

Mathieu Schaer and others added 5 commits November 9, 2026 20:55
The generic path of interp2poi and interp2grid (KDTree methods and griddata) ran in latitude/longitude
degrees whenever lat/lon coordinates were present, distorting distances (a degree of longitude is about
0.68 of a degree of latitude in Switzerland). Coordinates are now always in meters:

- native coordinates projected in meters are used first (get_coords_name_crs(prefer_projected=True));
- metric source or destination: its CRS (the larger grid if both are metric);
- both in lat/lon: LV95 if the destination is within lv95_max_offset (700 km) of the LV95 central
  northing, otherwise earth-centered earth-fixed coordinates (ECEF), which supports global grids;
- linear, linear_3D and cubic raise an error in the ECEF case;
- altitude is always the last coordinate column; height methods no longer force Swiss coordinates.

Other changes:

- geographic CRS on a sphere (e.g. CF rotated pole with earth_radius) are considered on WGS84, so that
  the datum shift (~180 m to LV95) is applied; parse_grid_mapping builds such CF mappings directly
  (pyproj.CRS.from_cf took ~0.3 s) and caches the other ones;
- set_grid_mapping and parse_grid_mapping support numpy values in grid mappings;
- create_dst_grid_from_src spans exactly the reprojected source domain;
- interp2grid raises a clear error for unstructured destination grids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With latitude/longitude source grids whose longitude convention (0..360 or -180..180) differs from the
one of the POIs or destination grid, the source was sliced to the wrong longitudes (wrong values in the
generic path, e.g. ERA5 or GFS to a -180..180 global grid) and the regular grid interpolation
(try_reg=True) returned NaN.

- target longitudes are brought into the convention of the source grid before slicing and before the
  regular grid interpolation;
- global grids are padded with one column across their 360 degree seam (only when targets fall in it)
  for the regular grid interpolation;
- when slicing a lat/lon source, the longitude buffer is widened by 1 / cos(latitude), since neighbors
  are searched in meters, and all longitudes are kept close to the poles or for targets spanning more
  than 180 degrees.

Operational cases (47, data4web POIs and seamless 2 km grid): same runtime; values identical except
tie-breaking between equidistant neighbors (ICON-CH2 LV95 to the 2 km grid offset by half a cell).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The temp method raised a rank-deficient error when all the k nearest grid points of a POI have the same
altitude, which happens close to the poles of global latitude/longitude grids (their pole rows repeat the
same point, e.g. GFS, ERA5) or on flat orography. Such POIs now use the existing fixed lapse rate fallback
(nearest grid point corrected with 6 K/km), like POIs far above or below their neighborhood.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rid)

- 05_interp2poi_gfs_global_temperature: GFS 0.25° 2 m temperature (single run and lead time, only the needed
  fields downloaded through the index byte ranges) interpolated to 33 stations worldwide in varied terrain
  with nearest, linear and temp, compared with NOAA GHCN-hourly observations;
- 06_interp2grid_era5_global_temperature: ERA5 0.25° 2 m temperature at a single time regridded to the
  global IFS 0.1° open data grid with nearest (regular and generic path), linear and temp_fixgrad.

Both illustrate the ECEF neighbor search, the 0..360 longitudes of the source grids and the poles.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ebook

- benchmark nearest through both the regular grid interpolation and the
  ECEF neighbor search, and compare their grid points at the stations
- label the stations whose model altitude differs by more than 300 m
- mention masking the sea grid points for coastal stations

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@matschaer matschaer self-assigned this Nov 10, 2026
Comment thread grid_process/data_model.py Resolved
Comment thread grid_process/grid_interp_utils.py Resolved
Comment thread test/test_data_model.py Resolved
Mathieu Schaer and others added 5 commits November 11, 2026 00:03
- test_get_coords_name_crs and test_get_coords_name_crs_prefer_projected
  checked independent fixtures in one test, through a parametrization with
  repeated values. One test per fixture, assertions unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- pyproj already converts float32 and integer coordinates to float64, so
  the cast only copied the full 2D source coordinates before outer_ring.
  The destination grids are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The fixgrad fallback applies to POIs more than 200 m above the highest or
  600 m below the lowest neighbor (the docstring said 300 m and 500 m).
- The z_fact default is 25 (the docstring said 50).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- is_metric_crs compared the axis unit names to "metre"/"meter". PROJ
  normalizes known units to "metre", but keeps custom names as written
  (e.g. UNIT["m",1] or UNIT["Meter",1] in a WKT without authority), which
  were missed. The conversion factor to the meter is now compared, so km,
  feet and the German legal metre are still excluded.
- test_is_metric_crs is parametrized, with the custom WKT units, km and the
  German legal metre added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hic CRS

- A projected CRS (e.g. LV95 or UTM) cannot represent the whole globe, so
  the destination grid of a global source would be meaningless.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@matschaer
matschaer merged commit e479e16 into main Nov 11, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants