Repository navigation
Never search neighbors or triangulate in latitude/longitude degrees - #8
Merged
Merged
Conversation
The generic path of interp2poi and interp2grid (KDTree methods and griddata) ran in latitude/longitude degrees whenever lat/lon coordinates were present, distorting distances (a degree of longitude is about 0.68 of a degree of latitude in Switzerland). Coordinates are now always in meters: - native coordinates projected in meters are used first (get_coords_name_crs(prefer_projected=True)); - metric source or destination: its CRS (the larger grid if both are metric); - both in lat/lon: LV95 if the destination is within lv95_max_offset (700 km) of the LV95 central northing, otherwise earth-centered earth-fixed coordinates (ECEF), which supports global grids; - linear, linear_3D and cubic raise an error in the ECEF case; - altitude is always the last coordinate column; height methods no longer force Swiss coordinates. Other changes: - geographic CRS on a sphere (e.g. CF rotated pole with earth_radius) are considered on WGS84, so that the datum shift (~180 m to LV95) is applied; parse_grid_mapping builds such CF mappings directly (pyproj.CRS.from_cf took ~0.3 s) and caches the other ones; - set_grid_mapping and parse_grid_mapping support numpy values in grid mappings; - create_dst_grid_from_src spans exactly the reprojected source domain; - interp2grid raises a clear error for unstructured destination grids. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With latitude/longitude source grids whose longitude convention (0..360 or -180..180) differs from the one of the POIs or destination grid, the source was sliced to the wrong longitudes (wrong values in the generic path, e.g. ERA5 or GFS to a -180..180 global grid) and the regular grid interpolation (try_reg=True) returned NaN. - target longitudes are brought into the convention of the source grid before slicing and before the regular grid interpolation; - global grids are padded with one column across their 360 degree seam (only when targets fall in it) for the regular grid interpolation; - when slicing a lat/lon source, the longitude buffer is widened by 1 / cos(latitude), since neighbors are searched in meters, and all longitudes are kept close to the poles or for targets spanning more than 180 degrees. Operational cases (47, data4web POIs and seamless 2 km grid): same runtime; values identical except tie-breaking between equidistant neighbors (ICON-CH2 LV95 to the 2 km grid offset by half a cell). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The temp method raised a rank-deficient error when all the k nearest grid points of a POI have the same altitude, which happens close to the poles of global latitude/longitude grids (their pole rows repeat the same point, e.g. GFS, ERA5) or on flat orography. Such POIs now use the existing fixed lapse rate fallback (nearest grid point corrected with 6 K/km), like POIs far above or below their neighborhood. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rid) - 05_interp2poi_gfs_global_temperature: GFS 0.25° 2 m temperature (single run and lead time, only the needed fields downloaded through the index byte ranges) interpolated to 33 stations worldwide in varied terrain with nearest, linear and temp, compared with NOAA GHCN-hourly observations; - 06_interp2grid_era5_global_temperature: ERA5 0.25° 2 m temperature at a single time regridded to the global IFS 0.1° open data grid with nearest (regular and generic path), linear and temp_fixgrad. Both illustrate the ECEF neighbor search, the 0..360 longitudes of the source grids and the poles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ebook - benchmark nearest through both the regular grid interpolation and the ECEF neighbor search, and compare their grid points at the stations - label the stations whose model altitude differs by more than 300 m - mention masking the sea grid points for coastal stations Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- test_get_coords_name_crs and test_get_coords_name_crs_prefer_projected checked independent fixtures in one test, through a parametrization with repeated values. One test per fixture, assertions unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- pyproj already converts float32 and integer coordinates to float64, so the cast only copied the full 2D source coordinates before outer_ring. The destination grids are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The fixgrad fallback applies to POIs more than 200 m above the highest or 600 m below the lowest neighbor (the docstring said 300 m and 500 m). - The z_fact default is 25 (the docstring said 50). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- is_metric_crs compared the axis unit names to "metre"/"meter". PROJ normalizes known units to "metre", but keeps custom names as written (e.g. UNIT["m",1] or UNIT["Meter",1] in a WKT without authority), which were missed. The conversion factor to the meter is now compared, so km, feet and the German legal metre are still excluded. - test_is_metric_crs is parametrized, with the custom WKT units, km and the German legal metre added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hic CRS - A projected CRS (e.g. LV95 or UTM) cannot represent the whole globe, so the destination grid of a global source would be meaningless. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The generic path of
interp2poiandinterp2grid(all KDTree methods, andgriddataforlinear/cubic) ran in latitude/longitude degrees whenever the grid had lat/lon coordinates, even for native LV95 grids. In Switzerland a degree of longitude is about 0.68 of a degree of latitude, so east–west distances counted about 1.45 times too much. The neighbors, the IDW weights and the triangulations were distorted.Coordinates are now always in meters:
get_coords_name_crs(prefer_projected=True), only in the generic path).lv95_max_offset(700 km, envGRID_PROCESS__lv95_max_offset) of the LV95 central northing, otherwise earth-centered earth-fixed coordinates (ECEF), which supports global grids.linear,linear_3Dandcubicraise a clear error in the ECEF case, since a triangulation is degenerate there. For regular lat/lon grids,try_reg=Trueremains available.Further fixes found along the way (separate commits):
earth_radius) are treated as WGS84, so the datum shift to LV95 (about 180 m) is applied.parse_grid_mappingbuilds such CF mappings directly (pyproj.CRS.from_cftook about 0.3 s per call) and caches the other ones.set_grid_mappingandparse_grid_mappingaccept numpy values (e.g. astandard_parallelarray).create_dst_grid_from_srcnow spans exactly the reprojected source domain. Before, it was e.g. 80 km too wide for ICON-CH2 in LV95 and missed 28 km of IFS.try_reg=Truereturned NaN. Global grids are now also interpolated across their 360° seam. The longitude buffer of the slicing is widened by 1/cos(latitude).temp. POIs whose neighbors all have the same altitude (pole rows of lat/lon grids, flat orography) now use the fixed lapse rate fallback instead of raising an error.interp2gridraises a clear error for an unstructured destination grid (not supported).Behaviour changes
linear/cubicresults. This includes native LV95 grids with lat/lon coordinates.linear,linear_3Dandcubicraise an error for geographic domains beyondlv95_max_offset.get_grid_crs/parse_grid_mappingreturn lat/lon and rotated-pole CF mappings on a sphere based on WGS84.create_dst_grid_from_srcproduces different (correct) bounds.set_grid_mappingwrites the grid mapping with sorted keys.tempno longer raises for flat neighborhoods.Benchmark (124 cases in fresh processes: the operational data4web POIs and seamlessweatherprovider grid cases, the notebook cases, the MIR comparison cases and others;
mainand the branch run back to back):main, within run-to-run noise. In the last run, first calls took 154 → 164 s and next calls 132 → 134 s (operational next calls 108.2 → 109.7 s). An earlier run gave the branch about 10 % faster.mainand the branch alternated over two rounds): the horizon withnearest(13.5 s) and 3 mosmix descriptors withlinear(1.4 s) are on par withmain, with identical values and peak memory. grid-process itself takes 0.03–0.06 s of each call in both versions; the rest is dask reading the zarr chunks.The benchmark scripts are kept outside the repository.
Related issue
None
Checklist