Repository navigation
Interpolate single-variable datasets without copying, coarsen skipping NaN - #11
Merged
Merged
Conversation
interp2poi and interp2grid stacked the data variables with Dataset.to_array, whose np.stack copies the data even for a single variable. ds_to_da gets a copy parameter (default True, unchanged): with copy=False, a single data variable spanning all the dimensions is returned as a view (expand_dims), identical to to_array. Both functions only read the array. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The coarsening of interp2grid (source downsampled to a coarser destination) used coarsen().reduce(np.mean), which gives NaN for a block with any NaN. _window_nanmean averages the valid values of each block with bottleneck.nanmean (only all-NaN blocks stay NaN), e.g. for the snow limit, NaN below the orography. With the window moved last and flattened, it is faster than np.mean (ICON-CH1 to the seamless 2 km grid 0.30 s -> 0.21 s). bottleneck sums float32 in float32: the error grows with the window size (about 1e-7 relative for 2 x 2, 2e-6 for 40 x 40). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cd24e93):interp2poiandinterp2gridcallds_utils.ds_to_da(..., copy=False). When there is one data variable and it spans all dimensions, they get a view viaexpand_dimsinstead of thenp.stackcopy made byto_array. The default stayscopy=True.01e36a0): both coarsening sites (_coarsen_interpandslice_coarsen) usecoarsen(...).reduce(_window_nanmean). This is a 2-line helper that moves the window axes last and callsbottleneck.nanmean. A coarsened cell with some NaN gets the mean of its valid values; only all-NaN cells stay NaN. This prepares for regridding the snow limit, which is NaN below the orography.Performance (operational grids, later calls with cached neighbors)
nearestnearest_altnearest_altKnown regression: ICON-CH2 in LV95 → seamless 2 km gets slower, from 0.23 to 0.29 s with
nearest_altand from 0.34 to 0.39 s withtemp_fixgrad. The source there is a sliced, non-contiguous view, and the reshape now copies it with a gather. Compacting the data asmaindid is slower in 9 of the 12 operational grid cases.Changes of results
Tests
test_ds_to_da_no_copy: checks the view, the default copy, multiple variables, and a dimension that only a coordinate has.test_window_nanmeanandtest_window_nanmean_large_window: check partial and all-NaN blocks, float32, integers equal tonp.mean, and a 10 × 10 window withinrtol=1e-6of the float64nanmean.🤖 Generated with Claude Code