Add lazy reindexer that can operate on ManifestArrays

[sharkinsspatial]

What I did

Added lazy reindex and reindex_like accessor methods that align a virtual dataset to new coordinate labels without materializing data, padding missing positions in the new target grid with null-path manifest entries so that the missing chunks are read as fill_value.

Acceptance criteria:

• Closes Support for sparsely tiled spatial grids? #882 Concatenation of virtual datasets fails due to missing Chunk Manager #382 Concatenation in time of (time, y, x)-dimentional data fails #883 ITS_LIVE data virtualization nasa-jpl/its_live_production#59
• Tests added
• Tests passing
• No test coverage regression
• Full type hint coverage
• Changes are documented in docs/about/releases.md
• New functions/methods are listed in an appropriate *.md file under docs/api
• New functionality has documentation

[codecov]

Codecov Report

❌ Patch coverage is 95.74468% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.89%. Comparing base (e1ebb4b) to head (8f42dcb).

Files with missing lines	Patch %	Lines
virtualizarr/manifests/array.py	87.50%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1031      +/-   ##
==========================================
+ Coverage   89.64%   89.89%   +0.24%     
==========================================
  Files          36       37       +1     
  Lines        2202     2296      +94     
==========================================
+ Hits         1974     2064      +90     
- Misses        228      232       +4     

Files with missing lines	Coverage Δ
virtualizarr/accessor.py	96.82% <100.00%> (+1.08%)	⬆️
virtualizarr/manifests/reindex.py	100.00% <100.00%> (ø)
virtualizarr/manifests/array.py	85.29% <87.50%> (+0.67%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[TomNicholas]

What would it take to support this type of reindexing in xarray?

[sharkinsspatial]

I did a few aborted investigations with Claude. Once I finalize this Draft I'll add some more context in the initial comment but I should start by saying my understanding of xarray internals is very limited. I'll have to look back through the results of my initial Claude session last week (I think I serialized as a memory) but IIUC

1. xarray's reindex operates at the element level rather than the array level so there would be no duck array hook for reindex.
2. Internally reindex is using a gather operation that operates at the element level. This element level gather could move elements within a chunk. We can only operate lazily at the chunk level without materializing the chunks so this would probably be undesirable.
3. Internally reindex also uses an element level where call that would require materializing.

My guess is that there may be some path to potentially updating xarray internals to handle this but I couldn't grok how that would be handled. Maybe you or @dcherian have insights here.

In the interim, users can call this accessor reindexer to mimic what would happen internally in xarray and then use xarray's native combination methods on the pre-aligned data.

[sharkinsspatial]

I'll actually probably try to add an additional align accessor method which would be

1. Calculate the union of the both ds indexes.
2. Reindex each ds onto these unioned indexes

To make the the user experience a bit more seamless.

[dcherian]

Xarray will provide the backing array library with an array indexer of ints with -1 indicating missing. What you can do is take that indexer, partition it by chunks and then for each partition assert either the partition has no -1 or is all -1 (that is the case you can support). if so, you should be able to translate that indexer to whatever approach is used here. If you support this, you will get Xarray's reindex & alignment machinery for free.

[sharkinsspatial]

@dcherian 🙇‍♂️ Thanks for this. I couldn't make the leap of logic for partitioning the indexer by chunk boundary. I'll try to investigate tomorrow, hopefully I can get this working for the simple case before we land #954 and then someone else will be on the hook for rectilinear partitioning 😆