perf: add fast path for local hdf5#1038
Draft
ianhi wants to merge 1 commit into
Draft
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1038 +/- ##
=======================================
Coverage 89.87% 89.87%
=======================================
Files 36 36
Lines 2202 2212 +10
=======================================
+ Hits 1979 1988 +9
- Misses 223 224 +1
🚀 New features to boost your workflow:
|
Member
|
@maxrjones does this seems sensible to you? |
ilan-gold
reviewed
Jul 14, 2026
Comment on lines
+5
to
+10
| Local files are walked with HDF5's own index-aware driver, so building a | ||
| manifest reads only the chunk index. For **remote** chunk-dense files (many | ||
| small chunks — e.g. sparse single-cell `.h5ad`), the default block reader | ||
| can instead fetch a large fraction of the whole file just to recover that | ||
| index. | ||
|
|
Contributor
There was a problem hiding this comment.
Not relevant here @ianhi but this is exactly what I was saying should be a target for doing in rust, especially over many files simultaneously
Contributor
|
This fix seems reasonable! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What I did
Added a fast path for parsing HDF files that are local. For these the Blockreader can cause massive perforamnce hits. Especially for a larger files (40 GB), you can actually end up reading more data than the file size! especially for b-trees where you end up revisting nodes.
draft for now - this is AI code I have not yet personally carefully reviewd.
attn @ilan-gold curious if this helps with the performance issues here: https://annbatch.readthedocs.io/en/stable/preshuffling.html
Acceptance criteria:
docs/about/releases.md*.mdfile underdocs/api