feat: make obs/var backend-agnostic with a DataFrameLike contract - #2516
Open
srivarra wants to merge 18 commits into
Open
feat: make obs/var backend-agnostic with a DataFrameLike contract#2516srivarra wants to merge 18 commits into
srivarra wants to merge 18 commits into
Conversation
Using narwhals as the conversion layer between DataFrame implementations, makes Dataset2D "dataframe-like" in the eyes of narwhals. Signed-off-by: Sricharan Reddy Varra <sricharanvarra@gmail.com>
for more information, see https://pre-commit.ci
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #2516 +/- ##
==========================================
- Coverage 87.83% 86.70% -1.13%
==========================================
Files 49 50 +1
Lines 7799 8183 +384
==========================================
+ Hits 6850 7095 +245
- Misses 949 1088 +139
|
Signed-off-by: Sricharan Reddy Varra <sricharanvarra@gmail.com>
Signed-off-by: Sricharan Reddy Varra <sricharanvarra@gmail.com>
Preserve eager dataframe backends while representing axis identity through an index or canonical obs_names and var_names columns. Signed-off-by: Sricharan Reddy Varra <sricharanvarra@gmail.com>
Normalize only the index name when axis values already match. Keep PyArrow helper coverage independent of Polars. Signed-off-by: Sricharan Reddy Varra <sricharanvarra@gmail.com>
Signed-off-by: Sricharan Reddy Varra <sricharanvarra@gmail.com>
Signed-off-by: Sricharan Reddy Varra <sricharanvarra@gmail.com>
n Signed-off-by: Sricharan Reddy Varra <sricharanvarra@gmail.com>
Signed-off-by: Sricharan Reddy Varra <sricharanvarra@gmail.com>
`nw.DataFrame` was rejected as `obs`/`var` while `nw.LazyFrame` was accepted, and the rejection contained nothing: both satisfy `DataFrameLike` structurally, so they slipped into `obsm`/`uns` and were stored as the wrapper, which IO then could not write. `unwrap_narwhals` peels the wrapper at the ingest boundaries. For `obs`/`var` it is a `_gen_dataframe` registration that unwraps and dispatches again, so a pandas-backed narwhals frame lands on the pandas handler and keeps its index instead of being treated as index-less. Move the `narwhals.plugins` entry point to `anndata._core.xarray`, next to the `Dataset2D` it describes, and drop the banner comments and other stylistic drift from `_dataframe_backend.py` so the module reads like the rest of the codebase.
# Conflicts: # src/anndata/acc/__init__.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Using narwhals as the conversion layer between DataFrame implementations, and makes Dataset2D "dataframe-like" in the eyes of narwhals.
DataFrameAPI forobsandvarkeys via runtime-checkableProtocol#2043I built a thin Narwhals translation layer between
AnnData.obs/AnnData.varand the DataFrame backends Narwhals supports..obsand .varnow accept native DataFrames instead of requiring pandas.The DataFrameLike contract builds on #2328. Indexed frames retain their native index, while index-less frames store the index in an
obs_namesorvar_namescolumn. DataFrame ops like slicing, copying, transposing, renaming categories, and updating names use this contract.I also added
AnnData.obs_as(...)andAnnData.var_as(...)to convert annotations to pandas, Polars, PyArrow, Modin, or cuDF. Concatenation preserves Polars or PyArrow when all inputs use the same backend, including annotations combined through the merge strategies. At write time, frames without a native writer convert to pandas so the existing HDF5 and Zarr writers continue to work.A small Narwhals plugin makes Dataset2D compatible with Narwhals, allowing conversion to other DataFrame formats. This conversion materializes the
Dataset2Das an in-memory pandas DataFrame during conversion. Lazy-to-lazy conversion should probably come in a separate PR, for example, convertingDataset2Dto a Dask DataFrame without loading it in memory. Right now, converting Dataset2D to an eager backend such as Polars still materializes the frame.I used AI tools to help with the implementation.
Remaining TODOS:
Dataset2D -> Narwhals LazyFrame somehow?Deferred for later