Repository navigation
Support PyArrow arrays and dataframes #2800
Description
Description of the desired feature
Apache Arrow is an in-memory format that is starting to become a common exchange format between different libraries in Python and other programming languages. For example:
- Pandas 3.0 will use
pyarrow.stringinstead ofobjectdtype for strings, see PDEP10, so we will eventually need to support PyArrow (at least for string dtypes) - Polars DataFrames can be zero-copy converted to Pyarrow via the
__dataframe__protocol, see https://arrow.apache.org/docs/python/interchange_protocol.html
This issue is to track compatibility and support of different PyArrow data types in PyGMT:
| Dtype | Implementation PR | Status | Notes |
|---|---|---|---|
| Numerical (uint/int/float) | #2774 | ✅ | |
| String | #2933 | 🚧 | May require modifying the put_strings method that currently uses np.char.encode |
| Date/Time | #2845 (pandas) and TODO (raw pyarrow) | 🚧 | May require modifying array_to_datetime that expects Python datetime or numpy-backed arrays, xref #242 and #3507 |
| Duration | TODO | ❌ | https://arrow.apache.org/docs/13.0/python/generated/pyarrow.duration.html, wait for #2884 also |
Special case: geopandas.GeoDataFrame with PyArrow dtype columns |
TODO | ❌ | See #2774 (comment) |
| GeoArrow geometry | TODO | ❌ | https://github.com/geoarrow/geoarrow-python |
Simplest way of integrating would be to just handle PyArrow-backed pandas.Dataframe objects as above.
Alternatively, we can also discuss about using PyArrow as the internal array representation (which would make pyarrow a hard dependency) since it may allow better interoperability across other Python libraries using Arrow, and this might be relevant for #1318 and #2731. My thought is to do this through the __dataframe__ protocol, see https://arrow.apache.org/docs/python/interchange_protocol.html
Further reading:
- https://voltrondata.com/codex/standards-over-silos#1-2-standardizing-on-arrow
- https://voltrondata.com/resources/dataframe-interoperability-python-pyarrow-enables-modular-workflows
- https://arrow.apache.org/docs/python/pandas.html#zero-copy-series-conversions
Are you willing to help implement and maintain this feature?
Yes, but help is welcome too!
Activity
[-]Support pyarrow arrays and dataframes[/-][+]Support PyArrow arrays and dataframes[/+]seisman commentedon Dec 7, 2023
Before supporting pyarrow-backed pandas objects like what you're doing in PR #2774 and #2845, maybe we should check/support passing pyarrow arrays directly to PyGMT? If all/most pyarrow dtypes work, then we can go on with pyarrow-backed pandas objects. Then if interested, we may support polars.
weiji14 commentedon Dec 7, 2023
Sure, I'd love to have direct support for PyArrow arrays too. I started with pyarrow-backed pandas objects because pandas 3.0 will eventually use PyArrow for string columns by default, but no reason we can't support passing a pyarrow.array object directly into PyGMT.
weiji14 commentedon Dec 9, 2023
Just opened a PR for this at #2864. Surprisingly, most PyGMT functions already work with
pyarrow.arrayorpyarrow.tablewithout any modification (I've testedblockm,info,nearneighbor,project,triangulate,xyz2grdso far), possibly because PyGMT can convert them internally tonumpy.array(see e.g. https://arrow.apache.org/docs/python/numpy.html#arrow-to-numpy). Will need to test out more complicated dtypes and check for edge cases, but it's looking promising!