Skip to content

Fix object array infer string perf 11470 #11484

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
JJFlorian wants to merge 6 commits into
base: main
Choose a base branch
from
+22 −4
Open

Fix object array infer string perf 11470 #11484

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
912275b
infer the dtype + cast where possible
JJFlorian Jul 30, 2026
e8d915a
add more cases to the test_as_compatible_data_writeable
JJFlorian Jul 30, 2026
acd4a4c
make if elif else logic more explicit
JJFlorian Jul 30, 2026
fe7c091
return values directly in else
JJFlorian Jul 30, 2026
9f176e5
add comment on datetime64 & timedelta64 cases
JJFlorian Jul 30, 2026
5514a0f
add comment on datetime64 & timedelta64 cases
JJFlorian Jul 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 19 additions & 4 deletions xarray/core/variable.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -210,8 +210,9 @@ def _maybe_wrap_data(data):

def _possibly_convert_objects(values):
"""Convert object arrays into datetime64 and timedelta64 according
to the pandas convention. For backwards compat, as of 3.0.0 pandas,
object dtype inputs are cast to strings by `pandas.Series`
to the pandas convention. Object dtype inputs that are inferred to be
strings are returned unchanged. For backwards compat, as of 3.0.0 pandas,
the remaining object dtype inputs are cast to strings by `pandas.Series`
but we output them as object dtype with the input metadata preserved as well.


Expand All @@ -220,8 +221,22 @@ def _possibly_convert_objects(values):
* pd.Timestamp
* pd.Timedelta
"""
as_series = pd.Series(values.ravel(), copy=False)
result = np.asarray(as_series).reshape(values.shape)
inferred = pd.api.types.infer_dtype(values.ravel(), skipna=True)

if inferred == "string":
return values
elif inferred == "datetime":
result = pd.to_datetime(values.ravel()).to_numpy().reshape(values.shape)
elif inferred == "timedelta":
result = pd.to_timedelta(values.ravel()).to_numpy().reshape(values.shape)
elif inferred in ["datetime64", "timedelta64"]:

@dcherian dcherian Jul 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's add a comment about infer_dtype dropping units

@JJFlorian JJFlorian Jul 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, I would say the cast drops them

# Casting drops unit info for these cases;
# fall back to pd.Series roundtrip, which preserves them.
as_series = pd.Series(values.ravel(), copy=False)
result = np.asarray(as_series).reshape(values.shape)
else:
return values

if not result.flags.writeable:
# GH8843, pandas copy-on-write mode creates read-only arrays by default
try:
Expand Down
3 changes: 3 additions & 0 deletions xarray/tests/test_variable.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,9 @@ def var():
[
np.array(["a", "bc", "def"], dtype=object),
np.array(["2019-01-01", "2019-01-02", "2019-01-03"], dtype="datetime64[ns]"),
np.array([datetime(2000, 1, 1), datetime(2000, 1, 2)], dtype=object),
np.array([timedelta(seconds=1), timedelta(seconds=2)], dtype=object),
np.array([1, "a"], dtype=object),
],
)
def test_as_compatible_data_writeable(data):
Expand Down
Loading