How to handle datetime arrays with timezone information? #3656

Activity

1. weiji14

   added

   questionFurther information is requestedFurther information is requested

   on Nov 28, 2024
2. seisman commented on Nov 28, 2024

   

   seisman

   on Nov 28, 2024

   Member

   More actions

   datetime arrays (e.g. pandas.Series with pandas.Timestamp, pyarrow.date32, pyarrow.date64, etc) that have non-UTC timezones

   pyarrow.date32/pyarrow.date64 are timezone-unware. As far as I know, only datetime.datetime, datetime.time, pandas.DatetimeTZDtype and pyarrow.timestamp are timezone-aware.

   ---

   I feel we should go with option 1, so the behavior is consistent with both GMT and the Python world (at least for matplotlib and pandas).

   import numpy as np
   import pandas as pd
   import matplotlib.pyplot as plt
   
   df = pd.DataFrame(
       {
           "datetime": pd.date_range(start="2024-01-01", periods=10, freq="H").tz_localize("Asia/Shanghai"), 
           "y": np.random.rand(10),
       }
   ) 
   print(df["datetime"])
   
   plt.figure(figsize=(10, 6))
   plt.plot(df["datetime"], df["y"], marker="o", linestyle="-", color="blue", label="y values")
   plt.title("Time Series Data (Hourly)")
   plt.xlabel("Datetime (UTC+8)")
   plt.ylabel("Y")
   plt.show()
   
   # Use pandas' plotting function
   df.plot(
       x="datetime",
       y="y",
       kind="line",
       marker="o",
       title="Time Series Data (Hourly)",
       figsize=(10, 6),
       grid=True,
       xlabel="Datetime (UTC+8)",
       ylabel="Y Values",
       legend=True
   )
   plt.show()

   
   

   👀React with 👀1Reacted by Wei Ji
3. weiji14 commented on Nov 28, 2024

   

   weiji14

   on Nov 28, 2024

   Last edited by weiji14

   MemberAuthor

   More actions

   datetime arrays (e.g. pandas.Series with pandas.Timestamp, pyarrow.date32, pyarrow.date64, etc) that have non-UTC timezones

   pyarrow.date32/pyarrow.date64 are timezone-unware. As far as I know, only datetime.datetime, datetime.time, pandas.DatetimeTZDtype and pyarrow.timestamp are timezone-aware.

   Yep, was typing this out too quickly. Have updated the top-post to clarify this.

   ---

   I feel we should go with option 1, so the behavior is consistent with both GMT and the Python world (at least for matplotlib and pandas).

   I'm trying to get some historical context on how matplotlib/pandas has handled timezone plotting before. It seems like matplotlib used to have a plot_date() function which allows setting of a tz parameter, but this was deprecated in matplotlib/matplotlib#18154 and it's unclear/confusing on what the regular matplotlib .plot function does now in terms of timezone handling.

   There's also an interesting case on what to do when plotting two or more arrays with different timezones (e.g. UTC and Europe/Berlin time) at matplotlib/matplotlib#8072 (comment). With option 2, everything is converted to UTC so things will be plotted 'correctly'; but with option 1, ignoring the timezones means the data would be plotted in a non-deterministic manner depending on which array (UTC or Europe/Berlin) was passed in first.
4. seisman commented on Nov 29, 2024

   

   seisman

   on Nov 29, 2024

   Last edited by seisman

   Member

   More actions

   it's unclear/confusing on what the regular matplotlib .plot function does now in terms of timezone handling.

   Isn't it clear that matplotlib's plot method is timezone-aware from my example above? The first point is plotted at 01-01 (Asia/Shanghai), not 12-31 (UTC).

   with option 1, ignoring the timezones means the data would be plotted in a non-deterministic manner depending on which array (UTC or Europe/Berlin) was passed in first.

   This won't be a case in GMT or PyGMT, since arrays can be processed independently. What makes more troubles is the handling of datetime with different timezones in arguments (although people are unlikely and unrecommended to use mixed timezones) like:

   region = [
       datetime.datetime(2024, 1, 1, 10, 0, 0, tzinfo=zoneinfo.ZoneInfo('Asia/Shanghai')),
       datetime.datetime(2024, 1, 2, 10, 0, 0, tzinfo=zoneinfo.ZoneInfo('UTC')),
       0, 
       10
   ]