-
Notifications
You must be signed in to change notification settings - Fork 2.4k
fix: unwrap_cast_in_comparison drops the timezone shift when unwrapping CAST(timestamp AS timestamptz) = literal #25099
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
6f6e5b4
ab0676b
df079a8
81b3579
156a6a2
d0ad665
b84ba80
e60c001
7a9760c
2cea9a2
838b442
340c693
72202c0
86a9cc8
f66c2a3
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -113,17 +113,54 @@ fn is_date_type(data_type: &DataType) -> bool { | |
| /// `Date64` carrying sub-day milliseconds would lose them. This is not a licence to | ||
| /// drop them - [`try_cast_numeric_literal`] returns `None` for a `Date64` value not | ||
| /// divisible by 86_400_000, so an inexact `Date64` -> `Date32` fold never happens. | ||
| /// | ||
| /// **Timezone Shifts:** | ||
| /// Conversions between timezone-naive and timezone-aware timestamps are | ||
| /// mathematically bijective (shifting the physical value by the timezone offset), | ||
| /// rather than many-to-one lossy. However, we return `true` here to block unwrapping | ||
| /// as an intentionally conservative guard. If we returned `false`, `unwrap_cast_in_comparison` | ||
| /// would strip the cast but fail to shift the underlying literal, returning incorrect | ||
| /// query results. (A robust alternative would be to allow the unwrap and shift the literal, | ||
| /// preserving pushdown and pruning.) Only UTC-equivalent timezones (where the shift is | ||
| /// exactly zero) are allowed to bypass this guard. | ||
| fn is_lossy_temporal_cast(from_type: &DataType, to_type: &DataType) -> bool { | ||
| if from_type == to_type { | ||
| return false; | ||
| } | ||
| if is_date_type(from_type) && is_date_type(to_type) { | ||
| return false; | ||
| } | ||
| if let (DataType::Timestamp(_, from_tz), DataType::Timestamp(_, to_tz)) = | ||
| (from_type, to_type) | ||
| { | ||
| match (from_tz, to_tz) { | ||
| (Some(tz), None) | (None, Some(tz)) | ||
| if !is_zero_offset_timezone(tz.as_ref()) => | ||
| { | ||
| return true; | ||
| } | ||
| _ => {} | ||
| } | ||
| } | ||
|
Comment on lines
+133
to
+144
|
||
| (is_date_type(from_type) && to_type.is_temporal()) | ||
| || (is_date_type(to_type) && from_type.is_temporal()) | ||
| } | ||
|
|
||
| /// Returns true if the timezone is known to have a fixed zero offset from UTC. | ||
| /// | ||
| /// This is used to determine if a cast between a timezone-aware and timezone-naive | ||
| /// timestamp is lossy. If the timezone is strictly UTC-equivalent, the cast is | ||
| /// a lossless re-labeling of the integer value. | ||
| fn is_zero_offset_timezone(tz: &str) -> bool { | ||
| match tz { | ||
| // Standard UTC identifiers | ||
| "UTC" | "Etc/UTC" | "GMT" | "Etc/GMT" | "Greenwich" | "Z" => true, | ||
| // Common fixed offset zero strings parsed by Arrow | ||
| "+00:00" | "-00:00" | "+0:00" | "-0:00" => true, | ||
| _ => false, | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is there no function in arrow-rs or chrono we could use for this?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The arrow::array::timezone::Tz enum only allows fetching the offset dynamically for a specific NaiveDateTime via offset_from_utc_datetime(). However, we cannot simply instantiate an arbitrary date (like the Unix Epoch) and check if the offset is zero because geographic timezones like Europe/London evaluate to an offset of 0 during winter time. This would create false positives during query planning. Because we need to evaluate this statically during optimization (where we don't want to instantiate values just to check offsets), explicitly whitelisting the known permanent zero-offset strings is currently the safest and most robust approach. |
||
| } | ||
| } | ||
|
|
||
| /// Returns true when casting a timestamp from `from_type` to `to_type` loses | ||
| /// timestamp precision. | ||
| /// | ||
|
|
@@ -998,6 +1035,31 @@ mod tests { | |
| assert!(is_lossy_temporal_cast(&ts, &DataType::Date32)); | ||
| } | ||
|
|
||
| #[test] | ||
| fn test_is_lossy_temporal_cast_timestamp_tz() { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This test examines Please add the queries from the issue to statement ok
set datafusion.execution.time_zone = 'Asia/Singapore';
statement ok
create table t as select TIMESTAMP '2024-11-01T00:00:00' as ts;
statement ok
create table u as select '2024-10-31T16:00:00Z'::timestamptz as tstz;
# 2024-11-01 00:00 in Singapore is 2024-10-31 16:00 UTC
query I
select count(*) from t where ts::timestamptz = '2024-10-31T16:00:00Z'::timestamptz;
----
1
query I
select count(*) from t where ts::timestamptz = '2024-11-01T00:00:00Z'::timestamptz;
----
0
# the same rewrite occurs for an implicit coercion
query I
select count(*) from t where ts = '2024-10-31T16:00:00Z'::timestamptz;
----
1
# control: a column against a column, thus the optimizer unwraps nothing
query I
select count(*) from t, u where t.ts::timestamptz = u.tstz;
----
1Please add two more cases:
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done! I've added all of these queries to datafusion/sqllogictest/test_files/datetime/timestamps.slt. I also added the two extra test cases (the symmetric guard check and the EXPLAIN block for the UTC session timezone) to ensure the optimization is still applied correctly when the offset is exactly zero. |
||
| let ts_naive = DataType::Timestamp(TimeUnit::Millisecond, None); | ||
| let ts_utc = DataType::Timestamp(TimeUnit::Millisecond, Some("UTC".into())); | ||
| let ts_etc_utc = | ||
| DataType::Timestamp(TimeUnit::Millisecond, Some("Etc/UTC".into())); | ||
| let ts_gmt = DataType::Timestamp(TimeUnit::Millisecond, Some("GMT".into())); | ||
| let ts_sgt = | ||
| DataType::Timestamp(TimeUnit::Millisecond, Some("Asia/Singapore".into())); | ||
|
|
||
| // Naive <-> UTC is NOT lossy (UTC offset is 0, so literal cast is exact) | ||
| assert!(!is_lossy_temporal_cast(&ts_naive, &ts_utc)); | ||
| assert!(!is_lossy_temporal_cast(&ts_utc, &ts_naive)); | ||
| assert!(!is_lossy_temporal_cast(&ts_naive, &ts_etc_utc)); | ||
| assert!(!is_lossy_temporal_cast(&ts_naive, &ts_gmt)); | ||
|
|
||
| // Naive <-> Non-UTC is lossy because it ignores session timezone | ||
| assert!(is_lossy_temporal_cast(&ts_naive, &ts_sgt)); | ||
| assert!(is_lossy_temporal_cast(&ts_sgt, &ts_naive)); | ||
|
|
||
| // Tz-aware <-> Tz-aware is not lossy (both are UTC under the hood) | ||
| assert!(!is_lossy_temporal_cast(&ts_utc, &ts_sgt)); | ||
| assert!(!is_lossy_temporal_cast(&ts_sgt, &ts_utc)); | ||
| } | ||
|
Comment on lines
+1038
to
+1061
|
||
|
|
||
| #[test] | ||
| fn test_timestamp_precision_narrowing_cast() { | ||
| let ts_ns = DataType::Timestamp(TimeUnit::Nanosecond, None); | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4301,6 +4301,10 @@ SELECT column1 FROM t_utc WHERE column1 < '2024-02-01T00:00:00' AT TIME ZONE 'Am | |
| query P | ||
| SELECT column1 FROM t_europe WHERE column1 = '2024-01-31T16:00:01' AT TIME ZONE 'America/Los_Angeles'; | ||
| ---- | ||
|
|
||
|
Comment on lines
4302
to
4304
|
||
| query P | ||
| SELECT column1 FROM t_europe WHERE column1 = '2024-01-31T15:00:01' AT TIME ZONE 'America/Los_Angeles'; | ||
| ---- | ||
| 2024-02-01T00:00:01+01:00 | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I verified this change on your branch. The empty result is correct, but the PR does not give the reason, and the description says the opposite (see the note from the Copilot review). Please put the reason in the PR description. The arithmetic:
An empty result is a weak assertion. A guard that is too strong also gives an empty result, and this test cannot see the difference. Please keep a row here. The instant I ran this query two times on your branch: one time with your change, and one time with the new guard removed. Without the guard it gives zero rows, which is incorrect. With the guard it gives the row above. It is thus a correct test for this bug. Please keep the empty case also.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for catching that! I have updated the PR description with the exact explanation. I also restored the original empty test case and added your suggested 15:00:01 query alongside it to ensure we have a strong assertion that the correct row is being selected. |
||
|
|
||
| query P | ||
|
|
@@ -5529,3 +5533,75 @@ query P | |
| SELECT date_bin(NULL, TIMESTAMP '2023-01-01 12:30:00', TIMESTAMP '2023-01-01 12:00:00') | ||
| ---- | ||
| NULL | ||
|
|
||
| # Issue 25095: Optimizer incorrectly unwrapping timestamp cast when session timezone is not UTC | ||
| statement ok | ||
| set datafusion.execution.time_zone = 'Asia/Singapore'; | ||
|
|
||
| statement ok | ||
| create table t_25095 as select TIMESTAMP '2024-11-01T00:00:00' as ts; | ||
|
|
||
| statement ok | ||
| create table u_25095 as select '2024-10-31T16:00:00Z'::timestamptz as tstz; | ||
|
|
||
| # 2024-11-01 00:00 in Singapore is 2024-10-31 16:00 UTC | ||
| query I | ||
| select count(*) from t_25095 where ts::timestamptz = '2024-10-31T16:00:00Z'::timestamptz; | ||
| ---- | ||
| 1 | ||
|
|
||
| query I | ||
| select count(*) from t_25095 where ts::timestamptz = '2024-11-01T00:00:00Z'::timestamptz; | ||
| ---- | ||
| 0 | ||
|
|
||
| # the same rewrite occurs for an implicit coercion | ||
| query I | ||
| select count(*) from t_25095 where ts = '2024-10-31T16:00:00Z'::timestamptz; | ||
| ---- | ||
| 1 | ||
|
|
||
| # control: a column against a column, thus the optimizer unwraps nothing | ||
| query I | ||
| select count(*) from t_25095, u_25095 where t_25095.ts::timestamptz = u_25095.tstz; | ||
| ---- | ||
| 1 | ||
|
|
||
| # A timezone-aware column against a timezone-naive literal | ||
| query I | ||
| select count(*) from u_25095 where tstz = TIMESTAMP '2024-11-01T00:00:00'; | ||
| ---- | ||
| 1 | ||
|
|
||
| # Set session timezone back to UTC and demonstrate that the cast IS unwrapped | ||
| statement ok | ||
| set datafusion.execution.time_zone = 'UTC'; | ||
|
|
||
| # The explain output should show that the cast s::timestamptz has been removed | ||
| # because we allow unwrap_cast_in_comparison for UTC offsets | ||
| query TT | ||
| EXPLAIN select count(*) from t_25095 where ts::timestamptz = '2024-11-01T00:00:00Z'::timestamptz; | ||
| ---- | ||
| logical_plan | ||
| 01)Projection: count(Int64(1)) AS count(*) | ||
| 02)--Aggregate: groupBy=[[]], aggr=[[count(Int64(1))]] | ||
| 03)----Projection: | ||
| 04)------Filter: t_25095.ts = TimestampNanosecond(1730419200000000000, None) | ||
| 05)--------TableScan: t_25095 projection=[ts] | ||
| physical_plan | ||
| 01)ProjectionExec: expr=[count(Int64(1))@0 as count(*)] | ||
| 02)--AggregateExec: mode=Final, gby=[], aggr=[count(Int64(1))] | ||
| 03)----CoalescePartitionsExec | ||
| 04)------AggregateExec: mode=Partial, gby=[], aggr=[count(Int64(1))] | ||
| 05)--------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1 | ||
| 06)----------FilterExec: ts@0 = 1730419200000000000, projection=[] | ||
| 07)------------DataSourceExec: partitions=1, partition_sizes=[1] | ||
|
|
||
| statement ok | ||
| drop table t_25095; | ||
|
|
||
| statement ok | ||
| drop table u_25095; | ||
|
|
||
| statement ok | ||
| RESET datafusion.execution.time_zone; | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I verified that this guard corrects the four queries in the issue, thank you. Four points on the block:
Please add the new rule to the doc comment of
is_lossy_temporal_cast. The comment gives the rules for identity casts, date casts andDate32/Date64casts in detail. It says nothing about timezones.A cast between a naive timestamp and a timezone-aware timestamp is not lossy. The cast is bijective: it moves the value by the offset of the timezone. The doc comment of this function describes a different problem, which is a cast that is many-to-one. The guard gives the correct result, but the name and the comment now disagree with the code. Please make the reason clear at this position.
The issue gives a second solution: keep the unwrap, but move the literal by the same offset. That solution keeps the optimization. The present solution stops the unwrap, and thus the engine loses the pushdown and the pruning for each of these comparisons. Did you examine the second solution? If you prefer the present solution, please add a comment that says that the guard is intentionally conservative.
unwrap()is safe here, becauseis_some() != is_some()makes sure that one side has a value. But amatchon the two options is more clear, and it removes theunwrap().There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the thorough review! I’ve updated the PR to address all four points:
is_lossy_temporal_castdoc comment to clearly explain the new timezone rules.truehere is an intentionally conservative guard. Shifting the literal would keep pushdown and pruning working, but blocking the unwrap is a safer immediate fix to guarantee correctness.match (from_tz, to_tz)statement and removed theunwrap().Please take a look at the latest commits!