fix: use session timezone for timestamp subtraction by kumarUjjawal · Pull Request #25094 · apache/datafusion

kumarUjjawal · 2026-09-09T04:28:56Z

Which issue does this PR close?

Closes datafusion.execution.time_zone is not used for basic time zone inference #13212.

Rationale for this change

When a timezone-aware timestamp is subtracted from a timezone-naive timestamp, DataFusion does not use datafusion.execution.time_zone to interpret the naive value.

For example, with the session timezone set to +08:00, subtracting 2024-11-01 00:00:00 from 2024-11-01 00:00:00+00:00 returns zero instead of eight hours.

PostgreSQL and DuckDB handle this by implicitly casting the timezone-naive operand to the session timezone. DataFusion already produces the expected result when that cast is written explicitly, so the missing behavior belongs in type coercion.

What changes are included in this PR?

This PR:

Passes the configured session timezone to the type-coercion analyzer.
Casts the timezone-naive operand of mixed timestamptz - timestamp expressions using the session timezone.
Preserves the timezone of the timezone-aware operand.
Coerces both operands to the same timestamp precision.
Applies the same behavior to both operand orders and nested subqueries.
Passes the session timezone through ExprSimplifier::coerce, ensuring SessionContext::create_physical_expr behaves consistently with SQL planning.

The change is limited to timestamp subtraction. Comparison operators retain their existing behavior and should be handled separately.

The existing DST-boundary limitation described in #25084 remains. Because this PR inserts the previously missing cast automatically, mixed timestamp subtraction can now encounter that limitation for ambiguous or nonexistent local times.

When no session timezone is configured, existing behavior is unchanged.

What is the testing strategy for this PR?

The added tests cover:

The reproduction from datafusion.execution.time_zone is not used for basic time zone inference #13212.
Literal and column expressions.
Both subtraction operand orders.
Different session and operand timezones.
EST and EDT offsets for a named session timezone.
The direct SessionContext::create_physical_expr API.
Timestamp operands with different precisions.

The following checks pass:

cargo fmt --all -- --check
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test -p datafusion-optimizer
cargo test -p datafusion --test core_integration

The extended workspace test suite from the contributor guide also passes, including all 512 SQL logic test files.

Are there any user-facing changes?

There are no public API changes.

Copilot

🟡 Changes recommended

The claimed nested-subquery coverage does not currently exercise subtraction inside a subquery.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Fixes #13212 by applying the session timezone when coercing mixed timezone-aware and timezone-naive timestamp subtraction.

Changes:

Propagates session timezone through type coercion, simplification, and subqueries.
Preserves timezone metadata while normalizing timestamp precision.
Adds analyzer, physical-expression, and SQL regression tests.

File summaries

File	Description
`datafusion/sqllogictest/test_files/datetime/timestamps.slt`	Tests mixed timestamp subtraction across timezones.
`datafusion/optimizer/src/utils.rs`	Uses the new rewriter constructor.
`datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs`	Supplies session timezone during expression coercion.
`datafusion/optimizer/src/scalar_subquery_to_join.rs`	Uses the new rewriter constructor.
`datafusion/optimizer/src/analyzer/type_coercion.rs`	Implements timezone-aware subtraction coercion and propagation.
`datafusion/core/tests/expr_api/mod.rs`	Tests direct physical-expression creation.

Review details

Files reviewed: 6/6 changed files
Comments generated: 1
Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

codecov-commenter · 2026-09-09T05:23:18Z

Codecov Report

❌ Patch coverage is 95.28302% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.80%. Comparing base (a5c809f) to head (4f209e5).
⚠️ Report is 13 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/optimizer/src/analyzer/type_coercion.rs	95.09%	4 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #25094      +/-   ##
==========================================
+ Coverage   81.74%   81.80%   +0.05%     
==========================================
  Files        1128     1130       +2     
  Lines      416644   417799    +1155     
  Branches   416644   417799    +1155     
==========================================
+ Hits       340592   341783    +1191     
+ Misses      55995    55883     -112     
- Partials    20057    20133      +76

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

adriangb · 2026-09-09T12:52:54Z

        let right_data_type = right.get_type(right_schema)?;
-        let (left_type, right_type) =
+        let (left_type, right_type) = if let Some(types) =
+            self.timestamp_subtraction_input_types(&left_data_type, &op, &right_data_type)


Could this go in BinaryTypeCoercer instead of the analyzer?

This special case lives in TypeCoercionRewriter, so the PR has to send the session timezone through four subquery call sites and through ExprSimplifier::coerce. One caller still does not get it: the coerce function in optimizer/src/utils.rs.

BinaryTypeCoercer in expr-common is the one source of coercion rules. The analyzer, the simplifier, the physical BinaryExpr::data_type, the statistics solver, and interval arithmetic all use it. A rule in BinaryTypeCoercer applies to all of them with no plumbing.

The cause of the bug is also in that file. In the arithmetic arm of signature_inner, the first branch asks arrow for a result type. Arrow accepts Timestamp(u, Some) - Timestamp(u, None) when the units are equal and reads the naive side as UTC. When the units differ, the pair falls through to temporal_coercion_strict_timezone, which casts the naive side to the aware side's timezone. This is what makes results depend on the units. A check before the arrow probe fixes that.

On main with SET TIME ZONE = '+08:00':

SELECT arrow_cast('2024-11-01T00:00:00Z', 'Timestamp(Nanosecond, Some("+08:00"))') - '2024-11-01T00:00:00'::timestamp; -- 0 hours (wrong) SELECT arrow_cast('2024-11-01T00:00:00Z', 'Timestamp(Millisecond, Some("+08:00"))') - '2024-11-01T00:00:00'::timestamp; -- 8 hours (right)

adriangb · 2026-09-09T12:52:54Z


 fn coerce(expr: Expr, schema: &DFSchema) -> Result<Expr> {
-    let mut expr_rewrite = TypeCoercionRewriter { schema };
+    let mut expr_rewrite = TypeCoercionRewriter::new(schema);


This caller does not get the session timezone, so the new rule does not apply here. See comment above.

adriangb · 2026-09-09T12:52:54Z

+        if op != &Operator::Minus {
+            return None;
+        }
+        let session_time_zone = self.session_time_zone?;


Can we add a test to assert the current/expected behavior when datafusion.execution.time_zone is None? Something like:

statement ok RESET datafusion.execution.time_zone statement ok SET datafusion.explain.logical_plan_only = true # With no session timezone the naive operand is still read as UTC: # 2024-11-01T00:00:00-04:00 is 04:00Z, and the naive value is taken as 00:00Z. statement ok CREATE TABLE no_session_tz AS SELECT arrow_cast('2024-11-01T00:00:00-04:00', 'Timestamp(Nanosecond, Some("America/New_York"))') AS ts_tz, '2024-11-01T00:00:00'::timestamp AS ts; query ?? SELECT ts_tz - ts, ts - ts_tz FROM no_session_tz; ---- 0 days 4 hours 0 mins 0.000000000 secs 0 days -4 hours 0 mins 0.000000000 secs query TT EXPLAIN SELECT ts_tz - ts FROM no_session_tz; ---- logical_plan 01)Projection: no_session_tz.ts_tz - no_session_tz.ts 02)--TableScan: no_session_tz projection=[ts_tz, ts] statement ok SET datafusion.explain.logical_plan_only = false

(I ran this against the PR branch: the values are 4 hours / -4 hours and no cast is inserted, so it records today's behaviour.)

adriangb · 2026-09-09T12:52:54Z

+            ),
+            _ => return None,
+        };
+        let DataType::Timestamp(unit, _) = comparison_coercion(left_type, right_type)?


Could we match on the two units directly, or make timeunit_coercion visible and call it instead of going through comparison_coercion?

adriangb · 2026-09-09T12:52:54Z

+----
+0 days 8 hours 0 mins 0.000000000 secs
+
+# The session timezone, not the aware operand's timezone, controls the cast.


The rule seems to diverge for - and =. On this branch:

SET datafusion.execution.time_zone = '+08:00'; CREATE TABLE t AS SELECT arrow_cast('2024-11-01T04:00:00Z', 'Timestamp(Nanosecond, Some("America/New_York"))') AS ts_tz, '2024-11-01T00:00:00'::timestamp AS ts; SELECT ts_tz = ts, ts_tz - ts FROM t;

returns true and 0 days 12 hours: = reads ts in America/New_York (so the two values are the same instant) while - reads it in the session timezone +08:00 (so they are 12 hours apart). Two values that compare equal yet differ by twelve hours.

fix: use session timezone for timestamp subtraction

c42a082

kumarUjjawal requested a review from adriangb September 9, 2026 04:29

github-actions Bot added optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Sep 9, 2026

adriangb requested a balanced review from Copilot September 9, 2026 04:30

Copilot started reviewing on behalf of adriangb September 9, 2026 04:32 View session

Copilot AI reviewed Sep 9, 2026

View reviewed changes

Comment thread datafusion/sqllogictest/test_files/datetime/timestamps.slt

test: cover timezone coercion in scalar subquery

4f209e5

adriangb reviewed Sep 9, 2026

View reviewed changes

adriangb mentioned this pull request Sep 9, 2026

fix: resolve DST-ambiguous and nonexistent local times when casting naive timestamps to a named timezone #25115

Closed

Navigation Menu

Uh oh!

fix: use session timezone for timestamp subtraction - #25094

fix: use session timezone for timestamp subtraction#25094
kumarUjjawal wants to merge 2 commits into
apache:mainfrom
kumarUjjawal:fix/13212-timezone-inference

kumarUjjawal commented Sep 9, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

codecov-commenter commented Sep 9, 2026

Uh oh!

adriangb Sep 9, 2026

Uh oh!

adriangb Sep 9, 2026

Uh oh!

adriangb Sep 9, 2026

Uh oh!

adriangb Sep 9, 2026

Uh oh!

adriangb Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

kumarUjjawal commented Sep 9, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

What is the testing strategy for this PR?

Are there any user-facing changes?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

codecov-commenter commented Sep 9, 2026

Codecov Report

Uh oh!

adriangb Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

adriangb Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

adriangb Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

adriangb Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

adriangb Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants