Skip to content

[WIP][SPARK-57421][SQL][CONNECT] Support @-syntax timestamp time travel on table names#56847

Open
sotikoug83 wants to merge 4 commits into
apache:masterfrom
sotikoug83:SPARK-57421-at-syntax-timestamp
Open

[WIP][SPARK-57421][SQL][CONNECT] Support @-syntax timestamp time travel on table names#56847
sotikoug83 wants to merge 4 commits into
apache:masterfrom
sotikoug83:SPARK-57421-at-syntax-timestamp

Conversation

@sotikoug83

Copy link
Copy Markdown

What changes were proposed in this pull request?

Add the @ time travel shorthand on table names so that

SELECT * FROM name@vN
SELECT * FROM name@<TS>

and

spark.read.format("format").table("name@vN")
spark.read.format("format").table("name@<ts>")

parse into the same plan as the existing VERSION AS OF / TIMESTAMP AS OF and option("versionAsOf") / option("timestampAsOf") clauses. Support for SQL, the DataFrame reader, and Spark Connect.

Why are the changes needed?

Spark has SQL AS OF and the versionAsOf/timestampAsOf reader options but not the compact @ suffix. Resolving it at parse time also simplifies the time travel entry points pipeline.

Does this PR introduce any user-facing change?

Yes, new syntax on table names gated by a conf (spark.sql.timeTravel.atSyntax.enabled) which allows time travel, default true. No longer gives a parse error when enabled and the syntax is used. Additionally, different from TIMESTAMP AS OF <TS> (which allows for arbitrary expressions in <TS>), the @<ts> timestamp only accepts the fixed 17-digit yyyyMMddHHmmssSSS Delta convention.

How was this patch tested?

New tests in PlanParserSuite, DataSourceV2SQLSuite, and SparkConnectPlannerSuite.

Was this patch authored or co-authored using generative AI tooling?

With help from Claude :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant