[SPARK-57741][PYTHON] Add timestamp_nanos to PySpark public API#56852
Open
jubins wants to merge 1 commit into
Open
[SPARK-57741][PYTHON] Add timestamp_nanos to PySpark public API#56852jubins wants to merge 1 commit into
jubins wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
Fixes SPARK-57741 (follow-up to SPARK-57526) — adds
timestamp_nanosto thePySpark API (
pyspark.sql.functionsand PySpark Connect), completing thenanosecond round-trip pair in Python.
timestamp_nanos(e)converts a nanoseconds-since-epoch integer to aTIMESTAMP_LTZ(9)value. It is the inverse ofunix_nanos(SPARK-57579).The SQL function and Scala API were added in SPARK-57526, but Python support
was explicitly deferred and tracked as a follow-up via
expected_missing_in_py:The round-trip pair is now complete
unix_nanostimestamp_nanosBrief change log
python/pyspark/sql/functions/builtin.pyAdded
timestamp_nanos(col)aftertimestamp_micros, decorated with@_try_remote_functions, with full docstring:versionadded:: 4.3.0python/pyspark/sql/connect/functions/builtin.pyAdded Connect-side wrapper for
timestamp_nanos, inheriting docstring from main module and following the same pattern astimestamp_microspython/pyspark/sql/functions/__init__.pyExported
timestamp_nanosin alphabetical order betweentimestamp_millisandtimestamp_secondspython/docs/source/reference/pyspark.sql/functions.rstAdded
timestamp_nanosentry betweentimestamp_millisandtimestamp_secondspython/pyspark/sql/tests/test_functions.pyRemoved
"timestamp_nanos"fromexpected_missing_in_py(set is now empty)Verifying this change
Covered by the existing parity test in
FunctionsTestsMixin:test_function_paritypreviously allowlistedtimestamp_nanosas an expected gapexpected_missing_in_pyensures the test will now fail iftimestamp_nanosis ever missing from the Python API againThe two doctests in the
timestamp_nanosdocstring verify:TIMESTAMP_LTZ(9)valueDoes this pull request potentially affect one of the following parts
@Public/@Evolving): Yes — new public PySpark functionDocumentation
pyspark.sql.functions.timestamp_nanosbuiltin.pyWas generative AI tooling used to co-author this PR?
All code was written, understood, and verified by the author.
Generated-by: Claude Opus 4.8