From 794c4ea8ab2d40008265b2515fd4502caa23057c Mon Sep 17 00:00:00 2001 From: Jubin Soni Date: Sun, 28 Jun 2026 08:46:41 -0700 Subject: [PATCH] [SPARK-57740][PYTHON] Fix unix_micros truncation disclaimer and add unix_nanos cross-references to unix_* See Also sections MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ### What is the purpose of the change Fixes SPARK-57740 — two related doc-correctness issues in the `unix_*` epoch-unit function family in `python/pyspark/sql/functions/builtin.py`: 1. **Missing truncation disclaimer in `unix_micros`.** `unix_millis` and `unix_seconds` both include `"Truncates higher levels of precision."` in their one-line description. `unix_micros` did not, despite silently dropping sub-microsecond digits when given a nanosecond-precision input (`TIMESTAMP_LTZ(p)` / `TIMESTAMP_NTZ(p)`, `p` in `[7, 9]`). This became a practical concern once `unix_nanos` (SPARK-57579) made nanosecond-precision timestamps usable in PySpark. 2. **Missing `unix_nanos` cross-references.** `unix_nanos` (SPARK-57579) correctly links to all three siblings in its `See Also` section. None of the three sibling functions were updated to link back to `unix_nanos`, breaking discoverability of the function family from the existing functions. ### Brief change log - `python/pyspark/sql/functions/builtin.py`: - `unix_micros`: added `"Truncates higher levels of precision."` to the one-line description - `unix_micros`, `unix_millis`, `unix_seconds`: added `:meth:\`pyspark.sql.functions.unix_nanos\`` to each `See Also` section ### Verifying this change These are documentation-only changes with no logic impact. Verified by inspection that: - All four `unix_*` functions now consistently include a truncation disclaimer in their description - All four `unix_*` functions now cross-reference each other in their `See Also` sections symmetrically ### Does this pull request potentially affect one of the following parts - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public`/`@Evolving`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: no - The S3 file system connector: no ### Documentation Does this pull request introduce a new feature? No — documentation fix only. ### Was generative AI tooling used to co-author this PR? Yes — Claude Code was used as a pair-programming assistant. All code was written, understood, and verified by the author. Generated-by: Claude Opus 4.8 --- python/pyspark/sql/functions/builtin.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/python/pyspark/sql/functions/builtin.py b/python/pyspark/sql/functions/builtin.py index bca4704962f96..de6683eabef51 100644 --- a/python/pyspark/sql/functions/builtin.py +++ b/python/pyspark/sql/functions/builtin.py @@ -11710,6 +11710,7 @@ def unix_date(col: "ColumnOrName") -> Column: @_try_remote_functions def unix_micros(col: "ColumnOrName") -> Column: """Returns the number of microseconds since 1970-01-01 00:00:00 UTC. + Truncates higher levels of precision. .. versionadded:: 3.5.0 @@ -11728,6 +11729,7 @@ def unix_micros(col: "ColumnOrName") -> Column: :meth:`pyspark.sql.functions.unix_date` :meth:`pyspark.sql.functions.unix_seconds` :meth:`pyspark.sql.functions.unix_millis` + :meth:`pyspark.sql.functions.unix_nanos` :meth:`pyspark.sql.functions.timestamp_micros` Examples @@ -11771,6 +11773,7 @@ def unix_millis(col: "ColumnOrName") -> Column: :meth:`pyspark.sql.functions.unix_date` :meth:`pyspark.sql.functions.unix_seconds` :meth:`pyspark.sql.functions.unix_micros` + :meth:`pyspark.sql.functions.unix_nanos` :meth:`pyspark.sql.functions.timestamp_millis` Examples @@ -11865,6 +11868,7 @@ def unix_seconds(col: "ColumnOrName") -> Column: :meth:`pyspark.sql.functions.unix_date` :meth:`pyspark.sql.functions.unix_millis` :meth:`pyspark.sql.functions.unix_micros` + :meth:`pyspark.sql.functions.unix_nanos` :meth:`pyspark.sql.functions.from_unixtime` :meth:`pyspark.sql.functions.timestamp_seconds`