Skip to content

[SPARK-57740][PYTHON] Fix unix_micros truncation disclaimer and add unix_nanos cross-references to unix_* See Also sections#56851

Open
jubins wants to merge 1 commit into
apache:masterfrom
jubins:j-SPARK-57740-added-cross-reference-to-unix-nanos
Open

[SPARK-57740][PYTHON] Fix unix_micros truncation disclaimer and add unix_nanos cross-references to unix_* See Also sections#56851
jubins wants to merge 1 commit into
apache:masterfrom
jubins:j-SPARK-57740-added-cross-reference-to-unix-nanos

Conversation

@jubins

@jubins jubins commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

What is the purpose of the change

Fixes SPARK-57740 — two related doc-correctness issues in the unix_* epoch-unit function family in python/pyspark/sql/functions/builtin.py:

  1. Missing truncation disclaimer in unix_micros. unix_millis and unix_seconds both include "Truncates higher levels of precision." in their one-line description. unix_micros did not, despite silently dropping sub-microsecond digits when given a nanosecond-precision input (TIMESTAMP_LTZ(p) / TIMESTAMP_NTZ(p), p in [7, 9]). This became a practical concern once unix_nanos (SPARK-57579) made nanosecond-precision timestamps usable in PySpark.

  2. Missing unix_nanos cross-references. unix_nanos (SPARK-57579) correctly links to all three siblings in its See Also section. None of the three sibling functions were updated to link back to unix_nanos, breaking discoverability of the function family from the existing functions.

Brief change log

  • python/pyspark/sql/functions/builtin.py:
    • unix_micros: added "Truncates higher levels of precision." to the one-line description
    • unix_micros, unix_millis, unix_seconds: added :meth:\pyspark.sql.functions.unix_nanos`to eachSee Also` section

Verifying this change

These are documentation-only changes with no logic impact. Verified by inspection that:

  • All four unix_* functions now consistently include a truncation disclaimer in their description
  • All four unix_* functions now cross-reference each other in their See Also sections symmetrically

Does this pull request potentially affect one of the following parts

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public/@Evolving: no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: no
  • The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? No — documentation fix only.

Was generative AI tooling used to co-author this PR?

  • Yes — Claude Code was used as a pair-programming assistant. All code was written, understood, and verified by the author.
    Generated-by: Claude Opus 4.8

…nix_nanos cross-references to unix_* See Also sections

### What is the purpose of the change

Fixes SPARK-57740 — two related doc-correctness issues in the `unix_*` epoch-unit
function family in `python/pyspark/sql/functions/builtin.py`:

1. **Missing truncation disclaimer in `unix_micros`.**
   `unix_millis` and `unix_seconds` both include `"Truncates higher levels of
   precision."` in their one-line description. `unix_micros` did not, despite
   silently dropping sub-microsecond digits when given a nanosecond-precision
   input (`TIMESTAMP_LTZ(p)` / `TIMESTAMP_NTZ(p)`, `p` in `[7, 9]`). This
   became a practical concern once `unix_nanos` (SPARK-57579) made
   nanosecond-precision timestamps usable in PySpark.

2. **Missing `unix_nanos` cross-references.**
   `unix_nanos` (SPARK-57579) correctly links to all three siblings in its
   `See Also` section. None of the three sibling functions were updated to link
   back to `unix_nanos`, breaking discoverability of the function family from
   the existing functions.

### Brief change log

- `python/pyspark/sql/functions/builtin.py`:
  - `unix_micros`: added `"Truncates higher levels of precision."` to the
    one-line description
  - `unix_micros`, `unix_millis`, `unix_seconds`: added
    `:meth:\`pyspark.sql.functions.unix_nanos\`` to each `See Also` section

### Verifying this change

These are documentation-only changes with no logic impact. Verified by
inspection that:

- All four `unix_*` functions now consistently include a truncation disclaimer
  in their description
- All four `unix_*` functions now cross-reference each other in their
  `See Also` sections symmetrically

### Does this pull request potentially affect one of the following parts

- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public`/`@Evolving`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: no
- The S3 file system connector: no

### Documentation

Does this pull request introduce a new feature? No — documentation fix only.

### Was generative AI tooling used to co-author this PR?

Yes — Claude Code was used as a pair-programming assistant. All code was
written, understood, and verified by the author.
Generated-by: Claude Opus 4.8
@MaxGekk

MaxGekk commented Jun 28, 2026

Copy link
Copy Markdown
Member

@jubins please, make PR's title concise

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants