[SPARK-57740][PYTHON] Fix unix_micros truncation disclaimer and add unix_nanos cross-references to unix_* See Also sections#56851
Open
jubins wants to merge 1 commit into
Conversation
…nix_nanos cross-references to unix_* See Also sections
### What is the purpose of the change
Fixes SPARK-57740 — two related doc-correctness issues in the `unix_*` epoch-unit
function family in `python/pyspark/sql/functions/builtin.py`:
1. **Missing truncation disclaimer in `unix_micros`.**
`unix_millis` and `unix_seconds` both include `"Truncates higher levels of
precision."` in their one-line description. `unix_micros` did not, despite
silently dropping sub-microsecond digits when given a nanosecond-precision
input (`TIMESTAMP_LTZ(p)` / `TIMESTAMP_NTZ(p)`, `p` in `[7, 9]`). This
became a practical concern once `unix_nanos` (SPARK-57579) made
nanosecond-precision timestamps usable in PySpark.
2. **Missing `unix_nanos` cross-references.**
`unix_nanos` (SPARK-57579) correctly links to all three siblings in its
`See Also` section. None of the three sibling functions were updated to link
back to `unix_nanos`, breaking discoverability of the function family from
the existing functions.
### Brief change log
- `python/pyspark/sql/functions/builtin.py`:
- `unix_micros`: added `"Truncates higher levels of precision."` to the
one-line description
- `unix_micros`, `unix_millis`, `unix_seconds`: added
`:meth:\`pyspark.sql.functions.unix_nanos\`` to each `See Also` section
### Verifying this change
These are documentation-only changes with no logic impact. Verified by
inspection that:
- All four `unix_*` functions now consistently include a truncation disclaimer
in their description
- All four `unix_*` functions now cross-reference each other in their
`See Also` sections symmetrically
### Does this pull request potentially affect one of the following parts
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public`/`@Evolving`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: no
- The S3 file system connector: no
### Documentation
Does this pull request introduce a new feature? No — documentation fix only.
### Was generative AI tooling used to co-author this PR?
Yes — Claude Code was used as a pair-programming assistant. All code was
written, understood, and verified by the author.
Generated-by: Claude Opus 4.8
MaxGekk
approved these changes
Jun 28, 2026
Member
|
@jubins please, make PR's title concise |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
Fixes SPARK-57740 — two related doc-correctness issues in the
unix_*epoch-unit function family inpython/pyspark/sql/functions/builtin.py:Missing truncation disclaimer in
unix_micros.unix_millisandunix_secondsboth include"Truncates higher levels of precision."in their one-line description.unix_microsdid not, despite silently dropping sub-microsecond digits when given a nanosecond-precision input (TIMESTAMP_LTZ(p)/TIMESTAMP_NTZ(p),pin[7, 9]). This became a practical concern onceunix_nanos(SPARK-57579) made nanosecond-precision timestamps usable in PySpark.Missing
unix_nanoscross-references.unix_nanos(SPARK-57579) correctly links to all three siblings in itsSee Alsosection. None of the three sibling functions were updated to link back tounix_nanos, breaking discoverability of the function family from the existing functions.Brief change log
python/pyspark/sql/functions/builtin.py:unix_micros: added"Truncates higher levels of precision."to the one-line descriptionunix_micros,unix_millis,unix_seconds: added:meth:\pyspark.sql.functions.unix_nanos`to eachSee Also` sectionVerifying this change
These are documentation-only changes with no logic impact. Verified by inspection that:
unix_*functions now consistently include a truncation disclaimer in their descriptionunix_*functions now cross-reference each other in theirSee Alsosections symmetricallyDoes this pull request potentially affect one of the following parts
@Public/@Evolving: noDocumentation
Does this pull request introduce a new feature? No — documentation fix only.
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Opus 4.8