feat: add Spark-compatible hypot function#23774
Conversation
|
cc @Jefffrey @andygrove — this is my first contribution, so CI is currently awaiting a committer's approval. Would really appreciate a trigger when you have a moment. This adds a Spark-compatible Thanks! |
| let [x, y] = take_function_args(self.name(), args.args)?; | ||
|
|
||
| // Broadcast scalars to arrays so one path covers every combination. | ||
| let x = x.to_array(num_rows)?; |
There was a problem hiding this comment.
make_scalar_function() handles this for us
e.g.
fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
make_scalar_function(hypot_array, vec![])(&args.args)
}
// outside impl
fn hypot_array(args: &[ArrayRef]) -> Result<ArrayRef> {
let [x, y] = take_function_args("hypot", args)?;
let x = x.as_primitive::<Float64Type>();
let y = y.as_primitive::<Float64Type>();
return Ok(Arc::new(binary(x, y, |a, b| a.hypot(b))))
}| ## PySpark 3.5.5 Result: {'HYPOT(3, 4)': 5.0, 'typeof(HYPOT(3, 4))': 'double', 'typeof(3)': 'int', 'typeof(4)': 'int'} | ||
| #query | ||
| #SELECT hypot(3::int, 4::int); | ||
| # Scalar: classic Pythagorean triples (3-4-5, 5-12-13) |
There was a problem hiding this comment.
could we add some more edge cases such as infinity inputs, nans, etc.
| // Spark only defines hypot over doubles; `exact` makes coercion | ||
| // guarantee both inputs are Float64 before `invoke` runs. |
There was a problem hiding this comment.
| // Spark only defines hypot over doubles; `exact` makes coercion | |
| // guarantee both inputs are Float64 before `invoke` runs. | |
| // Spark only defines hypot over doubles |
extra detail unnecessary
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #23774 +/- ##
=======================================
Coverage ? 80.71%
=======================================
Files ? 1090
Lines ? 368933
Branches ? 368933
=======================================
Hits ? 297789
Misses ? 53381
Partials ? 17763 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Which issue does this PR close?
datafusion-sparkSpark Compatible Functions #15914Rationale for this change
Spark provides
hypot(expr1, expr2), which returnssqrt(expr1^2 + expr2^2)computed without intermediate overflow or underflow. It was not yet implemented indatafusion-spark— only an auto-generated test stub existed atspark/math/hypot.sltwith its query commented out.What changes are included in this PR?
SparkHypot(implementingScalarUDFImpl) indatafusion/spark/src/function/math/hypot.rs, backed by Rust'sf64::hypot— the same overflow-safe algorithm as Java/Spark'sMath.hypot.datafusion/spark/src/function/math/mod.rs.hypot.sltsqllogictest.The signature is
exact(Float64, Float64) -> Float64, following thedatafusion-sparkconvention of only accepting types Spark supports. Computation uses the Arrowbinarykernel so NULL in either argument propagates to a NULL result, matching Spark.Are these changes tested?
Yes —
datafusion/sqllogictest/test_files/spark/math/hypot.sltcovers:hypot(3, 4)→ 5,hypot(5, 12)→ 13),hypot(3e200, 4e200)stays finite, whereas a naivesqrt(a^2 + b^2)would overflow toInfinity.Are there any user-facing changes?
Yes — adds the Spark-compatible
hypotscalar function todatafusion-spark. No breaking changes to public APIs.