diff --git a/docs/source/contributor-guide/roadmap.md b/docs/source/contributor-guide/roadmap.md index bae4ddba58..b8ae8a310f 100644 --- a/docs/source/contributor-guide/roadmap.md +++ b/docs/source/contributor-guide/roadmap.md @@ -24,15 +24,22 @@ helpful to have a roadmap for some of the major items that require coordination ## Window Expressions -Native window execution is currently disabled by default due to known correctness issues ([#2721], [#2841]). -In addition, dedicated window functions such as `rank`, `dense_rank`, `row_number`, `lag`, `lead`, `ntile`, -`cume_dist`, `percent_rank`, and `nth_value` are not yet implemented and fall back to Spark ([#2705]). The -goal is to enable windowed aggregates by default ([#4007]) and add the missing dedicated window functions. - -[#2705]: https://github.com/apache/datafusion-comet/issues/2705 -[#2721]: https://github.com/apache/datafusion-comet/issues/2721 -[#2841]: https://github.com/apache/datafusion-comet/issues/2841 -[#4007]: https://github.com/apache/datafusion-comet/issues/4007 +Native window execution runs by default (`spark.comet.exec.window.enabled`). The ranking functions (`rank`, +`dense_rank`, `row_number`, `percent_rank`, `cume_dist`, `ntile`), value functions (`lag`, `lead`, `nth_value`, +`first_value`, `last_value`), and the `count`, `min`, `max`, `sum`, and `avg` aggregates are accelerated. +Remaining work is to close the gaps that still fall back to Spark: statistical aggregates (`stddev`, `variance`, +`corr`, `covar`) and `collect_list` / `collect_set` as window functions ([#4766]), `GROUPS` frames ([#4836]), `RANGE` frames with explicit date or +decimal offsets ([#4834]), `first_value` / `last_value` on `RANGE` frames with a literal offset ([#4835]), +non-literal `lag` / `lead` default values ([#4268]), and `WindowGroupLimitExec` ([#4837]). See the +[window function compatibility guide](../user-guide/latest/compatibility/operators.md) for the complete list of +supported functions, frames, and fallback cases. + +[#4268]: https://github.com/apache/datafusion-comet/issues/4268 +[#4766]: https://github.com/apache/datafusion-comet/issues/4766 +[#4834]: https://github.com/apache/datafusion-comet/issues/4834 +[#4835]: https://github.com/apache/datafusion-comet/issues/4835 +[#4836]: https://github.com/apache/datafusion-comet/issues/4836 +[#4837]: https://github.com/apache/datafusion-comet/issues/4837 ## Lambda Expressions diff --git a/docs/source/user-guide/latest/compatibility/operators.md b/docs/source/user-guide/latest/compatibility/operators.md index a25d1d1db9..28c0954047 100644 --- a/docs/source/user-guide/latest/compatibility/operators.md +++ b/docs/source/user-guide/latest/compatibility/operators.md @@ -21,8 +21,42 @@ under the License. ## Window Functions -Comet's support for window functions is incomplete and known to be incorrect. It is disabled by default and -should not be used in production. The feature will be enabled in a future release. Tracking issue: [#2721](https://github.com/apache/datafusion-comet/issues/2721). +Comet runs `WindowExec` natively and it is enabled by default (`spark.comet.exec.window.enabled`). A broad set of +window functions is accelerated, and any shape Comet does not support falls back to Spark rather than producing an +incorrect result. When any single window expression in a `WindowExec` falls back, the entire operator runs on Spark. + +**Accelerated natively:** + +- Ranking functions: `row_number`, `rank`, `dense_rank`, `percent_rank`, `cume_dist`, `ntile`. +- Value functions: `lag`, `lead`, `nth_value`, `first_value` (`first`), `last_value` (`last`). `IGNORE NULLS` is + supported. +- Aggregate window functions: `count`, `min`, `max`, `sum`, `avg`. +- Frame units `ROWS` and `RANGE`, with `UNBOUNDED PRECEDING` / `UNBOUNDED FOLLOWING`, `CURRENT ROW`, and numeric + `PRECEDING` / `FOLLOWING` offsets. + +**Falls back to Spark:** + +- Aggregate window functions other than the ones listed above, including the statistical aggregates + (`stddev`, `stddev_pop`, `stddev_samp`, `var_pop`, `var_samp`, `corr`, `covar_pop`, `covar_samp`). These run + natively as plain aggregations but not as window functions + ([#4766](https://github.com/apache/datafusion-comet/issues/4766)). +- `min` / `max` on string, binary, timestamp-without-time-zone, interval, or nested (array / struct) input types, + and `sum` / `avg` on year-month or day-time interval input types. Windowed aggregates inherit the same input-type + support as the batch aggregates, so these fall back in both contexts. +- `sum` or `avg` on `DECIMAL` with a sliding (non ever-expanding) frame, because the sliding path would wrap on + overflow instead of returning Spark's `NULL` ([#4729](https://github.com/apache/datafusion-comet/issues/4729)). +- `RANGE` frame with an explicit offset when the `ORDER BY` column is `DATE` or `DECIMAL` + ([#4834](https://github.com/apache/datafusion-comet/issues/4834)). +- `first_value` / `last_value` on a `RANGE` frame with a literal offset + ([#4835](https://github.com/apache/datafusion-comet/issues/4835)). +- `lag` / `lead` with a non-literal default value ([#4268](https://github.com/apache/datafusion-comet/issues/4268)). +- A `ROWS` offset that is not an integer or long, or a `RANGE` offset that is not numeric. +- `GROUPS` frames ([#4836](https://github.com/apache/datafusion-comet/issues/4836)). `DISTINCT` aggregates over a + window are not supported by Spark either. +- Any `PARTITION BY` or `ORDER BY` expression that Comet cannot serialize. + +`WindowGroupLimitExec` (window-based limit pushdown) is not yet supported and falls back to Spark +([#4837](https://github.com/apache/datafusion-comet/issues/4837)). ## Round-Robin Partitioning diff --git a/docs/source/user-guide/latest/expressions.md b/docs/source/user-guide/latest/expressions.md index 2ca3a13c62..a67a5706e3 100644 --- a/docs/source/user-guide/latest/expressions.md +++ b/docs/source/user-guide/latest/expressions.md @@ -625,13 +625,16 @@ expression-level). The `outer` variants are wired but marked `Incompatible`; the ## window_funcs -Window functions run via `CometWindowExec`. Aggregate window functions -(`count`, `min`, `max`, `sum`, `avg`, `first_value`, `last_value`), -ranking functions (`row_number`, `rank`, `dense_rank`, `percent_rank`, -`cume_dist`, `ntile`), and value-shift functions (`lag`, `lead`, -`nth_value`) are all wired in the window serde and execute natively. -A handful of frame shapes still fall back — see the per-function notes -for the exact unsupported cases. +Window functions run via `CometWindowExec`, which is enabled by default. +Aggregate window functions (`count`, `min`, `max`, `sum`, `avg`, +`first_value`, `last_value`), ranking functions (`row_number`, `rank`, +`dense_rank`, `percent_rank`, `cume_dist`, `ntile`), and value-shift +functions (`lag`, `lead`, `nth_value`) are all wired in the window serde +and execute natively. Statistical aggregates such as `stddev`, `var_pop`, +`corr`, and `covar_pop` run natively as plain aggregations but fall back +to Spark when used as window functions. A handful of frame shapes also +fall back. See [window function compatibility](compatibility/operators.md) +for the full list of supported functions, frames, and fallback cases. | Function | Status | Notes | | --- | --- | --- | diff --git a/docs/source/user-guide/latest/operators.md b/docs/source/user-guide/latest/operators.md index 0bdf3ba919..a29ffa1355 100644 --- a/docs/source/user-guide/latest/operators.md +++ b/docs/source/user-guide/latest/operators.md @@ -99,10 +99,10 @@ omitted from the tables below and may be reconsidered based on demand: ## Window -| Operator | Status | Notes | -| ---------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | -| `WindowExec` | ⚠️ | Runs natively, but only a subset of window functions is accelerated. The rest fall back. See the [expression reference](expressions.md) (#2721). | -| `WindowGroupLimitExec` | 🔜 | Window-based limit pushdown falls back today. | +| Operator | Status | Notes | +| ---------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `WindowExec` | ⚠️ | Runs natively and is enabled by default. A broad set of window functions is accelerated; unsupported shapes fall back to Spark. See [window function compatibility](compatibility/operators.md). | +| `WindowGroupLimitExec` | 🔜 | Window-based limit pushdown falls back today ([#4837](https://github.com/apache/datafusion-comet/issues/4837)). | ## Generators and set operations