From cdf018887819b1bda124bb10004d7131bdbce6f8 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 6 Jul 2026 09:29:33 -0600 Subject: [PATCH 1/4] docs: correct window function support status and limitations Window execution runs natively and is enabled by default, but the compatibility guide and roadmap still described it as disabled by default and known to be incorrect, citing the now-closed epic #2721. Update the window docs to describe the current reality: which ranking, value, and aggregate window functions are accelerated, which frame units and bounds are handled, and the specific shapes that fall back to Spark. State the unsupported aggregate input types positively (the Spark types Comet does not accept) rather than by exclusion. --- docs/source/contributor-guide/roadmap.md | 18 +++++------ .../latest/compatibility/operators.md | 32 +++++++++++++++++-- docs/source/user-guide/latest/expressions.md | 17 ++++++---- docs/source/user-guide/latest/operators.md | 8 ++--- 4 files changed, 53 insertions(+), 22 deletions(-) diff --git a/docs/source/contributor-guide/roadmap.md b/docs/source/contributor-guide/roadmap.md index bae4ddba58..388df3dfb1 100644 --- a/docs/source/contributor-guide/roadmap.md +++ b/docs/source/contributor-guide/roadmap.md @@ -24,15 +24,15 @@ helpful to have a roadmap for some of the major items that require coordination ## Window Expressions -Native window execution is currently disabled by default due to known correctness issues ([#2721], [#2841]). -In addition, dedicated window functions such as `rank`, `dense_rank`, `row_number`, `lag`, `lead`, `ntile`, -`cume_dist`, `percent_rank`, and `nth_value` are not yet implemented and fall back to Spark ([#2705]). The -goal is to enable windowed aggregates by default ([#4007]) and add the missing dedicated window functions. - -[#2705]: https://github.com/apache/datafusion-comet/issues/2705 -[#2721]: https://github.com/apache/datafusion-comet/issues/2721 -[#2841]: https://github.com/apache/datafusion-comet/issues/2841 -[#4007]: https://github.com/apache/datafusion-comet/issues/4007 +Native window execution runs by default (`spark.comet.exec.window.enabled`). The ranking functions (`rank`, +`dense_rank`, `row_number`, `percent_rank`, `cume_dist`, `ntile`), value functions (`lag`, `lead`, `nth_value`, +`first_value`, `last_value`), and the `count`, `min`, `max`, `sum`, and `avg` aggregates are accelerated. +Remaining work is to close the gaps that still fall back to Spark: statistical aggregates (`stddev`, variance, +`corr`, `covar`) as window functions, `GROUPS` frames, non-literal `lag` / `lead` default values ([#4268]), and +`WindowGroupLimitExec`. See the [window function compatibility guide](../user-guide/latest/compatibility/operators.md) +for the complete list of supported functions, frames, and fallback cases. + +[#4268]: https://github.com/apache/datafusion-comet/issues/4268 ## Lambda Expressions diff --git a/docs/source/user-guide/latest/compatibility/operators.md b/docs/source/user-guide/latest/compatibility/operators.md index a25d1d1db9..4ced117a13 100644 --- a/docs/source/user-guide/latest/compatibility/operators.md +++ b/docs/source/user-guide/latest/compatibility/operators.md @@ -21,8 +21,36 @@ under the License. ## Window Functions -Comet's support for window functions is incomplete and known to be incorrect. It is disabled by default and -should not be used in production. The feature will be enabled in a future release. Tracking issue: [#2721](https://github.com/apache/datafusion-comet/issues/2721). +Comet runs `WindowExec` natively and it is enabled by default (`spark.comet.exec.window.enabled`). A broad set of +window functions is accelerated, and any shape Comet does not support falls back to Spark rather than producing an +incorrect result. When any single window expression in a `WindowExec` falls back, the entire operator runs on Spark. + +**Accelerated natively:** + +- Ranking functions: `row_number`, `rank`, `dense_rank`, `percent_rank`, `cume_dist`, `ntile`. +- Value functions: `lag`, `lead`, `nth_value`, `first_value` (`first`), `last_value` (`last`). `IGNORE NULLS` is + supported. +- Aggregate window functions: `count`, `min`, `max`, `sum`, `avg`. +- Frame units `ROWS` and `RANGE`, with `UNBOUNDED PRECEDING` / `UNBOUNDED FOLLOWING`, `CURRENT ROW`, and numeric + `PRECEDING` / `FOLLOWING` offsets. + +**Falls back to Spark:** + +- Aggregate window functions other than the ones listed above, including the statistical aggregates + (`stddev`, `stddev_pop`, `stddev_samp`, `var_pop`, `var_samp`, `corr`, `covar_pop`, `covar_samp`). These run + natively as plain aggregations but not as window functions. +- `min` / `max` on string, binary, timestamp-without-time-zone, interval, or nested (array / struct) input types. +- `sum` / `avg` on year-month or day-time interval input types. +- `sum` or `avg` on `DECIMAL` with a sliding (non ever-expanding) frame, because the sliding path would wrap on + overflow instead of returning Spark's `NULL` ([#4729](https://github.com/apache/datafusion-comet/issues/4729)). +- `RANGE` frame with an explicit offset when the `ORDER BY` column is `DATE` or `DECIMAL`. +- `first_value` / `last_value` on a `RANGE` frame with a literal offset. +- `lag` / `lead` with a non-literal default value ([#4268](https://github.com/apache/datafusion-comet/issues/4268)). +- A `ROWS` offset that is not an integer or long, or a `RANGE` offset that is not numeric. +- `GROUPS` frames and `DISTINCT` aggregates over a window. +- Any `PARTITION BY` or `ORDER BY` expression that Comet cannot serialize. + +`WindowGroupLimitExec` (window-based limit pushdown) is not yet supported and falls back to Spark. ## Round-Robin Partitioning diff --git a/docs/source/user-guide/latest/expressions.md b/docs/source/user-guide/latest/expressions.md index 2ca3a13c62..a67a5706e3 100644 --- a/docs/source/user-guide/latest/expressions.md +++ b/docs/source/user-guide/latest/expressions.md @@ -625,13 +625,16 @@ expression-level). The `outer` variants are wired but marked `Incompatible`; the ## window_funcs -Window functions run via `CometWindowExec`. Aggregate window functions -(`count`, `min`, `max`, `sum`, `avg`, `first_value`, `last_value`), -ranking functions (`row_number`, `rank`, `dense_rank`, `percent_rank`, -`cume_dist`, `ntile`), and value-shift functions (`lag`, `lead`, -`nth_value`) are all wired in the window serde and execute natively. -A handful of frame shapes still fall back — see the per-function notes -for the exact unsupported cases. +Window functions run via `CometWindowExec`, which is enabled by default. +Aggregate window functions (`count`, `min`, `max`, `sum`, `avg`, +`first_value`, `last_value`), ranking functions (`row_number`, `rank`, +`dense_rank`, `percent_rank`, `cume_dist`, `ntile`), and value-shift +functions (`lag`, `lead`, `nth_value`) are all wired in the window serde +and execute natively. Statistical aggregates such as `stddev`, `var_pop`, +`corr`, and `covar_pop` run natively as plain aggregations but fall back +to Spark when used as window functions. A handful of frame shapes also +fall back. See [window function compatibility](compatibility/operators.md) +for the full list of supported functions, frames, and fallback cases. | Function | Status | Notes | | --- | --- | --- | diff --git a/docs/source/user-guide/latest/operators.md b/docs/source/user-guide/latest/operators.md index 0bdf3ba919..e8694778c0 100644 --- a/docs/source/user-guide/latest/operators.md +++ b/docs/source/user-guide/latest/operators.md @@ -99,10 +99,10 @@ omitted from the tables below and may be reconsidered based on demand: ## Window -| Operator | Status | Notes | -| ---------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | -| `WindowExec` | ⚠️ | Runs natively, but only a subset of window functions is accelerated. The rest fall back. See the [expression reference](expressions.md) (#2721). | -| `WindowGroupLimitExec` | 🔜 | Window-based limit pushdown falls back today. | +| Operator | Status | Notes | +| ---------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `WindowExec` | ⚠️ | Runs natively and is enabled by default. A broad set of window functions is accelerated; unsupported shapes fall back to Spark. See [window function compatibility](compatibility/operators.md). | +| `WindowGroupLimitExec` | 🔜 | Window-based limit pushdown falls back today. | ## Generators and set operations From 6a9d1a6cbbd6e0c72e870bed5779980831f4d3b3 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 6 Jul 2026 09:34:39 -0600 Subject: [PATCH 2/4] docs: link window function gaps to tracking issues --- docs/source/contributor-guide/roadmap.md | 13 +++++++++--- .../latest/compatibility/operators.md | 20 ++++++++++++------- docs/source/user-guide/latest/operators.md | 2 +- 3 files changed, 24 insertions(+), 11 deletions(-) diff --git a/docs/source/contributor-guide/roadmap.md b/docs/source/contributor-guide/roadmap.md index 388df3dfb1..9a75b5cafc 100644 --- a/docs/source/contributor-guide/roadmap.md +++ b/docs/source/contributor-guide/roadmap.md @@ -28,11 +28,18 @@ Native window execution runs by default (`spark.comet.exec.window.enabled`). The `dense_rank`, `row_number`, `percent_rank`, `cume_dist`, `ntile`), value functions (`lag`, `lead`, `nth_value`, `first_value`, `last_value`), and the `count`, `min`, `max`, `sum`, and `avg` aggregates are accelerated. Remaining work is to close the gaps that still fall back to Spark: statistical aggregates (`stddev`, variance, -`corr`, `covar`) as window functions, `GROUPS` frames, non-literal `lag` / `lead` default values ([#4268]), and -`WindowGroupLimitExec`. See the [window function compatibility guide](../user-guide/latest/compatibility/operators.md) -for the complete list of supported functions, frames, and fallback cases. +`corr`, `covar`) as window functions ([#4766]), `GROUPS` frames ([#4836]), `RANGE` frames with explicit date or +decimal offsets ([#4834]), `first_value` / `last_value` on `RANGE` frames with a literal offset ([#4835]), +non-literal `lag` / `lead` default values ([#4268]), and `WindowGroupLimitExec` ([#4837]). See the +[window function compatibility guide](../user-guide/latest/compatibility/operators.md) for the complete list of +supported functions, frames, and fallback cases. [#4268]: https://github.com/apache/datafusion-comet/issues/4268 +[#4766]: https://github.com/apache/datafusion-comet/issues/4766 +[#4834]: https://github.com/apache/datafusion-comet/issues/4834 +[#4835]: https://github.com/apache/datafusion-comet/issues/4835 +[#4836]: https://github.com/apache/datafusion-comet/issues/4836 +[#4837]: https://github.com/apache/datafusion-comet/issues/4837 ## Lambda Expressions diff --git a/docs/source/user-guide/latest/compatibility/operators.md b/docs/source/user-guide/latest/compatibility/operators.md index 4ced117a13..28c0954047 100644 --- a/docs/source/user-guide/latest/compatibility/operators.md +++ b/docs/source/user-guide/latest/compatibility/operators.md @@ -38,19 +38,25 @@ incorrect result. When any single window expression in a `WindowExec` falls back - Aggregate window functions other than the ones listed above, including the statistical aggregates (`stddev`, `stddev_pop`, `stddev_samp`, `var_pop`, `var_samp`, `corr`, `covar_pop`, `covar_samp`). These run - natively as plain aggregations but not as window functions. -- `min` / `max` on string, binary, timestamp-without-time-zone, interval, or nested (array / struct) input types. -- `sum` / `avg` on year-month or day-time interval input types. + natively as plain aggregations but not as window functions + ([#4766](https://github.com/apache/datafusion-comet/issues/4766)). +- `min` / `max` on string, binary, timestamp-without-time-zone, interval, or nested (array / struct) input types, + and `sum` / `avg` on year-month or day-time interval input types. Windowed aggregates inherit the same input-type + support as the batch aggregates, so these fall back in both contexts. - `sum` or `avg` on `DECIMAL` with a sliding (non ever-expanding) frame, because the sliding path would wrap on overflow instead of returning Spark's `NULL` ([#4729](https://github.com/apache/datafusion-comet/issues/4729)). -- `RANGE` frame with an explicit offset when the `ORDER BY` column is `DATE` or `DECIMAL`. -- `first_value` / `last_value` on a `RANGE` frame with a literal offset. +- `RANGE` frame with an explicit offset when the `ORDER BY` column is `DATE` or `DECIMAL` + ([#4834](https://github.com/apache/datafusion-comet/issues/4834)). +- `first_value` / `last_value` on a `RANGE` frame with a literal offset + ([#4835](https://github.com/apache/datafusion-comet/issues/4835)). - `lag` / `lead` with a non-literal default value ([#4268](https://github.com/apache/datafusion-comet/issues/4268)). - A `ROWS` offset that is not an integer or long, or a `RANGE` offset that is not numeric. -- `GROUPS` frames and `DISTINCT` aggregates over a window. +- `GROUPS` frames ([#4836](https://github.com/apache/datafusion-comet/issues/4836)). `DISTINCT` aggregates over a + window are not supported by Spark either. - Any `PARTITION BY` or `ORDER BY` expression that Comet cannot serialize. -`WindowGroupLimitExec` (window-based limit pushdown) is not yet supported and falls back to Spark. +`WindowGroupLimitExec` (window-based limit pushdown) is not yet supported and falls back to Spark +([#4837](https://github.com/apache/datafusion-comet/issues/4837)). ## Round-Robin Partitioning diff --git a/docs/source/user-guide/latest/operators.md b/docs/source/user-guide/latest/operators.md index e8694778c0..a29ffa1355 100644 --- a/docs/source/user-guide/latest/operators.md +++ b/docs/source/user-guide/latest/operators.md @@ -102,7 +102,7 @@ omitted from the tables below and may be reconsidered based on demand: | Operator | Status | Notes | | ---------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `WindowExec` | ⚠️ | Runs natively and is enabled by default. A broad set of window functions is accelerated; unsupported shapes fall back to Spark. See [window function compatibility](compatibility/operators.md). | -| `WindowGroupLimitExec` | 🔜 | Window-based limit pushdown falls back today. | +| `WindowGroupLimitExec` | 🔜 | Window-based limit pushdown falls back today ([#4837](https://github.com/apache/datafusion-comet/issues/4837)). | ## Generators and set operations From f9997e77b09b035a79ffdb95347420c146a98d87 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 6 Jul 2026 10:50:35 -0600 Subject: [PATCH 3/4] Update docs/source/contributor-guide/roadmap.md Co-authored-by: Oleks V --- docs/source/contributor-guide/roadmap.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/contributor-guide/roadmap.md b/docs/source/contributor-guide/roadmap.md index 9a75b5cafc..2af0a56d53 100644 --- a/docs/source/contributor-guide/roadmap.md +++ b/docs/source/contributor-guide/roadmap.md @@ -27,7 +27,7 @@ helpful to have a roadmap for some of the major items that require coordination Native window execution runs by default (`spark.comet.exec.window.enabled`). The ranking functions (`rank`, `dense_rank`, `row_number`, `percent_rank`, `cume_dist`, `ntile`), value functions (`lag`, `lead`, `nth_value`, `first_value`, `last_value`), and the `count`, `min`, `max`, `sum`, and `avg` aggregates are accelerated. -Remaining work is to close the gaps that still fall back to Spark: statistical aggregates (`stddev`, variance, +Remaining work is to close the gaps that still fall back to Spark: statistical aggregates (`stddev`, `variance`, `corr`, `covar`) as window functions ([#4766]), `GROUPS` frames ([#4836]), `RANGE` frames with explicit date or decimal offsets ([#4834]), `first_value` / `last_value` on `RANGE` frames with a literal offset ([#4835]), non-literal `lag` / `lead` default values ([#4268]), and `WindowGroupLimitExec` ([#4837]). See the From 27d1eeb319a507d53ceab9c9eb2205be73b9c7d8 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 6 Jul 2026 13:30:48 -0600 Subject: [PATCH 4/4] docs: note collect_list/collect_set window gap under #4766 --- docs/source/contributor-guide/roadmap.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/contributor-guide/roadmap.md b/docs/source/contributor-guide/roadmap.md index 2af0a56d53..b8ae8a310f 100644 --- a/docs/source/contributor-guide/roadmap.md +++ b/docs/source/contributor-guide/roadmap.md @@ -28,7 +28,7 @@ Native window execution runs by default (`spark.comet.exec.window.enabled`). The `dense_rank`, `row_number`, `percent_rank`, `cume_dist`, `ntile`), value functions (`lag`, `lead`, `nth_value`, `first_value`, `last_value`), and the `count`, `min`, `max`, `sum`, and `avg` aggregates are accelerated. Remaining work is to close the gaps that still fall back to Spark: statistical aggregates (`stddev`, `variance`, -`corr`, `covar`) as window functions ([#4766]), `GROUPS` frames ([#4836]), `RANGE` frames with explicit date or +`corr`, `covar`) and `collect_list` / `collect_set` as window functions ([#4766]), `GROUPS` frames ([#4836]), `RANGE` frames with explicit date or decimal offsets ([#4834]), `first_value` / `last_value` on `RANGE` frames with a literal offset ([#4835]), non-literal `lag` / `lead` default values ([#4268]), and `WindowGroupLimitExec` ([#4837]). See the [window function compatibility guide](../user-guide/latest/compatibility/operators.md) for the complete list of