Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 16 additions & 9 deletions docs/source/contributor-guide/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,15 +24,22 @@ helpful to have a roadmap for some of the major items that require coordination

## Window Expressions

Native window execution is currently disabled by default due to known correctness issues ([#2721], [#2841]).
In addition, dedicated window functions such as `rank`, `dense_rank`, `row_number`, `lag`, `lead`, `ntile`,
`cume_dist`, `percent_rank`, and `nth_value` are not yet implemented and fall back to Spark ([#2705]). The
goal is to enable windowed aggregates by default ([#4007]) and add the missing dedicated window functions.

[#2705]: https://github.com/apache/datafusion-comet/issues/2705
[#2721]: https://github.com/apache/datafusion-comet/issues/2721
[#2841]: https://github.com/apache/datafusion-comet/issues/2841
[#4007]: https://github.com/apache/datafusion-comet/issues/4007
Native window execution runs by default (`spark.comet.exec.window.enabled`). The ranking functions (`rank`,
`dense_rank`, `row_number`, `percent_rank`, `cume_dist`, `ntile`), value functions (`lag`, `lead`, `nth_value`,
`first_value`, `last_value`), and the `count`, `min`, `max`, `sum`, and `avg` aggregates are accelerated.
Remaining work is to close the gaps that still fall back to Spark: statistical aggregates (`stddev`, `variance`,
`corr`, `covar`) and `collect_list` / `collect_set` as window functions ([#4766]), `GROUPS` frames ([#4836]), `RANGE` frames with explicit date or
decimal offsets ([#4834]), `first_value` / `last_value` on `RANGE` frames with a literal offset ([#4835]),
non-literal `lag` / `lead` default values ([#4268]), and `WindowGroupLimitExec` ([#4837]). See the
[window function compatibility guide](../user-guide/latest/compatibility/operators.md) for the complete list of
supported functions, frames, and fallback cases.

[#4268]: https://github.com/apache/datafusion-comet/issues/4268
[#4766]: https://github.com/apache/datafusion-comet/issues/4766
[#4834]: https://github.com/apache/datafusion-comet/issues/4834
[#4835]: https://github.com/apache/datafusion-comet/issues/4835
[#4836]: https://github.com/apache/datafusion-comet/issues/4836
[#4837]: https://github.com/apache/datafusion-comet/issues/4837

## Lambda Expressions

Expand Down
38 changes: 36 additions & 2 deletions docs/source/user-guide/latest/compatibility/operators.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,42 @@ under the License.

## Window Functions

Comet's support for window functions is incomplete and known to be incorrect. It is disabled by default and
should not be used in production. The feature will be enabled in a future release. Tracking issue: [#2721](https://github.com/apache/datafusion-comet/issues/2721).
Comet runs `WindowExec` natively and it is enabled by default (`spark.comet.exec.window.enabled`). A broad set of
window functions is accelerated, and any shape Comet does not support falls back to Spark rather than producing an
incorrect result. When any single window expression in a `WindowExec` falls back, the entire operator runs on Spark.

**Accelerated natively:**

- Ranking functions: `row_number`, `rank`, `dense_rank`, `percent_rank`, `cume_dist`, `ntile`.
- Value functions: `lag`, `lead`, `nth_value`, `first_value` (`first`), `last_value` (`last`). `IGNORE NULLS` is
supported.
- Aggregate window functions: `count`, `min`, `max`, `sum`, `avg`.
- Frame units `ROWS` and `RANGE`, with `UNBOUNDED PRECEDING` / `UNBOUNDED FOLLOWING`, `CURRENT ROW`, and numeric
`PRECEDING` / `FOLLOWING` offsets.

**Falls back to Spark:**

- Aggregate window functions other than the ones listed above, including the statistical aggregates
(`stddev`, `stddev_pop`, `stddev_samp`, `var_pop`, `var_samp`, `corr`, `covar_pop`, `covar_samp`). These run
natively as plain aggregations but not as window functions
([#4766](https://github.com/apache/datafusion-comet/issues/4766)).
- `min` / `max` on string, binary, timestamp-without-time-zone, interval, or nested (array / struct) input types,
and `sum` / `avg` on year-month or day-time interval input types. Windowed aggregates inherit the same input-type
support as the batch aggregates, so these fall back in both contexts.
- `sum` or `avg` on `DECIMAL` with a sliding (non ever-expanding) frame, because the sliding path would wrap on
overflow instead of returning Spark's `NULL` ([#4729](https://github.com/apache/datafusion-comet/issues/4729)).
- `RANGE` frame with an explicit offset when the `ORDER BY` column is `DATE` or `DECIMAL`
([#4834](https://github.com/apache/datafusion-comet/issues/4834)).
- `first_value` / `last_value` on a `RANGE` frame with a literal offset
([#4835](https://github.com/apache/datafusion-comet/issues/4835)).
- `lag` / `lead` with a non-literal default value ([#4268](https://github.com/apache/datafusion-comet/issues/4268)).
- A `ROWS` offset that is not an integer or long, or a `RANGE` offset that is not numeric.
- `GROUPS` frames ([#4836](https://github.com/apache/datafusion-comet/issues/4836)). `DISTINCT` aggregates over a
window are not supported by Spark either.
- Any `PARTITION BY` or `ORDER BY` expression that Comet cannot serialize.

`WindowGroupLimitExec` (window-based limit pushdown) is not yet supported and falls back to Spark
([#4837](https://github.com/apache/datafusion-comet/issues/4837)).

## Round-Robin Partitioning

Expand Down
17 changes: 10 additions & 7 deletions docs/source/user-guide/latest/expressions.md
Original file line number Diff line number Diff line change
Expand Up @@ -625,13 +625,16 @@ expression-level). The `outer` variants are wired but marked `Incompatible`; the

## window_funcs

Window functions run via `CometWindowExec`. Aggregate window functions
(`count`, `min`, `max`, `sum`, `avg`, `first_value`, `last_value`),
ranking functions (`row_number`, `rank`, `dense_rank`, `percent_rank`,
`cume_dist`, `ntile`), and value-shift functions (`lag`, `lead`,
`nth_value`) are all wired in the window serde and execute natively.
A handful of frame shapes still fall back — see the per-function notes
for the exact unsupported cases.
Window functions run via `CometWindowExec`, which is enabled by default.
Aggregate window functions (`count`, `min`, `max`, `sum`, `avg`,
`first_value`, `last_value`), ranking functions (`row_number`, `rank`,
`dense_rank`, `percent_rank`, `cume_dist`, `ntile`), and value-shift
functions (`lag`, `lead`, `nth_value`) are all wired in the window serde
and execute natively. Statistical aggregates such as `stddev`, `var_pop`,
`corr`, and `covar_pop` run natively as plain aggregations but fall back
to Spark when used as window functions. A handful of frame shapes also
fall back. See [window function compatibility](compatibility/operators.md)
for the full list of supported functions, frames, and fallback cases.

| Function | Status | Notes |
| --- | --- | --- |
Expand Down
8 changes: 4 additions & 4 deletions docs/source/user-guide/latest/operators.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,10 +99,10 @@ omitted from the tables below and may be reconsidered based on demand:

## Window

| Operator | Status | Notes |
| ---------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `WindowExec` | ⚠️ | Runs natively, but only a subset of window functions is accelerated. The rest fall back. See the [expression reference](expressions.md) (#2721). |
| `WindowGroupLimitExec` | 🔜 | Window-based limit pushdown falls back today. |
| Operator | Status | Notes |
| ---------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `WindowExec` | ⚠️ | Runs natively and is enabled by default. A broad set of window functions is accelerated; unsupported shapes fall back to Spark. See [window function compatibility](compatibility/operators.md). |
| `WindowGroupLimitExec` | 🔜 | Window-based limit pushdown falls back today ([#4837](https://github.com/apache/datafusion-comet/issues/4837)). |

## Generators and set operations

Expand Down